What criteria should be used for evaluating proposed plans?by Dan Power
In the process of developing plans and alternative courses of action (COA), multiple criteria or factors should be used to determine the validity of each proposed plan. The criteria should be assessed and considered prior to any systematic comparative evaluation of plans. This is true in business, military and general organizational planning situations. The criteria are often relevant in personal planning situations as well.
Any computerized planning DSS should provide a "checklist" or an organized set of rating scales that can be used to assess the validity of each alternative plan (COA) based on seven criteria: acceptability, adequacy, completeness, distinguishability, feasibility, suitability and variety. Ideally multiple analysts/planners would rate each proposed plan on each of these seven criteria.
Planners, decision makers and analysts should interpret “valid” as meaning worth further analysis and consideration. A valid plan or COA meets minimum standards and hence an invalid COA has failed on one or more of the seven criteria of "goodness" used to evaluate specific COA plans; a course of action plan that is deficient on one or more criteria should be revised or rejected.
The challenge in computerizing plan evaluation criteria, especially these seven criteria, is that the terms are familiar words that are hard to operationalize. Also, the terms used for the criteria are interrelated and overlapping concepts. In a computerized planning and decision support system the criteria need to be defined and used consistently and users of the system need a shared understanding of what the terms mean.
Progress must be made in reaching agreement on the meaning and operationalization of these terms if a computerized contingency planning system is to be useful.
The remainder of this analysis discusses each of the seven evaluation concepts (terms). The discussion draws upon glossary definitions, Roget's Thesaurus, and the American Heritage Dictionary (AHD). The concepts are intended as "attributes of a plan and COA" so one should be able to discern there presence or absence by examining a specific plan. An attribute is a descriptor of a planned course of action and hence an adjective should be used as the identifing term. The term used for the corresponding criterion should be a noun.
- Acceptable; the criterion is acceptability. A plan is acceptable if it accomplishes the given mission or purpose and it is deemed worth the risks. Acceptability determines whether a contemplated course of action (COA) is worth the cost of manpower, materiel, and time involved; and is consistent with legal constraints. Synonyms for acceptable include: desirable, expedient, tolerable, satisfactory. The dictionary defines acceptable as satisfactory or adequate.
A partial list of evaluation questions linked to acceptability include:
a) How likely is it that the plan will accomplish the mission and purpose?
b) Is the plan consistent with constraints and laws?
c) Is the cost of manpower, materiel, and time involved tolerable?
d) Is the level of risk involved supportable?
- Adequate; the criterion is adequacy. This criterion evaluates the scope and concept of planned activities to determine if they are sufficient to accomplish the task assigned. Synonyms for adequacy include ability, sufficiency, utility, satisfactoriness, effectual, tolerable. According to the dictionary, adequate means 1) able to satisfy a requirement; suitable or 2) barely satisfactory or sufficient. In COA plan development, adequacy must be assessed in terms of completing the mission and accomplishing decision maker's intent.
A partial list of evaluation questions linked to adequacy include:
a) How likely is it that the scope and concept of the COA plan is sufficient to accomplish the assigned task?
b) Does the COA plan satisfy the minimum requirements of the mission/task?
- Complete; the criterion is completeness. A COA plan is complete if it answers what, where, when, why, and how. Synonyms for completeness are wholeness, intactness. AHD defines complete as having all necessary or normal parts. Complete suggests the final stage in a process of assembling parts into a whole.
A partial list of evaluation questions linked to completeness include:
a) Are all of the major elements of a COA plan specified?
b) Have the parts of a COA plan been assembled into a coherent whole?
- Distinguishable; the criterion is distinguishability. A COA plan is distinguishable if it is significantly different from other COAs. Synonyms for distinguish are differentiate, discriminate, recognition. The disctionary defines distinguish as to recognize as being different or distinct. Distinct means 1) not similar; different; unlike; or 2) well-defined; explicit; unquestionable. Distinguishability means the distinguishing elements of a COA should be enumerated and explained.
A partial list of evaluation questions linked to distinguishability include:
a) Is the COA plan significantly different than other COAs that are being developed?
b) Is the COA plan distinct and recognizable?
c) Is the COA plan well-defined?
- Feasible; the criterion is feasibility. A COA plan is feasible if it may be accomplished with the resources currently available or obtainable. Synonyms for feasibility include practicable, expedient (especially in terms of timely, wise, fitting). AHD defines feasible as 1) capable of being accomplished or brought about; practicable; possible. or 2) logical; likely.
A partial list of evaluation questions linked to feasibility include:
a) Is it likely that the COA plan can be accomplished with the resources currently available and readily obtainable?
b) Is the COA plan practicable and fitting given the situation?
c) How likely is it that the COA plan can be accomplished and brought about?
- Suitable; the criterion is suitability. A COA plan is suitable if it accomplishes the mission. Synonyms for suitability include fitness, timeliness, expedience, adequate. AHD defines suitable as appropriate to a given purpose or occasion.
A partial list of evaluation questions linked to suitability include:
a) How likely is it that the COA plan will accomplish the mission or task?
b) How appropriate is the COA plan for accomplishing the given mission or task?
- Variable or varied; the criterion is variety. A COA plan is variable if it can be modified or altered systematically. Synonyms for variety are difference and assortment. AHD defines variety as 1) the condition or quality of being varied or 2) a different kind, sort or form. The two meanings of variety suggest possibly different ways to interpret the criterion during COA plan development. The focus should be how different or varied a specific COA plan is from other COAs under consideration. Variety among COA plans in approach is desirable.
A partial list of evaluation questions linked to variety include:
a) Can the COA plan be modified or altered systematically?
b) Is the COA of a different kind or sort than other COA under development?
c) Doe the COA enhance the variety in the choice set?
The above discussion should provide a starting point for systematically evaluating proposed plans. The criterion variety should be subjected to additional scrutiny during analysis to insure perceived variety among COA is meaningful and useful. There is a conceptual overlap among some criteria and some testing of any evaluation protocol or rubric is desirable.
In general, I think the likelihood questions will be most useful for evaluating the overall quality of plans during the development phase of planning. A likelihood scale from 0%-100% or a scale from very unlikely to very likely may be equally useful. On a subjective probability scale, I would recommend defining 3 zones, a “red zone” where the likelihood is <= 50%, a “yellow zone” >50% <= 75%, and a green zone where the likelihood is > 75%. Evaluating the Yes/No questions may also be useful as well to remind the planner/decision maker of important issues. The “appropriate” scale should range from very inappropriate to very appropriate.
The American Heritage Dictionary of the English Language, Boston: Houghton Mifflin Co., 1969.
Joint Publication 3-30, Command and Control for Joint Air Operations, June 5, 2003.
Roget's International Thesaurus (3rd edition) New York: Thomas Y. Crowell Co., 1962.