Naturalistic Decision-Making Research Paper

Naturalistic decision-making is a new field. The first studies were conducted in 1985 by Klein (1998), who examined decision-making by urban firefighters. This work was part of a new program initiated by the Army Research Institute because of the Army’s dissatisfaction with decision aids that had been developed on the basis of traditional decision theories. The dominant decision theories at that point grew out of economic models and emphasized rational choice to select the optimal decision option. Decision aids based on these models were laborious, requiring military commanders to make judgments of probability and value associated with a set of options. This process was incompatible with the thought processes typically used by experienced commanders and so was rejected. The Army decided that there must be a better way.

The ‘better way’ was to examine what experts do naturally when they make effective decisions under challenging conditions and to determine the basis for the quality of their decisions. Traditional research has sought to determine why human decision makers typically fall short of optimality and fail to use fully rational strategies. In contrast, NDM researchers recognized that experts in many fields, such as nurses, pilots, and military commanders, usually make decisions that achieve their goals, even if they are not optimal from a traditional perspective. The focus of NDM research is on understanding how experts make good decisions under difficult conditions, an under-standing that can then serve as the basis for training and aiding novices.

1. How Is Naturalistic Decision-Making Research Conducted?

Because of the NDM investigators’ interest in experts’ performance in real settings, the nature of the research settings and methods typically differ from those used in traditional decision research.

1.1 Research Settings

Harking back to its origins in military concerns, NDM research has been conducted in environments that share some common features that are quite distinct from the laboratory environment. They include:

1.1.1 Ambiguous Cues. The need to make a decision is prompted by cues that signal that a problem exists, such as a fire, a malfunction in an aircraft system, or a deterioration of a patient’s medical condition. These cues are sometimes ambiguous or subtle, or conflict with other cues, suggesting opposing courses of action. Thus, the nature of the problem may be difficult to determine, and expertise is needed to ‘read’ the cues and ‘size up’ the situation.

1.1.2 Dynamic Conditions. Conditions faced by decision makers in natural environments often change over time, and the human operator must update his or her understanding of the situation in order to respond appropriately. As conditions shift, goals also may shift, sometimes from maintaining the status quo to coping with an emergency. For example, when an urban fire commander arrives at a fire, his initial goal may be to extinguish the fire. However, when it appears that the roof is about to collapse, his goal shifts to evacuating all the firefighters.

1.1.3 High Risk. In most domains of concern to NDM researchers, the consequences of making a poor decision involve high costs, including possible loss of life or property. If an emergency room doctor misdiagnoses the presenting symptoms of an unconscious patient, the patient may die. Pilots’ misjudgments about the severity of weather in the vicinity of their destination airport can result in a crash. Fire-fighters’ failure to assess correctly the source and nature of a fire can result in application of the wrong fire suppressor, exacerbating rather than quelling a fire.

1.1.4 Time Pressure. Many decisions in NDM set-tings must be made quickly before a negative consequence occurs. Fires rage, patient conditions deteriorate, enemy aircraft or missiles approach swiftly. From the initial appearance of cues that signal a problem to the onset of serious consequences may be a matter of minutes. Often there is no time to conduct a full rational analysis of all possible response options, their probabilities, and their outcomes, which can take considerable time and computational power.

1.1.5 Team And/Organizational Contexts. Teams, rather than solo decision makers, are frequently the rule in NDM settings. While a single individual may have primary responsibility for the decision, other players provide critical information, alternative perspectives, and critiques of the decision. Teams usually function within an organizational context that establishes goals, norms, and procedures for operations, sets constraints on what is possible, and deter-mines what counts as a good decision.

Because NDM researchers are concerned with real-world settings that embody many of the above features, the methods they use also differ considerably from those employed by traditional decision researchers working in the lab.

1.2 Methods

Naturalistic decision-making research tends to be conducted in the field rather than in the laboratory. Thus, the predominant methods involve observations and interviews. Models of decision processes are built from the ground up, based on rich descriptions derived from multiple observations in field settings. Participants are interviewed about what they did and why. Laboratory experiments, on the other hand, typically involve top-down testing of hypotheses derived from theories. NDM research is young and has not yet been formalized to the point at which laboratory studies are common. However, there have been a number of studies to test predictions of NDM models.

A technique that has come to be associated with the NDM approach is called cognitive task analysis. This technique involves interviewing experts about critical decisions to find out what cues signaled the problem, what information was actually used, why particular options were considered and perhaps rejected, and why others were preferred. Highly structured interview protocols have been developed to assure reliability in procedures and responses. Several analyses of data collected using this method have shown it to be sound and reliable. Other approaches involve process tracing, which seeks to make explicit the knowledge used as experts work on a task and make decisions. In a team context, participants’ communication can pro-vide insight into how they shape the problem, what they think is important to attend to, their situation awareness, their plans and strategies, and often their rationales for choosing a course of action. These techniques may be applied to videotapes of individuals performing in real or highly realistic simulated situations, such as in high-fidelity aircraft simulators.

2. The Recognition-Primed Decision Model

Perhaps the most significant accomplishment within the NDM approach has been the development of a new decision process model. Based on his extensive investigations of fire fighters in both urban and wildfire situations, of neonatal nurses, and of military personnel, Klein (1998) proposed a model known as the Recognition-Primed Decision model (RPD). This model emphasizes the importance of recognizing cues in a situation that signal a particular type of problem. This recognition triggers retrieval of a response that has been associated in the past with a similar cue pattern and has led to a successful resolution.

In contrast, traditional analytic decision theories would require the decision maker to generate all possible options in that situation and then to evaluate each option in a concurrent manner, selecting the best option among them. In the recognition-primed model, the decision maker retrieves only one response option at a time and evaluates how well it satisfies the current goals. If the option is inadequate, then another option is retrieved and evaluated. Thus, options are considered in a serial rather than a concurrent manner. Note that many possible options are ignored in this strategy. It reflects what Herbert Simon called ‘satisficing,’ or choosing the first option that satisfies one’s goals and existing constraints, rather than searching for the best of all possible options.

The RPD model highlights several other findings that distinguish the naturalistic approach.

2.1 Importance Of Situation Assessment

The initial process involved in the RPD model is cue recognition and interpretation. A major distinction between naturalistic and traditional analytic decision approaches is that the NDM approach includes both a situation assessment process and choice of a response option, whereas traditional analytic approaches focus exclusively on the second component. In real-world situations, it is critical to recognize and understand the cues that signal a problem before choosing a response. If a problem has been assessed incorrectly, it is unlikely that an appropriate decision will be made. For example, pilots with an engine malfunction mistakenly thought the right engine was failing and shut it down. In fact, it was the left engine that had failed, and the plane crashed. In traditional laboratory studies, the situation assessment component of the process typically is minimal: the participant simply responds to the problem as designed and presented by the experimenter. A fixed set of options is presented from which the participant makes a choice.

2.2 Basis For Evaluating Options

Once the situation is understood and a response option has been retrieved, the option needs to be evaluated. In traditional analytic models, mathematical computation is used to combine preferences and probabilities associated with each option to choose the option that is mathematically optimal. For example, if you are buying a car, you may choose from many models. A variety of factors may influence your choice, e.g., cost, size, make, fuel consumption, and convenience features such as a sunroof. You need to decide how important each feature is to you and then evaluate how each car stacks up on each feature. The car you select presumably maximizes the combined values across all of the features. In naturalistic situations, experts typically use a different and less cumbersome process. The retrieved option is evaluated to determine if it will satisfy the decision maker’s current goal. If it does, it is chosen. If it does not, another option is retrieved and evaluated.

The basis for option evaluation in the NDM approach is mental simulation–the decision maker projects forward what is likely to happen if the candidate option is adopted. This includes identifying difficulties that may be encountered, likely results, and future decisions that may be required. If no significant impediments are foreseen, then the option is accepted. If limitations are found, a new option is retrieved or the original one is modified to get around the obstacle.

2.3 Role Of Knowledge

The findings just described reflect the importance of the decision maker’s knowledge to the decision-making process. In traditional laboratory studies of decision-making and other cognitive processes such as memory, knowledge has been treated as an unwanted source of variability. As such, efforts were made to design pared-down tasks that bypassed whatever knowledge the participants might have, thus eliminating the possibility that knowledge could give one participant an advantage over another. Hence, early studies of memory involved nonsense syllables or words such as ang or bwap. In naturalistic decision studies, knowledge is seen as the basis for making a decision, so effort is directed toward assessing participants’ knowledge and how that knowledge is used.

For example, knowledge is critical to situation assessment. Early studies have shown that performance differences between experts and novices can be traced to differences in their ability to recognize cue patterns, but not to differences in general problem solving or decision-making skills. For instance, when chess pieces on a board form a realistic pattern from a game, chess experts can remember the location of the pieces much better than novices following only a brief inspection. However, when the same pieces are randomly placed on the chessboard, the experts perform no better than novices. This illustrates that pattern recognition reflects experience with the game; it provides the expert with a basis for making more effective decisions about the next moves.

Likewise, knowledge plays a critical role in retrieval of appropriate options. One might question the serial option-evaluation strategy described above if the decision maker were randomly retrieving and evaluating all possible options. In fact, studies have shown that experts typically retrieve highly appropriate options first, based on past success in similar situations. For example, the quality of chess moves selected by both chess masters and intermediate players during a game was assessed by grandmasters. The players were asked to indicate the moves they thought of first, as well as subsequent moves they considered. More experienced players consistently thought of good moves as their first ones; subsequent moves were typically worse. This was not the case for novices, whose order of retrieval was less systematic.

Knowledge also serves as the basis for experts’ evaluation of options. Without knowledge of out-comes based on prior experience, one cannot know what outcomes are likely. Studies have documented that decision strategies used by experts in a field are quite different from those used by novices. Klein found that relatively inexperienced tank commanders frequently used analytic concurrent option-evaluation strategies, whereas more experienced commanders more often used a recognition-primed sequential option-analysis strategy. There is one exception to this finding, however: experts may revert to more general-purpose strategies if a situation is unfamiliar and they lack specific knowledge relevant to the decision.

3. Value Of The Naturalistic Approach

One may ask whether the recognition-primed decision model is appropriate to all circumstances or is limited to special cases. Basically, it is appropriate in any domain in which the decision maker has considerable experience, when time is limited and conditions are changing dynamically, and when problems are ill defined and goals are shifting, conditions that make an analytic approach difficult or impossible. An RPD approach represents an efficient use of knowledge and experience. It is less effortful and time consuming than most analytic approaches.

Analytic approaches are useful under certain conditions. These include situations in which the decision maker has little knowledge pertaining to the decision domain. Adequate time must be available in order to do a thorough evaluation of many options, when selecting the optimal solution is important. Analytic approaches are appropriate when conditions are stable, the parameters of the problem are clear, and all options are known. Analytic approaches yield a reliable, thorough outcome that can be justified to participants who may have competing goals and interests in the problem. For example, analytic approaches have been used to make decisions about where to site nuclear power plants. Analytic approaches are general-purpose tools that anyone can use when conditions permit.

4. Relations To Other Research Disciplines

Naturalistic decision researchers have borrowed techniques from several other disciplines to conduct their studies. They are closely aligned with much European research on work analysis, process tracing, and eco-logical psychology. Observational techniques have been drawn from ethnography, along with interview techniques for understanding the meaning of actions in context. Efforts to understand the role of expertise are grounded in cognitive psychology studies of experts, which involve verbal protocol analysis and process tracing.

In some cases the goals are similar but the approaches are different across related disciplines. For example, researchers from both the NDM and the judgment and decision-making (JDM) communities are interested in determining what cues are used to make decisions and their relative importance. While JDM researchers seek to determine cue weights, NDM researchers seek to know how the cues support situation assessment.

5. Methodological Issues

Because they deviate from traditional laboratory methods of science, NDM efforts have been challenged on several grounds. Most notable has been a challenge to the scientific rigor of field-based methods, namely, observations and interviews. NDM researchers maintain that a thorough description of the phenomena in situ is required in order to understand the role of contextual factors in decision processes and how knowledge is used. Field studies support the discovery phase of science in which new phenomena are identified and documented. Laboratory experiments may be suitable at a later point when specific hypotheses or model tests are desired.

Second, NDM research has not yet yielded formal models. Investigations are continuing in many do-mains and local models have been developed, but a formal general process model has not yet emerged. In part this reflects the newness of the field; in part it reflects the goals and interests of the investigators. Formal models typically are developed so that studies can be done to disconfirm the models and advance the theory. At the present stage, NDM researchers are still trying to understand the nature of phenomena of interest.

Third, one might question the use of expert models as the basis for prescribing training and decision aids. Experts do not always agree and experts are fallible, so how can models based on experts be used as a basis for training novices? First, the expert models are generalized, not based on a single expert. Second, NDM researchers emphasize the importance of building thorough domain knowledge in order to make better decisions, rather than training specific expert strategies. Training of novices is most often done via context-based scenario training, which emphasizes that (a) good decisions cannot be made on the basis of incomplete or incorrect situation assessment, and (b) option evaluation must take into account future projections, present constraints, and potential pitfalls, both of which depend on domain knowledge.

6. Future Directions In Theory And Research

At the beginning of the twenty-first century, efforts in the NDM community are moving in two directions. One is to formalize the theory so that predictions may be made and tested. The other is to expand the scope to explore issues related to adaptive decision-making and critical thinking. These include study of problem identification and structuring of decision problems, especially in dynamic conditions when goals shift and conflict. Ill-structured and unfamiliar problems may require analogical reasoning or creative problem solving, processes that go beyond the RPD approach. Finally, the roles of organizational and other con-textual features on decision processes need to be better understood.

7. Relevant Readings

To achieve a fuller understanding of the origins and development of the field of naturalistic decision-making, the reader is directed to the volumes resulting from a series of workshops that have been held since NDM research began. The first workshop was held in l989 and yielded the first book on naturalistic decision-making (Klein et al. 1993). Since then, four more workshops have been held. Each proceedings volume includes a critique by a researcher representing a traditional decision-making approach. Klein (1998) is a popular book that provides a deeper understanding of NDM and its practical applications, and Flin (1996) describes decision-making challenges to incident managers in high-risk industries like offshore oil production.

Bibliography:

1. Flin R 1996 Sitting in the Hot Seat. Wiley, New York
2. Flin R, Salas E, Strub M, Martin L (eds.) 1997 Decision Making Under Stress: Emerging Themes and Applications. Ashgate, Aldershot, UK
3. Friman H (ed.) 2000 How Professionals Make Decisions. CD-ROM. National Defense College, Stockholm
4. Klein G A 1998 Sources of Power. MIT Press, Cambridge, MA
5. Klein G E, Orasanu J, Calderwood R, Zsambok C E (eds.) 1993 Decision Making in Action: Models and Methods. Ablex, Norwood, NJ
6. Zsambok C, Klein G (eds.) 1997 Naturalistic Decision Making. Erlbaum, Mahwah, NJ