Protocol Analysis In Psychology Research Paper

1. Problems With Introspective Descriptions Of Thinking And Proposed General Solutions

Adults describe their subjective experience, such as dreams, memories, and mental slips, on a regular basis, so why are observations of one’s own experiences and thinking problematic as psychological data? The fundamental problem with descriptions of inner experience is that they are private and known only to the person who experienced them. There are no methods to verify the accuracy of these experiences independently. Consequently, protocol analysis is restricted to situations where the scientist has other types of information about the participant’s thinking that can be used to validate the verbal reports.

The other major objection to introspection is that the mere act of observation of one’s thoughts and experiences changes them. For example, when someone examines a single visual image appearing during thinking to search for specific associated details, this analysis tends to make the desired details emerge and thus changes the content and structure of the original image. This extended analysis also interrupts the normal flow of visual images, so it is impossible just to resume the thought process where it was interrupted. To minimize this type of interference in protocol analysis, research participants are instructed to think aloud while problem solving so they can concentrate on the experimental task, and let verbalization of thought be secondary.

2. Historical Background

Self-observation of thinking has a long history, which helps explain the controversies surrounding introspection, as well as the development of less controversial methods for eliciting verbalizations of thinking. The first systematic attempts to understand the structure of thinking by observing one’s own spontaneous thoughts have been traced back at least as far as the Greeks. Based on observation of his thoughts, Aristotle argued that thinking corresponds to a sequence of thoughts (see Fig. 1), where the brief transition periods between consecutive thoughts (see the arrows in Fig. 1) do not contain any reportable information.

Aristotle’s general account of thinking as a thought sequence has never been seriously challenged (Ericsson and Crutcher 1991). The controversies about introspection have all involved attempts to go beyond the immediately observable aspects. For example, Aristotle tried to infer the unobservable processes that control the thought sequence (see the arrows in Fig. 2). By examining his memory of which thoughts tended to follow each other, Aristotle concluded that previously experienced associations between thoughts were the primary determining factor. Philosophers in the seventeenth, eighteenth and nineteenth centuries (Ericsson and Crutcher 1991) asked questions that required more involved inferences and introspective analyses. For example, can every thought be described as an amalgam of memories? In order to evaluate these claims empirically, philosophers would typically relax and daydream, allowing their thoughts to wander. Once in a while they would try to freeze an image and inspect it carefully to assess its sensory modality and components. Based on such detailed observations, the famous British philosopher David Hume came to the surprising conclusion that thoughts were as detailed as original perceptions—we cannot have a visual image a printed page of text without imaging every letter on the page at the same time.

By the nineteenth century, philosophers generally rejected Hume’s claim and agreed that the process of analyzing a visual image changes its original appearance. For example, when a person has an image of a rose and then decides to inspect its petals, the whole mental image changes. Consequently, the process of examining mental images is fundamentally different from observing solid objects in the real world. A real rose can be looked at repeatedly from different directions without being changed, even its petals can be inspected with a looking glass without any change. In contrast, extended introspective analysis of a mental image changes its content and structure and cannot therefore be used to study thoughts as they emerge during spontaneous thinking.

In the beginning of the twentieth century, psychologists at the University of Wurzburg tried to minimize this form of reactivity by training introspective observers. These observers were given thought-provoking questions and asked to respond as fast as possible. Only after their answers had been given would the observers start to recall the thoughts that they had had while answering the questions. The observers were trying to recall the most basic elements of their thoughts and visual images. Most reported thoughts consisted of visual and auditory images, but some subjects claimed to have experienced thoughts without any corresponding imagery–imageless thoughts. The alleged existence of imageless thoughts had far-reaching theoretical implications and led to a heated exchange between the psychologists who claimed to have observed them and others who argued that these reports were artifacts of inappropriate reporting methods and theoretical bias of the observers. A devastating conclusion arose from this controversy: the existence of imageless thoughts could not be resolved empirically. This finding raised fundamental doubts about analytic introspection as a scientific method.

Reacting to the methodological and theoretical problems of the introspective method, psychologists redirected their research from conscious experience and thinking and towards observable Behavior, and focused on basic learning processes shared by adults, children, and animals. For example, some of the tasks involved learning the path from a starting location to a goal location in an arbitrary maze, where the benefits of prior experience and knowledge would be minimal. Similar learning tasks required participants to memorize ‘meaningless’ materials, such as nonsense syllables like XOK and ZUT, where the memory performance was measured objectively by scoring the sequence of recalled syllables for accuracy. The hypothesis that memory for nonsense syllables was based only on basic associations, and was later rejected in studies by cognitive psychologists showing that during memorization subjects reported thinking of words that were similar to the studied nonsense syllable, such as ‘cozy’ for KOZ and ‘payment’ for PYM (Prytulak 1971). When subjects were able to associate the nonsense syllable with a word, the speed and accuracy of memorization improved (Montague 1972), which provides validation for the reported thoughts.

The cognitive revolution in the 1960s renewed interest in higher-level cognitive processes and how thinking allows individuals to generate solutions to novel tasks. Cognitive theories describe how individuals are able to apply acquired knowledge and procedures to novel problems, such as mental multiplication of any combination of two two-digit numbers. Information processing theories (Newell and Simon 1972) proposed computational models that could reproduce the observable aspects of human performance on well-defined tasks through the application of explicit procedures.

One of the principle methods of the information processing approach is task analysis. Task analysis specifies the range of alternative procedures that people could use, in light of their prior knowledge of facts and procedures, to generate correct answers to a task. Let me illustrate how task analysis can be applied to mental multiplication. Most adults have only limited mathematical knowledge: they know their multiplication table and only the standard ‘pencil and paper’ procedure taught in school for solving multiplication problems. Accordingly, one can predict that they will solve a specific problem such as 36 × 24 by first calculating 4×36 = 144 then adding 20×36 = 720. More sophisticated subjects may recognize that 24 × 36 is equivalent to (30 + 6) × (30 – 6) and use the formula (a + b) × (a – b) = a² – b², thus calculating 36 × 24 as 30² – 6² = 900- 36 = 864. The choice of alternative procedures participants use to generate the answer can be inferred by the time needed and verbal reports of their thoughts during problem solving.

In conclusion, the theoretical and methodological controversies about verbal reports have never cast doubt people’s ability to recall part of their thought sequences. The controversies have centered around efforts to go beyond the sequence of thoughts (see Fig. 1), to analyze their detailed structure through introspection, and to infer the processes controlling the generation of new thoughts. In fact, all major theoretical frameworks concerned with thinking have advocated the use of verbally reported sequences of thoughts (Ericsson and Crutcher 1991). For example, the Behaviorist John B. Watson pioneered the use of ‘think aloud,’ and the gestalt psychologist Karl Duncker established it as a major method.

3. Protocol-analysis Methodology

The central assumption of protocol analysis is that it is possible to instruct subjects to verbalize their thoughts in a manner that doesn’t alter the sequence of thoughts mediating the completion of a task, and can therefore be accepted as valid data on thinking.

3.1 Elicitation Of Nonreactive Verbal Reports Of Thinking

Based on their theoretical analysis, Ericsson and Simon (1993) argued that the closest connection between thinking and verbal reports is found when subjects verbalize thoughts generated during task completion (see Fig. 2). When subjects are asked to think aloud, some of their verbalizations seem to correspond to merely vocalizing ‘inner speech,’ which would otherwise have remained inaudible. Nonverbal thoughts can also be often given verbal expression by brief labels and referents. For example, when one subject was asked to think aloud while mentally multiplying 36 by 24 on two test occasions one week apart, the following protocols were obtained:

(a) OK, 36 times 24, um, 4 times 6 is 24, 4, carry the 2, 4 times 3 is 12, 14, 144, 0, 2 times 6 is 12, 2, carry the 1, 2 times 3 is 6, 7, 720, 720, 144 plus 720, so it would be 4, 6, 864.

(b) 36 times 24, 4, carry the—no wait, 4, carry the 2, 14, 144, 0, 36 times 2 is, 12, 6, 72, 720 plus 144, 4, uh, uh, 6, 8, uh, 864.

In these two examples, the reported thoughts are not introspectively analyzed into their perceptual or imagery components, but merely verbally expressed and referenced, such as ‘carry the 1,’ ‘36,’ and ‘144 plus 720.’ Similarly, subjects are not asked to describe or explain how they solve these problems. Instead, they are asked to remain focused on solving the problem and merely to give verbal expression to those thoughts that emerge in attention while generating the solution under normal (silent) conditions.

If the act of verbalizing subjects’ thought processes does not change the sequence of thoughts, then subjects’ task performance should not change as a result of thinking aloud. In a comprehensive review of dozens of studies, Ericsson and Simon (1993) found no evidence that the sequences of thoughts (accuracy of performance) were changed when subjects thought aloud as they completed the tasks, compared with subjects who completed the same tasks silently. However, some studies showed that think-aloud subjects would take somewhat longer to complete the tasks— presumably due to the additional time required to produce the overt verbalization of the thoughts.

The same theoretical framework can also explain why other types of verbal report procedures consistently change cognitive processes. For example, when subjects are instructed to explain or carefully describe their thoughts, they are not able merely to verbalize each thought as it emerges, they must engage in additional cognitive processes to generate the thoughts corresponding to the required explanations and descriptions. This additional cognitive activity changes the sequence of mediating thoughts. Instructions to explain and describe the content of thought are reliably associated with changes in ability to solve problems correctly (Ericsson and Simon 1993).

In sum, after brief training in giving verbal reports, subjects can think aloud without any systematic changes to their thought process (see Ericsson and Simon 1993 for detailed instructions and associated warm-up tasks recommended for laboratory research). This means that subjects must already possess the necessary skills for verbalization of thoughts.

3.2 Validity Of Verbalized Information From Think Aloud

When adults are able to perform tasks while thinking aloud without sacrificing accuracy and speed, the verbalized information would almost have to reflect some aspect of the regular cognitive processes. By analyzing the information expressed as verbalized thoughts, it is possible to assess the validity of the verbalized information. In protocol analysis, the verbalized thoughts are compared with intermediate results generated by different strategies that are specified in a task analysis (Ericsson and Simon 1993). The sequence of thoughts verbalized while multiplying 24 by 36 mentally (reproduced above) agrees with the sequence of intermediate thoughts specified by one, and only one, of the possible strategies for calculating the answer.

Even when subjects think aloud with its close connection between thoughts and reports, correspondence between verbalized thoughts and intermediate products predicted from the task analysis is not perfect (see Fig. 2). The lack of one-to-one correspondence is due primarily to the fact that not all thoughts which pass through attention are verbalized and some processing steps (thoughts) may be short-circuited with acquired skill. However, there is persuasive evidence of validity for the thoughts that are verbalized (Ericsson and Simon 1993). Even if a highly skilled participant’s think-aloud report in the multiplication task only consisted of ‘144’ and ‘720,’ the reported information would still be sufficient to reject many alternative strategies, because these strategies did not involve generating both of the reported intermediate products. The general finding that a task analysis can identify a priori the specific intermediate products that are later verbalized by subjects during their problem solutions, provides the strongest evidence that concurrent verbalization reflects the processes that mediate the actual generation of the correct answer.

More generally, verbal reports are only one indicator of the thought processes that occur during problem solving. Other indicators include reaction times (RTs), error rates, patterns of brain activation, and sequences of eye fixations. Given that each kind of empirical indicator can be separately recorded and analyzed, it is possible to compare the results of such independent data analyses. In their review, Ericsson and Simon (1993) found that longer RTs were associated with verbal reports of a larger number of intermediate thoughts than those corresponding to shorter RTs. Furthermore, there seemed to be close correspondence between subjects’ thoughts and what information that they looked at—when subjects verbalized thoughts about objects in the environment they very frequently looked at them.

Finally, the validity of verbally reported thought sequences depends on the time interval between the occurrence of a thought and its verbal report, where the highest validity is observed for concurrent, think aloud verbalizations. For tasks with relatively short response latencies (less than 5–10 s), subjects are able to recall their sequences of thoughts accurately immediately after the completion of the task, and the validity of this type of retrospective reports remains very high. However, for cognitive processes of longer duration, the problems of accurate recall of prior thoughts increases, with a corresponding decrease in validity of the verbal reports.

3.3 Explanations Of Validity Problems With Other Types Of Verbal Reports

Protocol analysis specifies the constrained conditions necessary for valid, nonreactive verbalizations of thinking. Many of the problems with verbally reported information obtained by other methods can be explained as violations of protocol analysis methodology. The first problem arises when the investigators ask subjects to give more information than is contained by the their recalled thought sequences. For example, some investigators ask subjects why they responded in a certain manner. Sometimes subjects will have deliberated and their recalled thoughts will provide a sufficient answer, but typically the subjects need to go beyond any retrievable memory of their processes to give an answer. As subjects can only access the end products of their cognitive processes during perception and memory retrieval, they cannot report why only one of several logically possible thoughts entered their attention, so they must confabulate to answer such questions. In support of this argument, Nisbett and Wilson (1977) found that subjects’ responses to ‘why’ questions were in many circumstances as inaccurate as those given by other subjects who merely observed and tried to explain another subject’s Behavior.

Second, scientists are interested primarily in the general strategies and methods subjects use to solve a broad class of problems in a domain, such as mathematics or text comprehension. They often ask subjects to describe their general methods after solving a long series of different tasks, which often leads to misleading summaries or after the fact reconstructions of what subjects think they must have done. In the rare cases when subjects have deliberately and consistently applied a single general strategy to solving the problems, they can answer such requests easily by recalling their thought sequence from any of the completed tasks. However, subjects typically employ multiple strategies, and their strategy choices may change during the course of an experimental session. Under such circumstances subjects would have great difficulty describing a single strategy that they used consistently throughout the experiment, thus their reports of such a strategy would be poorly related to their averaged performance. Hence, general strategy descriptions are usually inadequate, even when reports after each trial are valid descriptions of thought sequences.

Similar problems have been encountered in interviews of experts (Hoffman 1992). When experts are asked to describe their general methods in professional activities, they have difficulties, and there is often poor correspondence between the described methods and their observed Behavior. However, when experts perform challenging representative problems from their domain of expertise while thinking aloud, their verbalizations consistently reveal a valid and informative account of their thought sequences during complex thinking and planning (Ericsson and Lehmann 1996). In sum, to obtain the most valid and complete trace of thought processes, scientists should strive to elicit those verbalizations that most directly reflect the subjects’ spontaneous thoughts generated while they perform tasks that are representative of the studied phenomenon.

4. Range Of Application

Protocol analysis has emerged as one of the principal methods for studying thinking in cognitive psychology (Crutcher 1994), cognitive science (Simon and Kaplan 1989), and Behavior analysis (Austin and Delaney 1998). As further evidence of its validity, protocol analysis now plays a central role in applied settings, such as in the design of surveys and interviews (Sudman et al. 1996) and user testing of computer software (Henderson et al. 1995). Finally, several interesting adaptations of verbal-report methodology are emerging in the study of text comprehension (Pressley and Afflerbach 1995) and education (Renkl 1997).

Bibliography:

1. Austin J, Delaney P F 1998 Protocol analysis as a tool for Behavior analysis. Analysis of Verbal Behavior 15: 41–56
2. Crutcher R J 1994 Telling what we know: The use of verbal report methodologies in psychological research. Psychological Science 5: 241–44
3. Ericsson K A, Crutcher R J 1991 Introspection and verbal reports on cognitive processes-two approaches to the study of thinking: A response to Howe. New Ideas in Psychology 9: 57–71
4. Ericsson K A, Lehmann A C 1996 Expert and exceptional performance: Evidence on maximal adaptations on task constraints. Annual Review of Psychology 47: 273–305
5. Ericsson K A, Simon H A 1993 Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA
6. Henderson R D, Smith M C, Podd J, Varela-Alvarez H 1995 A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics 38: 2030–44
7. Hoffman R R (ed.) 1992 The Psychology of Expertise: Cognitive Research and Empirical AI. Springer-Verlag, New York
8. Montague W E 1972 Elaborative strategies in verbal learning and memory. In: Bower G (ed.) The Psychology of Learning and Motivation. Academic Press, New York, Vol. 6, pp. 225–302
9. Newell A, Simon H A 1972 Human Problem Solving. PrenticeHall, Englewood Cliffs, NJ
10. Nisbett R E, Wilson T D 1977 Telling more than we can know: Verbal reports on mental processes. Psychological Review 84: 231–59
11. Pressley M, Afflerbach P 1995 Verbal Protocols of Reading: The Nature of Constructively Responsive Reading. L Erlbaum Associates, Hillsdale, NJ
12. Prytulak L S 1971 Natural language mediation. Cognitive Psychology 2: 1–56
13. Renkl A 1997 Learning from worked-out examples: A study on individual diff Cognitive Science 21: 1–29
14. Simon H A, Kaplan C A 1989 Foundations of cognitive science. In: Posner M I (ed.) Foundations of Cognitive Science. MIT Press, Cambridge, MA, pp. 1–47
15. Sudman S, Bradburn N M, Schwarz N (eds.) 1996 Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. Jossey-Bass, San Francisco, CA