Experimenter And Subject Artifacts Research Paper

1. The Nature Of Artifacts In Scientific Inquiry

The sociologist H. H. Hyman wisely cautioned researchers not to equate ignorance of error with the lack of error. Because all scientific investigation is subject tOverror, it is prudent to be aware of this, to expose the sources in an attempt to control them, and to estimate the magnitude of such errors when they occur (Hyman 1954). In the social and behavioral sciences, researchers study the conduct of participants (or ‘subjects’) to further our understanding of the complexities of human behavior. Some of the complexity of subjects’ behavior, however, results from specifiable but uncontrolled extraneous variables, often social in nature. The effects of such variables are conceptualized as a set of systematic errors (or ‘artifacts’) to be isolated, quantified, taken into account in interpreting the results, and, when possible, eliminated. Thus artifacts are not merely inconsequential effects in the research design, but may jeopardize the validity of the results. To be sure, today’s artifact could become tomorrow’s independent variable, as conditions discounted as ‘nuisance’ variables at one time may be redefined as substantive variables in their own right in some future study. For example, the placebo effect in medical research, where an inert substance or nonspecific treatment produces a ‘healing’ effect, can be defined as an artifact, but can also be exploited as a potential ‘treatment’ for chronic pain disorder (Turner et al. 1994).

Experimenter and subject artifacts, because they are conceptualized as systematic errors, are further distinguished from random errors. To explain the difference, Rosnow and Rosenthal (1997) envisioned an imaginary grocer who weighed the same bunch of grapes a number of times in a row. In an ideal world, no matter how precise the measurements, they would always come out a bit differently. Some of them would slightly overestimate the correct weight of the grapes, while other measurements would slightly underestimate the correct weight. Because the various overestimates and underestimates occur haphazardly (rather than in a patterned fashion), we can assume that the random fluctuations (or random errors) will cancel one another over the long run. Thus we can simply average the measurements to estimate the correct weight of the grapes. But suppose the grocer always weighed the grapes with his thumb on the scale, thereby introducing a systematic error by always tacking a few extra ounces onto the true weight. This is essentially the nature of artifacts, inasmuch as they tend to push measurements in the same direction, up or down, systematically causing the ‘average value’ to be consistently too large or too small. The problem for researchers is to determine the direction in which a particular artifact might push the results, to try to improve the accuracy of the measurements, and when possible, to eliminate the source of systematic error and therefore the artifact.

The discovery of artifacts in the social and behavioral sciences has a long history, but it was not until the 1950s and 1960s that investigation of experimenter and subject artifacts began in earnest (see Rosnow 1981 for a detailed discussion of the historical context and wider implications of this work). Claims about the existence of experimenter and subject artifacts were sometimes met with skepticism, but by the late 1960s there was ample evidence to show the magnitude of the artifact problem in social and behavioral research (see Rosenthal and Rosnow 1969). Seminal investigations conducted by R. Rosenthal and M. T. Orne, working independently of one another, provided the initial impetus behind a flood of relevant work in the 1960s and 1970s. On the assumption that typical subjects often try to penetrate the nature of the study in which they are participating, Orne proposed the expression ‘demand characteristics of the experiment’ (which he derived from K. Lewin’s concept of Aufforderungscharaktere to describe the ‘demand value’ of uncontrolled task-orienting cues in an experiment that exert an influence on subjects’ ideas about its purpose. About the same time, Rosenthal showed that the experimenter’s tacit hypothesis or expectancy can sometimes become a self-fulfilling prophecy of the subjects’ experimental responses. Well before this work appeared, however, there had been intriguing clues about the possible existence of artifacts caused by uncontrolled variables when sentient subjects are the objects of investigation.

2. Earlier Clues About Artifacts

An early indication that artifacts may be lurking in the investigative procedures used in animal research involved a horse named ‘Clever Hans.’ At the turn of the twentieth century, an eminent German psychologist, O. Pfungst (1911), carried out a six-week investigation of this remarkable horse owned by a mathematics teacher named von Osten. Hans, who responded by tapping his hoof according to a code taught to him by von Osten, was purported to be able to reason and to answer questions put to him in German, including solving arithmetic problems. It was unlikely that von Osten had fraudulent intent because he readily allowed others (even in his absence) to question Hans, and also did not profit financially from the horse’s talents. Thus it was possible to rule out intentional cues from von Osten as the reason for the horse’s cleverness. In a series of elegant experiments, Pfungst demonstrated that Hans’s accuracy diminished when: (a) he was fitted with blinders so he could not see his questioners; (b) the distance between Hans and his questioners was increased; or (c) the questioner did not know the answer. These results implied that Hans’s performance was due to something other than his capacity to reason. Pfungst found that Hans was responding to subtle cues given by his questioners, not just intentional cues, but unwitting movements and mannerisms. This classic research provided an object lesson in the susceptibility of behavior (even animal behavior) to unconscious suggestion.

A half century later, another influential development fostered the suspicion that human subjects behaved in special ways because they knew they were ‘subjects’ of investigation. This proposition, called the ‘Hawthorne effect,’ grew out of a series of human factors experiments begun in the 1920s by a group of industrial researchers at the Hawthorne works of the Western Electric Company in Cicero, Illinois (Roethlisberger and Dickson 1939). The experiments were intended to study how workers’ productivity and job satisfaction might be affected by specific workplace conditions (lighting, temperature, rest periods, and so on). The conclusion reached, however, was that increases in productivity were due not to the manipulated experimental conditions, but to the uncontrolled effects of being singled out for participation and to adventitious cues intrinsic in the experimental situation. More recently, this research has been taken to task because of ambiguities and irregularities that make it difficult to interpret; it is also charged that secondary accounts of the research were tarnished by sweeping generalizations embroidered by overly zealous authors (see, e.g., R. Gillespie’s chapter in Morawski 1988, and also Parsons 1974 for an interesting behavioral interpretation of previously unreported circumstances of the Hawthorne research). Nonetheless, the ‘Hawthorne effect’ entered into the vocabulary of the social and behavioral sciences as implying a kind of placebo effect in social research with human subjects (see Sommer 1968). As a consequence, researchers were advised to employ ‘Hawthorne control groups,’ which received special attention or a meaningless task. Presumably, by examining obtained differences in performance between the Hawthorne control group and a no-treatment control group, any Hawthorne effects should be detectable.

Another important development, which went largely unnoticed for many years, involved a conceptual advance. S. Rosenzweig (1933) published an insightful critique of uncontrolled aspects of psychology experiments. In particular, he identified three distinct sources of artifacts, which he theorized were associated with (a) the experimenter’s ‘observational attitude’; (b) the subjects’ motivational states; and (c) the experimenter’s demeanor when interacting with the subjects. Rosenzweig sketched procedures that he thought might obviate these ‘errors,’ such as the use of ‘simple deceptions’ to distract subjects from the true purpose of the experiment. He cautioned, however, that it might be unclear whether the experimenter or the subject was the ‘true deceiver,’ because subjects typically carry on a train of thought in which they attempt to guess the purpose of the experiment, and then model their own behavior accordingly. Interestingly, early psychologists referred to their subjects as ‘reagents,’ implying that they resembled the ingredients that chemists mixed in clean test tubes to produce predictable reactions. Rosenzweig argued that the ‘test tubes’ (i.e., research settings) and ‘reagents’ (the subjects) of psychological researchers were sullied by the needs, anxieties, self-deceptions, and intentions of human beings who knew very well that their behavior was being scrutinized as part of a scientific investigation.

3. The Role Of Experimenter Artifacts

Rosenthal (1966) divided experimenter artifacts into two general classes, called ‘noninteractional’ and ‘interactional.’ The noninteractional kind operate, so to speak, in the mind, in the eye, or in the hand of the scientist, but do not directly affect the subjects’ responses. Within this class are three subtypes, known as observer, interpreter, and intentional effects. ‘Observer effects’ refer to overestimates or underestimates of results during the observation and recording phase of an experiment, which are controlled by the use of mechanical recording devices or by independent replications to discover if similar results are observed. ‘Interpreter bias’ refers to an error in the interpretation of data, which is ostensibly controlled by allowing other scientists to analyze and interpret the original data for themselves (see also MacCoun 1998). ‘Intentional effects’ imply that the data were fabricated or fraudulently manipulated. Such distortions are devastating to science because they undermine the integrity of the literature on which the advancement of science depends; safeguarding against dishonesty is presumed to be the ethical responsibility of every scientist.

Interactional artifacts, the second general class, are those due to extraneous experimenter-related variables that can directly influence the reactions of the research subjects. These include ‘biosocial effects’ (biases due to biological or social attributes of researchers, such as the sex, age, ethnicity, and race of the researcher), ‘psychosocial effects’ (e.g., the researcher’s personality and comportment), ‘situational effects’ (biases attributable to uncontrolled aspects of the setting), and modeling effects (i.e., the example set by the investigator in the role of a ‘scientist’). Replications across settings, subjects, and experimenters are used to isolate interactional artifacts, although procedural and sampling differences may not be easy to interpret. Moreover, scientists, like all humans, are susceptible to the biases imposed by human limitations of perception and cognition.

One additional interactional artifact is well known in the social and behavioral sciences, called the ‘experimenter expectancy effect’ by Rosenthal, its discoverer. It has spawned a large number of studies (Rosenthal and Rubin 1978), with various applications in education, the law, and clinical practice (see Blanck 1993). An ‘experimenter expectancy bias’ means that the experimenter’s hypothesis leads unintentionally to behavior toward the subjects that, in turn, increases the likelihood that the hypothesis will be confirmed (reminiscent of the Clever Hans phenomenon). One solution to this problem is to keep experimenters unaware of (‘blind’ to) which subjects are to receive the experimental treatment and which the control treatment. If the experimenters do not know what treatment the subject has received, then they cannot accurately communicate expectancies about the nature of the particular treatment. The necessity of keeping experimenters blind is well recognized in randomized drug trials, for example. In fact, no drug trial is taken completely seriously unless it has followed elaborate ‘double-blind procedures,’ in which neither the subjects nor the experimenters know who is in the experimental and who is in the control group.

4. The Role Of Subject Artifacts

In focusing on subject artifacts, researchers have generally been interested in the role motivations that guide the research subject’s behavior. Orne (1962) described the role of the ‘good subject,’ that is, the participant who is particularly attentive to, and compliant with, demand characteristics. In one fascinating demonstration study, Orne asked the subjects to add hundreds of thousands of two-digits numbers— which they kept doing until finally the experimenter gave up! Even when the subjects were told to tear each worksheet into a minimum of 32 pieces before going on to the next, they persisted. Orne explained this behavior as the ‘role enactment’ of individuals who reasoned that, no matter how trivial and inane the assigned task seemed to them, it must have some important scientific purpose or they would not have been asked to participate in the first place. Thus, Orne argued, they complied with the demand characteristics of the experiment in order to ‘further the cause of science.’

Not all research participants are willing to play the ‘good subject’ role, but Rosenthal and Rosnow (1975) reported that many volunteers for research participation are likely to conform to this expectation. Volunteers for research are also usually better educated, higher in social class status, more intelligent, more approval-motivated, and more sociable than non- volunteers. These and other demographic and personality characteristics may result in ‘volunteer subject bias’ in that the results of studies using subjects who specifically volunteered may not be simply generalizable to the population at large. For example, standardizing the norms of an intelligence test on typical volunteer subjects would produce biased estimates of true values in a more general population. Rosenthal and Rosnow described some strategies for improving recruitment procedures and the generalizability of results, and also logical procedures for estimating the direction of volunteer subject biases.

Other subject motivations may be operating, however, including what M. J. Rosenberg called ‘evaluation apprehension,’ referring to an anxiety-toned state in which subjects worry that the experimenter plans to evaluate an aspect of their psychological competence. As a result, they develop their own hypotheses about how to win the experimenter’s approval or avoid disapproval. Experiments in which evaluation apprehension appears likely to occur are those that contain an element of surprise or have an aura of mystery. The more explicit the cues, the more control the experimenter has in granting positive evaluation, or the less effortful the subjects’ responses, the greater may be the resulting response bias due to subjects’ evaluation apprehensions (see Rosenberg’s discussion in Rosenthal and Rosnow 1969). One solution to this problem is to ensure the confidentiality of subjects’ responses, on the assumption that individual subjects will then be less apprehensive, and more forthcoming in their responses. In research on sensitive topics, such as studies of sexual behavior, self-reported data nevertheless may be suspected of measurement error and participation bias (see Catania et al. 1990 for detailed discussion).

Other subject artifacts have been studied in the field of assessment, including a variation on evaluation apprehension known as ‘social desirability bias.’ In this case, the respondent to a personality inventory answers items in ways that seem likely to elicit favorable evaluation. One way to assess this problem is by means of a questionnaire that was specifically designed to tease out this response bias (Crowne and Marlow, 1964). Various other subject biases in assessment have been identified, and particular strategies have been developed to isolate or avoid those problems, such as the development of ‘lie scales’ to detect prevarication. Orne (see his chapter in Rosenthal and Rosnow 1969, pp. 143–79) suggested a strategy to ferret out demand characteristics by using ‘quasi-control subjects’ in addition to the usual control groups. Quasi-controls are individuals who are asked to step out of the traditional subject role; they reflect on the experiment in which they participated and suggest how their own experimental behavior was influenced by extraneous variables as opposed to only the controlled independent variable. It is now argued, however, that demand characteristics also exert an influence on behavior beyond the experimental setting, and should be recognized as a pervasive influence on all human interaction (see Orne and Bauer-Manley 1991, for the role of demand characteristics in the context of psychotherapy).

5. Artifacts And Ethics In Collision

Until World War II, the popular conception of science as an ‘endless frontier’ unencumbered by moral constraints remained largely unshaken. Currently, however, investigators in even the most benign areas of the social and behavioral sciences are required to comply with stringent ethical guidelines laid down in professional and statutory codes overseen by review boards (see Rosnow et al. 1993). This situation can at times lead to knotty moral dilemmas as researchers try to strike a balance between ethical accountability and the technical demands of sound scientific practice (see Blanck et al. 1992, Kimmel 1996, Schuler 1980). The effort to control for artifacts may, for example, come into collision with ethical imperatives such as informed consent and full disclosure. By way of illustration, in clinical trials using double-blind placebo-control designs, complying will full disclosure implies that the potential participants will be told they could be in a condition that will be deprived of possible benefits of the experimental treatment. Once they become aware of this fact, however, some subjects might surreptitiously share doses in the hope that this will increase the likelihood that each of them will receive at least some access to the experimental treatment. Such a situation could render any subsequent comparison of the experimental group and the ‘untreated’ control group meaningless. A strategy that can be used to avoid this problem, if there is an alternative treatment that is better than a placebo, is to provide the control group with this treatment rather than a placebo. This approach avoids the ethical bind of depriving control patients of any effective treatment, keeps the design double-blind, and focuses the research on other comparisons of practical and scientific importance (e.g., efficacy, relative cost, and side effects). As moral dilemmas proliferate in the sciences, the challenge faced by researchers is to expand our existing knowledge with valid data and yet abide by an evolving social contract that is responsive to current ethical sensitivities (see Rosnow 1997 for further discussion).

Bibliography:

1. Blanck P D (ed.) 1993 Interpersonal Expectations: Theory, Research and Applications. Cambridge University Press, New York
2. Blanck P D, Bellack A S, Rosnow R L, Rotheram-Borus M J, Schooler N R 1992 Scientific rewards and conflicts of ethical choices in human subjects research. American Psychologist 47: 959–65
3. Catania J A, Gibson D R, Chitwood D D, Coates T J 1990 Methodological problems in AIDS behavioral research: Influences on measurement error and participation bias in studies of sexual behavior. Psychological Bulletin 108: 339–62
4. Crowne D P, Marlow D 1964 The Approval Motive. Wiley, New York
5. Hyman H H 1954 Interviewing in Social Research. University of Chicago Press, Chicago
6. Kimmel A J 1996 Ethical Issues in Behavioral Research: A Survey. Blackwell, Oxford, UK
7. MacCoun R J 1998 Biases in the interpretation and use of research results. Annual Review of Psychology 49: 259–87
8. Morawski J G (ed.) 1988 The Rise of Experimentation in American Psychology. Yale University Press, New Haven, CT
9. Orne M T 1962 On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implication. American Psychologist 17: 776–83
10. Orne M T, Bauer-Manley N K 1991 Disorders of self: Myths, metaphors, and demand characteristics of treatment. In: Strauss J, Goethals G R (eds.) The Self: Interdisciplinary Approaches. Springer Verlag, New York, pp. 93–106
11. Parsons H M 1974 What happened at Hawthorne? Science 183: 922–32
12. Pfungst O 1911 Clever Hans (The Horse of Mr. on Osten). H. Holt, New York, [reissued 1965 by Holt, Rinehart & Winston, New York]
13. Roethlisberger F J, Dickson W J 1939 Management and the Worker. Harvard University Press, Cambridge, MA
14. Rosenthal R 1966 Experimenter Effects in Behavioral Research. Appleton-Century-Crofts, New York
15. Rosenthal R, Rosnow R L (eds.) 1969 Artifact in Behavioral Research. Academic Press, New York
16. Rosenthal R, Rosnow R L 1975 The Volunteer Subject. Wiley, New York
17. Rosenthal R, Rubin D B 1978 Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences 3: 377–86
18. Rosenzweig S 1933 The experimental situation as a psychological problem. Psychological Bulletin 68: 337–54
19. Rosnow R L 1981 Paradigms in Transition: The Methodology of Social Inquiry. Oxford University Press, New York
20. Rosnow R L 1997 Hedgehogs, foxes, and the evolving social contract in psychological science: Ethical challenges and methodological opportunities. Psychological Methods 2: 345–56
21. Rosnow R L, Rosenthal R 1997 People Studying People: Artifacts and Ethics in Behavioral Research. W.H. Freeman, New York
22. Rosnow R L, Rotheram-Borus M J, Ceci S J, Blanck P D, Koocher G P 1993 The institutional review board as a mirror of scientific and ethical standards. American Psychologist 48: 821–86
23. Schuler H 1980 Ethische Probleme psychologischer Forschung. Hogrefe, Gottingen, Germany
24. Sommer R 1968 Hawthorne dogma. Psychological Bulletin 70: 592–95
25. Turner J A, Deyo R A, Loeser J D, Von Korff M, Fordyce W E 1994 The importance of placebo effects in pain treatment and research. Journal of the American Medical Association 271: 1609–14