Reflexivity Method And Evidence Research Paper

According to Hans-Georg Gadamer, the principal difference between the natural and human sciences, is not one of method but of the aims of knowledge (Gadamer 1987, Schutz, 1973a). Despite this fundamental difference in purpose, reflexivism and the ‘mainstream’—our shorthand for social scientists who embrace the unity of science position—share a significant number of methodological practices. These include a clear differentiation of premises from conclusions, respect for the canons of inference, recognition that some standards of validation must be established for data and source materials, differentiation of correlation from cause; the recognition of spuriousness problems when we rely on correlations, reliance on logic in establishing warrants for our assertions, and the belief that all findings can be contested. The deeper division about method is within the reflexivist community, between phenomenological, interpretevist, and hermeneutic approaches on the one hand and some postmodern or critical approaches on the other which do not subscribe to the above criteria. Jean-Francois Lyotard (1987), a representative of the latter tradition, asserts that ‘narrative knowledge certifies itself in the pragmatics of its own transmission without having recourse to argumentation and proof.

Because reflexivists are concerned with the social order and its reproduction through the behavior of actors, they have to be attentive to questions of context and meaning within which these actions take place. Context and meaning are principally shaped by language, and reflexivists accordingly recognize the importance of norms and problems of interpretation. This recognition has important implications for the constitution of facts and their evidentiary value, the problem of truth and its role in ‘theory building.’

1. First: What Is Evidence

Alfred Schutz (1973a) observed that all facts are created by cognitive processes. John Searle (1995) distinguished between ‘brute,’ or observable facts (e.g., a mountain), and ‘social,’ or intentional and institutional facts (e.g., an encyclopedia article). For the mainstream, facts are created by interpretation because they need to be detached from their context by intellectual abstraction. For reflexivists, they need to be considered in their particular setting. Neither the mainstream nor reflexivists can describe, let alone explain and theorize, without importing meaning. Reflexivists accept this as an epistemological given, not a corrupting intrusion to be ignored, controlled for, or explained away. Mainstream approaches, especially quantitative ones, often seem oblivious to the complications that arise from the impossibility of a pure empiricism. James Coleman has nevertheless shown that every measurement procedure that assigns a numerical value to a phenomenon has to be preceded by a qualitative comparison. While the assignment of numbers may permit powerful mathematical trans-formations, it is illicit to make such assignments if the antecedent qualitative comparison has not or cannot be completed (Coleman 1964). It is here that reflexivist and mainstreamers clash. The latter trying—illegitimately in the eyes of reflexivists—to apply the traditional measurement techniques and scientific operations to the social world, while the mainstream assert that even if the preconditions for successful measurements or causal modeling are not present, the ‘scientific method’ should serve as a regulative idea.

For the mainstream, the reality that matters most is usually that of the theorist who looks at the facts from an ‘objective’ point of view. Phenomenology, interpretivism and hermeneutics emphasize the subject’s story, but treat this evidence differently. Interpretivists may or may not situate themselves in their own account of the evidence, whereas hermeneuticans are consistently reflexivist. Hermeneuticans may treat the social structure as a more or less autonomous evidence, and may or may not become evidence them- selves as a result of changes they experience through the act of interpreting the subject.

The first move of the interpretivist is phenomenological. It is to gather inductively evidence that allows [the researcher] to evaluate the problem as understood by the actor (Ricoeur 1984, Berger and Luckmann 1966). This is not significantly different from applying a garden variety decision making approach to a social science problem. But the reconstruction of an actor’s perspective and understanding may be only the first step. Treatments of evidence acquire validity when the subjects themselves recognize that understanding as adequate (Schutz 1973b, p. 64). Researchers must read back their interpretations to those who have been interpreted. This standard of validation is not uncontested, as it too is based on imputation. Paul Ricoeur insists that such interpretations as ‘configurational acts’—might not be intelligible to the actors themselves, since they rep- resent constructions that no actor or witness could have put together when the events occurred (Ricoeur 1984, p. 157).

Why not just ask people what their statements and behavior mean? Interpretivists and hermeneuticans insist that unobservable social structures exist in all social contexts, and make the meaning of a peoples actions and words not theirs alone to interpret or control (Ricoeur 1971). The researcher’s task is to reproduce both the subject’s actions and the encompassing frames and social structures within which this action took place. Imagine asking a white woman why she moved her handbag from one side of her body to the other when a group of black men were approaching on the other side of the sidewalk. She would probably deny her action was racist. But irrespective of her actual intentions, her behavior may have reproduced a particular racist structure by typecasting those young black men as potentially dangerous felons.

Hermeneuticans go a step further and insist that the role of researchers must be included in accounts of actions. (That this is a big step away from and beyond phenomenology is obvious in Schutz’s (1973a, 1973b) uncritical treatment.) Researchers must comprehend the evidence through their own conceptions, while simultaneously realizing the perspectives of actors (1973a, pp. 104–11). Gadamer describes this process as placing oneself in the position of the other not to generate empathy or to better apply one s own criteria, but to attain a higher universality that overcomes, not only our own particularity, but also that of the other. This is the fusion of horizons of observer and subject (Habermas 1994). The hermaneutic assumption of intertexuality calls for an understanding of an actor’s story in its complete relationship to all other stories available in a site, as well as in its relationship to the observer.

2. Second: Where Is Evidence Found?

In a certain sense it is unfair to compare main stream and interpretivist evidence selection strategies, because they have different objectives. The mainstream wishes to test theories, interpretivism intends to generate understandings. Reflexivism wants to expand the field of inquiry. Instead of focusing only on ‘why’ questions that automatically privilege causal accounts, reflexivists insist that ‘how possible’ and ‘what questions are also legitimate objects of inquiry. ‘Why’ questions direct attention to the antecedent causes in a phenomenon, whereas ‘how possible’ and ‘what’ questions are what makes a phenomenon what it is. (For a fundamental discussion of the differences between ‘what’ and ‘why’ questions, see Alexander Wendt 1999.) Reflexivists who study sovereignty are not interested, at least not primarily, in what brought it about but rather in the generative logic of the concept, its various instantiations in institutions, and the constraining and enabling force of its norms that allow actors to reproduce domestic and international structures.

To the extent that mainstream rules are designed to promote the assessment of alternative theoretical accounts of a phenomenon, whether in a single case, or across some universe, they do not address interpretivist concerns. Because reflexivists do not usually engage in the competitive testing of theories, they are on the whole uninterested in the method of difference, crucial cases, and statistical significance. While the mainstream generally treats the knowledge of what constitutes a case as self-evident, reflexivists contend that case selection pre-judges ‘facts’ not yet in evidence.

This difference in orientation has implications for the choice of methods. While the mainstream stresses the deductive rigor in the models they employ, or the somewhat more problematic justifications for inductive inference, reflexivists stress the role of analogies and of conceptual frames that are antecedent to any causal argument or inductive inference. Given its choice of methods, the mainstream must consider counterfactual alternatives to which it rarely assigns ‘weight’ to the several causes considered relevant and apply the standard statistical controls to large enough samples whose cases are independent of one another. In practice, the mainstream decision rules that establish a warrant for a particular interpretation resemble more a procedure of sequential exclusion of alternatives than a ‘deductive’ demonstration or a generalizing inference.

Reflexivists do not believe it possible to establish complete analytical or theoretical control over a subject and consequently regard the ‘bivalence’ principle of logic (either something is so, or not so, and no third possibility exists) as generally inapplicable. The best our explanations can do is offer plausible accounts, even though some accounts might be more persuasive than others. Here a variety of criteria might interact and any proposed solution is not only ‘preliminary’ but contestable from the outset. By reversing the default drive of the mainstream theorists, who believe that a case or subject is theoretically identical until shown to be meaningfully different, reflexivists assume such differences among subjects until they can be persuaded otherwise. Reflexivists expect to find heterogeneity in their evidence; this is not a problem to be solved, but justification for emphasizing interpretative methods. They do not expect to find evidence that would allow predictions about future behavior within a case, let alone about a class of subjects across the historical universe. They argue that their stance allows for a more appropriate heuristics and thereby for a stronger warrant for their assertions. Reflexivist also believe that the traditional trade-off between rigor and detail misstates the underlying epistemological problem.

To elaborate on this last point, let us consider the problem of predictive success as a warrant for ‘science.’ Some mainstream approaches assert that the only warrant for a theory is predictive success, and that concern for the appropriateness of characterization is misguided. Reflexivists—scientific realists as well as logical positivists—counter this claim by pointing out that accurate predictions are sometimes made by demonstrably false theories, and that making predictions the only criterion for warranted knowledge discourages systematic search for mistakes when predictions fail. To that extent we are in danger of substituting ad hocery for disciplined inquiry. In short, reflexivists recognize control as a problem that is very hard to solve, and so reduce their claims about their theoretical constructs. The mainstream tends to see control as a technical problem, and so does not adjust its aspirations and expectations about evidence. (See, for example, the high level of attention ‘omitted variable bias’ receives in King et al. 1994, pp. 168–82, but how easily it is expected to succumb to simple techniques. On p. 172, we are reassured that it is possible to account for omitted variables because ‘fortunately, in most cases, researchers have considerable information about variables outside their analysis.’ Interpretivists recognize that such information is often impossible to obtain.)

3. Third: Evidence And Theory-Building

The secret to mainstream confidence in using evidence to test theories and develop generalizations for use in other places and times is the implicit assumption of uniformity. This assumes that things, concepts, and people are meaningfully identical across time and space. Votes are votes, wars are wars, and power is power—everywhere and for all time. (This problem has long been recognized in political science. In 1970, Giovanni Sartori pointed out the perils of doing comparative research. He warned against ‘stretching’ the meanings of variables across contexts in order to achieve some contrived generalization.) Some mainstream scholars acknowledge that unit homogeneity is often impossible to attain, but argue that understanding the degree of unavoidable heterogeneity helps to estimate the degree of uncertainty or likely biases of inferences (King et al. 1994, pp. 93–4). They allow the theory being tested to determine if a feature is meaningful. But what if the theory—and in social science this is virtually always the case—is not perfectly specified so as to avoid omitting a meaningful source of variance?

For reflexivists, this misstates the problem and so offers a misguided solution. If we knew which untheorized features of the field were relevant to understanding social outcomes, presumably we would have accounted for them in the a priori theory. Reflexivists see no solution to this problem because it is impossible, logically, empirically, and theoretically, to specify a causal model that could possibly account for everything that is excluded outside its boundaries, and everything that is purportedly causally effective within those boundaries. Nevertheless, reflexivism is subject to the same tension between homogenization and uniqueness. It objects to mainstream homogenization without often explicitly acknowledging that it also uses forms of homogenization to make sense of the social world. Intersubjectivity is really a way of homogenizing various parts of society or of historical developments through common webs of meaning. This is why common understandings and the practices based on institutional rules are so important. If people understand the same social practices in the same way, then they offer an example of sameness, of homogeneity or at least of comparability. It should be clear that we are not dealing with ‘constant conjunctions’ a la Hume, or antecedent (efficient) causes as in mainstream analysis. Common understandings or norms that are constitutive of practices (institutions) do not cause ‘behavior.’ They may provide weighty reasons, guide or inspire behavior, rationalize or justify it, or they may be ignored. They do not effect cause in the sense that a bullet through the heart causes death or an uncontrolled surge in the money supply causes price inflation (Kratochwil and Ruggie 1986, p. 767).

Reflexivists need a methodological technique to deal with both difference and similarity. (The difference between positivist and reflexivist homogenization of social reality is that positivism simply assumes homogeneity or uses prior theoretical categories to impose it on difference, while reflexivism theorizes about and uncovers homogeneity, treating it as a social phenomenon to be discovered and understood, not as one to be assumed and inflicted; see Habermas 1994, p. 108.) The image of a text, requiring both interpretation and an audience to which it is addressed and which brings to it certain pre-understandings that frame it, provides here their heuristic metaphor. Instead of viewing actors as simple ‘throughputs’ for norms and institutional rules, reflexivists point to the constraining and enabling functions of norms that cannot be reduced to a structure that determines behavior. Here the term ‘actor’ who has to act out or perform something rather than serve as a ‘mechanism’ or conveyor belt of forces is an alternative metaphor with considerable heuristic power. Reflexivist interrogation of mainstream homogenization is not always successful, but encourages us to guard against adducing identity where meaningful difference exists and vice versa.

The differences between the mainstream and reflexivism on this issue are easily observed when we consider how each handles the anomaly, a piece of evidence that confounds its theory or understanding of some phenomenon. The mainstream response is similar to how nineteenth century naturalists and anatomists dealt with physical ‘monstrosities.’ Rather than indicating systematic chaos or contingency, an interpretation with which reflexivists would feel comfortable, these anomalies instead provided opportunities to reaffirm the natural order and to define it with more precision. They were incorporated into more elaborate and powerful taxonomies (Ritvo 1997, p. 137). (Based on an empirical evaluation of the history of science, Pierre Duhem 1962) pointed out the power of prior theories, methodological commitments, and other elements of the scientific project to ensure the rejection of anomalous findings. Lakatos, Kuhn, and Feyerabend have also offered different accounts of why theories are so sticky in the face of countervailing evidence. As if these were not sufficient, there are excellent motivated and cognitive reasons to restore balance and persevere in ones prior beliefs.) While interpretivists try to place any given historical event within a narrative, within a contextualized bounded social structure, the mainstream places that same event within the framework of an opposition between the particular and the universal (Ricoeur 1984, p. 112). It treats the historical event according to whether it is part of a more general story to tell about the world, or an idiosyncratic ‘outlier’ that needs to be explained away.

Mainstream theories are sticky, in the Lakatosian or Kuhnian sense. Single pieces of contrary evidence do not and must not be used to change the observer’s account of the situation (Kuhn 1970). But, ironically, and perhaps only in this regard, reflexivism is truer to Popperian (logical) positivism than the mainstream which adopts the practical and sociological defenses of Lakatos and Kuhn. Popper offers an account of falsification that resembles that of an open market or Darwinian system. Explanations fail when they are no longer competitive, according to established falsification criteria, relative to the other available explanations. In other words, one’s understandings are far more responsive to new evidence, no matter how anomalous, than would be the case in either a Lakatosian or Kuhnian world. The product could be called a relative working truth, or, as Popper called it, ‘situational certainty’ (Popper 1979, pp. 78–81). It is a relative truth in that its validity exists only in comparison to other possible accounts. It is a ‘working truth’ in the sense that its validity is acknowledged to be a pragmatic convention, rather than an absolute fact that is simply part of the world.

Since we never test our theories against a reality, but only against other theoretical interpretations of the world, two additional problems arise. The first is that truth is no longer a property of ‘the world out there,’ but of theoretically informed assertions for which we need warrants. Consequently, the bivalence principle that links data and theories is no longer applicable, since most questions in science remain undecidable. This makes all scientific statements subject to revisions and introduces a second important element of pragmatics into theorizing. A statement, or ‘fact,’ is true only in the sense that can be accepted in daily practice until something more satisfactory comes along. It is a ‘truth’ in the sense that it is believed to be contextually valid, within the prior two constraints.

Reflexivism is doubly Popperian. It resists both theoretical priors, sociological and institutional consensus, and motivated and cognitive impulsion toward premature closure. It treats every anomaly as if it is, at least potentially, meaningful. (Of course, Popper was never the naıve falsificationist who argued that a single disconfirmation meant the abandonment of the imperfect theory. Disconfirmation had to be accompanied by an alternative theory that could account for all that the failed theory had accomplished, and then some.) The mainstream, on the contrary, rejects treating every disconfirming instance as a serious challenge, instead appearing to don the protective belts offered by a Lakatosian research program, or the Kuhnian mantle of normal science toiling on within a settled paradigm.

Bibliography:

1. Berger P L, Luckmann T 1966 The Social Construction of Reality. A Treatise in the Sociology of Knowledge, 1st edn. Doubleday, Garden City, NY
2. Coleman J S 1964 Introduction to Mathematical Sociology. Free Press of Glencoe, New York
3. Duhem P 1962 The Aim and Structure of Physical Theory. Atheneum, New York
4. Gadamer H-G 1987 Foreword. In: Baynes, Bohman, McCarthy (eds.) Truth and Method, After Philosophy, 2nd edn. MIT Press, Cambridge, MA, p. 340
5. Habermas J 1994 On the Logic of the Social Sciences. MIT Press, Cambridge, MA, pp. 108, 151
6. Lyotard J-F 1987 The postmodern condition. In: Baynes K, Bohman J, McCarthy T (eds.) After Philosophy. End or Transformation? The MIT Press, Cambridge, MA, p. 80
7. King G, Keohane R O, Verba S 1994 Designing Social Inquiry: Scientific Inference and Qualitative Research. Princeton University Press, Princeton, NJ, pp. 93–4; 168–82
8. Kratochwil F V, Ruggie J G 1986 International organization: a state of the art on the art of the state. International Organization 40: 753–76
9. Kuhn T 1970 The Structure of Scientific Revolutions. 2nd. edn. University of Chicago Press, Chicago
10. Popper K 1979 Objective Knowledge: an Evolutionary Approach. Oxford University Press, Oxford, UK, pp. 78–81
11. Ricoeur P 1971 The model of the text: meaningful action considered as a text. Social Research 38(4): 534
12. Ricoeur P 1984 Time and Narrative. University of Chicago Press, Chicago, Vol. 1, pp. 129–130, 157
13. Ritvo H 1997 Platypus and the Mermaid, and Other Figments of the Classifying Imagination. Harvard University Press, Cambridge, MA, p. 137
14. Sartori G 1970 Concept misformation in comparative politics. American Political Science Review 64(4): 1033–53
15. Schutz A 1973a Common-sense and scientific interpretation of human action. In: Natanson M (ed.) Collected Papers, The Problem of Social Reality. Martinus Nijhoff, The Hague, Vol. I, pp. 6; 36–8; 104–11
16. Schutz A 1973b Concept and theory formation. In: Natanson M (ed.) Collected Papers, The Problem of Social Reality. Martinus Nijhoff, The Hague, p. 64
17. Searle J R 1995 The Construction of Social Reality. The Free Press, New York
18. Wendt A 1999 Social Theory of International Politics. Cambridge University Press, Cambridge, UK