Counterfactual Reasoning Research Paper

1. The Unavoidability Of Counterfactual Reasoning

The ferocity of the critics is a bit unnerving. Moreover, they are certainly right that counterfactual inference is often dauntingly difficult. But they are wrong to say that one can avoid counterfactual reasoning at acceptable cost. And they are wrong that all counterfactuals are equally absurd because they are equally hypothetical (Fisher 1970). Analysts can avoid counterfactuals only if they eschew all causal inference from historical data and limit themselves to strictly noncausal narratives of what actually happened (no smuggling in causal claims under the guise of verbs such as ‘influenced,’ ‘responded,’ ‘triggered,’ ‘precipitated,’ and the like). Putting to the side the question of whether any coherent and compelling narrative can be noncausal, this prohibition would prevent observers from drawing exactly the sorts of lessons from history that scholars and policy makers regularly draw concerning such topical areas as the best ways to encourage economic growth, to preserve peace, and to cultivate democracy. Without counterfactual reasoning, how could observers know whether state intervention accelerated or retarded growth in a set of developing countries, whether deterrence policies pre-vented or provoked aggression, or whether the loan policies of international institutions such as the World Bank reduced the risk of financial contagion or increased the risk of moral hazard? In brief, counter-factual reasoning is a prerequisite to any form of learning from history. To paraphrase Robert Fogel’s (1964) reply to the critics of counterfactual history in the 1960s, everyone does it and the alternative to an open counterfactual model is a concealed one.

This research paper surveys the diverse roles that counter-factual arguments play in policy analysis and in political debate. A useful place to begin is by clarifying what is meant by counterfactual reasoning. A reason-ably precise philosophical definition is that counterfactuals are subjunctive conditionals in which the antecedent is known or presumed for purposes of argument to be false (Goodman 1983). As such, an enormous array of politically consequential arguments qualify as counterfactual. Consider the following rather representative sample of counterfactuals that have loomed large in recent policy controversies:

If Boris Yeltsin had followed the fiscal and monetary advice of the advocates of shock therapy policies in early 1992, Russian inflation in 1993 would have been a small fraction of what it was;

If the USA had not dropped atomic bombs on two Japanese cities in August 1945, the Japanese would still have surrendered roughly when they did;

If it were not for the affirmative action programs that have existed in the USA since the 1970s, the black middle class would have grown much more slowly than it did;

If Reagan had not adopted such a hardline posture toward the USSR in the early 1980s, it is very unlikely that the USSR would have been as conciliatory as it was in the late 1980s;

If the liberal social welfare policies of the 1960s had not implicitly encouraged single parenthood, the percentage of out-of-wedlock births in the USA in the 1980s would have been much lower than it was.

2. Distinguishing Sound From Unsound Counterfactuals

What standards of evidence and proof might be used for judging the quality of the counterfactual arguments that underlie so many causal claims in policy debates? In a comprehensive review of the methodological and philosophical literature bearing on counterfactual inference, Tetlock and Belkin (1996) identify six classes of criteria for making such judgments. These are outlined below.

2.1 Clarity

Good counterfactual arguments should specify and circumscribe the independent and dependent variables (the hypothesized antecedent and consequent). This initial recommendation might strike readers as obvious. Like actual experiments, thought experiments should manipulate one cause at a time, thereby isolating pathways of influence. Although excellent advice in principle, implementing it is often deeply problematic. As Jervis (1997) documents, there is no way to hold ‘all other things equal’ when we perform thought experiments on social systems that are densely interconnected. To invoke the terminology of experimental design, the ‘independent’ variable in inter-connected systems cannot be manipulated without creating ripple effects that alter the values taken on by other potential causes in the historical matrix, thereby creating ‘confounding’ variables that render the interpretation of the original thought experiment problematic. For example, if Boris Yeltsin had lived in a Russia in which he felt it was feasible to pursue shock therapy policies in 1992, there is a good case to be made that this hypothetical Russia differs from the actual Russia in a host of other ways (less corruption and reduced influence of special interests) that might also have been more conducive to better economic performance.

2.2 Logical Consistency Or Cotenability

Good counterfactual arguments should specify the connecting principles that link the antecedent with the consequent with sufficient precision that it is possible to gauge whether those connecting principles are cotenable with the original antecedent. Every counter-factual claim represents a condensed or incomplete argument that requires connecting principles that sustain, but do not imply, the conditional assertion. When explicitly articulated, these connecting principles are often complex, even in the case of such deceptively simple counterfactuals as ‘if the match had been struck, it would have lighted.’ The connecting principles specify, within reasonable limits, everything else that would have to be true to sustain the counterfactual, including the necessary amount of friction generated by the contact of the match against the surface, the chemical composition of the match, the absence of water, and the presence of oxygen.

These usually invisible connecting principles are sometimes in direct tension with the historical antecedent that has been altered in the counterfactual. One example is Jon Elster’s (1978) critique of Robert Fogel’s (1964) counterfactual assertion that if the railroads had not existed, the American economy in the nineteenth century would have grown only slightly more slowly than it did. Elster did not show that Fogel was wrong but he did show that it is nonsensical to postulate as a supportive connecting principle the proposition that the internal combustion engine would have been invented earlier in a USA without railroads because that proposition presupposes a theory of technological innovation that undercuts the original antecedent. If we have a theory of innovation that permits the invention of cars 50 years earlier, why does it also permit blinding people to the potential of railroads? In a similar vein, some critics have taken to task John Mueller (1988)for his counterfactual claim that, even in the absence of nuclear weapons, the Cuban missile crisis would not have escalated into war. This is an even more straightforward violation of cotenability of antecedent and connecting principles. Why would the USSR go to all the trouble of placing conventionally armed intermediate-range missiles in Cuba? Why take so large a risk for so small an advantage?

Taken together, the previous two standards—the logical clarity and cotenability standards—are helpful for screening out ambiguous and oxymoronic counterfactuals. But they define a bare minimum for quality-control purposes. Additional, historical and theoretical standards are needed for winnowing out other forms of ‘what-if’ foolishness.

2.3 The Minimal Rewrite Rule

Good counterfactual arguments should specify antecedents that could possibly have been implemented given the political and economic constraints at work in the situation. As shall soon be evident, there are occasions when scholars and policy analysts want to override this rule. However, historians do frequently apply it and many policy makers sympathize. What is the point of wondering about the effects of a policy that never had the remotest chance of being implemented? There is indeed little point in doing so if one is interested solely in judging whether policy makers selected the best of the politically feasible options at that historical moment. But there is good reason for doing so if the goal is to delineate the conditions under which particular policy interventions are likely to have particular effects. For instance, political leaders in industrialized countries during the Great Depression may have correctly surmised that they had little choice but to resort to protectionist policies. To suppose that they could have acted otherwise may be historically naive. But from the standpoint of drawing useful policy lessons for the future, it is important to assess to what degree the economic downturn of the 1930s was exacerbated by the proliferation of beggar-thy-neighbor tariffs that thwarted application of the logic of comparative advantage.

2.4 Theoretical Consistency

Compelling counterfactuals should rest on connecting principles that are consistent with well-established theoretical generalizations relevant to the hypothesized antecedent–consequent linkage. Otherwise, analysts cannot rule out counterfactuals that start from reasonable antecedents but end in far-fetched consequences by invoking preposterous principles of causality such as: ‘If Oswald had not shot Kennedy, then someone else would have done so, because Kennedy was astrologically fated to die by assassination’ or ‘If North Korea had conquered South Korea in 1950, the economy of the South would have grown even more rapidly than it actually did because of the wisdom of the policy of self-sufficiency of the Great Leader Kim Il Sung.’

Ideally, all counterfactual inferences could be grounded in extensively validated scientific laws of the sort drawn upon in the match-lighting conditional. Occasionally, this is possible: Witness physicist Richard Feynmann’s compelling thought experiment (with the assistance of a glass of ice water) that demonstrated the key role that cold-sensitive O-rings played in the explosion of the space shuttle Challenger in 1986 (Vaughan 1996). More often, however, policy analysts must rely on more fragile—and controversial—connecting principles drawn from the behavioral and social sciences.

2.5 Statistical Consistency

Compelling counterfactuals should rest on connecting principles that are consistent with well-established statistical generalizations bearing on the antecedent– consequent linkage. In many contexts, we rely not on theoretical laws but on statistical generalizations to fill in ‘what would have happened if this rather than that event had occurred.’ One obvious form such reasoning takes is reliance on base rates and patterns of co-variation. For instance, an observer might justify the counterfactual, ‘If Bill Clinton had lost the presidential election of 1992, he would have been disappointed,’ by observing in a two-by-two contingency table that when people fail to achieve a goal for which they have worked long and hard, the overwhelming majority experience disappointment, but when people do achieve their goals, there is markedly less disappointment. This discovery is hardly startling; statistical counterfactual arguments can, however, be made more controversial. Econometric modeling techniques can be used to reconstruct how the US economy might have performed if the invention of railroads had been delayed or whether slavery might have disappeared as a result of the operation of market forces (no Civil War necessary?). Or regression and time-series techniques might be used to infer how extensive out-of-wedlock births would have been in late twentieth century USA if welfare benefits had been lower or higher across states since the 1970s (inferences that rest on controversial but at least well-defined analytical assumptions).

2.6 Projectability

Good counterfactual arguments should lead to test-able implications that help to make sense of antecedent–consequent links in other contexts. Theory evaluation and counterfactual evaluation are inextricably entangled. Sound counterfactuals require sound theories that provide the law-like generalizations that fill in the missing data points in our thought experiments. How can social scientists judge, however, whether these law-like generalizations are robust enough to support counterfactual inferences? Here Nelson Goodman’s (1983) concept of project-ability is helpful. Goodman draws a sharp distinction between coincidental generalizations that just happen to be true at a particular time and place (and are therefore unprojectable) and robustly law-like generalizations that hold up over a range of circumstances and permit projection into the past and future. An example of a merely coincidental generalization is ‘All the coins in my pocket yesterday were silver.’ Nothing follows from this observation— certainly not ‘If this penny were in my pocket yesterday, it would be silver.’ The counterfactual fails because ‘if this penny were in my pocket yesterday,’ the reader would simply assume that the original generalization—‘all the coins in my pocket yesterday were silver’ was false. By contrast, a robustly law-like generalization—such as that oxygen is a necessary but not sufficient condition for fire—inspires confidence when the reader moves either backward in time (if there had been no oxygen, the Great Fire of London would not have occurred) or forward in time (if we cut off any future fire’s source of oxygen, the fire will expire).

Many social scientists view projectability as the acid test of scientific legitimacy. The same causal principles that allow observers to retrodict the past should make it possible to predict the future. Indeed, the strong Popperian form of this argument asserts that observers should take counterfactual claims seriously if and only if the law-like generalizations supporting the claims yield falsifiable forecasts. This classic philosophical argument has resurfaced in the recent methodological advice of King et al. (1994), who urge scholars to search aggressively for the observable implications of their causal constructs by regularly asking themselves, ‘If my argument is correct, what else should be true?’ Counterfactuals that are devoid of testable implications in the actual world leave historical observers marooned in hypothetical worlds of their own subjective making. Projectability, from this vantage point, stands as the pre-eminent criterion for judging the value of counterfactual speculation.

3. ‘Consciousness Raising’

There are special classes of counterfactual arguments that thoughtful people may not want to subject to the conventional historical and social science tests— counterfactuals that are designed primarily to open otherwise closed minds. Tetlock and Belkin (1996) dub such arguments ‘gadfly counterfactuals.’ Here it is, however, worth noting the intense skepticism of many psychologists. They doubt the capacity of the human mind to escape its preconceptions when it undertakes mental simulations of possible worlds. It is just too easy for political partisans to conjure up answers to ‘what-if’ questions that reinforce their prejudices: what could be more natural than for liberals to fill in the missing counterfactual control conditions of history with ideologically consonant scenarios (‘of course, subtract out the Reagan presidency and replace it with a two-term Carter presidency and a Mondale follow-up, and the USSR disintegrates pretty much in the same way it did’) and for conservatives to reach the opposite counterfactual conclusion. Indeed, such beliefs are often held with such conviction that they feel factual, not counter-factual.

But there are also psychological reasons for suspecting that people are not always fated to be prisoners of their preconceptions (Tetlock 1998, 1999). Here it is worth dwelling on some intriguing parallels between the mental process of simulating what would have happened in possible worlds and that of constructing computer simulations of natural or social phenomena (Cederman 1997). Both the computer software and the mental wetware exercises require making explicit previously implicit and ill-thought-through assumptions. In the course of running these computer or mental simulations, one can in principle learn a great deal about hitherto concealed gaps and contradictions in one’s knowledge. As Kahneman (1996) points out, mental simulations derive their persuasive force and power to surprise people by revealing unnoticed tensions between explicit, conscious beliefs and implicit, unconscious ones. In this sense, people discover aspects of their political belief systems in mental simulations that would otherwise have gone undiscovered. It is useful to distinguish three specific ways in which mental simulations can yield insights into one’s own thought processes: by revealing double standards in moral judgment, contradictory causal beliefs, and the influence of unwanted biases such as certainty of hindsight.

3.1 Counterfactual Morality Tales

Mental simulations can compel people to acknowledge embarrassing inconsistencies in their application of moral rules. The paradigmatic example is the identity-substitution thought experiment that manipulates either the perpetrator or victim of a policy act and asks the audience to contemplate whether they had the same emotional reaction to what actually happened as they would have to various hypothetical events. For example, ‘if Bosnians were bottle-nosed dolphins (or if Rwandans were white or if Chechnyans were Lithuanians …), the public never would have tolerated the slaughter of innocence so long.’ Insofar as the audience detects a discrepancy in their reactions to the scenarios, and insofar as the audience firmly believes that the mentally manipulated cause should be ir-relevant, the audience will deem the discovery of a differential emotional reaction to be a disturbing fact about themselves, that is, to be cognitively dissonant. Of course, not all political leaders fold when they are accused of moral inconsistency: The famously anti-Bolshevik Winston Churchill responded to those who probed how uncomfortable he felt about being aligned with Stalin in World War II by invoking a counter-factual argument of his own to the effect that if Hitler were to invade Hell, Churchill would find a way of making a favorable reference to the Devil in the House of Commons.

3.2 Counterfactual Consistency Probes

These mental simulations reveal contradictions be-tween causal beliefs that may have previously co-existed within a belief system. The paradigmatic example is the syllogism that traces the logical implications of one set of beliefs to the point where the contradiction becomes undeniable. For example, ‘if you really believe there is that much indeterminacy at the microlevel (factors that determine the outcome of battles, the success of firms, the policies of bureaucratic units of government), even invocations of the ‘miracle of aggregation’ do not allow you to argue for so strongly deterministic a position at the macrolevel (the outcomes of wars, the performance of economies, the political stands of governments).’ Or, ‘if you adopt an extreme structural-realist position that denies any significant role to domestic politics in shaping proper-ties of the international system, then you would have to belief that even if the USSR had been a democracy in 1945, the Cold War would still have occurred.’ In these cases, the thinker has the dissonance-reduction options of changing one or both sets of beliefs, introducing new cognitions that neutralize the con-tradiction, or disengaging from the simulation exercise by simply ignoring the contradiction.

3.3 Counterfactual Exercises As Means Of Stimulating The Imagination

Retrospective scenario generation is a form of mental stimulation in which the goal is to prevent premature cognitive closure induced by certainty of hindsight. Some scholars (e.g., Weber [1905] 1949) have long suspected, and cognitive psychologists have now demonstrated (e.g., Fischhoff 1975), that ‘outcome knowledge’ contaminates our understanding of the past. Once people learn the outcome of an event, they not only perceive that outcome as more likely ex post than they did ex ante (which might be defended as a rational Bayesian updating of subjective probabilities), they systematically misrecall their ex ante assessments as more consistent with that actual out-come than those assessments really were (the ‘I-knew-it-all-along’ effect).

Counterfactual thought exercises can check the ‘creeping determinism’ of certainty of hindsight. Asking people to think of how things could have worked out differently becomes a means of preventing the world that did occur from blocking our views of the worlds that might well have occurred if some antecedent condition had taken on a different value. Indeed, there is a ‘debiasing’ literature in experimental psychology that assesses the usefulness of encouraging people to imagine that the opposite outcome occurred. There is also a literature in literary studies on the concepts of sideshadowing, foreshadowing, and back- shadowing in narratives that makes a strikingly similar normative point (for reviews, see Tetlock and Belkin 1996). Sideshadowing calls attention to what could have happened, thereby locating what did happen in the context of a range of possibilities that might, with equal or even greater likelihood, have taken place instead. Sideshadowing serves as a valuable check on foreshadowing (the tendency, in extreme form, to reduce all past events to harbingers of the future) and back-shadowing (the even more insidious tendency to judge historical actors as though they too should have known what was to come).

4. Closing Remarks

The topic of counterfactual thought experiments in public policy leaves many social scientists feeling a bit uneasy, even defensive. To be blunt, it feels like epistemological slumming. There is, in the behavioral and social sciences, a prestige hierarchy of methods for drawing causal inferences. At the top of the pecking order is experimentation in which investigators can manipulate hypothesized causes and then either hold everything else constant (ceteris paribus) or randomize extraneous influences across treatment conditions. Experimental control of this sort is occasionally possible in the domain of public policy: There have been some well-known experimental attempts to assess the impact of various drug treatment programs and social welfare programs. By and large, however, experimentation is hopelessly impractical. We cannot rerun the tape of history to assess the causal impact of Gorbachev’s policies of glasnost and perestroika or to assess the impact of Reagan’s defense build-up or tax cuts.

Social scientists often resort to statistical control when experimentation is ethically or practically problematic. But statistical arguments invariably rest on counterfactual assumptions and are extraordinarily difficult to justify in many policy contexts. For example, what kind of regression or time series analysis will allow us to estimate the causal contribution of nuclear weapons to the ‘long peace’ between the USA and USSR between 1945 and 1991? There are too many confounding variables—a problem that can be alleviated but not eliminated through judicious selection of comparison cases and meticulous examination of decision-making protocols (Fearon 1991).

So where does this leave the field? Probably still feeling uneasy: Social scientists and policy analysts seem to be stuck with quite literally a third-rate method, i.e., counterfactual thought experimentation. The control groups exist—if ‘exist’ is the right word— only in the imaginations of the policy analysts who are left with the daunting task of trying to reconstruct how history would have unfolded if causal variables from the past had taken on different values from the ones that they actually did. The whole exercise starts to look hopelessly subjective, circular, and nonfalsifiable. What is to stop ideologues from simply inventing counterfactual outcomes that serve to justify their political biases and predilections? There appear to be large classes of policy questions for which experimental control is out of the question and statistical control is of limited usefulness (assuming that one can find suitable comparison cases and can reliably operationalize the key conceptual constructs). These questions are too important to ignore but apparently too difficult to answer in a fashion that transcends traditional ideological divisions.

Too often, the response to the dilemma is to embrace extreme solutions (Strassfeld 1992): either to reject all counterfactual arguments as fanciful suppositions, mere conjecture, and frivolous figments of policy analysts’ imaginations or to assume confidently that we know exactly what would have happened if we’d gone down an alternative path, sometimes going so far as to project several steps deep into hypothetical causal sequences. The former response leads to futile efforts to exorcize all counterfactuals from historical inquiry; the latter response leads at best to error (we ignore the compounding of subjective probabilities at our peril) and at worst to the full-scale politicization of counterfactual argumentation (as advocates claim carte blanche to write hypothetical histories that advance their ideological agenda).

This research paper has articulated a principled compromise between these extremes. On the one hand, it is necessary to acknowledge that counterfactual thought experiments inevitably play key roles in the causal arguments that animate many policy controversies. On the other hand, it is necessary to acknowledge that counterfactual thought experiments are often suffused with error and bias. But, that said, it is not appropriate to conclude that things are hopeless—that it is impossible to draw reasonable causal inferences from historical data when experimental and statistical control are not available. Rather, the preferred conclusion of this research paper is that disciplined use of counterfactuals—grounded in explicit standards of evidence and proof—can be enlightening in specific policy settings.

Bibliography:

1. Carr E H 1961 What is History? Knopf, New York
2. Cederman L E 1997 Emergent Actors in World Politics: How States and Nations Develop and Dissolve. Princeton University Press, Princeton, NJ
3. Elster J 1978 Logic and Society: Contradictions and Possible Worlds. Wiley, New York
4. Fearon J D 1991 Counterfactuals and hypothesis testing in political science. World Politics 43: 169–95
5. Fischhoff B 1975 Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance 1: 288–99
6. Fisher D H 1970 Historians’ Fallacies: Toward a Logic of Historical Thought. Harper and Row, New York
7. Fogel R 1964 Railroads and American Economic Growth: Essays in Econometric History. Johns Hopkins University Press, Baltimore, MD
8. Goodman N 1983 Fact, Fiction, and Forecast. Harvard University Press, Cambridge, MA
9. Hanson V 2002 One man saves the West. In: Tetlock P E, Lebow R N, Parker G (eds.) Unmaking the West: Exploring Alternative Histories and Counterfactual Worlds. Columbia University Press, New York
10. Jervis R 1997 Systems: Dynamics and Eff Princeton University Press, Princeton, NJ
11. Kahneman D 1996 Varieties of counterfactual thinking. In: Roese N J, Olson J M (eds.) What Might Ha e Been: The Social Psychology of Counterfactual Thinking. Erlbaum, Mahwah, NJ, pp. 408–24
12. King G, Keohane R O, Verba S 1994 Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton Uni-versity Press, Princeton, NJ
13. Mueller J 1988 The essential irrelevance of nuclear weapons: Stability in post-war world. International Security 13: 55–79
14. Strassfeld R N 1992 If … Counterfactuals in the law. George Washington Law Review 60: 339–416
15. Strauss B 2002 The resilient West: Salamis without Themistocles. Classical Greece without Salamis, and the West without Classical Greece. In: Tetlock P E, Lebow R N, Parker G (eds.) Unmaking the West: Exploring Alternative Histories and Counterfactual Worlds. Columbia University Press, New York
16. Tetlock P E 1998 Close-call counterfactuals and belief system defenses: I was not almost wrong but I was almost right. Journal of Personality and Social Psychology 75: 639–52
17. Tetlock P E 1999 Theory-driven reasoning about possible pasts and probable futures: Are we prisoners of our preconceptions? American Journal of Political Science 43: 335–66
18. Tetlock P E, Belkin A 1996 Counterfactual Thought Experiments in World Politics. Princeton University Press, Princeton, NJ
19. Vaughan D 1996 The Challenger Launch Decision. University of Chicago Press, Chicago
20. Weber M [1905] 1949 Objective possibility and adequate causation in historical explanation. In: The Methodology of the Social Sciences. Free Press, Glencoe, IL, pp. 163–88