Questionnaires and Subjective Expectations Research Paper

This research paper focuses on a single topic about questionnaires—how to elicit respondents’ subjective expectations of future events. For example, demographers ask women to predict their future fertility and political scientists ask citizens to predict their voting behavior in upcoming elections. Researchers use the responses to expectation questions to predict fertility, voting behavior, and other future events. They also use the responses to assess how individuals’ perceptions of the future are related to the decisions they make today. Subjective expectations of future events seem to be measured with increasing frequency. Yet the social and behavioral sciences have not formed a common view on how subjective expectations should be conceptualized and measured. A particularly intriguing controversy, described in this research paper, concerns the relative merits of verbal and probabilistic forms of questioning.

1. Subjective Concepts And Measures

A US National Research Council (NRC) panel study on Sur eying Subjecti e Phenomena has distinguished subjective from objective phenomena as follows (Turner and Martin 1984, p. 8):

Subjective phenomena are those that, in principle, can be directly known, if at all, only by persons themselves, although a person’s intimate associates or a skilled observer may be able to surmise from indirect evidence what is going on ‘inside.’ Objective phenomena are those that can be known by evidence that is, in principle, directly accessible to an external observer.

The NRC panel acknowledged that the sharp dichotomy just drawn is not always tenable, but the essential idea clearly has content: subjective phenomena are not amenable to external verification.

The social and behavioral sciences have a common interest in understanding subjective phenomena, but the various disciplines have not been able to agree on concepts or measures. The conceptual problem is to characterize what is going on ‘inside.’ A reader of social and behavioral science research encounters an enormous variety of subjective concepts in use, with concepts that are central to one discipline often playing no role in others. A modest sense of this variety can be obtained by scanning the introductory chapter of Turner and Martin (1984), which mentions affect, aspirations, attitudes, disappointments, dispositions, emotions, frustrations, fulfillment, happiness, hopes, intentions, motives, opinions, plans, preferences, quality of life, satisfaction, and values.

The measurement problem is to obtain empirical evidence on what is going on ‘inside.’ A reader of empirical research in sociology, social psychology, or political science finds much analysis of subjective data elicited from survey respondents through questionnaires. Yet a reader of anthropology or economics finds little reliance on questionnaire responses. Anthropologists, who tend to recoil from the tightly scripted structure of questionnaires, have preferred to elicit information through conversations. Economists, who tend to be skeptical of subjective data of any kind, have sought to infer subjective phenomena from observations of behavior and experiences.

To illuminate the diversity of perspectives on subjective phenomena, this research paper examines how different disciplines have used questionnaires to study subjective expectations about potential future events. How persons perceive the future is a shared concern of all social and behavioral science disciplines. Yet there is a wide spectrum of views on how expectations should be conceptualized and measured.

Section 2 examines the prevailing practice in economics of inferring expectations from realization data, and recalls early efforts to collect data on economic expectations through questionnaires. Section 3 describes qualitative attitudinal research as practiced by sociologists and social psychologists. Section 4 describes the growing body of research in economics and cognitive psychology eliciting subjective probabilities of future events. Section 5 considers the relative merits of qualitative and probabilistic modes of questioning. Section 6 poses directions for future research on subjective expectations. See Dominitz and Manski (1999) for a more detailed exposition of the subjects discussed below.

2. Inference On Expectations In Economics

Early in their careers, economics students are taught that a good economist believes only what people do, not what they say. Skeptical of subjective data, economists have sought to infer expectations of future events by combining data on past experiences with assumptions about processes of expectation formation. This is a daunting task. To infer expectations from data on past experiences, economists must speculate about what information persons possess and how they use the available information to form their expectations.

Consider, for example, the matter of income expectations. Economists often assume that persons use their own past incomes to forecast their future incomes. Perhaps so, but how do persons form expectations of future income conditional on past income? Economists often assume that persons have rational expectations; that is, they know the actual stochastic process generating their income streams. Perhaps so, but what is this stochastic process? The practice has been to specify the income process up to some parameters and use available data on income realizations to estimate the parameters. Authors differ, however, in their specifications. So the literature displays competing models of expectations formation, with no clear way of distinguishing among them. See Dominitz and Manski (1997a).

If inference on expectations from data on experiences is so challenging, why do economists not collect and analyze data on expectations? The answer lies at least partly in the history of a scientific controversy that began in the 1940s and persisted until the 1960s. In the 1940s, the US Federal Reserve Board began to fund an annual Survey of Consumer Finances, conducted by the University of Michigan Survey Research Center (SRC), that elicited qualitative assessments of expected household finances. A typical question took this form:

How about a year from now—do you think you people will be making more money or less money than you are now, or what do you expect?

The usefulness of responses to such qualitative questions was controversial and the Federal Reserve Board appointed a committee to assess their value. The Federal Reserve Consultant Committee on Consumer Survey Statistics (1955) issued findings that questioned the predictive power of the SRC expectations data. The negative findings of the Committee were challenged by SRC researchers, notably George Katona, a leading proponent of research on consumer attitudes and expectations (Katona 1957). A contentious conference on expectations data followed (National Bureau of Economic Research 1960). Then Juster (1964) reported an intensive study, drawing largely negative conclusions, of the usefulness of qualitative expectations data in predicting individual behavior.

The Federal Reserve–SRC controversy only concerned the usefulness of a certain type of qualitative expectations data in predicting consumer behavior. Academic economists, however, appear to have drawn the broader conclusion that all expectations data are suspect. Economists also failed to take note when Juster (1966) proposed elicitation of probabilistic expectations of consumer durable purchases and reported favorable empirical evidence. Exploration of this idea had to wait a quarter century, as described in Sect. 4.

3. Qualitative Attitudinal Research

The Federal Reserve–SRC controversy may have soured most economists on the usefulness of qualitative expectations data, but the collection and analysis of such data have long been important activities in other behavioral and social science disciplines. Many questions in common use have a structure similar to that of this General Social Survey (GSS) question eliciting expectations of job loss (Davis and Smith 1994):

Thinking about the next twelve months, how likely do you think it is that you will lose your job or be laid off—very likely, fairly likely, not too likely, or not at all likely?

This GSS question illustrates well a basic persistent problem that researchers face in interpreting qualitative expectations data—assessment of the inter-and intrapersonal comparability of responses. Respondents are often asked to report whether a specified event is ‘expected,’ ‘likely,’ or ‘probable’ to occur. Sometimes they are asked to report their strength of belief in the event on a numerical scale with undefined units. Do different respondents interpret these verbal phrases and numerical scales in the same way? Does a given respondent interpret them in the same way when asked about different events? Cognitive research does not give reason to think that responses should be or are comparable. Indeed, the available empirical evidence indicates that interpretations of qualitative expectations questions vary substantially between persons (see Sect. 5).

Concern about interpersonal comparability comes into even sharper focus when considering the events about which respondents are asked. Attitudinal researchers often ask respondents about vaguely defined future outcomes, as in this SRC question (Curtin 1982):

Now looking ahead—do you think that a year from now you (and your family living there) will be better off financially, or worse off, or just about the same as now?

How should a respondent interpret the phrase ‘better off financially?’ Do different respondents interpret it the same way? The data provide no way of knowing.

A second persistent problem in qualitative attitudinal research is that the coarseness of the response options limits the information contained in the responses. Consider, for example, the fertility question asked female respondents in the annual June Supplement to the Current Population Survey of the US Bureau of the Census:

Looking ahead, do you expect to have any (more) children? Yes/no Uncertain

The three response options can express little of the richness of the uncertainty that women may perceive about their future childbearing. Savage (1971, p. 795) called attention to the problem when he observed that ‘Yes, No, or Maybe is Not Enough.’ Juster (1966) hypothesized that a person facing a Yes No expectations question responds as would a statistician asked to make a best point prediction of a future random event. Manski (1990) formalized this hypothesis and derived an upper bound on the information about expectations a researcher can extract from responses to Yes/No questions.

4. Elicitation Of Probabilistic Expectations

A growing body of research in economics and cognitive psychology elicits expectations in the form of subjective probabilities. For example, the Survey of Economic Expectations uses this question to elicit expectations of job loss (Dominitz and Manski 1997b):

What do you think is the percent chance that you will lose your job during the next 12 months?

The National Longitudinal Study of Youth: 1997 Cohort of the US Bureau of Labor Statistics elicits the fertility expectations of teenage [girls boys] as follows:

What is the percent chance that you will become the [mom/dad] of a baby sometime between now and when you turn 20?

Probabilistic questions of similar structure are posed in other national surveys, such as the Health and Retirement Study (Juster and Suzman 1995) and the Bank of Italy’s Survey of Household Income and Wealth, as well as in small-scale surveys such as the one described in Quadrel et al. (1993).

The idea that probabilistic elicitation of expectations might improve on the traditional qualitative approaches of attitudinal research developed independently in economics and in cognitive psychology. In economics, the idea appears to have originated with Juster (1966), who recommended elicitation of purchase probabilities for consumer durable goods. Market researchers were immediately attracted to Juster’s proposal, and elicitation of purchase probabilities has since become a common practice in market research (e.g., Jamieson and Bass 1989). However, the idea that expectations might be elicited probabilistically from survey respondents did not draw serious attention within economics until the early 1990s, when economists began to re-assess the discipline’s self-imposed prohibition on the collection and analysis of subjective data.

Elicitation of subjective probabilities has a more continuous history of research in cognitive psychology. Psychologists have long conducted calibration studies assessing the accuracy of elicited expectations reports when the outcome is known. Lichtenstein et al. (1982) reviewed findings from calibration studies dating back to 1906. McClelland and Bolger (1994) updated the review with findings from 1980 through 1994. Whereas the older studies mostly examined the accuracy of experts (e.g., weather forecasters’ reported probabilities of precipitation), the more recent research typically analyzes the expectations of nonexperts, especially students situated in a cognitive laboratory.

5. Qualitative vs. Probabilistic Expectations

As this research paper is written, a consensus has yet to emerge on the relative merits in practice of qualitative and probabilistic elicitation of subjective expectations. However, there is reasonably widespread agreement that elicitation of probabilistic expectations has several a priori desirable features. Perhaps the most basic attraction is that probability provides a well-defined absolute numerical scale for responses. Hence there is some reason to think that responses may be inter- and intrapersonally comparable.

A second attraction is that some empirical assessment of the internal consistency and external accuracy of respondents’ expectations is possible. A researcher can use the algebra of probability (Bayes Theorem, the law of total probability, etc.) to examine the internal consistency of a respondent’s elicited expectations about different events. In those cases where probability has a frequentist interpretation, a researcher can compare elicited subjective probabilities with known event frequencies, as is done in calibration studies, and reach conclusions about the correspondence between subjective beliefs and frequentist realities.

A third consideration is the usefulness of elicited expectations in predicting prospective outcomes. Suppose that respondents have reasonably accurate expectations about the likelihood of future events. Then, as argued by Juster (1966), Savage (1971), and Manski (1990), numerical responses to probability questions should have more predictive power than categorical responses to qualitative expectations questions.

These a priori desirable features of probabilistic expectations data must be weighed against a set of unresolved issues. Perhaps the most basic concern is that there is no direct way to assess how well elicited subjective probabilities reflect respondents’ thinking. Of course this concern is not specific to elicitation of probabilities. Interpretation of responses to subjective questions of any type runs up against the generic problem that a researcher cannot directly observe a respondent’s thinking.

A specific concern in the elicitation of probabilities is that respondents may not think probabilistically about uncertain events. A large and diverse literature suggests that persons may think about uncertain events using less than the full structure of modern probability theory (e.g., Camerer and Weber 1992). If so, then questions eliciting precise probabilities force respondents to give point responses when only interval responses may be meaningful.

Within cognitive psychology, there has long been controversy about the way in which humans internally represent their beliefs, and their ability and willingness to express their internal beliefs as numerical probabilities. Ferrell and McGoey (1980) posed models in which individuals may have some difficulty expressing beliefs as numerical probabilities, but nevertheless concluded that elicitation of numerical subjective probabilities is feasible. However, Zimmer (1983), who argued that humans process information using verbal rather than numerical modes of thinking, concluded that expectations should be elicited in verbal rather than numerical forms.

Wallsten et al. (1993) has reported that a majority of respondents prefer to communicate their own beliefs verbally and to receive the beliefs of others in the form of numerical probabilities. This asymmetry is intriguing but only marginally relevant to the design of expectations questions. The relevant question is not what communication mode respondents prefer to use, but rather what modes they are willing and able to use. Wallsten et al. (1993) report that virtually all of their respondents were willing to communicate their beliefs numerically, should the situation warrant it. Certainly the accumulating experience of economists using questionnaires to elicit probabilistic expectations has been that respondents are willing and able to communicate their beliefs in numerical probabilistic terms (e.g., Dominitz and Manski 1997a, 1997b, Hurd and McGarry 1995).

A related theme of cognitive research has been to learn how individuals associate numerical probabilities and verbal phrases such as ‘very likely,’ ‘probable,’ or ‘rarely’ (e.g., Wallsten et al. 1986). A central finding is that individuals vary considerably in the numerical probabilities that they associate with given verbal phrases. There is, however, controversy about the proper conclusion to be drawn. Should one conclude that individuals differ in their interpretation of verbal phrases, in their interpretation of numerical probabilities, or in their interpretation of both? The empirical findings are compatible with all of these conclusions.

6. Expectations Formation And Behavior

Empirical research on subjective expectations is still in its infancy. Basic questions about the conceptualization and measurement of expectations are not yet resolved. When sufficiently satisfactory answers to these questions are achieved, research will be able to move to yet more challenging and interesting matters.

Future research should seek to understand how persons form their expectations. Attitudinal researchers have given little systematic thought to this basic matter. Researchers who conceptualize expectations in probabilistic terms have not been able to agree on how persons update their subjective probabilities in light of new evidence. Economists and statistical decision analysts have commonly assumed that persons apply Bayes rule to update their expectations. Psychologists have reported intriguing empirical evidence to the contrary (e.g., Kahneman et al. 1982), but the practical import of this evidence remains unsettled. Many social scientists hypothesize that persons form expectations by observation of the behavior and experiences of others in their environments. However, we know precious little about the nature of social learning processes.

Research on subjective expectations should aim to understand how a person’s perception of the future affects behavior. Economists suppose that youth make schooling choices based on their expectations of the returns to schooling relative to labor market activity. They also hypothesize that uncertainty about future income induces individuals to increase savings, thereby decreasing consumption. Criminologists suggest that perceived risks of crime victimization may cause individuals to take precautions in their daily activities. Public health researchers speculate on how perceptions of health risks affect smoking, drug use, and unprotected sexual behavior. To shed light on these and similar questions of practical import will require much more intensive collection and analysis of expectations and behavioral data than the social and behavioral sciences have yet undertaken.

Bibliography:

1. Camerer C, Weber M 1992 Recent developments in modeling pBibliography: Uncertainty and ambiguity. Journal of Risk and Uncertainty 5: 325–70
2. Curtin R 1982 Indicators of consumer behavior: The University of Michigan Surveys of Consumers. Public Opinion Quarterly 46: 340–52
3. Davis J, Smith T 1994 The General Social Surveys, 1972–1994, Cumulative File. National Opinion Research Center, Chicago
4. Dominitz J, Manski C 1997a Using expectations data to study subjective income expectations. Journal of the American Statistical Association 92: 855–67
5. Dominitz J, Manski C 1997b Perceptions of economic insecurity: Evidence from the survey of economic expectations. Public Opinion Quarterly 61: 261–87
6. Dominitz J, Manski C 1999 The several cultures of research on subjective expectations. In: Smith J, Willis R (eds.) Wealth, Work, and Health: Essays in Honor of F. Thomas Juster. University of Michigan Press, Ann Arbor, MI
7. Federal Reserve Consultant Committee on Consumer Survey Statistics 1955 Smithies Committee report in Reports of the Federal Reserve Consultant Committees on Economic Statistics. Hearings of the Subcommittee on Economic Statistics of the Joint Committee on the Economic Report, 84th US Congress
8. Ferrell W, McGoey P 1980 A model of calibration for subjective probabilities. Organizational Behavior and Human Performance 26: 32–53
9. Fishbein M, Ajzen I 1975 Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. AddisonWesley, Reading, MA
10. Hurd M, McGarry K 1995 Evaluation of the subjective probabilities of survival in the health and retirement study. Journal of Human Resources 30: S268–92
11. Jamieson L, Bass F 1989 Adjusting stated intentions measures to predict trial purchase of new products: A comparison of models and methods. Journal of Marketing Research 26: 336–45
12. Juster F T 1964 Anticipations and Purchases. Princeton University Press, Princeton, NJ
13. Juster F T 1966 Consumer buying intentions and purchase probability: An experiment in survey design. Journal of the American Statistical Association 61: 658–96
14. Juster F T, Suzman R 1995 An overview of the health and retirement study. Journal of Human Resources 30: S7–56
15. Kahneman D, Slovic P, Tversky A (eds.) 1982 Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, New York
16. Katona G 1957 Federal Reserve Board Committee Reports on Consumer Expectations and Savings Statistics. Review of Economics and Statistics 39: 40–6
17. Lichtenstein S, Fischoff B, Phillips L 1982 Calibration of probabilities: The state of the art to 1980. In: Kahneman D, Slovic P, Tversky A (eds.) Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, New York
18. Manski C 1990 The use of intentions data to predict behavior: A best case analysis. Journal of the American Statistical Association 85: 934–40
19. McClelland A, Bolger F 1994 The calibration of subjective probabilities: Theories and models 1980–94. In: Wright G, Ayton P (eds.) Subjective Probability. Wiley, New York
20. National Bureau of Economic Research 1960 The Quality and Economic Significance of Anticipations Data. Princeton University Press, Princeton, NJ
21. Quadrel M, Fischhoff B, Davis W 1993 Adolescent (in)vulnerability. American Psychologist 48: 102–16
22. Savage L 1971 Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66: 783–801
23. Schuman H, Johnson M 1976 Attitudes and behavior. Annual Review of Sociology 2: 161–207
24. Turner C, Martin E (eds.) 1984 Surveying Subjective Phenomena. Russell Sage Foundation, New York
25. Wallsten T, Budescu D, Rapoport A, Zwick R, Forsyth B 1986 Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General 115: 348–65
26. Wallsten T, Budescu D, Zwick R, Kemp S 1993 Preferences and reasons for communicating probabilistic information in verbal or numerical terms. Bulletin of the Psychonomic Society 31: 135–8
27. Zimmer A 1983 Verbal vs. numerical processing of subjective probabilities. In: Scholz R (ed.) Decision Making Under Uncertainty. North-Holland, Amsterdam