Generalization In The Social Sciences Research Paper

Cronbach (1982) has perhaps the most comprehensive framework for understanding generalization. We use his theory, though slightly modifying it. He claims that social scientists typically draw conclusions about four not completely independent entities— human populations, physical settings, causes, and observables (including effects and moderators of cause–effect relationships). Time, understood as historical period, might also be added. Cronbach further contends that scientists work with samples, instances, cases, exemplars, or particulars (terms he uses almost interchangeably), and that these represent studyspecific populations, categories, entities, universes, classes, constructs, or categories (terms he also uses interchangeably). A major task of all scientists is to identify the class a given set of lower-order instances represents.

Cronbach also understands generalization in terms of inferences about classes that the sampling specifics manifestly do not represent. Such knowledge transfer is particularly salient, for instance, when animals are studied as proxies for humans or when laboratories substitute for other social settings. But transfer is also involved when research on American senior citizens, say, is used to extrapolate to Americans in mid-life or to seniors in Western Europe; or when findings about factories are extrapolated to schools on grounds that both are complex organizations. Extrapolation also pertains to causes and effects, as when studies of healthcare subsidies are applied to housing subsidies, or when a finding about a subsidy’s effect on hospital utilization is applied to effects on mental health. Cronbach believes that methods are needed both for justifying inferences from particulars to generals (as from samples to populations or from measures to constructs) and for justifying extrapolations beyond the categories studied.

His schema also includes subcategories—e.g., the racial groups within some human population or the components constituting a construct-like stress. Researchers routinely partition data both to test a relationship’s stability across the subcategories examined and to identify the conditions under which it varies. The more heterogeneous the subpopulations are over which a relationship holds, the greater its generality is usually assumed to be. In additon, the more specifically identified are the boundary conditionals limiting a relationship, the better its generality is understood. Campbell’s external validity (Campbell 1957) also emphasizes generalization across entities, but without restricting it to subtypes. For him, a relationship found among Austrians should be examined not just by partitioning Austrians into regional or generational subgroups, but also by examining the same relationship across other nations. Findings that dependably replicate across very different circumstances are the foundation of all general science.

Using the Cronbach framework, we structure this exposition into four parts: (a) methods based on formal sampling theory, as they are and are not used for generalizing to clearly designated universes of people, settings, times, causes, and effects; (b) purposive sampling methods, especially as they are used in validating constructs; (c) methods for assessing the range of circumstances over which a causal connection holds; and (d) methods for extrapolating knowledge to entities with different characteristics from those on which the knowledge was originally established.

This phrasing acknowledges that many social scientists use universe (and population) differently from construct, though the later Cronbach tends to use them almost interchangeably. In statistics, populations are ostensive; their elements are real and can be pointed to. But constructs are hypothetical and more obviously theory dependent. Moreover, the formal methods statisticians prefer when sampling elements from a population cannot be used with constructs, because the necessary enumeration and sampling of elements cannot be readily achieved with measures of abstract constructs. This is why Cook and Campbell (1979) used external validity to refer to people and settings and construct validity to refer to instances of more hypothetical causes and effects. However, the distinction is partly arbitrary. Constructs have constitutive elements theoretically specified as their components, and instances of any one construct vary in which components they incorporate. Moreover, human populations are not totally ostensive per se. Despite official definitions, there is still room to disagree about what being an Australian means: what about someone with an Australian passport who has always lived abroad, or the illegal immigrant who has always lived in Australia without a passport? There is room for legitimate debate about the definition of any human class and about whether certain individuals fit within that class. Also, whether human populations or abstract constructs are involved, the functional task is always the same: justifying inferences from particular cases to general categories. Hence, we see no fundamental reason for treating populations and constructs differently, though we do respect social science terminology and often use the one to refer to people and settings and the other to cause and effect variables as well as causal moderators.

1. Formal Sampling Theory

The best way to represent a population is to enumerate its members before selecting a random sample from that population. When properly implemented, this guarantees that the sample will formally represent the population within known limits of sampling error. The sample and population distributions will match on all measured and unmeasured attributes. This makes random sampling the centerpiece of survey research, often complemented by other sampling details. To reduce costs, cases can be clustered within a geographic area before being randomly selected. To ensure that subpopulations are reliably represented, stratification can also occur before random selection. Formal sampling theory provides an impeccable justification for generalization to well-designated populations. And methods for implementing are now so improved that its use in describing human populations and some settings is almost routine.

The problem with random selection is its own limited generality. It rarely applies to causes and other abstract constructs (or to time). One cannot readily imagine enumerating all instances of a specific causal agent before selecting a random sample of these instances for study. In practice, causal agents are purposively chosen because of a presumed correspondence between substantive theory and operational instance(s). Likewise, it is rare for all possible measures of a construct to be enumerated before some are randomly chosen. The selection of measures to represent a construct is again purposive. Also, it is hardly practical to sample historical periods at random for generalizing, if not to eternity, then at least to some meaningful interval. Detailed measures are not always available from the past; and anyway, many theories predict temporal contingency rather than constancy. This last justifies the purposive selection of time periods for the unique light each throws on some causal process.

Even when selection is at random, unambiguous inference still depends on large samples that are not always readily forthcoming. How often can we randomly sample 50 or 100 time points for assessment? Or 50 or 100 institutions, unless the aim is to describe a particular class of institutions? What about manipulating causal agents within institutions? What proportion of those asked to participate will agree to do so? What types are these volunteer organizations likely to be—the most typical or the most proud?

Random sampling rarely applies to assessing the conditions under which cause and effect are related. We know of only six experiments where individuals or households were randomly selected before being randomly assigned to a treatment (see Shadish et al. 2001). However, survey researchers often use random sampling to test causal hypotheses, using item responses to identify variation in a possible causal agent and to provide outcome data. But this is widely acknowledged to generate inferior causal conclusions compared to experimenter-controlled intervention. So how can causal relationships be better generalized via the random selection that surveys afford if these relationships are more uncertain because a survey was used? Moreover, surveys do not routinely use random selection for other generalization-relevant features of the research. For instance, data are typically collected at one or a few time points only, and then in one type of setting (usually the home),

Random selection was not developed for extrapolation, generalizing beyond the sampling particulars. So, it is not surprising that it is not relevant to this function. Indeed, none of the extrapolation methods we later detail depends on random sampling. Other premises are involved, as they are for generalizing causal relationships and for selecting measures to represent constructs. Thanks to its logical and empirical warrant, random selection is the method of choice for generalizing to populations of persons and sometimes settings. But it is rarely relevant to other tasks with which a comprehensive theory of generalization must struggle.

2. Purposive Selection And Construct Validity

Purposive samples are selected without randomization and require only that the selected cases have certain prespecified characteristics. If these characteristics coincide with those of a population, then a match is achieved between the details in the population description and the details achieved in the sampling particulars. Unfortunately, such proximal matching cannot guarantee that the cases and categories are matched on all unmeasured characteristics, let alone perfectly matched on the measured ones. Purposive selection is less well theoretically justified than random selection. By itself, it justifies claims about category membership but not category representativeness.

Yet purposive selection is used throughout the social sciences. Even passionate advocates of random sampling measure behavior by first describing the construct to be assessed and then ensuring that measures of it include all attributes in the theoretical description. Empirical patternmatching between constructs and measures is the basis for generalization in this case, and substantive theory provides the template for choosing how to measure (or manipulate). The correspondence between construct and operational representation is usually multidimensional, with the most important attributes being the prototypical ones that most clearly define the construct’s uniqueness (Brunswik 1956).

Such pattern matching (also called proximal similarity) is not restricted to measurement contexts. When random selection is infeasible it also applies to selecting persons, settings, and times. Thus, experimenters might partition their data into gender subgroups. The resulting subsamples clearly correspond to the categories ‘male’ and ‘female,’ as inferred from body shape or valid self-reports. Selecting this way ensures a proximal similarity between the class ‘female’ and the sampled instances of that class. But do they represent their gender if all attend the University of Berlin? In such a case, the gender specifics and the construct match on some prototypical attributes; but the sample is much more internally homogeneous than the population with respect to age, intelligence, geography, and culture, among other factors. Belonging in a class does not entail formally representing that class. Proximal similarity is necessary but not sufficient for inferring the representativeness of sampled instances. Sophisticated construct validation requires more.

One auxiliary principle concerns ‘the heterogeneity of irrelevancies’ (Campbell 1988), ensuring that the sample is not homogeneous on attributes that are irrelevant to the inference being made—such as age, intelligence, and nation in the case of females students in Berlin. To make these conceptual irrelevancies heterogeneous requires adding samples of women of different ages, social classes, and countries. In measurement contexts, heterogeneity requires multiple measures of each construct or varying multiple instances of a treatment, provided that each instance has unique irrelevancies and that no irrelevancy is shared by all instances. It also requires testing the degree of triangulation achieved across the sources of presumed irrelevancy.

Another way to deal with homogeneous irrelevancies requires focused comparison. If female students in Berlin are compared with their male counterparts, this holds constant irrelevancies like country, language, and city of testing, while also varying gender. But such internal comparisons are not always feasible; and groups may differ in other unmeasured irrelevancies that affect responding—e.g., if female students in Berlin have higher intelligence than males on the average, or if data collection there occurs in chemistry classes that attract different kinds of females than males. Whatever the construct being targeted— gender, intelligence, or being Australian—purposive sampling requires critically examining whether the instances selected are more homogeneous than their presumed population.

Construct validation also requires ‘discriminant validity’ (Campbell and Fiske 1959), demonstrating that a relationship obtained with constructs labeled one way is not due to another construct that is correlated but nonetheless conceptually unique constructs. Thus, the validity of an anxiety measure requires not only showing that its measured attributes behave as hypothesized but also that they, as a whole, behave differently from conceptually related concepts like depression. Likewise, testing a hypothesis about how new computer firms are unique requires sampling more than just this type of firm. Also needed are firms making new products in other sectors. Relationships that hold across sectors can be generalized to firms in general, including computer firms. However, only relationships limited to computer firms help define what is unique about computer firms.

All sophisticated construct validity is about testing unique predictions entailed by the substantive theory of a population. We always validate interpretations of categories rather than categories themselves, even with human populations (Cronbach 1989). Lawyers might pronounce that an Australian is anyone eligible for an Australian passport. But this definition is not adamantine. It includes individuals with Australian parents who have never lived in the country; it excludes illegal immigrants who have lived there for decades; and what about persons eligible for a passport who deliberately choose not to claim one despite living in the country for decades? No specification of a construct or population is beyond reasonable criticism and so each preferred interpretation must be defended. Usually, the interpretation can be formulated as hypotheses about some combination of components, antecedents, consequences, or correlates. Construct validation requires specifying and testing the nearest to unique of these hypotheses. Only then can we be truly confident about generalizing from purposive samples and measures to their target constructs rather than to some other entity correlated with them. However, constructing the arguments needed is more laborious and uncertainty-riddled than random sampling. Still, most social scientists sample purposively most of the time, though they vary greatly in how much auxiliary information they collect after a modicum of proximal similarity has been achieved.

3. Generalizing Causal Connections

Much effort goes into testing causal hypotheses— statements that variation in one construct leads to variation in another and that no alternative cause can account for their covariation. Generalizing causal statements depends on methods other than formal sampling theory, given that experiments with both random selection and random assignment are rare and surveys provide causal tests that are flawed when compared to experiments. So, how is generalization achieved when the object to be generalized is a causal claim?

One way is when a population has recently been described using measures that are also available on a purposive sample of persons or sites. Then, any causal relationships obtained at the sample level can be weighted to the population level, using the empirical relationship between sample and population for this purpose. But such weighting only approximates what the population effect size would have been if the sample had been randomly selected. Correct inference depends on the sample level effect sizes not being affected by unmeasured sample characteristics that are not fully captured by the shared, measured variables. Still, we can anticipate that weighting procedures will be used more often in the future as methods for social experiments and estimating missing data advance. After all, survey researchers now describe human populations from biased sample data using basically the same weighting technique.

Sometimes, substantive theory is specific enough to detail the conditions under which a causal relationship holds. Then, a program of research can test explicit hypotheses about specific sources of population, setting, or time variation in the strength or direction of the causal relationship. This involves deliberate attempts to falsify the hypothesis that a causal connection is general (Gadenne 1976). Rejecting the hypothesis entails identifying specific causal moderators; failure to reject implies that the hypothesis of a general cause–effect relationship remains viable. A less formal variant of this describes common practice in the laboratory sciences. A causal claim is made and then independently replicated, often under conditions that approximate the original ones. Successful replication leads to concluding that the causal relationship is provisionally true—that is, true until future failures to replicate indicate the need to invoke causal moderator variables. In both approaches, special emphasis is on identifying moderators that reduce the causal relationship to zero or that change its sign, as opposed to those that merely vary its size without affecting its sign.

Purposive sampling is also used in other less formal generalization probes. In clinical trials, hospitals recruit patients with a given disease and then adhere to the study’s intervention and data collection protocols. Hospitals are basically chosen for their willingness to comply and for their heterogeneity with respect to, say, region of the country, predominant ethnic group served, or type of hospital sponsorship. Statistical power is often only adequate for cross-site analysis, suggesting that the goal is not to identify site-level main and interaction effects, but to see if the causal estimate exceeds zero despite heterogeneity in the hospitals studied. If it does, the causal effect is held to be general. If it doesn’t, several alternative claims are viable and need to be tested, especially the following: there is no effect; the causal sign varies by hospital type and obscures a main effect; or the methods used cannot detect an effect of the size expected. Only increasing the sample of sites allows direct tests of which hospital types are involved in site-level interactions. Even so, relying on a purposive sample of sites that are heterogeneous on many attributes except for volunteering is only logically defensible if researchers know from other sources that no confounds are associated with all the hospitals being volunteers for the clinical trial.

Exploratory analyses often involve measuring many constructs from among a large array of possible causal modifiers in order to probe which ones affect the strength of a causal relationship. Without a priori hypotheses about modifiers, the approach can be time-consuming and run afoul of error rate problems due to the many tests conducted and the small sample sizes for some values on the moderator. Nonetheless, subcategory breakdowns are often used within experiments to probe how robust a relationship is across persons, settings, times, and measures. This can prevent the overgeneralization that occurs when researchers conclude that a relationship found with the entire sample holds in each subgroup within the sample. The entire sample allows one to conclude that the effect is found despite the internal heterogeneity achieved. It does not imply there is independent replication within any of the individual subgroups. The story is essentially the same when partitioning data from a single survey. With adequate sample sizes per group, one can probe the robustness of a relationship across the subgroups sampled. But only those potential conditional variables directly sampled can be assessed.

The best tests of causal conditionals come from synthesizing multiple studies on a topic rather than from subgroup breakdowns within a single study (Cooper and Hedges 1994). Experiments and surveys relevant to the same causal hypothesis accumulate and can be used in meta-analysis, the best-known form of synthesis. Meta-analysis combines data across multiple studies asking the same question, taking advantage of the between-study variation in measures, manipulations, populations of people, settings, and times. The dependent variable is each study’s standardized effect size(s) and these are manipulated (a) to generate an average effect size across all the studies and all the irrelevant sources of heterogeneity they contain, and (b) to identify the specific contingency factors responsible for variation in the effect sizes.

What logic buttresses meta-analysis? It is not random sampling since individual studies are selected for their relevance to the hypothesis and the hope is to achieve a census of all published and unpublished studies. However, the more relevant target population is all studies that could be conducted on the topic, not all those done on it. In meta-analytic practice, generalization depends on the sample of achieved studies including a wide range of human populations, setting types, historical periods, and operationalizations of the cause and effect. Each study is first coded to label its sampling particulars in terms of theoretical constructs and to examine the sources of irrelevancy with which the particulars are and are not collectively imbued. Thus in a hospital experiment, hospitals might be described as being university-based, public, or private, and as being large or small. Patients might be described in terms of diagnosis, age, and gender mix, etc. The treatment might be described in terms of its components, duration, and manner of delivery; and the outcome might be described in similar terms. Then researchers typically test whether the same causal relationship holds despite these sources of sampled irrelevancy.

Sample sizes permitting, researchers also test whether the relationship holds across individually identified sources of potential irrelevancy so as to generalize to hospitals in general, people diagnosed with cancer, or whatever the target inferences might be for a particular study. Important here are the number and especially the heterogeneity of the types examined. Of course, some sources of variation might have a priori substantive meaning, as when university hospitals are a special target. Then one has to do a discriminant validity test and show that the effect size for such hospitals is different from that for other hospitals. Otherwise, there is nothing unique about university hospitals. So, meta-analysis depends on the very tests for proximal similarity, heterogeneous irrelevancy, and discriminant validity that are so central in construct validation. In addition, to achieve such tests multiple replication is needed across studies that may each be individually very narrow in the range of people, settings, times, and cause and effect operations it includes. Central is the heterogeneity across studies and the pattern of results achieved despite this sampled heterogeneity.

4. Extrapolating To Nonsampled Entities

Some social scientists probably think it silly to use samples for generalizing to populations that are not even proximally similar to the samples used. So, many animal researchers abjure all talk of generalizing to humans. But their language is often anthropomorphic, and their research is usually funded because others think it is relevant to humans. So, elaborate reasoning has evolved over time to justify generalizing from animals to humans.

In this regard, consider the use of cats in visual research (Blake 1988). The rationale for their use is in two parts. The first is entirely pragmatic and stresses the conveniences of working with cats (or Drosophila in genetics research), usually because of cost and ethics. The second is theoretical and is based both on cats sharing some anatomical and physiological features with humans and on behavioral research showing that cats and humans behave similarly when confronted with similar visual stimuli. However, the empirical work necessary for showing such correspondences has also revealed that cats are much less like humans in some physical ways—especially features related to night vision—and that some behaviors are not replicated across species when the same stimulus is presented. So, a contingency theory has evolved of the conditions under which cats are better or worse substitutes for humans.

But when asked, honest vision researchers will avow that convenience is the paramount rationale for using cats. Would they rather work with chimpanzees? Undoubtedly, since the chimps’ visual system is much closer to humans’ than the cats’. But the expense of maintaining chimps precludes using them. Would the researchers rather use humans than chimps? Undoubtedly, if ethical research procedures are possible— which they often are not. However, now that new imaging technologies are available that allow observing the human eye reacting to planned stimuli in great micro-detail and real time, many vision researchers are turning to work with humans rather than cats because the pragmatic obstacles to using humans now hardly apply. Thus within the community of vision researchers the preference for generalization to humans via within-species proximal similarity is clear. So, too, is the distaste for arguments that depend on the (only partial) proximal similarity between cat and human on some vision-related attributes given their obvious dissimilarity in others. A similar story could be told for conducting research in the laboratory rather than in the field. Convenience is paramount, and the reasons given for preferring the lab often contain elements of rationalization. But even so, extrapolation across species and settings is surely superior when there is an empirically defensible contingency theory of the conditions under which the extrapolation is better and worse warranted.

How is extrapolation otherwise achieved? Causal explanation is often held to play a major role, explaining how or why two or more things are related (Mackie 1974). Once an explanatory process has been identified, the argument is that it can be reproduced in other settings, with other kinds of people, at future times, and with unique ways of instantiating the process. Indeed, this is the major public rationale for basic science. Once one knows why fire happens, one can replicate it with new kinds of materials (say, some newly invented polymers), in new settings (an uninhabited desert island), and with people who have never seen fire before (so long as they can understand the explanatory recipe). The problem with this view is that full explanation is hard to conceptualize and achieve. It is difficult enough to use data for discriminating among explanatory claims even at one level of explanation, let alone for simultaneously investigating factors at the biological, individual, and social levels. Even so, causal explanation is the holy grail of science, not just for its aesthetic parsimony, but also for the utilitarian causal generalization it promotes in the form of extrapolation.

Extrapolation is also achieved through careful induction. If patient education reduces the length of hospital stays and does so with patients suffering from 25 different diagnoses, then it is likely to do so with the 26th. Of course, the logical warrant for such induction is limited, and seems more reasonable the more numerous and heterogeneous are the patient types already examined. If all 25 were different forms of cancer diagnosis, or involved patients aged between 25 and 40, or were restricted to men, this limited heterogeneity would reduce our confidence in any inductive-empirical extrapolation. Meta-analysis is particularly important in this regard, especially when it permits replication across multiple different sources of irrelevancy. A relationship that has proven to be robust in whatever form it was operationalized is more likely to re-occur when other ways of operationalizing its components are attempted—more likely, but not certain. We see here the value traditionally assigned in science to discovering both general dependencies and the contingencies that describe when a given relationship does and does not hold.

The final context for extrapolation relates to formal modeling. When subgroups can be arrayed along a single continium, the functional form of a causal relationship can then be described. Is it stable? Is there a linear increase or decrease? If a pattern clearly emerges, one might want to argue that the strength of the relationship can be inferred for unobserved sub- populations with values that fall within the continuum range. Simple interpolation is involved here, while the strength of the relationship for values beyond the continuum is inferred by extrapolating from the observed function. But comfort with such extrapolation depends on how distant the nonobserved subpopulation values are from the sampled ones.

Model-based extrapolation is always tricky, but especially so when the models are more complex. Consider much of macroeconomics. Models of a given economic system are developed from theory and past findings, and changes in parameter values are introduced to see how the system reacts to this perturbation. This is basically an extrapolation exercise to discover what would happen if some details of the economic system were not as they actually are. Can we extrapolate from what we think we know, in order to learn what would happen under novel circumstances? The validity of any extrapolation depends on the model being correctly specified with respect to the variables included, the relationships postulated between them, and the strengths of these relationships. Be wrong about any of these assumptions and any conclusions about the effects of the novel perturbation will be biased to some unknown extent. The assumption that the model is complete and accurate is difficult to test, though crucial to the credibility of the results (Glymour et al. 1987). Still, the creation of valid multivariate models of causal interdependency is another major goal of science, and it is not difficult to see why. It promises the extrapolation of results beyond whatever circumstances were studied in the past.

5. Conclusion

In the behavioral and social sciences at the beginning of the twenty-first century, theory and practice are most developed when individuals or households are sampled to describe a human population. Here, best practice requires randomly sampling units from the population. Theory and practice are also well developed for generalizing from a measure to an abstract construct or from an experimental treatment to a more general causal agent. Here, best practice requires an explicit theory of construct validity that necessarily invokes proximal similarity, but preferably also the heterogeneity of irrelevancies, discriminant validity, and causal explanation. However, the purposive selection and theory-linked methods needed for construct validation are not as formally well supported as random selection is. Theory and practice are even less developed for generalizing to settings and historical time periods and for identifying the conditions under which a causal relationship holds. However, various deliberate sampling and replication strategies are relevant here, including meta-analysis. At their best, such strategies borrow much of the logic underpinning sophisticated construct validation. Least well developed are theory and practice for extrapolating results to nonstudied circumstances. But even here, understanding causal explanatory processes can permit re-creation of the same causal forces in novel settings. Also, there are now decades of research and reflection about using animals to extrapolate to humans and about using the laboratory to extrapolate to other social settings. In addition, substantive modeling is sometimes used to examine what would happen to a system if one or more of its parameters were changed. Of all the methods examined, only random sampling provides an impeccable formal rationale for generalization. But its reach is, alas, limited. It is a great irony that the best formal method for generalization in the behavioral and social sciences can be used for only a small part of the generalization tasks that practicing social scientists routinely face. Given this, practicing social scientists and theorists of method reflecting on these practices have cobbled together more complex methods of generalization that depend more heavily on substantive theory, purposive sampling, heterogeneity of irrelevancies and demonstrated replication and the identification of causal contingencies.

Bibliography:

1. Blake R 1988 Cat spatial vision. Trends in Neuroscience. 11: 78–82
2. Brunswik E 1956 Perception and the Representative Design of Psychological Experiments, 2nd edn. University of California Press, Berkeley, CA
3. Campbell D T 1957 Factors relevant to the validity of experiments in social settings. Psychological Bulletin 54: 297–312
4. Campbell D T 1988 Methodology and Epistemology for Social Science: Selected Papers [ed. Overman E S]. University of Chicago Press, Chicago
5. Campbell D T, Fiske D W 1959 Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56: 81–105
6. Cook T D 1993 A quasi-sampling theory of the generalization of causal relationships. In: Sechrest L, Scott A G (eds.) New Directions for Program Evaluation: Understanding Causes and Generalizing about Them (Vol. 57). Jossey-Bass, San Francisco
7. Cook T D, Campbell D T 1979 Quasi-experimentation: Design and Analysis Issues for Field Settings. Rand-McNally, Chicago
8. Cooper H, Hedges L V (eds.) 1994 The Handbook of Research Synthesis. Russell Sage Foundation, New York
9. Cronbach L J 1982 Designing Evaluations of Educational and Social Programs. Jossey-Bass, San Francisco
10. Cronbach L J 1989 Construct validation after thirty years. In: Linn R L (ed.) Intelligence: Measurement, Theory, and Public Policy. University of Illinois Press, Urbana, IL, pp. 147–71
11. Gadenne V Die Guttigkeit psychologischer Untersuchungen. Kohlhammer, Stuttgart
12. Glymour C, Scheines R, Spirtes P, Kelly K 1987 DiscOvering Causal Structure: Artificial Intelligence, Philosophy of Science, and Statistical Modeling. Academic Press, San Diego, CA
13. Mackie J L 1974 The Cement of the Universe: A Study of Causation. Oxford University Press, Oxford, UK
14. Shadish W R, Cook T D, Campbell D T 2001 Experimental and Quasi-experimental Designs for Generalized Causal Inference. Houghton-Mifflin, Boston