Later Statistical Methods Research Paper

During the early decades of the twentieth century, the theory of probability and of statistical methodology of the least-squares tradition from Laplace and Gauss continued to be developed and the two fields married permanently. Methods of regression, correlation, and analysis of variance found their final form. As did survey sampling, which was given a probabilistic footing. The discussion of Bayesian analysis and inverse probability continued and led to a split between the frequentist and the Bayesian schools of thought.

This period also saw the emergence of fundamentally new ideas and methodologies. Statistical inference, e.g., in the forms of hypothesis testing and confidence interval estimation, was identified as distinct from data description. Parametric models, allowing exact inference, even for small samples, were found, and led to constructive concepts of inductive logic and optimality for estimation and testing.

This impressive development in the concepts and methods peaked with Fisher’s 1922 paper. It was a paradigmatic revolution of statistics, along those of Laplace and Gauss around 1800 (Hald 1998). It took about 20 years and many papers for each of these authors to work out their basic ideas in detail and it took half a century for the rest of the statistical community to understand and develop the new methods and their applications.

The impact of statistics on society and the social sciences also changed and expanded fundamentally. Statistical thinking spread and was integrated in several of the social sciences as part of their basic thinking. The development of psychology, sociology and economics was influenced in essential ways by statistics and statistics itself benefited from ideas and methods developed in these fields, as well as in biology and other sciences.

2. Impacts Of Statistics On The Social Sciences

The term ‘statistics’ has its origin in the social sciences. Statistics meant the assemblage of numerical data describing the state, its population, institutions etc., with focus on the particulars. With Adolphe Quetelet this changed. Statistical mass phenomena attracted more and more interest from social scientists throughout the nineteenth century. It was asked whether distributions in society followed laws of chance, such as the binomial, the Poisson, and the normal distribution. Most phenomena were found to show ‘supernormal’ variation, and the question arose whether the population could be stratified to obtain ‘normal’ variation within strata.

The emphasis in statistical work changed around 1900 from the general study of variability to more specific issues. Statistical data were requested and brought forward into discussions of social policy and social science. In 1910, Karl Pearson and Ethel M. Elderton, for example, gathered data to see whether children of heavy drinkers were more disabled than other children. Their conclusion (Elderton and Pearson 1910) was in the negative:

the alcoholic propensities of parents seemed utterly unrelated to any measurable characteristics of the health or intelligence of off-springs, and what relations were found were as likely to be the reverse of what the temperance movement expected as not.

Stigler (1999) summarizes this paper and the ensuing discussion with the leading economist Alfred Marshall and others. Pearson (letter to the Times, 12 July 1910) scolded the economists for only arguing theoretically and not bringing forward empirical support:

to refute these data the only method is to test further equally unselected material … Professor Marshall suggested that the weaklings among the sober had grandparents who were profligate. This may be so but, as a statistician, I reply: ‘Statistics—and unselected statistics—on the table, please.’ I am too familiar with the manner in which actual data is met with the suggestion that other data, if collected, might show another conclusion to believe it to have any value as an argument. ‘Statistics on the table, please,’ can be my sole reply.

The call for focused unbiased statistics spread, and the growth in statistical data was impressive both in official statistics and in social and other sciences. The toolbox of statistical methods also grew and the concepts and methods were steadily used more widely.

Even though statistics and probability originated in the social sphere with gambling, insurance and social statistics, statistical thinking in the modern sense came late to the core social sciences. It was, in a sense, easier for Gauss to think statistically about errors in his measurement of the deterministic speed of a planet and for the experimental psychologist to randomize their design and to use corresponding statistical models in the analysis, than it was for Pearson to study how children were affected by their parents’ drinking and than it was for the economists to study business cycles. The difficulty is, according to Stigler (1999) that statistics has a much more direct role in defining the object of study in the social sciences. This relative unity of statistical and subject matter concepts was a taxing challenge that delayed progress. When statistical thinking took hold, however, the fields themselves changed fundamentally and received new energy.

This delayed but fundamental impact of statistics can be traced in various disciplines. Consider the study of business cycles. Good times follow bad times, but are these cycles of business regular in some way? While the nineteenth century economists mostly were looking for particular events that caused ‘crises’ in the economy, Karl Marx and others started to look for regular cyclic patterns. Moving averages and other linear filtering techniques were used to remove noise and trend from the series to allow cycles of medium length to be visible. Statisticians worried whether a cycle of wavelength 20 years were due to the data or to the filtering methods. George Udny Yule found in 1926 that ‘nonsense correlations’ between time series of finite length are prone to show up and stand conventional tests of significance simply because of internal auto-correlation. He had earlier coined the term ‘spurious correlation,’ but found that nonsense- correlation are flukes of the data that ‘cannot be explained by some indirect catena of causation’ (Yule 1926). Eugene Slutsky’s Russian paper of 1927 made things even worse for descriptive business cycle analysis. By a simulation experiment using numbers drawn in the People’s Commissariat of Finance lottery, he demonstrated that linear filters can produce seemingly regular cycles out of pure noise. It has later been found that the filter used to find the popular 20-year cycles tends to produce 20-year cycles even from random data.

Slutsky’s experiment and Yule’s analysis highlighted the limitations of model-free descriptive statistical analysis. It gradually became clear that economic data had to be interpreted in light of economic theory, and the field of econometrics emerged. This revolution peaked with Trygve Haavelmo’s Probability Approach in Econometrics (1944). Mary S Morgan (1990) holds further that the failure of descriptive business cycle analysis was instrumental in the development of dynamic models for whole economies, starting with the work of Ragnar Frisch and Jan Tinbergen.

Other disciplines saw similar developments of delayed but fundamental impact of statistical methodology, but perhaps not to the same extent as economics. The study of intelligence and personality was, for example, transformed and intensified by the advent of statistical factor analysis stemming from Spearman (1904) and Thurstone (1935).

3. Statistics Transformed

3.1 The Study Of Populations

Populations can be real, such as the population of Norway in 1894 with respect to the number of sick and disabled. Theoretical populations, on the other hand, are the hypothetical possible outcomes of an experiment, an observational study, or a process in society or nature.

For finite realized populations, Kiær proposed in 1895 to the International Statistical Institute that his representative method should be considered as an alternative to the method of the day, which was to take a complete census. The idea of survey sampling was extensively debated and was gradually accepted internationally. Results from Kiær’s 1894 survey of sickness and disability were, however, questioned in Norway by Jens Hjorth. He had no problem with sampling per se but he criticized the particulars of the survey and the questionnaire. From actuarial reasoning he claimed that Kiær had underestimated the number of disabled. At issue was a proposed universal insurance system for disability and Hjorth argued that a snapshot picture of the population obtained from a survey would be insufficient. Estimates of transition probabilities to disability were also needed. In the background of Hjorth’s reasoning was, in effect, a theoretical population, e.g., a demographic model, with disability probabilities by age, occupation group and other determinants as primary parameters. Equipped with the statistical competence of the day, Hjorth presented error bounds based on the binomial distribution for the fraction of disabled and he discussed the sample size necessary to obtain a given precision for various values of the true fraction. Kiær apparently did not understand probability. In 1906, he had to concede that the number of disabled was grossly underestimated in his 1894 survey, and he became silent on his representative method for the rest of his life.

This debate over survey sampling continued over the ensuing decades until Jerzy Neyman (1934) described in clear and convincing form the modern frequentist probability model for survey sampling including stratification, cluster sampling, and other schemes. Neyman’s model for a survey of a given finite population can be seen as a theoretical population consisting of the collection of all the possible samples that could be drawn, with a probability associated with each.

The distinction between sample quantities and theoretical parameters were not always observed. For example, instead of asking how data could be used to draw inferences about theoretical quantities, Karl Pearson and others proposed to assess the degree of association in two-by-two tables by means of various coefficients, e.g., the coefficients of mean square contingency and tetrachoric correlation, by emphasizing the importance of substantive theory, and the distinction between theoretical and empirical quantities.

R A Fisher cleared up the confusion caused by the wilderness of coefficients. By 1920, he had introduced the notion of ‘parameter’ for characteristics of the theoretical population and he used the word ‘statistics’ for estimates, test variables, or other quantities of interest calculated from the observed data. Fisher’s interest was not focused on describing data by statistics, but on using them to draw inferences about the parameters of the underlying theoretical population. In the mid-1920s, Fisher pursued these notions of inference and he incorporated his evolving principles of statistical estimation into a new chapter in the second 1928 edition of his pioneering 1925 book, Statistical Methods for Research Workers. Here he presented his new measure of the statistical information in the data with respect to a particular parameter and he stressed the importance of using efficient estimators. Fisher’s ideas were based on the concept of a parametric statistical model that embodies both the substative theory, e.g., relevant aspects of Mendel’s theory of genetics, and a probabilistic description of the data generating process, i.e., the variational part of the model. This probabilistic model led Fisher to formulate the likelihood function, which in turn led to a revolutionary new methodology for statistics.

Fisher also picked up on William S. Gosset’s 1908 development of the t-distribution for exact inference on the mean of a normal distribution with unknown variance. From a simulation experiment and guided by distributional forms developed by Karl Pearson, Gosset guessed the mathematical form of the t-distribution. Later, Fisher was able to prove formally the correctness of Gosset’s guess and then go on to show how the t-distribution also provided an ‘exact solution of the sampling errors of the enormously wide class of statistics known as regression coefficients.’ (Fisher 1925)

3.2 The Study Of Variation

There are two related aspects of the term variation: uncertainty in knowledge, and sampling variability in data, estimates or test statistics. In the tradition of inverse probability stemming from Thomas Bayes, no distinction was drawn between uncertainty and variation. This inverse-sampling probability tradition dominated well into the twentieth century, although unease with some of the implicit features was expressed repeatedly by some critics as early as the mid-nineteenth century.

Inverse probability is calculated via Bayes’ theorem, which turns a prior distribution of a parameter coupled with a conditional distribution of the data given the parameter into a posterior distribution of the parameter. The prior distribution represents the ex ante belief/uncertainty in the true state of affairs and the posterior distribution represents the ex post belief. The prior distribution is updated in the light of new data.

Before Fisher, statistical methods were based primarily on inverse probability with flat prior distributions to avoid subjectivity, if they were grounded at all in statistical theory. Fisher was opposed to the Bayesian approach to statistics. His competing methodology avoided prior distributions and allowed inference to be based on observed quantities only. When discussing Neyman’s paper on survey sampling Fisher (1934), stated:

All realized that problems of mathematical logic underlay all inference from observational material. They were widely conscious, too, that more than 150 years of disputation between the pros and the cons of inverse probability had left the subject only more befogged by doubt and frustration.

The likelihood function was the centerpiece of Fisher’s methodology. It is the probability density evaluated at the observed data, but regarded as a function of the unknown parameter. It is proportional to the posterior density when the prior distribution is flat. The maximum-likelihood estimate is the value of the parameter maximizing the likelihood. Fisher objected, however, to the interpretation of the normed likelihood function as a probability density. Instead, he regarded the likelihood function and hence the maximum likelihood estimator as a random quantity, since it is a function of the data which are themselves considered random until observed.

To replace the posterior distribution, he introduced the fiducial distribution as a means of summing up the information contained in the data on the parameter of interest. In nice models, the fiducial distribution spans all possible confidence intervals by its stochastic quantiles. A confidence-interval estimator for a parameter is a stochastic interval obtained from the data, which under repeated sampling bounds the true value of the parameter with prescribed probability, whatever this true value is. Neyman was the great proponent of this precise concept. He even claimed to have introduced the term ‘degree of confidence,’ overlooking Laplace, who had introduced the large sample error bounds and who spoke of ‘le degre de confiance’ in 1816 (Hald 1998). Fisher’s fiducial argument helped to clarify the interpretation of error bounds as confidence intervals, but Fisher and Neyman could not agree.

Neyman thought initially that confidence intervals, which he introduced in his (1934) paper on sampling, were identical to Fisher’s fiducial intervals and that the difference lay only in the terminology. He asked whether Fisher had committed a ‘lapsus lingua’ since Fisher spoke of fiducial probability instead of degree of confidence (Neyman 1941). Neyman’s point was that no frequentist interpretation could be given for the fiducial probability distribution since it is based on the observed data ex post and is a distribution over the range of the parameter. He shared with Fisher the notion of the parameter having a fixed but unknown value and to characterize it by a probability distribution made no sense to him. Fisher stressed, however, that his fiducial distribution is based on the logic of inductive reasoning and provides a precise representation of what is learned from the observed data. He could not see that the frequentist property of the confidence interval method allowed one to have ‘confidence’ in the actually computed interval. That intervals computed by the method in hypothetical repetitions of the experiment would cover the truth with prescribed frequency was of some interest. It was, however, the logic and not this fact that made Fisher say that the computed interval contained the parameter with given fiducial probability.

Harrold Hotelling (1931) staked out an intermediate position. He generalized Gosset’s t-distribution to obtain confidence regions for the mean vector of multivariate normal data with unknown variance-covariance matrix. He concluded

confidence corresponding to the adopted probability P may then be placed in the proposition that the set of true values [the mean vector] is represented by a point within this boundary [the confidence ellipsoid].

Although fierce opponents on matters of inference, Fisher and Neyman agreed that inverse probability had left the subject only more ‘befogged by doubt and frustration’ (Fisher 1934) and they united in their fight against Bayesianism. The tradition from Bayes survived, however. One solution to the flawed indifference principle is to go subjective. By interpreting the prior distribution as a carrier of the subjective belief of the investigator before he sees the data, the posterior distribution is interpreted correspondingly, as representing the subjective believes ex post data. This was the approach of Bruno de Finetti. Harold Jeffreys proposed another solution. He argued that the prior should be based on the statistical model at hand. For location parameters, the flat prior is appropriate since it is invariant with respect to translation. Scale parameters, like the standard deviation, should have flat priors on the logarithmically transformed parameter. These and other priors were canonized from invariance considerations.

The basic idea of statistical hypotheses testing had been around for some 200 years when Karl Pearson developed his chi-square test in 1900 (Hald 1998). The chi-square test and Gosset’s t-test initiated a renewed interest in the testing of statistical hypotheses. A statistical test is a probabilistic twist of the mathematical proof by contradiction: if the observed event is sufficiently improbable under the hypothesis, the hypothesis can reasonably be rejected. Fisher formulated this logic in his p-value for tests of significance.

A given statistical hypothesis can usually be tested by many different test statistics calculated from the data. Fisher argued that the likelihood ratio was a logical choice. The likelihood ratio is the ratio of the likelihood at the unrestricted maximum to the likelihood maximized under the hypothesis. Neyman and Egon S. Pearson, son of Karl Pearson, were not satisfied with this reasoning. They looked for a stronger foundation on which to chose the test. From Gosset, they took the idea to formulate the alternative to the hypothesis under test. To them, testing could result either in no conclusion (there is not sufficient support in the data to reject the null hypothesis), or in a claim that the alternative hypothesis is true rather than the null hypothesis.

The alternative hypothesis, supposed to be specified ex ante, reflects the actual research interest. This led Neyman and Pearson to ask which test statistic would maximize the chance of rejecting the null hypothesis if indeed this hypothesis and not the null hypothesis is true, when the probability of falsely rejecting the null hypothesis is controlled by a pre-specified level of significance. In nice problems, they found the likelihood ratio statistic to produce the optimal test in 1933. This result is the jewel of the Neyman–Pearson version of frequentist inference. As important as their optimality result is, perhaps, their emphasis on the alternative hypothesis, the power of the test, and the two types of error (false positive and false negative conclusion) that might occur when testing hypotheses.

Gauss asked in 1809 whether there exists a probability distribution describing the statistical variability in the data that makes the arithmetic mean the estimator of choice for the location parameter. He showed that the normal distribution has this property and that his method of least squares for linear parameters is indeed optimal when data are normally distributed. This line of research was continued. The distribution that makes the geometric mean the estimator of choice was found and John M. Keynes discussed estimating equations and showed that the double exponential distribution makes the median an optimal estimator. In 1934 Fisher followed up with the more general exponential class of distributions. When the statistical model is within this class, his logic of inductive inference gives unambiguous guidance.

Probability remained as the underpinning of formal statistical methods and the probability theory saw many advances in the early twentieth century, particularly in continental Europe. The Central Limit Theorem, which is at the heart of Laplace’s approximate error bounds, was strengthened. Harald Cramer (1946) summed up this development and he exposed its statistical utility since many estimators and other statistics are approximately linear in the observations and thus are approximately normally distributed. Related to the Central Limit Theorem is the use of series expansions to find an approximation to the distribution of a statistic. Thorvald N. Thiele completed his theory of distributional approximation by way of half invariants, which Fisher later rediscovered under the name of cumulants. Francis Y Edgeworth proposed another series expansion in 1905 (see Hald 1998). Probability theory is purely deductive and should have an axiomatic basis. Andrei N Kolmogoroff gave probability theory a lasting axiomatic basis in 1933, see Stigler (1999) who calls 1933 a magic year for statistics.

3.3 The Study Of Methods Of The Reduction Of Data

Raw statistical data need to be summarized. Fisher (1925) says

no human mind is capable of grasping in its entirety the meaning of any considerable quantity of numerical data. We want to be able to express all relevant information contained in the mass by means of comparatively few numerical values … It is the object of the statistical process employed in the reduction of data to exclude the irrelevant information, and to isolate the whole of the relevant information contained in the data.

It is theory that dictates whether a piece of data is relevant or not. Data must be understood in the light of subject matter theory. This was gradually understood in economics, as well as in other fields. Fisher (1925) showed by examples from genetics how theory could be expressed in the statistical model through its structural or parametric part. The likelihood function is, for Fisher, the bridge between theory and data. Fisher introduced the concept of sufficiency in 1920: a statistic is sufficient if the conditional distribution of the data given the value of the statistic is the same for all distributions within the model. Since statistics having a distribution independent of the value of the parameter of the model is void of information on that parameter, Fisher was able to separate the data into its sufficient (informative) statistics, and ancillary (noninformative) statistics.

The likelihood function is sufficient and thus carries all the information contained in the data. In nice cases, notably when the model is of the exponential class, the likelihood function is represented by a sufficient statistic of the same dimensionality as the parameter. It might then be possible to identify which individual statistic is informative for which individual parameter. If, say, the degree of dependency expressed by the theoretical odds ratio in a two by two contingency table is what is of theoretical interest, the marginal counts are ancillary for the parameter of interest. Fisher suggested therefore conditional inference given the data in the margins of the table. In the testing situation, this he argued leads to his exact test for independence. Conditional inference is also often optimal in the Neyman–Pearson sense.

4. Conclusion

Statistical theory and methods developed immensely in the first third of the twentieth century with Fisher as the driving force. In 1938, Neyman moved from London to Berkeley. The two men were continents apart and kept attacking each other. The question is, however, whether this was more grounded in personal animosity than real scientific disagreement (Lehmann 1993). In the US, where Neyman’s influence was strong, there has been a late-twentieth-century resurgence in Fisherian statistics (Efron 1998) and even more so in Bayesian methods. Fisher and Neyman both emphasized the role of the statistical model. Both worked with the likelihood function, and suggested basically the same reduction of the data. Fisher gave more weight to the logic of inductive reasoning in the particular case of application, while Neyman stressed the frequentistic properties of the method. To the applied research worker, this boils down to whether the level of significance or the confidence coefficient should be chosen ex ante (Neyman), or whether data should be summarized in fiducial distributions or pvalues. These latter ex post benchmarks are then left to the reader of the research report to judge.

Over the period, probabilistic ideas and statistical methods were slowly dispersed and were applied in new fields. Statistical methods gained foothold in a number of sciences, often through specialized societies like Econometric Society and through journals such as Econometrica from 1933 and Psychometrica from 1936.

Statistical inference was established as a new field of science in our period, but it did not penetrate empirical sciences before World War II to the extent that research workers in the various fields were required to know its elements (Hotelling 1940). Statistical ideas caught the minds of eminent people in the various sciences, however, and new standards were set for research. Fisher’s Statistical Methods for Research Workers has appeared in 14 editions over a period of 47 years. His ideas and those of Neyman and others spread so far and so profoundly that C Radhakrishna Rao (1965) declared, perhaps prematurely, statistics to be the technology of the twentieth century.

Bibliography:

1. Cramer H 1946 Mathematical Methods of Statistics. Princeton University Press, Princeton, NJ
2. Efron B 1998 R. A. Fisher in the 21st Century. Statistical Science 13: 95–114
3. Elderton E M, Pearson K 1910 A First Study of the Influence of Parental Alcoholism on Physique and Ability of the Off The Galton Laboratory, London
4. Fisher R A 1934 Discussion of Neyman. Journal of the Royal Statistical Society 97: 614–19
5. Fisher R A 1922 On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, Series A, 222: 309–68
6. Fisher R A 1925 Statistical Methods for Research Workers. Reprinted in Statistical Methods, Experimental Design, and Scientific Inference. Oxford University Press, Oxford, 1990
7. Hald A 1998 A History of Mathematical Statistics From 1750 to 1930. Wiley, New York
8. Haavelmo T 1944 The probability approach in econometrics. Econometrica 12(Supplement): 1–118
9. Hotelling H 1931 The generalization of Student’s ratio. Annals of Mathematical Statistics 2: 360–378
10. Hotelling H 1940 The teaching of statistics. Annals of Mathematical Statistics 11: 457–70. Reprinted (1988) as ‘Golden Oldy,’ Statistical Science 3: 57–71
11. Lehmann E L 1993 The Fisher, Neyman–Pearson theories of testing hypotheses: One theory or two? Journal of American Statistical Association 88: 1242–9
12. Morgan M S 1990 The History of Econometric Ideas. Cambridge University Press, Cambridge, UK
13. Neyman J 1934 On two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society 97: 558–625
14. Neyman J 1941 Fiducial argument and the theory of confidence intervals. Biometrika 32: 128–50
15. Rao C R 1965 Linear Statistical Inference and its Applications. Wiley
16. Spearman C 1904 General intelligence, objectively determined and measured. American Journal of Psychology 15: 201–93
17. Stigler S M 1999 Statistics on the Table. The History of Statistical Concepts and Methods. Harvard University Press, Cambridge, MA
18. Thurstone L L 1935 The Vector of Mind. University of Chicago Press, Chicago, IL
19. Yule G U 1926 Why do we sometimes get non-sense correlation between time-series? A study in sampling and the nature of time-series. Journal of the Royal Statistical Society 18: 1–63