Early Statistical Methods Research Paper

1. From Political Arithmetic To Statistics, 1660–1830

The name ‘statistics’ was coined in eighteenth-century Germany for a descriptive science of states. Whether or not statistical descriptions should involve numbers was left open at first. After 1800, in the polarizing era of the French Revolution, this question became more ideological, and the mere purveying of numerical tables was denounced for its slavishness, and for viewing the state as lifeless and mechanical. It thus entailed a change of mentality, suggestive also of new structures of government and administration, when this ‘state science’ began to be defined in terms of the collection and deployment of numbers. That happened first in Britain and France, where it was largely settled by 1840. The new science of statistics was closely related to censuses and other official information agencies, which expanded enormously in this same period (Hacking 1990).

The new statistics, commonly called a science, was practiced mainly by reformers and administrators. Its methods involved organization and classification rather than mathematics. Procuring reliable numbers, then as now, was no small achievement. Among those who hoped to move this social science in the direction of finding laws and abstract relationships, the highest priority was to standardize categories. In this way, they hoped, the ‘statist’ could compare rates of crime, disease, or illegitimacy across boundaries, and try to discover how they varied according to nationality, language, religion, race, climate, and legal arrangements. Such figures might also enable them to predict the potential for improvement if, for example, laws were changed or educational systems were put in place. A series of statistical congresses beginning in 1853 undertook to negotiate statistical uniformity, with very little success.

The comparative neglect of mathematics among the social statisticians was not simply a consequence of a lack of appropriate tools. Social numbers had been collected since the late seventeenth century under the rubric of ‘political arithmetic,’ a bolder and more freewheeling enterprise than statistics. John Graunt, a tradesman by origin, was elected to the Royal Society of London for his ‘Observations upon the Bills of Mortality’ (1662), a compilation and analysis of mortality records. William Petty, rather more daring, used what figures he could procure to calculate such quantities as the total wealth of Britain, including the monetary value of its human lives. On this basis he even proposed modestly to increase the wealth of the nation by transporting the Irish to England. In the eighteenth century, probability theory was deployed by the most illustrious of mathematicians to investigate social numbers. Jakob Bernoulli’s Ars conjectandi (1713) addressed the convergence of observed ratios of events to the underlying probabilities. A discussion was provoked by the discovery that male births consistently outnumber female ones, in a ratio of about 14 to 13. In 1712, John Arbuthnot offered a natural theology of these birth numbers. The ratio was nicely supportive of belief in a benevolent God, and indeed of a faith without priestly celibacy, since the excess of male births was balanced by their greater mortality to leave the sexes in balance at the age of marriage. The odds that male births should exceed female ones for 82 consecutive years by ‘chance’ alone could be calculated, and the vanishingly small probability value stood as compelling evidence of the existence of God. Against this claim, Nicolas Bernoulli of the famous Swiss mathematical family showed that the steadiness of these odds required no providential intercession, but was about what one would anticipate from repeated tosses of dice with—he revised slightly the ratios—18 sides favoring M and 17 F (Hacking 1975).

Pierre Simon Laplace investigated birth ratios in the 1780s, and also discussed how to estimate a population from a partial count. He was party to an alliance in France between mathematicians and administrators, which also included the mathematician and philosopher Condorcet (Brian 1994). Laplace’s population estimate was based on a registry of births for all of France, and on the possibility of estimating the ratio of total population to births through more local surveys. Proceeding as if the individuals surveyed were independent, and the birth numbers accurate, Laplace used inverse probabilities to specify how many should be counted (771,469) in order to achieve odds of 1,000 to 1 against an error greater than half a million. The estimated population, which had to be fed into the previous calculation, was 25,299,417, attained by multiplying the average of births in 1781–2 by a presumed ratio of 26 (Gillispie 1997).

Laplace and Condorcet wrote also about judicial probabilities. Supposing a jury of 12, requiring at least 8 votes to convict, and that each juror had a specified, uniform probability of voting correctly, these mathematicians could calculate the risk of unjust convictions and undeserved acquittals. They also determined how the probabilities would change if the number of jurors were increased, or the required majority adjusted. The study of judicial probabilities reached its apex in the 1830s when Simeon-Denis Poisson made ingenious use of published judicial records to inform the calculations with empirical content. By this time, however, the whole topic had become a dinosaur. The use of mathematics to model human judgment now seemed implausible, if not immoral, and inverse probabilities (later called Bayesian) came under challenge (Daston 1988). Political arithmetic had come to seem weirdly speculative, and probabilistic population estimates both unreliable and unnecessary. In the new statistical age, sound practice meant a complete census. On this basis, some of the more mathematical methods of statistical reasoning were rejected around 1840 in favor of empirical modesty and bureaucratic thoroughness. Not until the twentieth century was sampling restored to favor.

One important application of probability to data grew up and flourished in place of these old strategies and techniques. That was the method of least squares. The problem of combining observations attracted a certain amount of attention in the eighteenth century. The first formal method to gain wide acceptance was worked out between about 1805 and 1810. It was initially published in 1805 by the French mathematician Adrien-Marie Legendre, who was then at work on the problem of measuring a line of meridian in order to fix the meter. Carl Friedrich Gauss asserted soon afterwards that he had already been employing least squares for several years, though there are reasons not to take his claim at face value (Stigler 1999, Chaps. 17 and 20). From 1809 to 1811, Gauss and then Laplace published probabilistic derivations of the method, and provided interpretations that shaped much of its subsequent history. Particularly crucial was its connection to what became known as the astronomer’s error law, and later as the Gaussian or normal distribution. Gauss derived least squares by assuming that errors of observations conform to this law. Laplace showed how the error curve would be produced if each observation was subject to a host of small disturbances, which could equally well be positive or negative. The mathematics of error, and of least squares, formed the topic for a line of mathematical publications throughout the century. It provided a means of analyzing quantitative data in astronomy, meteorology, surveying, and other related observational sciences. Later, through a paradoxical transformation, the same mathematics was used to describe and analyze variation in nature and society.

2. Social Science And Statistical Thinking

The mathematical tools of statistics worked out by Laplace and Gauss were extended but not decisively changed in the period from 1830 to 1890. They engaged the attention of some excellent mathematicians, including Auguste Bravais, I.-J. Bienayme, and A.-L. Cauchy (Hald 1998). The work of the Russian school of probability theory, initiated by P. L. Chebyshev, had consequences for statistical methods. But the really decisive development in this period involved statistics as a form of reasoning, worked out mainly in relation to social science, and involving, for the most part, comparatively rudimentary mathematics. This statistical thinking arose first of all in relation to the social science that bore the name ‘statistics.’ Statistics as a method was conceived by analogy to this substantive science. It was, to rely here on anachronistic terms, less a mode of inference than a family of models. They involved, at the most basic level, the production of collective order from the seemingly irregular or even chaotic behavior of individuals.

Like most discoveries, this one had ostensible precedents. Since, however, the Belgian Adolphe Quetelet had in the 1820s learned probability from Laplace, Poisson, and Joseph Fourier in Paris, and published several memoirs on vital statistics, his astonishment in 1829 should count for something. In that year, inspired by newly published French statistics of criminal justice, he attached a preface on crime to a statistical memoir on the Low Countries. He declared himself shocked at the ‘frightening regularity with which the same crimes are reproduced,’ year after year. He introduced a language of ‘statistical laws.’ At first he worried that such laws of moral actions might conflict with traditional doctrines of human free will, though he eventually rejected the idea of a ‘strange fatalism’ in favor of interpreting these laws as properties of a collective, of ‘society.’ His preoccupation with statistical regularities reflected the new emphasis in moral and political discussion on society as an autonomous entity, no longer to be regarded as subordinated to the state. But there was also something more abstract and, in the broadest sense, mathematical, in Quetelet’s insight. It implied the possibility of a quantitative study of mass phenomena, requiring no knowledge at the level of individuals (Porter 1986).

Quetelet was an effective publicist for his discovery, which indeed was not uniquely his. Statistical laws were widely discussed for half a century in journalism, literature, and social theory by authors like Dickens, Dostoevsky, and Marx. Moralists and philosophers worried about their implications for human freedom and responsibility. Natural scientists invoked socialstatistical analogies to justify the application of this form of reasoning to physical and biological questions. The creation of a statistical physics by James Clerk Maxwell, Ludwig Boltzmann, and Josiah Willard Gibbs, and of statistical theories of heredity and evolution by Francis Galton and Karl Pearson, attests to the increasing range of these statistical theories or models. Precisely this expansion made it increasingly credible to define statistics as a method rather than a subject matter. At the same time, a new body of analytical tools began to be formulated from within this discourse, drawing from probability and error theory, but applying and adapting them to issues in social statistics.

The most pressing of these issues was precisely the stability of statistical series. A particularly forceful and uncompromising restatement of Quetelet’s arguments in Henry Thomas Buckle’s 1857 History of Civilization in England provoked an international debate about statistics and free will. One strategy of response was to explore this stability mathematically. The presentation and analysis of time series had long been associated with economic and population statistics, and with meteorology (Klein 1997). These debates provided a new focus. Was there really anything remarkable in the much-bruited regularities of crime, marriage, and suicide? Robert Campbell, schoolboy friend of the physicist Maxwell, published a paper in 1859 comparing Buckle’s examples with expectations based on games of chance. He found the social phenomena to be much less regular than were series of random events. In Germany, where statistics was by this time a university-based social science, this question of the stability of social averages formed the core of a new mathematical approach to statistics. Wilhelm Lexis set out to develop tools that could simultaneously raise the scientific standing of statistics and free it from the false dogmas of ‘French’ social physics.

The normal departure of measured from true ratios in the case of repeated independent random events was expressed by Poisson as (2pq n), where p and q are complementary probabilities and n the number of repetitions. Lexis began his investigation of the stability of statistical series in 1876 with a paper on the classic problem of male to female birth ratios. In this case, the tables of annual numbers were reasonably consistent with Poisson’s formula. He called this a ‘normal dispersion,’ though it appeared to be unique. Nothing else—be it rainfall or temperature, or time series of births, deaths, crimes, or suicides, whether aggregated or broken down by sex or age—showed this much stability. He understood Buckle’s deterministic claims as implying ‘subnormal dispersion,’ that is, an even narrower range of variation than the Poisson formula predicted. This could easily be rejected. Lexis’s limits of error, defined by three times the Poisson number, corresponded with a probability of about 0.999978. Virtually every real statistical series showed a ‘supernormal’ dispersion beyond this range.

Laplace and Poisson seemed often to use legal and moral questions as the occasion to show off their mathematics. Lexis was serious in his pursuit of a quantitative social science, one that would reveal the social and moral structure of society. Against Quetelet, who had exalted the ‘average man’ as the type of the nation, he undertook to show that human populations were highly differentiated—not only by age and sex, but also by religion, language, and region. His program for social science was to begin by disaggregating, seeking out the variables that differentiate a population, and using statistical mathematics to determine which ones really mattered. Society, it could be shown, was not a mere collection of individuals with identical propensities, but a more organic community of heterogeneous groups. The wider than chance dispersions revealed that society was not composed of independent atomic individuals.

3. Origins Of The Biometric School

Institutionally, the crucial development in the rise of modern methods of statistics was the formation of the biometric school under Karl Pearson at University College, London. The biometric program began as a statistical theory or model of natural processes, not a method of inference or estimation. Francis Galton, who pioneered the statistical study of heredity and evolution, derived his most basic conceptions from the tradition initiated by Quetelet. He was impressed particularly by the error curve, which Quetelet had applied to distributions of human measurements such as height and circumference of chest. Since he regarded the applicability of the error law as demonstrating the reality of his ‘average man,’ he minimized the conceptual novelty of this move. The mathematics, for him, showed that human variation really was akin to error. Galton strongly rejected this interpretation. The ‘normal law of variation,’ as he sometimes called it, was important for statistics precisely because it provided a way to go beyond mean values and to study the characteristics of exceptional individuals. The biometricians used distribution formulas also to study evolution, and especially eugenics, to which Galton and Pearson were both deeply committed (MacKenzie 1981).

Galton attached statistics to nature by way of a particulate theory of inheritance, Darwin’s ‘Pangenesis.’ His statistical concepts were also, at least initially, biological ones: ‘reversion’ meant a partial return of ancestral traits; ‘regression’ a tendency to return to a biological type; and ‘correlation’ referred to the tendency for measures of different bodily parts of one individual to vary in the same direction. By 1890, however, he recognized that his statistical methods were of general applicability, and not limited to biology. Pearson drew his statistical inspiration from Galton’s investigations, though he may well have required the mediation of Francis Edgeworth to appreciate their potential (Stigler 1986). Edgeworth, an economist and statistician, was, like Pearson, a skillful and even original mathematician of statistics, but no institution builder.

Thus it was Pearson, above all, who created and put his stamp on this new field. Interpreting it in terms of his positivistic philosophy of science, he strongly emphasized the significance of correlation as more meaningful to science than causation. While he was committed above all to biometry and eugenics, he also gave examples from astronomy, meteorology, and education. Beginning in the mid-1890s, he attracted a continuing stream of postgraduate students, many but not all intending to apply statistics to biology. Among the first was George Udny Yule, an important statistician in his own right, who worked mainly on social and economic questions such as poverty. Pearson fought with many of his students, and also with other statisticians, most notably R. A. Fisher, and his reputation among modern-day statisticians has suffered on this account. Nevertheless, the 1890s were decisive for the history of statistical methods. Their transformation was not exclusively the work of the biometricians. This is also when the Norwegian A. N. Kiaer drew the attention of statisticians once again to the possibilities of sampling, which Arthur Bowley began to address mathematically. These trends in social statistics, along with the new mathematics of correlation and the chi-square test, were part of something larger. This was the modern statistical project, institutionalized above all by Pearson: the development of general mathematical methods for managing error and drawing conclusions from data.

Bibliography:

1. Brian E 1994 La mesure de l’Etat: Administrateurs et geometres au XVIIIe siecle. Albin Michel, Paris
2. Daston L 1988 Classical Probability in the Enlightenment. Princeton University Press, Princeton, NJ
3. Desrosieres A 1998 The Politics of Large Numbers: A History of Statistical Reasoning. Harvard University Press, Cambridge, MA
4. Gillispie C C 1997 Pierre-Simon Laplace, 1749–1827: A Life in Exact Science. Princeton University Press, Princeton, NJ
5. Hacking I 1975 The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical inference. Cambridge University Press, London, UK
6. Hacking I 1990 The Taming of Chance. Cambridge University Press, London, UK
7. Hald A 1998 A History of Mathematical Statistics from 1750 to 1930. Wiley, New York
8. Klein J 1997 Statistical Visions in Time: A History of Time-Series Analysis, 1662–1938. Cambridge University Press, Cambridge, UK
9. Kruger L, Daston L, Heidelberger M (eds.) 1987 The Probabilistic Revolution, Vol. 1: Ideas in History. MIT Press, Cambridge, MA
10. MacKenzie D 1981 Statistics in Britain, 1865–1930: The Social Construction of Scientific Knowledge. Edinburgh University Press, Edinburgh, UK
11. Matthews J R 1992 Mathematics and the Quest for Medical Certainty. Princeton University Press, Princeton, UK
12. Porter T M 1986 The Rise of Statistical Thinking, 1820–1900. Princeton University Press, Princeton, NJ
13. Stigler S M 1986 The History of Statistics: The Measurement of Uncertainty Before 1900. Belknap Press of Harvard University Press, Cambridge, MA
14. Stigler S 1999 Statistics on the Table: The History of Statistical Concepts and Methods. Belknap Press of Harvard University Press, Cambridge, MA