Statistics as Legal Evidence Research Paper

Shortly after the birth of mathematical probability in the seventeenth century, mathematicians such as Leibniz and James Bernoulli examined the legal implications of the subject (the latter in his masterpiece, the Ars conjectandi) and, in the eighteenth and nineteenth centuries, analyses of the reliability of testimonial evidence became commonplace in the treatises of the day on mathematical probability. But it was only in the hands of the French mathematicians Condorcet, Laplace, and Poisson that models of jury deliberation were first proposed, analyzed, and fitted to data. The models of Condorcet and his successors became of renewed interest in the 1970s when some United States jurisdictions considered permitting juries to either have fewer than 12 members, or non-unanimous verdicts. Subsequently, critics such as Comte and Bertrand in France and Mill and Venn in England questioned whether the subjective, qualitative, and individual could be analyzed by methods that were supposed to be objective, quantitative, and based on the collective; and in the end such academic discussions had little practical legal impact.

But in the twentieth century, and especially in the years following World War II, when the uses and applications of statistics found ubiquitous applications throughout science, commerce, government, and society, the law began to turn increasingly to the methods of statistical inference for assistance in a wide variety of fields. Such applications now range from sampling, proof of discrimination, and the determination of damages, to the establishment of paternity, human identification, and the prediction of future violent behavior. The applications that occur fall into two natural classes: in some cases, statistical methods are used to determine specific facts relevant to a particular case being decided (did this employer discriminate against that employee? did the defendant commit the offense charged?); in other cases, statistical methods are used are answer more general questions thought to be relevant to underlying law (does capital punishment deter homicides?).

2. Sampling

The use of sampling in civil and administrative cases represents a natural and early application of statistical methods. A company seeking a refund for overpaid taxes but wishing to avoid the time and expense of examining each and every sales receipt at issue, might instead draw a sample from the population of all receipts; a government regulatory agency, suspecting inappropriate behavior on the part of a company trading commodity futures might conduct an audit of a random selection of accounts, rather than examine and analyze potentially tens of thousands of such accounts; an insurance company engaged in an arbitrated dispute might agree to accept the results of a sample of policies drawn by a neutral third party. Such applications are natural and represent some of the earliest serious uses of statistics in the law precisely because the benefits were obvious and the methodology (if properly conducted) difficult to challenge.

3. Random Selection

A properly drawn scientific sample is accepted as representative because the randomness in question has been imposed by the statistician (using either a computer, random number table, pseudo-random number generator, or physical randomization). Sampling practitioners know however that true random- ness is often difficult to achieve and that samples not drawn in accord with appropriate protocols may turn out to exhibit serious biases. Thus in some legal applications of statistics, whether or not a sample has been randomly drawn is itself the question of interest. Casinos reporting unusually low cash receipts during a year are sometimes challenged on the basis of a statistical analysis of possible variation in yearly revenue. Governments also resort to the use of random selection or lotteries, for example, in jury selection, the military draft, random audits of tax returns, and spot checks of baggage or individuals at airports, in ways that may be subjected to statistical scrutiny and legal challenge. One famous example of this is the 1968 United States draft lottery (which determined the order in which individuals were drafted into the army): individuals born later in the year were more likely to be drafted before individuals born earlier in the year. Sometimes the purpose of a government lottery is not just to achieve randomization, but also to ensure a popular perception of fairness. Aristotle, in his Athenian Constitution, describes a complex multistage procedure for juror selection that presumably satisfied both of these functions.

Even if the mechanism for selection of a jury or grand jury does not employ randomization, there may be a legal requirement that the results of whatever method of selection has been used should at least appear to be random. In any discussion of random selection, it is important to distinguish between the process used to generate an outcome, and the absence of pattern in the outcome. The process may employ a true element of physical randomness, yet the outcome might by chance display some form of pattern; while the output of some deterministic processes, such as a pseudo-random number generator, may appear to be pattern free. Thus in the United States, for example, juror pools are often based on samples of administrative convenience but may, as a matter of law, be required, however selected, to be a representative cross-section of the population. Jury verdicts are therefore sometimes challenged on the ground of possible discrimination in the selection of the jury, it being alleged that either minorities or other groups had been excluded or underrepresented. Thus, for example, in the trial of Dr. Benjamin Spock for advocating resistance to the draft, his attorneys (believing that women would be more favorably disposed towards their client) appealed his conviction in part on the ground that women had been disproportionately excluded during one stage in the jury selection process and based this contention on a statistical analysis.

4. Employment Discrimination

In scientific sampling, the model of random selection arises from the actions of the statistician and, in jury selection, the randomness of the outcome may be a legal requirement. But in some instances the model of random selection is a benchmark imposed by the analyst. In employment discrimination cases, for example, it may be claimed that racial, ethnic, sexual, or age discrimination took place in the hiring, promotion, or termination of individuals. In the workplace, such discrimination, if it does take place, is seldom explicit today and resort is often made to statistical methods of proof to document the underrepresentation of a protected class.

In the United States, the law distinguishes two basic classes of employment discrimination: disparate treatment and disparate impact. In disparate treatment, intentional discrimination on the part of the employer is charged and statistical proof of underrepresentation may be introduced to support such a claim. In disparate impact cases, in contrast, the intent of the employer is not at issue: it is charged that, for whatever reason, the hiring, promotion, or termination practices of the employer impact to a greater extent on one or more protected classes, for example, Hispanics and African-Americans, than others. Rulings of the US Supreme Court in the 1970s found that, in certain circumstances, disparate impact was enough to sustain a finding of discrimination, unless the employer can defend the practice at issue on the grounds of ‘business necessity’; for example, a promotional examination may have a disparate impact but test for a necessary job skill. Such rulings were of great importance: by creating a purely statistical category of employment discrimination, they resulted in a rapid assimilation of statistical methods into the law, including many areas other than employment discrimination, due in large measure to the imprimatur given them by the Supreme Court.

After statistical methods of proof of employment discrimination became routinely accepted in US courts, increasingly sophisticated methods began to be introduced. In cases of salary discrimination, for example, complex regression models are sometimes used in an attempt to argue that a minority group or other protected class is being underpaid even after the salary of the individual is adjusted for all variables (such as experience, skill, and credentials). Because both the hiring and promotion of individuals and the salaries they are paid are not the outcome of a lottery but the result of human choice and agency, the appropriateness of using as a yardstick the random selection model underlying all such statistical models has been questioned. One serious concern is that as the methods used become more complex and in consequence more removed from the intuition of the lay trier of fact, the potential for misunderstanding and misuse increases. One illustration of this is the not uncommon interpretation of a mere finding of statistical significance as being a direct proxy for one of substantive significance.

It is not always obvious at what level an analysis should take place in a discrimination case. Simpson’s paradox refers to situations where a disparity between two groups is found at one level of analysis, but disappears at another. For example, in the 1960s, the Graduate School of the University of California at Berkeley discovered that a substantially higher percentage of men than women were being admitted for graduate studies. In an attempt to identify those departments responsible for the disparity, the Graduate School examined the records of individual departments and found that in each department women were doing at least as well as, and sometimes better than, their male counterparts. There is a simple explanation for the phenomenon in this case: women were applying preferentially to departments that had a lower admission rate, men were applying preferentially to departments that had a higher admission rate.

5. Forensic Applications

Much of forensic science is devoted to the identification of substances or individuals, and in both cases important statistical issues arise. Classical gunpowder residue, blood alcohol level, and drug tests, three examples of tests for the suspected presence of substances, must all take into account the statistical variability inherent in the testing process. In the case of human identification, additional issues arise. The determination of human identity has classically exhibited two extremes. In fingerprint analysis, it is usually accepted that an adequate fingerprint exhibiting enough characteristics suffices to uniquely identify its source. In classical serological testing, in contrast, determining the ABO type of a biological specimen (usually blood, saliva, or semen) merely suffices to narrow down the class of possible donors to perhaps 25 percent to 1 percent of the population. Such evidence, although probative, obviously cannot by itself establish identity. It is only recently, with the advent of sophisticated methods of DNA testing, that the potential for human identification has dramatically increased.

In a DNA profile, one or more locations (loci) on the human chromosome are examined and typed. Because human chromosomes come in pairs, two alleles are present for each locus and the list of allele pairs for each locus constitutes the DNA profile for the individual. Because the loci used in forensic applications are typically hypervariable (many possible alleles can occur at each locus), the resulting DNA profiles can be highly discriminating (few individuals in the population will have the same profile) and the typical profile may have a frequency in the population ranging from 1 in several thousand, to 1 in several million, billion, or more (depending on the system used). Such impressive frequency estimates necessarily make certain statistical or population genetic assumptions (independence of alleles within and between loci, sometimes termed either ‘gametic’ and ‘zygotic,’ or ‘Hardy-Weinberg’ and ‘linkage’ equilibrium), assumptions whose validity must be tested and may vary substantially from one population to another. More sophisticated approaches estimate parameters that attempt to correct for the presence of possible subpopulations.

Initially the introduction of such frequency estimates was attended by considerable controversy, but the introduction of polymerase chain reaction (PCR) methods in recent years has resulted in the ability to type increasing numbers of loci. At first, the FBI laboratory only typed four restriction fragment length polymorphism (RFLP) loci but in the not too distant future, the resulting DNA profiles may well come to be viewed as essentially establishing identity.

Nevertheless, even in situations where the computation of a profile frequency is not regarded as being by itself controversial, a number of statistical complexities still arise:

(a) the calculation of profile frequencies requires knowledge of the underlying allele frequencies, and knowledge of these requires either a true random sample, or at least, if a sample of convenience is used, one thought to be representative of the population,

(b) profile frequencies can vary considerably from one racial or ethnic group to another and information for some population subgroups may not be readily available (for example, some American Indian populations),

(c) in some cases, especially sexual assaults, a mixture of two or more sources of DNA may be present and such complex mixture cases can present formidable difficulties in both interpretation and appropriate statistical analysis,

(d) if a DNA profile is identified by searching through a large databank of such profiles (for example, profiles of convicted felons), the statistical weight afforded to such a match needs to take this into account. (From a frequentist perspective, it is some- times argued that this should be done by adjusting for the multiple comparisons being performed; from a Bayesian perspective, the adjustment would reflect the differing priors in a situation where a suspect is in hand and a suspectless crime resulting in a databank search.)

(e) matching DNA profiles are much more common among relatives than unrelated individuals in a population, the computation of profile frequencies for relatives leads naturally to another important application of DNA testing.

6. Paternity Testing

Closely related to its use in criminal forensics, DNA profiles are also used in establishing paternity. Suppose the profiles of a child, mother, and putative father have been determined. If the putative father is not excluded as a possible father, then two statistics are commonly encountered to measure how unusual such a failure to exclude is. One statistic, termed the ‘random man not excluded’ (RMNE), computes what fraction of the male population would similarly not be excluded if their DNA profiles were also tested. The second statistic, termed the ‘paternity index,’ computes the likelihood ratio for two hypotheses conditional on the observed evidence (the three profiles): the first hypothesis, that the putative father is indeed the father; the second, that the true father is a random member of the population. Such calculations were initially performed using knowledge of the ABO and other blood groups, but in the current era of DNA testing often suffice to effectively establish paternity.

There are a number of variant calculations that can also be performed. One can compute paternity indices if the profile of the mother is unknown or in more complex situations where the profiles of relatives other than the mother are known. Or one can attempt to identify the source of biological material when the profile of the individual is unavailable but the profiles of relatives are available. This is often the case in body identification, in particular when mass disasters are involved.

7. Violence Prediction

Society often has a stake in attempting to predict future violent behavior: are there adequate grounds for confining a mentally ill person on the grounds that they pose a significant risk if left at liberty; if a prison inmate is paroled, is it likely that they will commit another crime after release; if a sexual predator is released after their prison sentence is completed, how likely is it that they will commit another sexually violent offense after their release? Such predictions, especially in the last category, have traditionally suffered from the same difficulties as testing for a rare disease: even if a test for a disease has both high sensitivity and specificity, when the disease is rare enough, most of the positives are false positives. Despite such difficulties, in recent years some psychologists have attempted to construct predictive ‘instruments’ (tests) using complex logistic regression models, and have analyzed their performance using survival analysis techniques.

8. Interpretation

Statistical analyses may be performed by highly competent individuals but their results must be interpreted and used by persons lacking formal training or quantitative understanding. This presents challenges both in successful presentation and in meeting the important responsibilities placed on the statistician to ensure that his results are not overvalued. Terms such as ‘statistical significance’ are easily and frequently misunderstood to imply a finding of practical significance; levels of significance are all too often interpreted as posterior probabilities, for example, in the guise that, if a DNA profile occurs only 1 in 10,000 times, then the chances are 9,999 to 1 that an individual having a matching profile is the source of the profile. As a result, individuals may tend to focus on the quantitative elements of a case, thereby overlooking qualitative elements that may in fact be more germane or relevant.

9. Other Applications

The last decades of the twentieth century saw a remarkable variety of legal applications of statistics: attempting to determine the possible deterrent effects of capital punishment, estimating damages in antitrust litigation, epidemiological questions of the type raised in the book and movie A Civil Action. Some of the books listed in the Bibliography: discuss a number of these.

Bibliography:

1. Bickel P, Hammel E, O’Connell J W 1975 Is there a sex bias in graduate admissions? Data from Berkeley. Science 187: 398–404. Reprinted In: Fairley W B, Mosteller F (eds.) Statistics and Public Policy. Addison-Wesley, New York, pp. 113–30
2. Chaikan J, Chaikan M, Rhodes W 1994 Predicting Violent Behavior and Classifying Violent Off In: Understanding and Preventing Violence, Volume 4: Consequences and Control. National Academy Press, Washington, DC, pp. 217–95
3. DeGroot M H, Fienberg S E, Kadane J B 1987 Statistics and the Law. Wiley, New York
4. Evett I W, Weir B S 1998 Interpreting DNA Evidence: Statistical Genetics for Forensic Scientists
5. Federal Judicial Center 1994 Reference Manual on Scientific Evidence. McGraw Hill
6. Fienberg S E 1971 Randomization and social affairs: The 1970 draft lottery. Science 171: 255–61
7. Fienberg S E (ed.) 1989 The Evolving Role of Statistical Assessments as Evidence in the Courts. Springer-Verlag, New York
8. Finkelstein M O, Levin B 1990 Statistics for Lawyers. SpringerVerlag, New York
9. Gastwirth J L 1988 Statistical Reasoning in Law and Public Policy, Vol. 1: Statistical Modeling and Decision Science; Vol. 2: Tort Law, Evidence and Health. Academic Press, Boston
10. Gastwirth J L 2000 Statistical Science in the Courtroom. Springer-Verlag, New York
11. Harr J 1996 A Civil Action. Vintage Books, New York
12. Lagakos S W, Wessen B J, Zelen M 1986 An analysis of contaminated well water and health effects in Woburn, Massachusetts. Journal of the American Statistical Association 81: 583–96
13. Meier P, Zabell S 1980 Benjamin Peirce and the Howland Will. Journal of the American Statistical Association 75: 497–506
14. Tribe L H 1971 Trial by mathematics. Harvard Law Review 84: 1329–48
15. Zeisel H, Kaye D H 1997 Prove It With Figures: Empirical Methods in Law and Litigation. Springer-Verlag, New York