Elicitation Of Probabilities Research Paper

1. Subjective Opinion Within A Bayesian Framework

Subjective opinion is actually employed in several parts of any statistical analysis, Bayesian or frequentist (Lad 1996). The Bayesian model of decision making and inference is that prior beliefs about a particular attribute or state of nature are updated through data, and then used together with utilities to decide on a course of action. Formally, θ represents a (possibly multiattributed) state of nature such that θϵΘ; x represents observed data; and

Under a personal and subjective view of Bayesian inference sometimes called subjective expected utility (Savage 1954), a decision maker will choose the course of action, a, that, based on prior beliefs updated by collected data, will have the least adversarial consequences when compared with other possible decisions.

Elicitation is concerned with mapping subjectively held opinions into prior probabilities, f (θ), and (subjective) preferences and value judgments into utilities (or exchangeably, loss functions) L(θ|a). These prior distributions and utilities represent an amalgamation of information available in addition to the data collected for a particular problem. As such, the most valuable elicited opinions are those of experts in the area of the problem, who may (or may not) have any training in statistical methods. Meyer and Booker (2000) and Chaloner (1996) outline reasons why subjective expert opinions are valuable in Bayesian analysis. These include the a priori information that can be quantified in the investigation of new, rare, complex, or poorly understood phenomena; when collecting large amounts of data is not feasible; when collected data alone is insufficient evidence for decision making; in designing experiments; in forecasting future events; and incorporating historical data and experiences that may not be well documented.

Spetzler and Stael von Holstein (1975) identify three stages of an elicitation process: the deterministic phase, which corresponds to choosing a functional form for a statistical model, the probabilistic phase where the subjective assessments are made, and the informational phase where verification checks are carried out. The primary focus of elicitation methods is on the middle phase (probabilistic). It is assumed generally, at least in statistical practice, that the first phase is fixed by the likelihood chosen for the data; the third phase is addressed more in the research on cognitive aspects of elicitation than the statistical aspects of encoding subjective beliefs and preferences.

2. Cognitive Aspects Of Elicitation

In reporting subjectively held beliefs and preferences, there are several psychological heuristics that can lead to misrepresentation. For more detailed discussion on these, early work on the subject is found in Kahneman et al. (1982), Kyberg and Smokler (1980), Hogarth (1987); updated coverage is detailed in Poulton (1986) and in Wright and Ayton (1994). The most common problems in eliciting subjective opinions come from:

(a) Overconfidence. Probability assessors tend to underestimate variability and the tails of the distribution.

(b) Hindsight bias. Most assessors believe they would have predicted correctly the outcome of an event; thus only the outcomes that actually occurred are viewed as having nonzero probability of occurrence.

(c) Representativeness. Expert judgments can be based on the synthesis of previously observed data. If that data came from small samples, it may not be representative. Other terms often used in conjunction with this heuristic are base-rate neglect, small-sample fallacy, and misperception of randomness. Another well-known aspect of representativeness is the conjunction fallacy, where higher probability is given to a well-known event that is a subset of an event to which lower probability is assigned.

 (d) Availability. The probability of an event is judged by the frequency with which an event can be recalled in memory. The classic example of this is in the elicitation of beliefs about likely causes of death; botulism, which typically gets a great deal of press attention, is usually overestimated as a cause of death, whereas diabetes, which does not generate a great deal of media attention, is underestimated as a cause of death.

(e) Adjustment and anchoring. When an initial assessment is made, elicitees often make subsequent assessments by adjusting from the initial anchor, rather than using their expert knowledge.

To overcome possible biases introduced in the elicitation of probabilities and utilities by these heuristics, Kadane and Wolfson (1998) summarize several principles for elicitation:

(a) Expert opinion is the most worthwhile to elicit.

(b) Experts should be asked to assess only observable quantities, conditioning only on covariates (which are also observable) or other observable quantities.

(c) Experts should not be asked to estimate moments of a distribution (except possibly the first moment); they should be asked to assess quantiles or probabilities of the predictive distribution.

(d) Frequent feedback should be given to the expert during the elicitation process.

(e) Experts should be asked to give assessments both unconditionally and conditionally on hypothetical observed data.

3. Eliciting Prior Distributions

Eliciting prior distributions entails quantifying the subjective beliefs of an individual into a statistical distribution. Kadane and Wolfson (1998) discuss the distinction between structural and predictive methods of doing this. In structural methods, an expert is asked to directly assess parameters or moments of the prior distribution; in predictive methods, elicitees are asked questions about aspects of the prior predictive distribution; the predictive distribution is

and when no data has been observed (i.e., x={Ø}), this is known as the prior predictive distribution. In most complex problems, the predictive method is preferable, since it is more suited to the prescription for good elicitation methods outlined in Sect. 2. The functional form of the prior distribution is usually selected to be conjugate to the likelihood chosen to model the data.

The elicitation is generally executed by asking assessors to identify quantile-probability pairs (q, p) in some sense, where p=P_X(X<q). Questions can be asked by specifying either a point or interval value for p or q, and responses can be given directly, as values, probabilities, or odds, or indirectly where preferences are stated between two alternatives.

To illustrate, consider various ways in which a prior distribution for the probability parameter in a binomial model might be elicited. The beta distribution is the usual choice for modeling the prior distribution, and functionally, the model is

In this instance, x is the number of successes observed in n trials; from the way in which the prior parameters are updated by the posterior distribution, α+β can be viewed as a ‘prior sample size’ or measure of prior strength, and α/(α+β) as a point estimate of the mean prior probability of a success. So one way to structurally elicit a prior would be to ask an elicitee to specify a prior sample size and probability of success, and this would uniquely define the parameters of the prior distribution. This, however, does not provide any feedback to the elicitee, and will almost surely be overconfident because of the way in which the variability of the distribution is never elicited directly. It also does not address the issue of the skewness of the distribution. Chaloner and Duncan (1983) propose a predictive method for eliciting this same prior distribution, where the elicitee is asked to specify the modal number of successes for given prior samples sizes, and then is asked several questions about the relative probability of observing numbers of successes greater than or less than the specified mode. The elicitee is also shown (graphically) the implied predictive distributions for future observations specified by their current answers, and then walked through a bisection algorithm to adjust the spread of the distribution.

While the predictive method just described is more difficult to implement, it is also more appropriate in terms of having the elicitee answer questions that will effectively allow them to communicate both their knowledge and their uncertainty within the constraints of the model. To illustrate, suppose that in the structural method, an elicitee specified a prior sample size of 15 and a prior mean of 1/3. In the predictive method, the same elicitee specified, for a sample size of 15, a predictive mode of five successes; the odds of observing four rather than five successes was specified as 0.8, and the odds of observing six rather than five success was specified as 0.7. The two priors, in the structural case, a beta (5,10) distribution, and in the predictive case, a beta (9.67,18.74), are shown in Fig. 1. This illustrates that way in which the predictive method allows for better representation of the actual state of knowledge the elicitee has.

Existing protocols for eliciting prior distributions are described in some detail in Kadane and Wolfson (1998). Elicitation protocols are described as being either general or application-specific. General methods are those that are meant for specific classes of models, such as the beta-binomial, the normal linear regression model, the dirichlet-multinomial, and others. Application-specific methods are ones tailored to the problem under study; several examples of these are described both in Kadane and Wolfson (1998) and French and Smith (1997). Application-specific methods are used typically for complex situations, where there may be no conjugate or standard prior that is suitable, or where the elicitation process must be decomposed in order to provide sensible assessments.

4. Eliciting Utilities

The ultimate goal of eliciting utilities is to make decisions that minimize expected loss, under the constraints of uncertainty specified by the prior distribution and likelihood function. Determining expected loss means constructing loss functions that consist of outcomes based on a quantity of interest, L, and the loss associated with each outcome. The loss can be fixed or random. In general, there is a set A of possible actions that can be taken, and for each element of A, there are losses that may or may not depend on A. The set A must be exhaustive; in other words, all possible actions must be enumerated, and each action must be exclusive of every other action.

The first stage of the elicitation process should be to choose the action set A and the quantity of interest L. The set of actions should be well defined and must contain at least two elements. These actions are usually dependent on some quantity of interest. For example, if the action set consists of choosing which family planning program to implement, the quantity of interest might be some quantification of the state of infant mortality. In choosing the quantity of interest, it is important to make sure that L is a quantity that incorporates the uncertainty inherent in the problem under study. Generally, L will be a random quantity defined by a statistical model, and as such, it can have several dimensions.

Once the action set A and the quantity of interest L have been determined, the second stage of the elicitation is to construct loss functions based on them. This is the most difficult stage of the elicitation process. If several stakeholders are involved in the decision-making process, they may construct these separately or as a group; regardless, those assessing the loss functions must give careful thought to the consequences resulting from each action in A for any value of L. For useful overviews of loss function elicitation strategies, see works by Clemen (1996), Zeleny (1982), Edwards (1992), and Saaty (1980). In general, it is necessary to specify the form the loss functions should take; this is sometimes specified in advance, but it is advisable to consider this as part of the elicitation process.

There are several ‘standard’ loss functions. One of the best-known examples of a loss function is the 0–1 loss function generally applied to statistical hypothesis testing. In this case, the set of actions is given by A=a : reject the null hypothesis, a : do not reject the null hypothesis (Berger 1985). An extension of this is the 0–k loss function, which is typically written (letting L=θ)

There are many possible variants on this loss function that could be considered, and identifying the ‘form’ of the loss function is the more controversial of the tasks that must be performed in eliciting utilities. Given the conceptual framework of a 0–k loss function, consider the next elicitation task—identifying the values of k₁, k₂. Because k₁, k₂ can represent rather intangible quantities, it is often advantageous to specify their relative rather than absolute values. Consider that the ‘optimal’ decision in this example is to choose a₁ when

Then the remaining elicitation task needed for decision-making is to elicit the ratio k₁/k₂, which simplifies the elicitation task from eliciting k₁, k₂ directly or independently. The ratio on the right-hand side above can be viewed as the ratio of the posterior probabilities that the parameter lies outside the ‘rejection region’ (θϵΘ₂) to the posterior probability that the parameter lies within the region (or θϵΘ₁). From this, a general relationship can be drawn between the functional form of the model and the two quantities for which preferences need to be elicited; this is illustrated in Fig. 2.

Exploiting the relationships between the functional form of the model and loss function and the quantities about which preferences and value judgments must be elicited is a useful strategy. Many examples of the application of evaluating utilities and preferences are found in journals in the fields of risk analysis, accounting, and economics.

5. Other Issues

It is sometimes advisable or necessary to elicit the opinion of more than one individual. Genest and Zidek (1986) review methods for combining the opinions of several experts; Cooke (1991), Meyer and Booker (2000), and Morgan and Henrion (1990) provide guidelines for structuring elicitation tasks with groups of experts. French and Smith (1997) have several examples of analyses where elicitation of more than one expert is used. In general, there are two approaches used in group elicitation—the collective opinion of the experts is elicited in the whole or the opinions of several experts can be combined in some fashion.

Another issue that is discussed in the literature (see Kadane and Wolfson 1998 for further discussion) is that of validating and comparing elicitation methods. There are several criteria that might be considered; the most important of these is coherence, that the elicited of errors-in-elicitation. Some cognitive aspects of this opinions obey the laws of probability. In general, this should be considered a minimal requirement for any elicitation method. Other considerations are the reliability of the elicitation methods in producing stable reflections of the elicitee’s beliefs and preferences; also, the elicitation method can be validated by an examination of whether it obeys proper scoring rules (i.e., does not systematically encourage the misrepresentation of beliefs and preferences). Some practitioners of elicitation, particularly in the field of meteorology, advocate the calibration of expert opinion, whereby elicited opinions are adjusted to reflect the elicitee’s ability to correctly predict events. Subjective Bayesians typically object to calibration because subjectively held beliefs and preferences are not meant to be infallible. A final consideration given in the validation and comparison of elicitation methods should be given to the modeling are described in Poulton (1996), but very little statistical literature is available on the subject.

Bibliography:

1. Chaloner K M 1996 The elicitation of prior distributions. In: Berry D A, Stangl D K (eds.) Case Studies in Bayesian Biostatistics. Marcel-Dekker, New York, pp. 141–56
2. Chaloner K M, Duncan G T 1983 Assessment of a beta prior distribution: PM elicitation. The Statistician 32: 174–80
3. Clemen 1996 Making Hard Decisions, 2nd edn. Duxbury Press, Belmont, CA
4. Cooke R M 1991 Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, New York
5. Edwards W (ed.) 1992 Utility Theories: Measurements and Applications. Kluwer Academic Publishers, Boston
6. French S, Smith J Q (eds.) 1997 The Practice of Bayesian Analysis. Arnold, London
7. Genest C, Zidek J V 1986 Combining probability distributions: A critique and an annotated Bibliography:. Statistical Science 1: 114–48
8. Hogarth R M 1987 Judgment and Choice. Wiley, New York
9. Kadane J B, Wolfson L J 1998 Experiences in elicitation (with discussion). The Statistician 47: 3–19
10. Kahneman D, Slovic P, Tversky A (eds.) 1982 Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge, UK
11. Kyberg H E Jr., Smokler H E (eds.) 1980 Studies in Subjective Probability, 2nd edn. Krieger, New York
12. Lad F 1996 Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction. Wiley, New York
13. Meyer M A, Booker J M 2000 Eliciting and Analyzing Expert Judgement: A Practical Guide. Society for Industrial and Applied Mathematics and American Statistical Association, Philadelphia, PA
14. Morgan M G, Henrion M 1990 Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, New York
15. Poulton E C 1994 Behavioral Decision Theory: A New Approach. Cambridge University Press, Cambridge, UK
16. Saaty T L 1980 The Analytic Hierarchy Process. McGraw-Hill, New York
17. Savage L J 1954 The Foundations of Statistics. Wiley, New York
18. Spetzler C S, Stael von Holstein C A S 1975 Probability encoding in decision analysis. Management Science 22: 340–58
19. von Winterfeldt D, Edwards W 1986 Decision Analysis and Behavioral Research. Cambridge University Press, Cambridge, UK
20. Wright G, Ayton P (eds.) 1994 Subjective Probability. Wiley, New York
21. Zeleny M 1982 Multiple Criteria Decision Making. McGrawHill, New York