Compliance with the Experiment Design Research Paper

1. Difficulties With Noncompliance

The complication of noncompliance necessitates the clear distinction between treatment assignment and treatment exposure. In randomized experiments with perfect compliance, these are the same; but when some subjects are noncompliant, assignment and exposure are different for those noncompliers. This distinction represents the crux of the problem induced by noncompliance: if a study is analyzed without regard to differences in assignment and exposure (e.g., using an intention-to-treat analysis) then the research question that is answered may not be the one that is posed.

Boruch (1997) takes a strong stand for analyzing studies as they are randomized (i.e., without accounting for noncompliance), since compliance behavior during a randomized trial is likely to approximate that same behavior in the general population were the interventions under study to be applied more broadly. Indeed, substantial noncompliance in itself may speak to the efficacy of assignment to the interventions under study. Nevertheless, understanding the relative efficacy of exposure to interventions and possible relationships between compliance behavior and subject outcome warrants a deeper examination of compliance.

Consideration of the statistical problems induced by subject noncompliance has taken place primarily in the field of clinical trials in cases where subjects take less of their assigned drug treatment and/or supplement their treatments with medications or activities other than the ones under study. A direct translation of the compliance problem from the medical setting to the behavioral or education setting comes from considering attendance at sessions of experimental interventions as a ‘dose’ or ‘treatment level.’ Alternatively, non-compliance with behavioral interventions might be as simple as subjects not taking advantage of an offered program, such as a family’s decision to keep their child in public school even after they have been selected by lottery to attend a private school (see Barnard et al. 1999).

Regardless of the manifestation of noncompliance, questions regarding causal effects of interventions under study remain. Holland (1986) provides an extensive overview of causal inference in statistics, making use of the counterfactual model, the seeds of which date at least to Fisher (1918) and Neyman (1923), and which has been refined over several years by Rubin (1978, 1980). This model, relating exposure and outcome, is summarized here to provide a framework for illustrating the noncompliance problem and for describing methods for describing existing methods for addressing that problem (Dawid 2000 argues for causal modeling without using counterfactual outcomes, though addressing noncompliance is greatly facilitated by their use).

2. The Counterfactual Model

The foundation of the counterfactual model is the notion of potential outcomes, which are the collection of outcomes that would be observed for a subject if that subject were exposed to all possible treatments. The words counterfactual and potential suggest quantities that are not observed—outcomes ‘counter to the fact,’ or that ‘might be observed.’ Consider N subjects, and let T_i denote the treatment assignment and X_i the treatment exposure for subject i. For simplicity, let T_i and X_i assume binary values, 0 and 1, representing a control and active treatment, respectively.

The specific quantities of interest for causal inference are subject outcomes under each treatment: Y_i(0) is the outcome that would be observed for subject i if he or she were exposed to treatment 0, and Y_i(1) is the outcome that would be observed for the same subject if he or she were instead exposed to treatment 1. Typically, only one of Y_i(0) or Y_i(1) is observed for each subject. The observed outcome for subject i, Y_i (or Y_i(X_i)), is linked to the counterfactuals via the consistency relation

that is, the observed outcome is the potential outcome under the observed treatment exposure.

It should be noted that Y_i(0) and Y_i(1) are reduced from their general form as functions of the entire vector of treatment exposures via the stable unit treatment value assumption (SUTVA) of Rubin (1980). This assumption asserts that each subject’s potential outcomes depend only upon his or her own treatment exposure and not those of any other subjects. In clinical trial settings, where subjects typically do not interact, SUTVA may be a reasonable assumption. In studies of behavioral or educational interventions, however, where subjects may interact with each other extensively, this assumption must be examined more carefully, and the causal conclusions that are based on it may be tenuous (Gitelman 1999).

The potential outcomes just defined, and SUTVA, are sufficient for defining an individual causal effect (ICE). This is simply the effect, for subject i, of exposure to treatment 1 vs. exposure to treatment 0

Alternative individual causal effects such as the ratio of these individual potential outcomes or the difference of their logs could be considered, though the causal parameter obtained by taking the expectation of the difference in Eqn. (2) argues for using this simpler form. The average causal effect (ACE) is the individual causal effect averaged over the space of subjects

The probability distribution over which these expectations are taken is induced by the process by which subjects enter, or are selected for, the study.

In general, θ is nonidentifiable without further assumptions. Rubin (1978) showed, for instance, that θ is identifiable when treatment exposure is ‘ignorable,’ meaning that each subject’s treatment exposure is stochastically independent of his or her collection of potential outcomes (this independence is expressed using the symbol Π, following Dawid (1979)

Under randomized treatment assignment and complete compliance, treatment exposure ignorability holds, since X_i =T_i for all subjects i. In this case

where the second equality follows from ignorability and the third from the consistency relation in Eqn. (1). α is called the apparent treatment effect, and it can be estimated by the difference in sample means of the group of subjects exposed to each treatment, perhaps appropriately scaled by a standard error estimate. Notice that α is typically identifiable regardless of treatment exposure ignorability—it is its equivalence to θ that is afforded by ignorability.

If treatments are randomly assigned, but some subjects are noncompliant, then treatment exposure is probably not ignorable. In that case, the apparent treatment effect is not a good estimate of the average causal effect. Specifically, there may be some characteristic of the subject (e.g., in a study comparing educational interventions, his or her academic ability) that predisposes him or her to be noncompliant with treatment assignment. This very characteristic may also be reflected in his or her potential outcomes under that treatment, and therefore his or her outcome cannot be attributed solely to the treatment in question.

One option when faced with noncompliance is to estimate the intention-to-treat effect, and use it as an estimate of the average causal effect. The intention-to-treat effect is given by

which can be estimated by the difference in sample means of the two treatment groups. This effect explores the relationship between assignment and outcome. Alternatives to this approach that seek to illuminate the relationship between exposure and outcome are described next.

3. Simple Noncompliance

In the case of binary treatment assignment and binary treatment exposure, noncompliance with treatment assignment manifests as ‘treatment switching.’ That is, a subject assigned to the active treatment is noncompliant if he or she switches to the inactive treatment (for instance by not taking the treatment at all). Similarly, a subject assigned to the control treatment is noncompliant by switching to the active treatment. This is essentially the situation described in Barnard et al. (1999), wherein families participated in a lottery to receive funding and other support for private school attendance. Some families, even though they ‘won’ the lottery, decided to keep their children in public schools—hence, these families are the ‘noncompliers.’

3.1 Modification Of The Counterfactual Model

The counterfactual outcomes described previously depend upon treatment exposure only, whereas to account for noncompliance they must be altered to depend also on treatment assignment. Following Angrist et al. (1996), let X_i(0) and X_i(l) denote potential or counterfactual exposures for subject i; these are the treatments subject i would be exposed to (i.e., based on his or her compliance behavior) under assignment to treatments 0 and 1, respectively. Let Y_i (T_i, X_i(T_i)) denote the potential outcome of subject i, under treatment assignment T_i and exposure X_i(T_i). Expression of both the potential exposures and potential outcomes again rely on Rubin’s SUTVA. In the binary assignment and binary exposure case, each subject has four potential outcomes, one for each assignment/exposure pair

For instance, Y_i(1, 0) is the outcome subject i would experience had he or she been assigned to the active treatment and been noncompliant with that treatment, thereby receiving the control treatment.

As in the case of the simpler potential outcomes Y_i(0) and Y_i(1), the outcomes in Eqn. (7) are related to the observed outcome via a consistency relation

Where I_(·) is the indicator function. As before, the observed outcome is the potential outcome that corresponds to the observed treatment assignment/exposure pair. Nonidentifiability is more drastic in this setting, however, as there are now four counterfactual outcomes to consider for each subject instead of just two.

The causal parameters of interest are the expected values (with expectations taking over the space of subjects) of the potential outcomes in (7); namely

With noncompliance, exposure is not ignorable, and so it is not necessarily the case that E [Y(0, 0)] equals E [Y(1, 0)] or that E [Y(0, 1)] equals E [Y(1, 1)]. That is, the average subject outcome of a noncompliant subject assigned to the treatment arm will not, in general, be the same as that of a compliant subject assigned to the control arm. Methods to address this complication are described next.

3.2 Bounds For ACE Using An Instrumental Variable

A variable T is an instrumental variable if the distribution of another variable, Y, depends on T only through a third variable, X. Instrumental variables have been popularized in the econometrics literature; they have been used to address noncompliance in other settings (e.g., Angrist et al. 1996). Ideally, T and X should be correlated, and T and Y uncorrelated conditionally on X. Then T can be used as a surrogate or instrument for X. An instrumental variable, then, is a variable that is partly defined by a conditional independence relationship, known as the exclusion restriction, or instrumental variable assumption

and partly by the required correlation between T and X. Angrist et al. (1996) translate this assumption into the language of potential outcomes, using assignment as an instrument for exposure

that is, a subject’s potential outcomes do not depend on the treatment assignment directly, but only through that assignment’s effect on the treatment exposure. Angrist et al. (1996) show the equivalence of the counterfactual and structural (or simultaneous) equation models.

The instrumental variable assumption in Eqn.(10)has the effect of reducing the number of potential out-comes from four, as in Eqn. (7) to two, simply Y_i(0) and Y_i(1). For each subject, however, each of Y_i(0) and Y_i(1) can be observed in one of two ways, either by the subject’s compliance with assigned treatment or their noncompliance with the opposing treatment. By using assignment as an instrument for exposure the average causal effect, θ can be bounded by taking minimums or maximums over these two ways of observing each treatment exposure (of course, these bounds also depend upon the responses having bounded expectations). Balke and Pearl (1994) produce the tightest possible bounds for θ (under binary assignment, binary exposure, and binary outcome) using the instrumental variable assumption and linear programming results.

Angrist et al. (1996) use the instrumental variable assumption, combined with a monotonicity assumption, to estimate θ rather than bound it. This monotonicity assumption applies to the counterfactual exposures—it is assumed that a subject will only be noncompliant by switching from the active treatment to the inactive control, but not vice versa, whereby Y_i(1, 0) is eliminated from the collection in Eqn. (7). Because it relates exposure under an active treatment to that under a control treatment, this monotonicity assumption only is applicable in trials that compare an active treatment (or treatments) to a control.

4. Noncompliance As Underdosing

A more complicated noncompliance situation occurs when noncompliant subjects take a lower-than-intended dose of their assigned treatment (or attend fewer-than-intended sessions of their assigned intervention), rather than switching between treatments. For simplicity, treatment assignment remains binary in this discussion. The counterfactual exposures, X_i(0) and X_i(l), now take categorical or even continuous values, however, and there is no switching between treatments except perhaps in the case of a subject taking a zero dose of an active treatment. In a randomized dose-response experiment, subjects are randomly assigned to different doses of a treatment (i.e., dose is ignorable); but with noncompliance the subjects choose those doses (i.e., compliance 1evels), and so the resulting treatment outcomes cannot be attributed to dose or compliance levels alone.

In a direct extension of the simple compliance case discussed previously, Goetghebeur and Molenberghs (1996) model counterfactual compliance-response curves (similar to dose-response curves) for categorical exposure levels and binary outcomes, Their model relies on a different monotonicity assumption (this one asserts that increased compliance cannot lead to decreased probability of a successful outcome); an instrumental variable type of assumption and a constant global odds ratio assumption. A subsequent contribution, Goetghebeur et al. (1998), relaxes this last assumption somewhat. An essential difficulty with these models, however, is that due to the need for modeling counterfactual (i.e., unobservable) outcomes, rather strong, untestable assumptions are needed. For example, the constant global odds ratio assumption is untestable, and it dictates an artificial relationship between the counterfactual compliance response curves.

Efron and Feldman (1991) take the approach of modeling compliance behavior as a fixed attribute of the subject, rather than as a response to treatment assignment. As a fixed attribute, each subject’s compliance behavior remains the same, regardless of the treatment to which he or she is assigned—this eliminates the problem of having to address counterfactual exposure levels. Compliance-response curves are estimated with this approach, and with additional modeling assumptions (e.g., linearity of the dose response relationship), portions of the ‘true’ dose response curves are recovered, Again, the true dose-response curves can be fully estimated only if dose level is ignorable.

Robins (1994) explores the slightly different setting of sequential randomized trials, in which subjects are assigned to treatment regimes in series, interspersed with assessment of their medical condition. In this work it is assumed that, conditional on all observable covariates, the treatment exposure regime is ignorable.

5. Addressing Noncompliance In Practice

When subjects are noncompliant with assignment to experimental interventions, a study loses the solid footing afforded by randomized treatment assignments. The essential question of interest regarding the effect of exposure to interventions is now compromised by the subject’s selection of their intervention and/or intervention level. The counterfactual model provides a foundation for examining the relationship between exposure and outcome, though careful consideration of the assumptions of both that model and the additional modeling components needed for identifying causal parameters of interest are crucial and may depend upon the type of study.

When faced with compliance problems, or the possibility of compliance problems, the researcher may be best served by careful planning of intervention delivery with an eye toward averting attendance problems or other lapses in the integrity of the intended treatment (Meyer and Fienberg 1992). Furthermore, by collecting as many relevant covariates as possible, differential, subject-selected treatment exposure can be alleviated, in part by adjusting for these covariates using, for example, propensity score matching (Rosenbaum and Rubin 1983). If sufficient subject covariates are observed, it may be reasonable to assume that treatment exposure level is ignorable conditional on these covariates, whereby the causal parameters of interest can be identified. For compliance problems that cannot be addressed a priori, however, the bounding (e.g., Balke and Pearl 1994) and modeling (e.g., Angrist et al. 1996, Barnard et al. 1999) methods described herein provide useful options.

Bibliography:

1. Angrist J D, Imbens G W, Rubin D B 1996 Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91: 444 –72
2. Barnard J, Frangakis C, Hill J, Rubin D 1999 School choice in NY City: A Bayesian analysis of an imperfect randomized experiment. In: Kass R E (ed.) Proceedings of the Fifth Case Studies in Bayesian Statistics Workshop, Vol. 5. SpringerVerlag, New York
3. Boruch R F 1997 Randomized Experiments for Planning and Evaluation, A Practical Guide. Sage, Thousand Oaks, CA
4. Balke A, Pearl J 1994 Counterfactual probabilities, computational methods, bounds and applications. In: Lopez de Mantaras R, Poole D (eds.) Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann, San Mateo, CA
5. Dawid A P 1979 Conditional independence in statistical theory. Journal of the Royal Statistical Society, Series B 41: 1–31
6. Dawid A P 2000 Causal inference without counterfactuals. Journal of the American Statistical Association 95: 407–24
7. Efron B, Feldman D 1991 Compliance as an explanatory variable in clinical trials. Journal of the American Statistical Association 86: 9–26
8. Fisher R A 1918 The causes of human variability. Eugenics Review 10: 213–20
9. Gitelman A I (1999) Treatment integrity concerns in comparative education studies. Ph.D. thesis, Carnegie Mellon University
10. Goetghebeur E, Molenberghs G 1996 Causal inference in a placebo-controlled clinical trial with binary outcome and ordered compliance. Journal of the American Statistical Association 91: 928–34
11. Goetghebeur E, Molenberghs G, Katz J 1998 Estimating the causal effect of compliance on binary outcome in randomized controlled trials. Statistics in Medicine 17(3): 341–56
12. Helgeson V S, Cohen S, Schulz R, Yasko J 1999 Education and peer discussion group interventions and adjustment to breast cancer. Archives of General Psychiatry 56: 340–47
13. Holland P W 1986 Statistics and causal inference. Journal of the American Statistical Association 8: 945–70
14. Meyer M M, Fienberg S E 1992 The Case of Bilingual Education Strategies. National Academy Press, Washington, DC
15. Neyman J 1923 On the application of probability theory to agricultural experiments. Essay on principles. Koczniki Nauk Roiniczych, Tom X: 1–51. [Reprinted in Statistical Science 5: 463–80]
16. Robins J M 1994 Correcting for noncompliance in randomized trials using structural nested mean models. Communications in Statistics A 23: 2379–412
17. Rosenbaum P R 1995 Observational Studies. Springer, New York
18. Rosenbaum P R, Rubin D B 1983 The central role of the propensity score in observational studies for causal eff Biometrika 70: 41–55
19. Rubin D 1974 Estimating causal effects of treatment in randomized and non-randomized studies. Journal of Experimental Psychology 66: 688–701
20. Rubin D 1977 Assignment of treatment group on the basis of a covariate. Journal of Educational Statistics 21: 1–26
21. Rubin D B 1978 Bayesian inference causal effects: The role of randomization. The Annals of Statistics 6: 34 –58
22. Rubin D B 1980 Discussion of ‘Randomization analysis of experimental data: The Fisher randomization test,’ by D. Basu. Journal of the American Statistical Association 75: 591–93