Confounding variables and evaluation design

The validity of a research design is a measure of the degree to which its conclusions reflect real phenomena.

"Internal validity" of a design refers to the extent to which the detected outcome changes can be attributed to the intervention or treatment rather than to other causes. Unless the internal validity of a design is high, the finding that a particular relationship is causal will not be particularly convincing. Some of the major threats to the internal validity (i.e. confounding factors) are summarized in table 1.2. The primary reason for the proper choice of a comparison group and for statistical adjustment techniques is to control for these threats as best as possible when randomized allocation is not feasible. The expression "gross outcome" refers to a measured change in the outcome variable in the population without controlling for the threats to internal validity. Gross outcome does not eliminate the effects of confounding variables and therefore does not enable the evaluator to distinguish between change that occurred as a result of the programme and change that would have occurred anyway because of other factors. "Net outcome," however, does explicitly address those factors, other than the programme, that bring about measured changes in outcome variables. Net outcomes thus control for the numerous threats to internal validity.

In addition to internal validity, evaluators must be concerned with the external validity of the evaluation. External validity refers to the generalizability of the conclusions drawn to other populations, settings, and circumstances. Both the internal and external validity of an evaluation are fundamentally functions of the design chosen. Simply, each of the commonly-employed designs for evaluation displays a different ability to control for threats to internal and external validity.

While it is beyond the scope of this chapter to discuss the conventional non-experimental, quasi-experimental, and true experimental designs, and the extent to which they address confounding variables, table 1.3 diagrams six conventional designs. The reader who is unfamiliar with or uncertain about the array of available designs is urged to consult directly standard works on this subject such as those by Cook and Campbell (10), Poister (11), or Judd and Kenny (12). However, let it suffice to suggest that choosing from among the conventional techniques involves a trade-off between the difficulties of data collection, first on comparison groups and, second, over time, with the plausibility of the causal inference drawn. They also depend to some extent on the analytical capacity available, as discussed in the next section.

TABLE 1.3. Conventional Evaluation Designs

* X= treatment. For items 1-3, 0 = observation of outcome; for items 4-6,
O = observation of both outcome and confounding variables