Using the randomized design

The analysis. To take advantage of the randomized design, one must use village as the unit of analysis because villages and not individuals were randomized. Most publications describing differences in outcomes between Atole and Fresco villages use the child or mother as the unit of analysis. Thus, the error term used to test these differences has many-fold more degrees of freedom and therefore will result in greater statistical signincance than in analyses using village as the unit of analysis. These analyses do not give the statistical significance relating the treatment to the difference and instead only provide the statistical significance that. the difference is not due to chance. The difference might be due to intrinsic village differences and not due to the treatments themselves. For example, the children of Espíritu Santo, the small Fresco village, had smaller head circumferences than the children in the other villages. When all the children from the Fresco villages are compared with all the children in the Atole villages, the systematically lower values of Espíritu Santo exaggerate the statistical significance of the difference between groups compared with using village as the units of comparison.

Martorell et al. (1982) developed an approach using the consistency of the response to supplementation across the two village sizes and two genders, (i.e., four sex-size groups for each treatment). This paper showed that the lengths of 3-y olds who had lived in Fresco villages their entire lives after the supplementation program began did not differ significantly (P ³ 0.05) in any of the four sex-village size groups compared with 3-y-old children measured before the study in the same groups. The range of change was -0. 7 cm to 1.1 cm with a mean of 0.45 cm. In contrast, the change in the Atole villages relative to baseline values was statistically significant for all four sex-size groups (P<0.05). The range of this secular change was 2.5-3.6 cm with a mean of 2.90 cm. Of course, the statistical probability of the change within each of the Atole sex-village size groups is not the probability that this was caused by Atole. However, the consistency of the changes across the Atole groups compared with the negligible changes observed in the Fresco groups over time make, the inference that the Atole improved growth credible.

TABLE 1 Length¹ of 3-y-old children before end after supplementation by village size and type of supplement

Overall difference in change: mean = 2.45 ± 0.10, t-test = 24.50, P<0.005 (Two-tailed probability; df = 2).

¹ Means of sex-specific data calculated from Table 3 in Martorell et al. (1982).
² Born between 1969 and 1973.
³ Measured in 1965.

A more rigorous statistical test can be made of the above-mentioned changes in Atole (A) and Fresco (F) villages by using village as the unit of analysis (Table 1). This analysis is true to the randomized design and deals with potential intrinsic differences between villages within each pair of similar sized villages by incorporating them into the statistical probability statement.

According to this analysis, the difference in net change (Atole minus Fresco) in the large villages was 2.55 cm and in the small villages it was 2.35 cm (Table 1). The mean of these differences is 2.45 ± 0.10 cm (mean ± SD). Even though the standard deviation only has 2 deg of freedom, the t-test is 24.5 with a twotailed probability of P<0.005. It is well known that the probability statement, P<0.005, means that there is only one chance in a thousand that this difference could be due to chance. What is less well understood is that such a probability statement, except in a randomized design, does not exclude the likelihood, often strong, that the difference is due to something other than the intervention. Only a randomized design incorporates the potential effects of confounding factors into the probability statement. Thus, one can infer, with very little chance of being wrong (P<0.005), that the difference in growth between Atole and Fresco villages was due to difference in the interventions and not to chance or to confounding.

The probability of the t-test shown above is for a twotailed test. However, there is such a clear expectation that the effect of Atole will be beneficial compared with Fresco that it may be more appropriate to use a one-tailed test. In this case, P would be <0.0025.

Potential biases. It is generally well understood that the statistical significance of the above impact cannot be due to initial village differences because these are included in the error term of the test statistic. Similarly, differential changes that occurred among the villages during the period of supplementation also are included in the error term so long as these are not associated with the supplementation.

Also, the effect of the intervention on growth cannot be explained by self-selection to ingest the supplement. A repeated criticism of the study is that children who came for supplementation may have had parents who were more concerned about child health and nutrition and thus, would have grown and performed better anyway. However, this self-selection hypothesis also would predict that the village mean growth would remain unchanged. This, as seen above, was not the case. Therefore, these and similar factors associated with ingestion of supplementation within a village could not affect the comparison across Atole and Fresco groups as presented above. Even differential selfselection where, for instance, the better off children in the Atole villages and the worst off children in the Fresco villages ingested the supplement, would not bias the results in Table 1. Thus, self-selection for ingestion of the supplement within the villages cannot introduce, by itself, biases into the analyses performed appropriately for the randomized design.

The causal statistical significance for an effect of the intervention is impressive, both in its statistical significance and in its exclusion of other factors related to the villages and to those who ingested the supplement. It is important, however, to remember that it does not specify what aspect of the intervention is responsible for the effect. Anything done in the villages that was associated with the supplement could have caused the effect seen. This is why care was taken to spread the INCAP presence equally across the villages through designing and implementing all interventions similarly in all villages, and through rotation of all personnel (Martorell et al. 1995).

One effect associated with supplementation across villages that could not be excluded is the effect of knowing the kind of supplementation a village received. The villagers were, for all practical purposes, "blinded" to this fact because of the distances and the lack of communication among the villages. However, the measurers could not be "blinded". All field workers knew that both supplements were good for mothers and children, so one might expect them not to have been biased. Nevertheless, this possibility must be excluded as described below when discussing the dose response to supplementation.

Another measurement effect that could be associated with supplementation across villages is differential participation in the measurement of outcomes. This could happen, for instance, if better off and worse off participants to Atole and Fresco, respectively, came to be measured. This has been investigated and no evidence of this kind of bias has been found, but this must be kept in mind and verified in each analysis.

Another way that other interventions could have been associated with ingestion of the supplements is if attendance rates were different between Atole and Fresco villages. This is, indeed, the case. Attendance rates were much higher from birth to 3 y in Atole than in Fresco villages (Schroeder et al. 1992). As noted above, this presents no problem if this was because of self- selection within a village. However, differential attendance can result in differential exposure to programmatic influences other than the supplements. For instance, it could have been that those who came to the feeding centers also received better medical care because the clinic and the feeding centers were in the same building. Or, maybe, the socialization experienced in the feeding centers fostered better scores in the behavioral tests. Fortunately, these influences due to differential attendance rates can be taken into account because, in this data set, it is possible to differentiate between nutritional ingestion from the supplements and differential attendance rates. All analyses carried out to date on various outcomes indicate ingestion remains significant after controlling for attendance (see below).

Summary. The randomized design permits a strong inference (P<0.005) that the intervention caused improvements in the outcomes. The component of the intervention that caused the impact must be elucidated by other analyses.