Cover Image
close this bookEarly Supplementary Feeding and Cognition (Society for Research in Child Development, 1993, 123 pages)
close this folderIV. Methodological and substantive considerations
View the documentInherent limitations of the longitudinal and follow-up studies
View the documentSummary

Inherent limitations of the longitudinal and follow-up studies

The complexities of conducting an experimental intervention study are well known. Quasi-experimental designs, in particular, have limitations, and the inferences that can be drawn from their findings are severely restricted by the degree to which underlying assumptions have been met. In the case of the Guatemala longitudinal study, its mere duration over 10 years presents difficulties of control. Moreover, knowledge gained during the course of the study resulted in ongoing changes in its design and in the interpretation of accumulating findings.

In addition to the limitations of the original study design, there are several considerations related specifically to the conditions of the follow-up. The intervention was carried out between 1969 and 1977; for all practical purposes, the study teams had no further contact with the villages from the end of that period until 1987, when the follow-up was being planned. Major life changes for which no data were available could have occurred during this 10-year gap. In addition, the fact that the original subjects were now adolescents and young adults presented additional methodological difficulties. This chapter addresses some of the most serious concerns regarding the design of the longitudinal study as well as issues that relate specifically to the follow-up.

Equivalence among Villages

Ideal experimental conditions prevail when the only relevant variable that distinguishes the experimental and the control groups is exposure to treatment. One of the most notorious risks associated with quasi-experimental studies is the nonequivalence of groups at pretest (Campbell & Stanley, 1963). Any non-treatment variable that is correlated with the treatment could explain any of the differences in outcome variables, which cannot then be attributed unequivocally to exposure. Similarly, if the treatment and control groups differ along the outcome variables before exposure, then differences observed after exposure cannot be attributed to the treatment.

One way to avoid such potential differences between treatment and control groups is the randomization of treatment. In the Guatemala study, randomization occurred at the village level: one large and one small village were randomly assigned, respectively, to Atole and Fresco. With four villages and randomization occurring only within pairs, it is very likely that errors were not distributed randomly across villages.

Apart from the randomization, villages were assumed to be equivalent in terms of important variables related to the intervention, and the validity of inferences about the effects of treatment rests on the validity of these assumptions. In the case of the Guatemala study, sociodemographic, behavioral, and nutritional equivalence was necessary to control for the potential confounding effects of these factors. Data establishing the initial equivalence of children in Atole and Fresco villages in terms of height, weight, and home diet have been reported (Martorell, Klein, & Delgado, 1980); however, there are no previous analyses testing whether the sociodemographic or cognitive characteristics of the population were equivalent.

Sociodemographic Characteristics

To ascertain initial equivalence in socioeconomic status, we performed comparisons between villages using a 2 (treatment) x 2 (size) analysis of variance with unbalanced cells (General Linear Model; SAS, 1988) for the continuous measures and a chi-square analysis of treatment within village size for the categorical variables. Because of the original matching of villages by size and the possibility that socioeconomic characteristics would be in part a function of size, village size was included in these analyses.

The measures used in these analyses were derived from 1967 census data (specifics of their derivation are given in Chap. V) and reflected family background variables believed to have theoretical associations with the developmental outcomes: parents' education and literacy levels, house quality, and father's occupation. Since only 5% of mothers reported being employed outside the home, mother's occupation was not analyzed. These measures were constructed for all residents of each village.

Our analysis showed that, in 1967, fathers in the Atole villages had significantly higher-ranked occupations than those residing in the Fresco villages, F(1, 508) = 5.61, p <.05. While Atole fathers were more likely to farm their own land (62% vs. 27% for Atole and Fresco, respectively), fathers in Fresco villages more frequently worked as tenant farmers (50% vs. 27% for Fresco and Atole, respectively). In contrast, both mothers and fathers in the Fresco villages were more likely to be literate than those in the Atole villages (34% vs. 26% for mothers, c²[1] = 3.72, p <.05; 53% vs. 38% for fathers, c²[1] = 12.49, p <.01). There were no differences in house quality by village type.

In sum, the data indicate that the original assumption of the equivalence of social and economic characteristics between villages was not justified: fathers in the Atole villages had higher occupation levels, while both mothers and fathers in the Fresco villages were more likely to be literate. Village size did not covary in any consistent way with any of the socioeconomic variables and was therefore excluded from all further analyses reported in the Monograph.

Cognitive Abilities

Family economic background and parents' education are known to be positively associated with children's cognitive development (Garcia-Coil, 1990; Laosa, 1984; Sigman et al., 1988). Given that the villages were not equivalent in terms of the relevant social and economic parameters, it is reasonable to suspect that these pretreatment between-village differences in SES could have been associated with pretreatment cognitive differences. If this were the case, then such differences could be related to differences in cognitive outcome in the adolescent period.

A substitute for the unavailable pretreatment information concerning cognitive abilities is the psychological test scores of the 5-7-year-old children from the Atole and the Fresco villages that were obtained in 1969, during the first year of the study. In comparison to other cohorts, these children would be less likely to have benefited from the supplement because of their late exposure to it. About 50 children from each pair of villages were compared at each of the three ages on the 12 preschool tests administered during this first study year. No consistent differences were found in favor of either group. Among the 5-year-olds, two tests (Incidental and Intentional Learning) favored Fresco children; at 6 years, there were two differences (Digit Memory and Verbal Inferences) in favor of Atole, and, at 7 years, the one difference (Digit Memory) favored Atole as well.

In sum, although social and economic differences between Atole and Fresco villages did exist, the comparisons of preschool cognitive test performance point to no systematic difference in favor of either group; out of 36 comparisons, 8% favor Atole and 6% Fresco. We conclude, therefore, that pretreatment preschool cognitive differences between Atole and Fresco subjects could not account for any differences observed in the adolescent period.

Changes in the Villages - the 10-Year Hiatus

The lack of contiguity between the time of exposure to the treatment and the time of assessment of the adolescent outcome variables makes it nearly impossible to discard the "history" (Cook & Campbell, 1979) threat to validity since a different rate of social and economic growth from village to village could affect differences in the cognitive growth of residents (Bronfenbrenner, 1979). In the following discussion, we contrast data on different parameters of social and economic growth that occurred between 1967 and 1987. 4

4 Suzan Carmichael was responsible for the data analysis of SES changes in the villages. We gratefully acknowledge her contributions to this chapter.

In two of the villages, the main water source had not changed noticeably over the intervening 20 years, and more than 90% of the families still obtained their water from a public well or spigot. In the other two villages, where over 90% of the families had obtained their water from a river or hand-dug wells in 1967, almost all had access to a public spigot or well by 1987, and about half had homes with piped connections to the water system (see Table 5).

In spite of these improvements in water sources, by 1987 only about 5% of the households in each village were hooked up to a piped network for drainage purposes. Similarly, methods of human waste disposal were still not well developed, with only about half the families having any means of sanitation, such as rudimentary septic tanks or latrines in their homes

Several of the changes that occurred in house quality are notable. As a result of a major earthquake in 1976, many tile roofs were replaced with sheet metal. Cement floors replaced dirt floors in over one-third of the homes in each village (Table 5). As represented by a factor score, the changes in house quality displayed in Table 6 indicate that overall quality increased over time and at similar rates among the villages, although less improvement was evident in one Atole village. Whereas electricity was unavailable in 1967, by 1987 two-thirds to three-quarters of all families had electricity in their homes. Although no differences in house quality were reported in 1967, by 1987 Fresco villages had significantly higher house factor scores than Atole villages. However, although the number of all household possessions had increased, their availability still remained low in 1987. Only 5% - 10% of families owned bicycles, refrigerators, or record players, while 15%-30% owned television sets.

The primary means of income for most villagers continued to be agricultural production, with most heads of households employed as tenant farmers or small landowners (see Table 7). The main change in occupation over the years was that the percentage of men involved in skilled trades had increased from very few in 1967 to 47% in 1987 in one Fresco village and to between 20% and 30% in the other three villages. By 1987, the previous differences between Atole and Fresco in occupation levels among fathers were no longer statistically significant. Women continued infrequently to report having occupations.

TABLE 5: CHARACTERISTICS OF HOMES BY VILLAGE AND YEAR (%)


Fresco 03

Atole 06

Fresco 08

Atole 14

Percentage of homes with:

Some means of disposal of human waste:


1967

7

8

1

5


1974

15

19

24

12


1987

46

65

54

66

Well in house or hookup to public network:


1967

0

3

7

0


1974

<1

21

15

<1


1987

7

82

71

9

Nondirt floors:


1967

5

5

12

2


1974

13

13

16

10


1987

62

27

38

28

Adobe walls:


1967

59

31

11

71


1974

83

47

20

83


1987

82

73

7

65

Separate kitchens:


1967

64

40

56

77


1974

87

61

69

87


1987

94

85

82

88

NOTE. - Villages are identified by census codes.

TABLE 6: CHANGES IN SES INDICATORS BY VILLAGE AND YEAR


Fresco 03

Atole 06

Fresco 08

Atole 14

House factor scores:


1967

2.08

1.77

1.64

2.15


1974

3.18

2.64

2.39

2.97


1987

4.02

3.52

2.39

3.46

Mothers' literacy (%):


1967

40

25

26

29


1974

51

37

35

39


1987

63

58

58

63

Fathers' literacy (%):


1967

47

38

60

38


1974

56

49

67

44


1987

65

66

76

61

Population:


1967

811

854

549

469


1974

995

1,073

760

672


1987

1,636

1,640

1,120

1,135

NOTE. - Villages are identified by census codes.

TABLE 7: PRINCIPAL OCCUPATION OF FATHERS (%)


Fresco 03

Atole 06

Fresco 08

Atole 14

Agricultural Wage labor:


1967

6

3

21

15


1974

5

7

29

4


1987

5

5

38

2

Tenant farmer:


1967

58

28

42

26


1974

54

33

30

34


1987

29

19

22

28

Small landowner:


1967

26

68

28

56


1974

27

53

28

55


1987

17

52

15

40

Skilled trade:


1967

8

<1

7

0


1974

14

7

11

4


1987

47

21

21

28

Merchant:


1967

1

1

2

2


1974

1

0

2

3


1987

2

2

4

3

NOTE. - Villages are identified by census codes.

Self-reported literacy increased over time at similar rates in all villages, with the same villages maintaining the highest literacy levels. The gap between the literacy rates of women and men was greatly reduced by 1987, except in one Fresco village, where the 76% literacy rate for men was higher than either male or female literacy rates in any of the other villages (see Table 6). The mean level of schooling increased at a much slower rate, with the average years of schooling, like literacy, being higher for men than for women. Differences in parents' literacy rates reported in 1967 continued to be evident, and both mothers' and fathers' levels of education were significantly higher in Fresco than in Atole villages.

Table 6 presents the change from 1967 to 1987 in the total population of each village. Growth rates for the villages appeared similar, with the population in each approximately doubling since 1967. Nuclear family size (mother, father, and offspring living with them) remained fairly stable across the 20-year span (4.6-4.7 in 1967 and 4.6-5.0 in 1987).

In summary, life in the villages had changed, and in many ways improved, over the 20 years since the initial study: electricity and water were more readily available, house quality improved, and illiteracy rates declined. Nonetheless, the change was slow and may not represent substantial reductions in the risks posed to child welfare. For example, although the average number of years of schooling of the adult population in 1987 had approximately doubled since 1967, this change represented an increase of only 1 year of schooling. Similarly, half the families still carried all their water, and in none of the villages was there adequate gray water or sewage disposal.

The history factor cannot be fully discarded in the present study, and anecdotal data suggest that between-village differences in social and economic development may in fact have existed. For instance, the Institute for Cultural Affairs in Guatemala began a large development project in one of the Atole villages shortly after the supplementation study finished in 1977. Bank loans were made available to invest in drip irrigation, running water was installed, and, more recently, a preschool was begun, enabling women to work outside their homes. However, the possibility of between-village differences in social and economic growth must be tempered by the fact that the data collected on social and economic changes in the villages do not point in that direction. There were no marked differences between villages in improvements in water sources, changes in house quality, household possessions, occupation, or parents' education.

Implementation of Treatment

As noted above, an essential element of experimental design is the assumption that the only difference between the experimental and the control groups lies in exposure to treatment. In the case of the Guatemala study, the only difference between groups should have been the nutritional aspects of the intervention. In fact, there were potentially quite important nonnutritional differences between the two treatments: beyond differences in both its real and its perceived nutritional value, Atole differed in name, appearance, and taste.

A second issue relates to the implementation of the intervention. Were there any nonnutritional factors in the delivery of the treatment that might have had effects on the outcomes similar to those expected from the nutritional intervention? Two factors are relevant here: the treatment was not blind, and it was extended to include the mothers.

Because the treatment was not blind, there is a possibility that the field-workers in the Atole and Fresco villages behaved differently toward the respective participants. For instance, field-workers might have given greater encouragement to the mothers to attend the feeding station in the Atole than in the Fresco villages. Consequently, rates of attendance at the feeding stations might have differed between villages. This, in turn, could have resulted in more frequent social interactions between participants and field-workers in the Atole than in the Fresco stations, which could have had a differential effect on the children in the two groups. Although no data are available on interactions between field-workers and subjects or on actual promotion of participation, attendance data are available and will be discussed in a later section.

Distribution of the supplement to the mothers who accompanied the children to the feeding station poses another problem: Atole, but not Fresco, may have had a salutary effect on caretaking behavior, which, in turn, could have fostered cognitive growth in the offspring. This issue, however, cannot be fully addressed because treatment effects on the mothers were not assessed and relevant records (e.g., mothers' weight changes) are restricted to specific periods (i.e., pregnancy and lactation). Hence, it is not possible to test whether treatment effects on mothers occurred and, if so, whether they might act as potential cofounders.

Patterns of Attendance

The original study design assumed that provision of Fresco controlled for the possible confounding effects of attendance and that any differences in outcomes between the Atole and the Fresco subjects could therefore be attributed solely to the nutritional intervention. However, this assumption is valid only if attendance patterns proved to be similar across villages and if the correlates of attendance were also similar in all four sites. We noted earlier that the perception of the benefits of Atole over Fresco and/or selective encouragement by field-workers due to nonblind treatments may have affected participation. Here, we examine data to assess across-village similarities in patterns of attendance and associations between socioeconomic variables and attendance and between attendance and outcome variables.

Attendance rates were computed as the number of times (morning and/or afternoon) attended per year for each year of the child's life (1-7). A summed composite of the seven individual years was also computed. Atole/Fresco differences in attendance were examined by t tests for each of the 7 years and for the composite; the results of these analyses are shown in Table 8.

During the first 5 years of life, attendance at the feeding stations in the Atole villages was significantly greater than in the Fresco villages; this pattern was reversed in year 7, when attendance became higher in the Fresco villages. It is consequently possible that any differences in outcome found between Atole and Fresco villages may be explained in part by attendance rather than by nutritional differences. The findings also suggest that the motivation for going to the respective treatment centers may have differed. Thus, mothers in Atole villages may have taken their small children more frequently to the treatment center in the belief that the Atole had nutritional benefits, while mothers in the Fresco villages may not have perceived the Fresco as being advantageous to their children. Conversely, since older children were able to attend on their own, the higher attendance in Fresco villages at age 7 might be explained by the children's preference for the taste of the Fresco over the Atole.

TABLE 8: MEAN NUMBER OF TIMES ATOLE AND FRESCO SUBJECTS ATTENDED A FEEDING STATION BY YEAR OF PARTICIPATION

Attendance Year

N

M

SD

t

Year 1:


Atole

340

179.51

109.56

15.42***


Fresco

318

71.98

65.15


Year 2:


Atole

395

191.52

110.68

9.26***


Fresco

369

123.40

92.42


Year 3:


Atole

440

207.34

107.91

7.52***


Fresco

415

153.14

102.64


Year 4:


Atole

510

203.84

103.55

3.95***


Fresco

462

177.79

101.75


Year 5:


Atole

546

206.27

100.10

2.87**


Fresco

520

188.29

103.97


Year 6:


Atole

556

188.15

98.53

.08


Fresco

534

187.69

104.97


Year 7:


Atole

515

182.78

99.23

2.51**


Fresco

505

198.91

105.88


ATTEND: a


Atole

772

833.21

643.67

4.20***


Fresco

728

704.26

543.28


a Attendance summed over all participation years.

**p <.01.

***p <.001.

Attendance and SES Measures

In general, the associations in both Atole and Fresco villages between SES indices and the attendance measures were negative, small, and often not statistically different from zero. Mother's years of schooling was the SES variable most highly correlated with attendance, and the correlations reached significant levels at 1, 2, and 3 years in the Atole villages (r = -.20, -.21, and -.23, respectively).

Attendance and Preschool Factor Scores

Correlational analyses yielded significant positive associations (ranging from r =.11 to r =.28) between Factor 1 scores (see Chap. II) obtained at 3, 4, 5, and 7 years and attendance at most ages in the Fresco villages. Conversely, in the Atole villages, the associations were small and nonsignificant.

Significant correlations were obtained in the Fresco villages between attendance at 2, 3, and 4 years of age and Factor 2 scores at age 3 years (r =.15,.18, and .22, respectively). In the Atole villages, the only significant associations reflected a negative relation (ranging from r = -.10 to r = -.21) between attendance from 1 to 6 years and Factor 2 scores at age 5 years.

Subsequent regression analyses of the preschool factor scores on attendance, controlling for socioeconomic status, indicated that attendance was significantly associated with Factor 1 scores and that these associations differed by treatment. The coefficients suggested that, for every 1,000 days of attendance in the Fresco villages, the factor score increased by four-tenths of a standard deviation, reflecting a moderate effect size. At the average rate of attendance at the supplementation centers - which was 182 days per year - 5.5 years would be required to observe an effect size of this magnitude.

In summary, between-village differences in attendance did exist. In comparison to the Fresco sites, subjects in the Atole villages attended the station significantly more often during the first 5 years of life. The opposite held for the oldest (7-year-old) children, who were more likely to attend the Fresco than the Atole stations. Attendance, in turn, maintained a different relation with test performance in the two groups, being positively related to the two Preschool Battery factor scores in the Fresco but not in the Atole villages. Although the probability of observing an effect of attendance was limited, we nevertheless decided that further analyses should control for the potential confounding of attendance.

The possible reasons for the observed differences in attendance are numerous, and it is likely that they varied across time as a function of the treatment and its interaction with the individuals. In terms of available data, a recent analysis of children's participation in the Guatemala study between 1969 and 1977 indicated that proximity to the feeding center and larger family size were significant predictors of attendance. Lower socioeconomic status also proved to be associated with attendance in Atole but not in Fresco villages (Schroeder, Kaplowitz, & Martorell, 1992).

Assumption of a True Treatment

Any test of the effects of a nutritional intervention on developmental outcomes is grounded in the assumption that an intervention in fact occurred. It is possible, however, that, even though subjects indeed consumed the supplement, they may have concurrently reduced their home intakes. If this were the case, then it is conceivable that the net intake of nutrients of the Atole and Fresco subjects did not in fact differ. Lechtig et al. (1975) examined this issue in connection with assessing the energy intake of pregnant women enrolled in the longitudinal study and concluded that the supplement did result in a net increase of total calories. Further, we also examined the dietary data of children whose high Atole intake placed them at particularly high risk of substitution and found that the energy intake from the home diets of subjects whose Atole supplement intake was above 200 kcal per day did not differ from that of those whose intake was below 200 kcal per day. Taken together, these results suggest that the supplement was not used as a substitute to replace home diet.

A second level of analysis aimed at establishing the presence of a nutritional experiment centers on whether variables known to be sensitive to changes in nutritional status did indeed change as expected. In the absence of laboratory data, the answer to this question can be found only in an analysis of physical growth. In the case of the Guatemala study, Martorell et al. (1980) reported that greater supplement intake was associated with improved growth in the length, weight, head circumference, and arm length of preschool-aged children.

In summary, we concluded that the Atole was indeed a nutrient intervention and that variables that are sensitive to nutritional status (e.g., growth) were, in fact, affected by the intervention.

Different Histories

Another potential threat to the internal validity of the study arises from differences in the history of each village, as well as of each subject, in the period between the time the study was initiated (1969) and the follow-up (1988). Of particular importance are any educational differences between villages and between subjects. By the time of the follow-up, all the subjects were past both the legal (7 years) and the average (8.3 years) age at which children first enroll in school. In the initial hypotheses of the follow-up study, we conjectured that, if observed, beneficial effects of the treatment would be consistent at the individual level across measures of school performance and psychoeducational test performance. In addition, given that schooling is an important contributor to the development of cognitive abilities (Ceci, 1991), any assessment of cognitive skills requires controlling for school experiences, particularly in light of the vast individual differences in levels of school performance among the village subjects.

One potential problem with this approach is that any of the possible differences between the schools could be responsible for differences attributed to the treatment. Data on school quality would provide information directly related to differences in performance achieved in different schools; however, data of this nature are unfortunately not available. Furthermore, with only one school per village, statistical control for between-school differences would not be possible even if the relevant data were on hand.

We do know that all four schools were of low quality. None had the administrative infrastructure that is common in the industrialized world, teachers had no more than a secondary education, classrooms included more than one grade, ages within a grade spanned several years, and the decisions as to who should be promoted were not based on standardized criteria.

Under such conditions, it is highly unlikely that cognitive skills and abilities would be the sole determinants of school advancement (Gorman & Pollitt, 1993). Furthermore, it has also been reported that certain school indicators varied as a function of work-related tasks and opportunities available in the given village (Balderston, 1981). We also know from the data on parents' education levels that both fathers and mothers from Fresco villages had attended school longer and were more likely to be literate than Atole mothers and fathers.

Data on different school indicators immediately point to sizable differences that favor the Fresco rather than the Atole villages. For example, at the school level, first-grade promotion rates in Fresco villages averaged .54 over an 8-year period (1975-1982), whereas the rate for Atole villages was only .37. Similarly, repetition and dropout rates were higher in Atole villages (.42 and .21, respectively) than in Fresco villages (.32 and .14). At the individual level, a higher percentage of Fresco subjects (94.8) enrolled in school than Atole subjects (88.3), and significant differences in the number of grades passed and failed and the maximum grade attained also favored Fresco villagers.

When taken together, these data suggest - albeit tentatively - that there were substantial differences between the village schools that would work in the opposite direction from that hypothesized on the basis of the intervention. We are aware of the fact that these comparisons are based on individual level data rather than more desirable independent school quality indicators and hence could be given alternative interpretations. The parents' education and literacy data, however, are consistent with and provide strong support for our interpretation of these data.

The final concern in establishing between-village equivalence is whether selective attrition and/or recruitment for the follow-up may have differentiated between the Atole and the Fresco sites. This issue is examined in the methods section of the following chapter.