| Food Composition Data: A User's Perspective (1987) |
|A system for evaluating the quality of published nutrient data: Selenium, a test case|
The main purpose of evaluating nutrient data is to eliminate poor-quality data, leaving only reliable information for the calculation of a mean value to be used in tables and data bases. A CC of a can mean that only three studies, two of them excellent, have been published or that a great deal of minimally acceptable data exists. In the case of Se data in general, a CC of a signifies the second situation. Most of the studies reporting the Se content of foods were assigned QIs between 1.0 and 2.0 out of a possible 3.0. A CC of c implies few data of minimally acceptable quality exist. A CC of b indicates data quality falling between a and c.
Combining and interpreting data from different studies presents some unique challenges to the nutrient data evaluator and statistician. Specifically, the biases of each study must be taken into account: biases based on different samples, analytical method, reagents, instrumentation, analysts' performance, and degree of accuracy and precision for each study. Usually these biases for a given study are not quantified or documented. Differences in the mean values for various studies cannot easily be evaluated when laboratories analyse samples obtained from different sources, and use different handling techniques, reagents, etc. The calculation of a mean nutrient value across studies can be performed in several ways. Weighting strategies were of particular interest. Weighting the mean towards the number of samples in the studies is one approach that was considered: data from studies that reported analyses of the largest numbers of samples would be weighted most heavily. However, this approach would attach greater significance to the number of samples category than to the others. Another approach might be to weight most heavily data from those studies with the smallest variance. However this is not always possible because a standard deviation or standard error is sometimes not reported. A third approach can weight most heavily data from the study with the highest QI. This was also rejected owing to the narrow range of QIs and the subsequent lack of resolution in the scale. In view of the limitations discussed, weighting was deemed undesirable, and it was agreed to calculate a simple mean Se value at this time. As documentation and data quality improve, a weighting strategy could be considered for calculation of the mean nutrient value.
The derivation of the QI and consequent CCs also can be approached in several ways. A conservative scheme would be based on the assumption that the quality of a study is only as great as its weakest aspect, as was the system in the iron table [X]. From this viewpoint, the QI for each study would be equal to the lowest of the five ratings. However, applying this method of scoring to the existing Se studies would have resulted in very few acceptable data, since 0 is a frequent rating, especially in analytical quality control. Also, making the QI equivalent to the lowest rating would weight the QI toward the category with the greatest number of zero ratings. To avoid these consequences, a less conservative approach was taken which considered that: (a) sometimes quality-control measures are taken during the course of research, but not reported; and (h) the actual values found in Se papers with no mention of quality control often fall within or close to the range of values reported in Se papers that report appropriate qualitycontrol measures. This holds true for the examples shown here. For the purpose of having enough acceptable data, standards have been adjusted. However. considering this compromise on the derivation of the Ql, one safety feature was added to the calculation of the mean Se value: the exclusion of values with an index smaller than 1.0. This feature requires a study to meet a minimum level of overall quality for its data to be included. The minimum acceptable Ql was set at 1.0 because that seemed to be a reasonable cut-off point, i.e. a higher cut-off point would eliminate the majority of studies.
Users of these data should be aware that the mean Se value for each food may not be representative of average levels found in the nation's food supply. Acceptable mean values were derived from available data from one or several studies. Individual criteria were not weighted, and even low ratings for sampling plan would not disqualify a study, depending on the other ratings. However, in each case the mean value represents the best present estimate of Se in a given food item.