Cover Image
close this book Food Composition Data: A User's Perspective (1987)
close this folder Other considerations
close this folder A system for evaluating the quality of published nutrient data: Selenium, a test case
View the document (introductory text)
View the document Introduction
View the document Background
View the document Procedure
View the document Criteria
View the document Calculation of the mean SE value and confidence code
View the document Results
View the document Discussion
View the document Implications
View the document Acknowledgements
View the document Disclaimer
View the document References

Criteria

Criteria

Number of Samples

Statistical rigour requires that the appropriate number of samples for a study be a function of the nutrient variability within the population of each food item [6].

Table 1. Data quality criteriaa

Ratings 3 2 1 0
Number of samples >10; SD, SE, or raw data reported 3-10 1-2; explicitly stated or not specified -
Analytical method Official fluoro-metric (ref. Given) or other method documented by a complete published write-up with validation studies for foods anaIysed, including use of appropriate SRM where available, 95 105 per
cent recoveries on food similar to sample analysed in same or other paper; Se concentration above quantitation limit of the method
Modified fluoro-metric or other method, some documentation, incomplete validation studies for
foods analysed; must include 9W 110 per cent recoveries on food similar to sample anaIysed (or good recovery but no statistics given), and/or use of other method (official fluorometric, isotope dilution or NAA) on same sample with good agreement (within 10 per cent)
Non-fluorometric method, partially described; 80-90 per cent or > 110 per cent recoveries on food similar to sample; or use of comparison method or recoveries on food only some-what related to sample (animal/plant) No documentation of method, no ref. or in accessible ref. given, no validation studies, or poor agreement (>10 per cent) of test method with comparison method on same sample
Sample handling Complete documentation of procedures, including
analysis of edible portion only, validation of homogenization method, details of food preparation, and storage and moisture changes monitored
Pertinent procedures documented, including analysis of edible portion only; procedures seem reasonable but some details not reported Limited description of procedures, including evidence of analysis of edible portion only Totally inappropriate procedures or no documentation of criteria pertinent to food analysed
Sampling plan Multiple geographical sampling with complete description; sample is representative of brands/varieties commonly consumed or commercially used 1 or 2 geographic areas sampled; sample is representative Sample representative of small percentage of US and/or origin not clear Not described or sample not representative
Analytical quality control Optimum accuracy and precision of method monitored and indicated explicitly by data Documentation of assessment of both accuracy and precision of method; acceptable accuracy and precision Some description of minimally acceptable accuracy and precision of method No documentation of accuracy and/or precision

a. See text for complete description of criteria.

Both intrinsic and extrinsic sources of variation affect the levels of Se in foods. However, when evaluating published data, one rarely has access to the magnitude of specific sources of variation for a given food. The standard deviation, when given, is an indication of total variance. Some variability is assessed indirectly by the other categories of criteria and includes: systematic error intrinsic to the analytical methodology or sample handling, variability attributable to differences in brands and varieties of foods analysed, and errors in analytical accuracy and precision in the execution of the analytical method. A statistical formula can be used to estimate appropriate sample size given a mean, standard deviation, and level of acceptable error [6]. However, the error term selected would depend upon the concentration of the nutrient in a given food item and the detection and quantitation limits [11] for the analytical method. Such a judgement could be made given sufficient data for each food. Using a predetermined coefficient of variation of 20 per cent as the limit of acceptable error, we estimated appropriate sample size for a limited number of studies where standard deviations were reported. Our estimate and the actual number of samples analysed in each study were comparable.

In the absence of adequate information for many studies, particularly standard deviation data, we chose to make a somewhat subjective judgement on the sample size limits for each rating: a rating of 1 for one to two samples or when number of samples is not specified; a rating of 2 for three to ten samples; and a rating of 3 for greater than ten samples and inclusion of the standard deviation, standard error, or raw data from which a standard deviation can be calculated. A rating of 0 is not applicable in this category. As documentation improves, it will be possible to evaluate the appropriateness of the number of samples analysed based on statistical considerations.

Analytical Method

Several issues related to the method of analysis are important with regard to ratings on analytical method. Documentation of whatever method is used is primary: suitability of method cannot be determined if no description or reference to details in another paper is included. A rating of 0 is assigned where the method is not described, no reference is given, or a reference is generally unavailable. In some cases the use of "official" methods for the analysis of the specific foods merits a higher score. The use of the official fluorometric method for Se analysis, as published in the handbook of the Association of Official Analytical Chemists (AOAC) [27], is rated a minimum of 2. However, use of an "official" method does not preclude attention to a second important issue: validation of the method.

Validation of the test method for the general matrix (e.g. meat, grain, fat), and preferably for the specific food item in question, is necessary to show that accurate results can be obtained. Use of recovery trials on the same or a similar food is one aspect of method validation. Higher ratings are earned for recovery close to 100 per cent and similarity to the food analysed of the food on which the recovery trials are done. The use of a highly regarded second analytical method on the same or a similar food is another aspect of method validation. In the case of Se, highly regarded analytical methods are the fluorometric method approved by the AOAC, neutron activation analysis, and isotope dilution-mass spectrometry. The analytical results of this second method must be in good agreement with the results of the test method, or a 0 is assigned. Good agreement is defined according to our adaptation of Stewart's general recommendation that the values obtained by comparable methods should be within 10 per cent of each other if a daily intake of the food provides greater than 5 per cent of the US RDA for that nutrient [25]. Since there is no US RDA for Se, we have used 5 per cent of the lower end of the estimated safe and adequate daily dietary intake of 50 200,ug of Se, as recommended by the Food and Nutrition Board of the National Research Council [7].

The analysis of a standard reference material (SRM) by the method in question and comparison of the value obtained with the certified value and the range of estimated uncertainty of the SRM is useful for validation when the matrix and Se concentration of the SRM are similar to that of the food in question [28]. However, the small number of SRMs that are certified for Se has confined this aspect to a rating of 3 for analytical method. As the availability of SRMs with a variety of matrices increases, this aspect of the criterion will be required for ratings below 3.

Finally, evidence that the analysis is carried out above the quantitation limit of the method, as defined by the American Chemical Society Committee on Environmental Improvement [11], is required to assure that the method can determine expected levels of Se in the food to be analysed. For a rating of 3, the quantitation limit must be defined and be below the Se level reported for the food in question.

Other issues with regard to analytical method are of some concern, but were considered secondary to the main points. One such issue is that of the size of the samples that are analysed. Sample size must be adequate relative to the sensitivity of the method so that (a) the concentration is above the quantitation limit, and (b) the sample analysed is representative of the whole food item. This concern was not included in analytical method requirements.

In summary, three concerns must be satisfied for data to be rated 3 in analytical method: (a) a complete description of the method in the same or another accessible publication; (b) validation studies for the food in question, which can consist of either recovery trials with 95 per cent to 105 per cent recovery of Se or comparable results with use of a second method that is highly regarded, as well as use of an appropriate SRM when available; and (c) reported analytical values above the defined quantitation limit of the method.

Sample Handling

How a sample is handled from the time of acquisition to the time of analysis is critical for general nutrient stability. For example, preventing the loss of volatile components is important for maintaining relative nutrient concentration, a factor of importance for Se. Therefore, documentation of sample handling protocol is essential to evaluate data pertaining to the nutrient composition of a food. Lack of documentation of sample handling procedures or use of inappropriate procedures is rated 0. Se contamination of food samples via utensils, cooking ware, grinder, or containers is not a problem, in contrast to the analysis of other inorganic nutrients such as zinc and chromium. However, details of homogenization, temperature control, and other aspects of sample preparation must be known to evaluate the representativeness of a sample aliquot taken from a large batch of prepared material.

Analysis of the edible portion must be reported for a rating of 1 or higher. For example, some canned foods must be drained, raw fruits and vegetables must be peeled or cored, and meats must be boned and trimmed of fat if these foods are generally eaten that way. Thorough homogenization of the food is critical for food items with diverse constituents. Examples of such foods are: breaded and fried fish or poultry, food mixtures, and fruits or vegetables eaten with skin or seeds. Ideally, thoroughness of homogenization is checked by analysing portions from various parts of the final mixture. Additional factors that should be reported for the highest rating are: detailed description of the food, including processing methods (e.g. whether rice was polished, unenriched, or instant); cooking method (if any); general storage conditions, e.g. frozen foods kept frozen, fresh foods analysed soon after pick-up; and measurements of moisture/volatiles content.

Sampling Plan

The sampling plan of the study reflects the representativeness of the samples with regard to the brand or cultivar, method of preparation, and geographic origin of the food. Is the particular food item typical of what many Americans eat? No description of the sampling plan or the use of a non-representative sample is rated 0. This would include a food grown under experimental soil conditions, food grown in someone's home garden, or food prepared in an unusual way. Data from Canadian studies were evaluated because some foods sold in Canada are grown in the United States - e.g. fruits and some grains - and therefore can be representative of what Americans eat. Data for foods grown on Canadian soil but not exported to the United States were given a lower score (0 or 1) for sampling plan since the concern of this work was foods consumed by Americans. The use of popular brands and frequently consumed forms of foods was rated 2. Obtaining representative samples from supermarkets in more than two wellpopulated areas was rated 3. For fresh foods obtained from growers or producers, representativeness of cultivar and geographic source was assessed by referring to Agricultural Statistics [29].

Analytical Quality Control

Information that details acceptable accuracy and precision in the day-to-day execution of an analytical method is necessary for evaluating the quality of nutrient data. Accuracy and precision are each rated as separate aspects of analytical quality control. For each datum, the lower rating of the two aspects determines the rating in this category.

Accuracy is the degree to which an analysed value represents or estimates the "true" value. An investigator must demonstrate that the method is capable of accurately determining the nutrient level in a particular food item; that is, a method must be validated for each general matrix, as described in the analytical method section above. Once the method has been validated, it must be carried out appropriately each time an analysis is performed. Accuracy in the day-to-day use of a method is one of the two elements that must be monitored and reported for a study's data to be rated favourably in analytical quality control.

Day-to-day accuracy is monitored by analysis of a quality-control material that is similar in matrix and nutrient concentration to the test sample. Analysis of such a material should be included with each batch of unknowns or on each day of analysis if several batches are run in a day. Quality-control materials can be SRMs such as those available from the National Bureau of Standards (NBS), which are certified for specific nutrients, or they can be secondary reference materials, i.e. materials developed especially for a study and characterized by one or more methods, including reference methods. NBS SRMs currently available and certified for use in evaluating the accuracy of a method for the determination of Se in foods are: Orchard Leaves 1571 [17], Wheat Flour 1567 [20], Rice Flour 1568 [19], Bovine Liver 1577a [15], Oyster Tissue 1566 [18], and Non-Fat Milk Powder 1549 [16].

Table 2. Accuracy requirements for secondary reference materials

Analysed mean value of secondary reference material should fall within the reference mean value ± To receive a rating of
2 standard deviations 3
2 1/2 standard deviations 2
3 standard deviations 1

When an NBS SRM is used to judge accuracy, the analysed mean value must fall within the mean plus or minus the estimated uncertainty, as stated in the NBS certificate, for the study to be rated a 3. Use of an SRM with results falling outside the range of estimated uncertainty is rated 0. This apparently stringent rule is based on the sources of SRM values. Many certificates of analysis do not label the uncertainty in statistical terms such as confidence intervals or a certain number of standard deviations. For example, the Rice Flour (no. 1568) certificate states [19]:

The estimated uncertainty is based on judgement and represents an evaluation of the combined effects of method imprecision, possible systematic errors among methods, and material variability for samples 400 mg or more. (No attempt was made to derive exact statistical measures of imprecision because several methods were involved in the determination of the constituents.)

When a secondary quality-control material is used, a reference mean and standard deviation should be obtained by analysis of the material by the same or another laboratory using a reference method. To monitor a method's accuracy, the formulated reference material should be analysed with each batch of unknowns or on each day of analysis, comparing the results to the reference mean and standard deviation as shown in table 2.

The review of Se references carried out for this work revealed that a reference material was frequently analysed at the outset of a study, serving to validate the method before analysis of unknowns begins. Occasionally a mean and standard deviation for a quality-control material were reported, but usually documentation was not sufficient to determine whether the material was analysed with each batch of unknowns. In such cases it was assumed that the investigator used the reference material only to validate the test method rather than to measure day-to-day accuracy. In this instance, an accuracy rating of no higher than 1 was assigned.

The other half of analytical quality control is the level of precision; the aspect of concern here is the amount of variability about the mean value associated with the day-to-day execution of a particular method. The indication of day-to-day variability can be determined only when the analytical method is monitored continuously through the use of a quality-control material similar in matrix to the unknowns to be analysed. Like accuracy, the day-to-day precision of a method is matrix-dependent.

Precision is usually measured by calculating the per cent coefficient of variation (% CV), also known as per cent relative standard deviation (% RSD), from the mean and standard deviation (SD) of several replicates of a sample: % CV or % RSD = SD divided by mean x 100 per cent. The lower the % CV, the more precise the analysis. Our limits for rating % CV are: 5 per cent or less for a 3, 10 per cent or less for a 2, and 15 per cent or less for a 1. The % CV, calculated from replicates analysed within a given laboratory, includes variability attributable to instrument and technician performance and to the method. The precision of a method could be poor in the hands of one investigator and acceptable in the hands of another.

Table 3. Assignment and meaning of confidence codes

Sum of quality indices Confidence code Meaning of confidence code
>6.0 a The user can have considerable confidence in this value.
2.4 to 6.0 b The user can have confidence in this value, however, some problems exist regarding the data on which the value is based.
1.0 to <3.4 c The user can have less confidence in this value due to limited quantity and/or quality of data.

When the ratings for accuracy and precision are the same, that rating becomes the analytical quality-control rating. When accuracy and precision ratings differ, the lower rating is assigned for the overall analytical quality-control rating for the study. Data from a study with incomplete documentation are rated no higher than 1 in this category. Lack of any documentation or unacceptable precision or accuracy earns a 0 in analytical quality control.