Results

Results of the Case Scenario Test

Ninety-three of the 100 case scenarios were injury-related cases; the remaining seven did not meet the definition of an injury case. The latter case scenarios were purposely included to see if coders would be able to distinguish injury-related cases from non-injury cases. The ability to classify a case as an injury versus non injury varied substantially among coders. Among the four Massachusetts coders, two recognized 5 of 7 non-injury cases and two recognized only 1 of the 7 non-injury cases. Also, 1 or 2 of 93 injury-related case scenarios were classified as non-injury cases by two of the four Massachusetts coders. Among the seven NEISS coders, three recognized all 7 non-injury cases, and the others recognized between 1 and 5 of the 7 non-injury cases. From 1 to 3 of 93 injury-related case scenarios were classified as non-injury cases by 3 of the 7 NEISS coders.

For injury-related cases, we found no significant differences in percent observed agreement or kappa statistics for case scenarios coded before or after the field test. Therefore, all 93 injury-related case scenarios were combined for further analysis.

For the 93 injury-related case scenarios, the percent observed agreement among MA ED-SCIP coders and between MA ED-SCIP coders and the gold standard for five major data elements (i.e., work-relatedness, locale of injury incident, activity at time of injury, intent of injury, and mechanism of injury) were observed to be quite high (Table 1). The observed inter-coder agreement ranged from 85.7% to 97.8%. The average percent observed agreement for coder and gold standard code sets ranged from 80.6% to 94.1%. However, a better measure of reliability can be obtained by using the kappa statistic, a measure of percent agreement beyond what can be expected by chance alone.

The average kappa (in percent) between coder and gold standard code sets and among coders for the five major short ICECI data elements were all in the good to excellent range for both the Massachusetts ED-SCIP and NEISS substudies (Figure 1, Tables 2 and 3). Kappas for MA coders were consistently higher than for NEISS coders for all five major short ICECI data elements (Table 4). Overall, excellent agreement was observed among the MA coders and between the MA coders and the gold standard.

Intra-coder reliability for the Massachusetts ED-SCIP substudy, as measured by average kappas, was also found to be in the excellent range or borderline excellent range for all five major data elements (Table 5).

Results of the Field Test

Inter-coder reliability, as measured by kappa, for randomly selected pairs of MA coders for a set of 127 injury-related ED cases shows percent agreement in the good to excellent range for most of the categories reported within four of the five major data elements (Table 6). Intent of injury was not included in this Table because all but four of the 127 cases were unintentional injuries; therefore the expected agreement was very high. For mechanism of injury, inter-coder reliability achieved 100% agreement for motor vehicle occupant, pedestrian-vehicle crash, poisoning, and foreign body. The lowest kappa was 83.5 for struck by/against.

The median coding time for abstracting data from ED records and completing the data collection form was 3 minutes for the Massachusetts ED-SCIP substudy and 4 minutes for the NEISS substudy. Average coding time was significantly higher for NEISS coders (mean= 4.9 minutes, 95% CI= [4.7%, 5.1%], range=1–15 minutes) than for Massachusetts coders (mean=3.4 minutes, 95% CI= [3.3%, 3.5%], range=1–18 minutes). NEISS coders were much more likely to record multiple mechanisms of injury (i.e., 2 or more mechanisms for the same case) than MA coders. For NEISS coders, 22.9% of 705 injury-related ED cases were coded with multiple mechanisms of injury. For MA coders, 2.6% of 1,710 injury-related ED cases were coded with multiple mechanisms of injury. Most of the cases with multiple mechanisms involved combinations of “struck by/against” and “fall” or of “struck by/against” and “cut/pierce.” This difference in recording multiple mechanisms of injury did not affect our calculations of percent observed agreement or kappa statistics for mechanism of injury (presented in Tables 1–6), because we limited our analysis to only the immediate, or most direct, cause of injury.

A description of short ICECI data for 1,841 injury-related ED patients obtained from review of medical records at 16 Massachusetts ED-SCIP hospitals is presented in Table 7. (NOTE: This number is higher than the 1,710 reviewed by MA coders; it includes an additional 131 cases reviewed by CDC coinvestigators during a site visit to two of the 16 hospitals.) A majority of data abstractions took between 2 and 5 minutes to complete. The sex and age of the patient were readily available in the medical record. Almost 15% of the cases abstracted were determined to be work-related. However, about 20% of the time, no information was available to determine work-relatedness. Locale of injury incident and activity at the time of injury could not be classified in about 40% of cases. Intent of injury and mechanism of injury could be classified in over 98% of cases. Of 200 assault cases, the relationship of perpetrator to victim and the reason for assault could be classified for about 55% of injury incidents. For 218 motor vehicle occupant-related cases, traffic-relatedness, type of motor vehicle in which the patient was riding, and the occupant status of the patient could be determined for over 90% of the injury incidents. Counterpart for the motor vehicle crash was classified in 60% of cases. The number of pedestrian, motorcyclist, and pedal cyclist cases was relatively small in this sample. However, counterpart for the injury incident could be classified in over 90% of these types of transportation-related cases. Traffic-relatedness could not be determined for 55% of pedal cyclist injuries. For cases where the mechanism of injury was struck by/against, the source of force and type of force could be determined in over 95% of injury incidents. The type of firearm used in gunshot injuries was recorded for only about half of the cases. Injuries where the mechanism of injury was stab/cut/pierce, fire/burn, or poisoning could be further characterized by type in over 89% of cases. Other specified mechanisms of injury were predominantly bites and stings.

Use of safety equipment was indicated for 238 of the 1,841 Massachusetts ED-SCIP cases. Most of these cases were associated with seat belt use and air bag deployment. Of the 218 motor vehicle occupant-related cases, seat belt use was indicated for 127 cases (58.3%), and air bag deployment was recorded for 19 cases (8.7%).

A listing of consumer products and a narrative description of the injury incident were provided, where applicable, on completed short ICECI data collection forms. NEISS and MA coders, in general, provided good details about the injury incident that further characterized the intent of injury, mechanisms of injury, and other circumstances of the injury incident. NEISS coders routinely recorded the principal diagnosis of the injury in the narrative field, as this is common practice in their injury surveillance activities.