Validity and reliability of a method to estimate the potential harm of medication errors by considering both the likelihood and degree of harm

Objectives: To develop a tool to ‘risk score’ the potential harm of a medication error by estimating the probability of a range of potential consequences, and assess its feasibility, validity, and reliability. Methods: The risk score tool described five levels of potential harm developed from an existing risk matrix. Judges estimated the likelihood of harm matching each level, from which a risk score (0–10) was calculated. Thirty judges (doctors, nurses, and pharmacists) used this risk score and the existing Dean and Barber scale to estimate the potential harm of 50 medication errors, 15 with a known outcome. Two weeks later, the judges re-scored ten of the errors. Reliability was investigated using generalizability theory. Key findings: Fifty medication errors were judged in a mean of 49 minutes with ≤0.7% missing data. There were no significant differences in the judges’ rating of the realism and ease of use of the two tools. Fifty-nine per cent of judges preferred the risk score tool. For both tools, there was a clear relationship between mean score and known outcomes, with no overlap between outcome categories, confirming discriminative validity. The correlation between scores from both tools (R ² = 0.99) confirmed the concurrent criterion validity of the risk score. For both tools, at least three judges would need to score an error to obtain a generalizability coefficient of ≥0.8 using the mean score as an indicator of potential harm. Conclusions: The risk score was feasible, valid, and reliable. Its performance was comparable with, but did not exceed, the Dean and Barber scale.