Validity and reliability of a method to estimate the potential harm of medication errors by considering both the likelihood and degree of harm

Objectives: To develop a tool to ‘risk score’ the potential harm of a medication error by estimating the probability of a range of potential consequences, and assess its feasibility, validity and reliability.

Methods: The risk score tool described five levels of potential harm developed from an existing risk matrix. Judges estimated the likelihood of harm matching each level, from which a risk score (0-10) was calculated. Thirty judges (doctors, nurses and pharmacists) used this risk score and the existing Dean and Barber scale to estimate the potential harm of 50 medication errors, 15 with a known outcome. Two weeks later, the judges re-scored ten of the errors. Reliability was investigated using generalisability theory.

Key findings: Fifty medication errors were judged in a mean of 49 minutes with ≤0.7% missing data. There were no significant differences in the judges’ rating of the realism and ease-of use of the two tools. 59% of judges preferred the risk score tool. For both tools, there was a clear relationship between mean score and known outcomes, with no overlap between outcome categories, confirming discriminative validity. The correlation between scores from both tools (R2=0.99) confirmed the concurrent criterion validity of the risk score. For both tools, at least three judges would need to score an error to obtain a generalisability coefficient of ≥0.8 using the mean score as an indicator of potential harm.

Conclusions: The risk score was feasible, valid and reliable. Its performance was comparable with, but did not exceed, the Dean and Barber scale.