Inter-rater and Intra-rater Validation of a Pharmacist Intervention Risk Assessment Tool
Purpose: To determine the validity and reliability of a pharmacist intervention risk assessment tool by assessing the differences in risk stratifications of pharmacist interventions by clinicians over an 18-month time lapse.
Methods: In 2005, a multidisciplinary group of clinicians from the Peter MacCallum Cancer Centre independently reassessed the risk (low, moderate, high or extreme) of 100 sample pharmacist interventions using a pharmacist intervention risk assessment tool; this was a repeat of an identical study conducted 18 months prior. Risk ratings among clinicians were compared to determine the inter-rater agreement which was then contrasted with that of 2004. The agreement within clinicians (intra-rater agreement) was also explored. The levels of agreement were calculated using kappa statistics and the mode.
Results: Nine pharmacists participated in the study (7 staff pharmacists from the original 18 clinicians in 2004 plus two external pharmacists with risk management expertise). Exact agreement (the mode) was obtained in an average of 60% (range 29-100%) of clinicians for any one sample intervention, which was similar to results in 2004 (58%, range 35-100%). 95% and 91% of all ratings were within one risk category of the mode in 2004 and 2005 respectively. High intra-rater agreement was achieved, with up to 97% of the scenarios being rated within one risk category of each other (kappa=0.22-0.45). No difference in risk ratings were observed in clinicians of varying clinical experience. There were tendencies to overestimate risk by the doctor (2004) and the two external pharmacists (2005) when compared with modal agreements of staff pharmacists.
Conclusions: The pharmacist intervention risk assessment tool appeared to be both valid and reliable as demonstrated by the high inter-rater and intra-rater agreements in both 2004 and 2005. The use of the tool should be investigated in a practical setting.