6. Has the HEARTSMAP tool’s inter-user reliability been evaluated?

Yes. HEARTSMAP’s inter-user reliability was evaluated among pediatric ED clinicians locally at BC Children’s Hospital in 2014, and among ED clinicians in general EDs in 2016.

At BC Children’s Hospital, data from 104 ED mental health presentations was extracted to construct clinical narratives. A panel of ED clinician evaluators reviewed each of these narratives. The panel included: 2 pediatric emergency medicine physicians, 1 nurse practitioner, and 1 bedside nurse. The results below summarize the agreement among the clinician reviewers with regard to consulting psychiatry in the ED:

Comparisons Kappa Value Interpretation¹
Pediatric ED physician vs. Pediatric ED physician 0.7 Substantial Agreement
Pediatric ED physician vs. Nurse practitioner 0.6 Moderate Agreement
Pediatric ED physician vs. Bedside nurse 0.5 Moderate Agreement

Eleven ED clinicians from 5 health authorities – Provincial Health Services Authority, Providence Health Care, Vancouver Coastal Health, Fraser Health Authority, and Interior Health Authority validated the HEARTSMAP tool in 2016 (3). These ED clinicians, including physicians, social workers, and psychiatric liaison nurses, were asked to use the HEARTSMAP tool to evaluate 50 fictional narratives describing a variety of pediatric mental health cases. Their agreement with regards to consulting psychiatry in the ED or referring to a community based program such as a Crisis Response Team, a Child and Youth Mental Health Team (CYMHT), social services through the Ministry of Child and Family Development, or a youth health clinic, is summarized below:

Resource Kappa Value Interpretation²
Psychiatry consultation in the ED
Outpatient Crisis Response Team
Outpatient CYMHT
MCFD Social Services
Youth Health Clinic/Substance & addiction services
  1. Viera AJ, Garrett JM. Understanding Interobserver Agreement : The Kappa Statistic. Fam Med. 2005;37(5):360-363.
  2. Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.
  3. Virk, P., Stenstrom, R., & Doan, Q. (2018). Reliability testing of the HEARTSMAP psychosocial assessment tool for multidisciplinary use and in diverse emergency settings. Paediatrics & Child Health, 1–6. https://doi.org/10.1093/pch/pxy017

2 Replies to “6. Has the HEARTSMAP tool’s inter-user reliability been evaluated?”

Leave a Reply

Your email address will not be published. Required fields are marked *