Overview

This project at an academic health system utilized machine learning to analyze software-generated outlier prescription alerts from a dataset of EHR records to systematically evaluate the accuracy and clinical value of the software-generated alerts in order to alleviate issues related to medication error EHR alert fatigue.

Organization Name

Brigham Women’s Hospital, Massachusetts General Hospital

Organization Type

• Academic Medical Center
• Integrated healthcare system/network

National/Policy Context

• Medication errors – such as prescribing the wrong dose of a drug or giving drugs to the wrong patient – cause significant patient morbidity and mortality.
• Prescription errors also lead to excessive  healthcare costs estimated at more than $20 billion annually in the United States. 
• Current approaches use clinical decision support systems, which only identify a small fraction of errors and suffer from high alert rates, creating “alert fatigue”.
  • Clinical decision support systems also overlook alerts related to medical concerns that may not be anticipated and programmed into the software rules.

Patient Population Served and Payor Information

• Outpatients of all ages who had at least one encounter with a Brigham & Women’s Hospital or Massachusetts General Hospital-affiliated clinician during the two year period from January 1, 2012 to December 31, 2013. 

Funding

• MedAware, Ltd. funded this research, but were not involved in any coding development, data analysis, or manuscript preparation.

Research + Planning

• The patient cohort in this retrospective study was split into two smaller groups with similar demographic traits: one used for training to generate MedAware’s individual medication models, and one used to test model performance through simulation.
• MedAware analyzed the number and types of alerts generated on an enriched sample of 300 alerts. Innovators manually selected a set of patient charts that represent the distribution of alert categories (e.g. clinical, time-dependent, and dosage outliers) across the full dataset.
  • Within each category, frequency counts were established for each alert type. MedAware then selected a random sample by identifying the 10 most frequently occurring alert types within each category.
• Once MedAware established a random sample of charts, Brigham & Women’s staff utilized study IDs and medical record numbers to identify specific patient charts for review in the EHRs.
• Patient charts were used to determine:
  • If alerts were accurate based on structured and coded information provided in the data to MedAware
  • If alerts were clinically valid based on the clinical data in the patient’s EHR.
  • If alerts were clinically useful by contributing additional information to patient care that could influence the caregiver to change medications. 

Tools or Products Developed

• MedAware (Raanana, Israel) is a commercial software screening system developed to identify and prevent prescription errors.
  • MedAware uses a machine learning algorithm to analyze historical EHR data, generating a model that displays the clinical environment in which a medication is likely to be prescribed.
  • The model identifies prescriptions as significant statistical outliers given each patient’s clinical situation to be flagged as potential medication errors. 
  • Each MedAware alert includes a short description to provide user-friendly explanations allowing clinicians to understand the reason underlying the alert.
    • Clinical outliers: medication is a marked outlier based on patient characteristics (for example, prescribing birth control for an infant boy).
    • Time-dependent irregularities: changes in blood test results indicate a current medication is an outlier from a patient’s profile.
    • Dosage outliers: dosage differs greatly from the machine-learned dosage distribution of medication in the general population and in the patient’s history.

Tech Involved

• Electronic medical record
• Software Program
• MedAware

Team Members Involved

• Data Analyst

Workflow Steps

• Retrospective clinical data – including demographics, diagnoses, problem lists, outpatient and inpatient encounters, encounter clinicians, clinician specialties, procedures, medications, allergies, vital signs, and selected blood tests – was extracted from existing databases of EHR records between  January 1, 2012 to December 31, 2013 for patients included in the study.
  • Patient and clinician names, as well as medical record numbers, were removed from the dataset and replaced by random study IDs.
  • Through a secure transfer, a limited dataset was sent to MedAware for analysis.

Budget Details

The following sources of costs were estimated by the CareZooming team:

• Labor costs for Brigham & Women’s Hospital and Massachusetts General Hospital research staff
• Labor costs for IT team members to manage the integration of the MedAware system
• Cost of clinicians’ time to respond to the increased amount of medication alerts

Outcomes

• MedAware’s machine learning approach found clinically useful information regarding prescription errors –  analyzing a total of 747,985 patient records that generated 15,692 alerts in a simulation cohort consisting of 373,992 patients.
  • Those alerts represented 1706 unique alert types with an overall distribution of 29.3% clinical outliers, 66.8% time dependent, and 3.9% dosage outliers.
• 76.2% of alerts generated by MedAware’s machine learning approach were found to be valid with potential medication errors.
  • 75.0% were found to be clinically useful in flagging potential issues, with 18.8% classified with medium clinical value and 56.2% assigned high clinical value.
• The below figure displays chart reviews classified as data-related vs. clinically valid

Future Outcomes

• This interventional program developed a novel rating system – defining alerts as “accurate” or “valid” with levels of “clinical value” – but has not formally validated this tool, which could be valuable for use by future researchers.
• Future studies may attempt to determine precise additional benefits beyond those of existing clinical decision support systems by comparing outlier-based alerts to existing rule-based alerts.

Benefits

• Through careful evaluation, the MedAware system was found to generate potentially useful alerts with a modest rate of false positives.
• MedAware was able to generate alerts that were otherwise missed by existing clinical decision support systems with a reasonably high degree of alert usefulness when reviewing patient’s clinical contexts. 
• MedAware’s self-learning and self-adaptive capability allow it automatic and continuous search for patient and institutional-based novel outlier patterns that could represent medication errors. 

Unique Challenges

• Because Brigham & Women’s Hospital and Massachusetts General Hospital has a homegrown EHR system under Partners Healthcare, it is unclear if the findings can be generalized to other EHR systems. 
  • However, the quality of data from this study is currently stronger than that from commercial systems
• Although chart reviewers were carefully trained with a clearly developed coding manual, each chart required a degree of judgment from the reviewer and overall research team. 
• Numerous challenges exist when working with clinical data structured in EHR systems:
  • Medication start/stop dates did not always accurately reflect active prescriptions.
  • Sometimes, the care provided within Partners HealthCare was limited to a single specialty (e.g. orthopedics), and the lack of additional clinical information made it difficult to assess alert accuracy. 
  • Some diagnoses were discussed in free-text notes but not added in structured data fields, so MedAware could generate technically accurate alerts that were not valid for the clinical situation.

Sources

• This primer was developed by the CareZooming team based on our analysis of a research article found and accessed through public sources. Authors were able to review the contents of this primer before publication, and all requested edits have been incorporated into the primer as presented above.
• Schiff, G. D., Volk, L. A., Volodarskaya, M., Williams, D. H., Walsh, L., Myers, S. G., & Rozenblum, R. (2017). Screening for medication errors using an outlier detection system. Journal of the American Medical Informatics Association, 24(2), 281-287. https://doi.org/10.1093/jamia/ocw171