CHARTwatch, a machine learning model, shows promise in reducing patient mortality and improving outcomes in hospital settings, according to new CMAJ study.

Study: Clinical evaluation of a machine learning–based early warning system for patient deterioration. Image Credit: LALAKA/Shutterstock.com

In a recent study published in the Canadian Medical Association Journal (CMAJ), researchers clinically evaluated CHARTwatch, a model that predicts patient deterioration based on machine learning.

Background

Estimating, preventing, and reacting to the clinical deterioration of hospitalized individuals is critical to increasing patient safety. Unidentified clinical deterioration is the primary cause of unnecessary admissions to the intensive care unit (ICU), resulting in prolonged stays and increased fatality. Despite the extensive usage of prediction tools, the evidence for their usefulness is inconsistent.

A Kaiser Permanente study of 19 hospitals in Northern California revealed that an automated risk estimation model with remote nurse monitoring and on-the-ground actions by quick response teams reduced 30-day mortality by 16%. However, the technological and clinical characteristics of advanced early alert systems that might enhance clinical outcomes are unknown.

About the study

The present study investigated whether CHARTwatch could improve patient deterioration-related clinical outcomes.

The program predicts patient deterioration by using real-time data from electronic medical records. The time-aware multivariate adaptive regression spline (MARS) technique considered risk score projections from past encounters, changes in risk ratings since previous assessments, and time-series summaries.

The model communicated to nurses and physicians via texts and email, and it included a clinical route for the high-risk patient category, such as physician evaluation within an hour, increased vital sign monitoring, and alerts for palliative care consultations.

Patients admitted to St. Michael’s Hospital’s general internal medicine (GIM) unit received the intervention between 1 November 2020 and 1 June 2022. The pre-interventional period was between 1 November 2016 and 1 June 2020.

Propensity score-based weighting compared intervention recipients to individuals admitted before the intervention. Difference-indifferences assessment compared intervention recipients in the general internal medicine unit and non-recipients in the respiratory, nephrology, and cardiology units.

The primary endpoint was within-hospital mortality from non-palliative care, defined as fatalities that did not result from a recorded palliative care treatment.

Secondary endpoints were palliative deaths, total deaths, and transfers (a composite measure of deaths among palliative care recipients or shifts to inpatient palliative care units), ICU transfer, a composite measure of transfer to ICUs or mortality, and hospital stay length.

The International Classification of Diseases, tenth revision, Canadian version (ICD-10-CA), ascertained patient diagnosis. Researchers retrospectively calculated model predictions for control group patients.

Clinicians received alerts only in the interventional period for GIM unit patients. The study excluded individuals with coronavirus disease 2019 (COVID-19) or influenza and those with preadmission palliative care comorbidities. Logistic regressions estimated propensity scores for the GIM and subspecialty cohorts.

Researchers calculated the relative risk (RR) for analysis, adjusting for study covariates. Poisson regressions compared binary outcomes, and linear models compared continuous outcomes.

Study covariates included age, gender, comorbidities, hospitalizations in the prior six months, hospitalization month, vital signs, homelessness, neighborhood racial and new populations, neighborhood material resources, and admission to the ICU before transfer to subspecialty wards or GIM units.

Results

The analysis comprised 13,649 GIM unit admissions and 8,470 subspecialties unit admissions. In the general internal medicine unit, 482 patients became high risk in the interventional period, and 1,656 patients became high risk during the control period.

Non-palliative mortality was significantly lower during the interventional period than before the intervention among GIM patients (1.60% vs. 2.10%; RR, 0.7) but not among subspecialty unit patients (1.90% vs. 2.10%; RR, 0.9).

Among GIM patients at high risk of deterioration for whom CHARTwatch provided one or more alerts, the non-palliative mortality rates were 7.1% during the interventional period and 10% before the intervention (RR, 0.7).

The team found no significant difference in the subspecialty groups (10% vs. 11%; RR of 0.98). Difference-indifference assessments yielded an RR reduction of 0.8 for mortality from non-palliative care in the general internal medicine unit.

In the held-out testing data, the model demonstrated 53% sensitivity and 31% positive predictive value (PPV) in detecting clinical deterioration during hospitalization (death or transfer to the ICU, step-up care, or palliative care unit).

Compared to the pre-interventional period, the intervention resulted in considerably more antibiotic and corticosteroid prescriptions and increased vital sign monitoring. These data indicate that the intervention was related to enhanced patient monitoring and therapies that might slow deterioration.

Conclusion

The study showed that deploying CHARTwatch for GIM admissions was related to a decreased probability of mortality from non-palliative care compared to the preintervention period.

The results show that early alert systems based on machine learning are potential technologies to improve healthcare outcomes.

However, one should interpret findings cautiously due to the potential unmeasured confounding. Future studies will assess equity-related factors of the intervention and the qualitative perspectives of clinical team members.