Machine learning models boost children's suicide-risk prediction
New study shows why many predictive algorithms may miss out on children at risk of self-harm.
By UCLA Health
A new study from UCLA Health researchers finds that the typical ways health systems store and track data on children receiving emergency care miss a sizable portion of those who are having self-injurious thoughts or behaviors. The researchers also found that several machine learning models they designed were significantly better at identifying those children at risk of self-harm.
Amid a nationwide youth mental health crisis, mental health providers are trying to improve their understanding of which children are at-risk of suicide or self-harm so providers can intervene earlier. However, health systems often do not have a full understanding of who is coming through their doors for self-injurious thoughts or behaviors, meaning that many risk-prediction models designed to flag children at future risk are based on incomplete data, limiting prediction accuracy.
Suicide is the fourth leading cause of death among 15-29 year-olds. Credit: 13582375
“Our ability to anticipate which children may have suicidal thoughts or behaviors in the future is not great – a key reason is our field jumped to prediction rather than pausing to figure out if we are actually systematically detecting everyone who is coming in for suicide-related care,” said Juliet Edgcomb, MD, PhD, the study’s lead author and associate director of UCLA’s Mental Health Informatics and Data Science (MINDS) Hub. “We sought to understand if we can first get better at detection.”
Depression or anxiety
Many risk-prediction models for suicide and self-harm rely on how providers categorize care they’ve provided through diagnostic codes known as International Classification of Diseases, version 10 (ICD-10). However, this may exclude many children who have self-injurious thoughts or behaviors but have been coded in their health records for an underlying mental health diagnosis, such as depression or anxiety. Another commonly used method for flagging at-risk patients is the chief complaint, a brief statement provided at the beginning of a health care visit describing why a patient is seeking care, but children may not always report suicidal thoughts and behaviors when they first come into the emergency department.
Experts reviewed clinical notes for 600 emergency department visits for children ages 10-17 at a large health system to understand how well ICD-10 codes and chief complaint identify children with self-injurious thoughts or behaviors.
Experts who reviewed the patients’ clinical notes found that ICD codes missed 29% of children who came to the emergency department for self-injurious thoughts or behaviors, while the chief complaint missed over half (54%) of those patients. Using the ICD code and the chief complaint together still missed about 22% of those patients.
Screening methods that relied on ICD codes or chief complaint were also more likely to miss male children than female children, as well as preteens compared to teens. There was also a signal that Black and Latino youth were more likely to be left out, raising concerns that these groups could be disproportionately underrepresented in risk prediction models.
The machine learning models were more likely to flag patients not at risk of self-harm, but Edgcomb said there is little downside to using these more sensitive screening tools. Credit: artbykleiton
Researchers designed three different machine learning models to test whether an automated system could do a better job of flagging children with self-injurious thoughts or behaviors. The most comprehensive model incorporated 84 data points available in a patient’s electronic record, including previous medical care, medications, demographic information, and whether the child lives in a disadvantaged neighborhood, among others. A second model used all diagnostic codes for mental health, rather than just the suicide-related codes that come from CDC’s suicide surveillance program, and a third looked at other indicators, such as a patient’s medications and lab tests.
All three machine learning models were better at identifying children with self-injurious thoughts and behaviors than just ICD codes and chief complaint alone. No machine learning model performed significantly better than any of the others, indicating that health systems could improve their ability to flag at-risk patients without having to build especially sophisticated models.
“Adding more information helps, but you don’t necessarily need a bells-and-whistles approach to get better detection,” Edgcomb said.
The machine learning models were more likely to flag patients not at risk of self-harm, but Edgcomb said there is little downside to using these more sensitive screening tools. “Depending on the situation, it may better to have some false positives and have a medical records analyst double-check those charts that screen positive, than to miss many children entirely,” she said.
Provided by UCLA Health
Reference: Edgcomb JB, Tseng C, Pan M, Klomhaus A, Zima BT. Assessing Detection of Children With Suicide-Related Emergencies: Evaluation and Development of Computable Phenotyping Approaches. JMIR Ment Health (2023). DOI: 10.2196/47084
