Stroke risk prediction tools are meant to guide how doctors approach a potentially deadly condition, using factors like heart disease and high blood pressure to get a handle on which patients might benefit from a particular treatment.
For years, doctors have used several different algorithms to try to capture the true risk of stroke, including newer models that use machine learning. A new analysis, led by researchers at Duke University School of Medicine, compared several of those algorithms head-to-head — and found that novel machine learning models weren’t much more accurate at predicting the risk of stroke than simpler algorithms based on self-reported risk factors and an older methodology. Alarmingly, the study also found all the algorithms were worse at stratifying risk for Black men and women than for white.
“We got the shocking result — to me, shocking — that the measure of discrimination, the ability to rank them, was much better… for white participants than Black participants,” said Michael Pencina, director of Duke AI Health and one of the lead researchers on the study.
advertisement
Researchers compared two previously developed models used to determine whether a patient is at high or low risk for a stroke, as well as a third that estimates the 10-year risk rates of events like heart attacks and strokes, to see how well it worked specifically at stroke prediction.
Once they’d gathered all the data those models were based on, Pencina said, the researchers asked themselves: “What if we developed fancy machine learning algorithms on the data that we have?’”
advertisement
So the researchers developed two machine learning models that used data from multiple U.S.-based cohort studies that had been used to develop the existing algorithms. Then, they validated the models using data from another cohort study that has been used in stroke research. Given the limitations of the datasets they had to work with, the researchers decided to look specifically at how accurate the models were at predicting stroke risk in white adults and Black adults, who have about a 50% higher risk of stroke than their white peers. Patients who self-identified as Black made up 29% of the total dataset used to analyze the performance of the models. All patients were 45 years and older and had never suffered a stroke or temporary blockage of blood flow to the brain.
Researchers then examined how the models did at risk ranking, or ordering patients according to their risk of stroke. Despite the potential for machine learning models, they fared relatively poorly.
“I can joke that it’s almost like Chat GPT, it really is the best method,” said Pencina, comparing the new machine learning models to the buzzy new technology. . “And … nothing happened. They couldn’t help us. It did not improve performance. It did not reduce bias.”
With all models, the accuracy was worse for Black men and women than their white peers. That means that Black Americans — who have a much higher probability of suffering from a stroke — are also less likely to get an accurate prediction of their stroke risk. In practice, that could lead to Black patients getting fewer resources for stroke prevention and treatment — even when they are more at risk — or unnecessary intervention for those at low risk.
One of the simpler algorithms, which used self-reported data, did slightly better at not overestimating or underestimating the risk. This finding gives Pencina hope that simple algorithms, taken together with better data, including social determinants of health, will help develop more equitable stroke risk assessments.
Alonso Alvaro, professor of epidemiology at the Rollins School of Public Health at Emory University, pointed out that one of the problems with these models is that the majority of the studies’ population are white — which can skew their performance in other populations.
“So in the end, even though you have a risk score that was derived in a population that has people from multiple racial backgrounds, it’s overweighted to be more helpful for people that are of white race,” Alvaro said.
Chere Gregory, the medical director for neurosciences at the Winston- Salem-based Forsyth Medical Center, also pointed out that, as race is a social construct, “any medical model or algorithm that uses race as a variable will frequently fail to appropriately predict outcomes.”
To Pencina, the issue isn’t with how advanced the stroke predictive technology is but with the datasets themselves, which don’t include information on stroke risk factors like chronic stress and social determinants of health.
Experts pointed to literature showing that applying machine learning algorithms to traditional epidemiologic data begets similar results — biases and all — if the patient information available hasn’t changed.
“The algorithm can only predict as much as the data it is fed so it is not surprising that it is not accurately predicting risk in communities of color. We need a dataset that is more comprehensive that includes variables from structural bias to discrimination to neighborhood pollution,” Bernadette Boden-Albala, director of the UC Irvine Program in Public Health, told STAT in an email.
James Meschia, a professor of neurology at the Mayo Clinic who specializes in stroke, said he’s hopeful that further research can look into these non-biological risk factors. But, he added, the challenge is how to establish whether there’s a direct correlation between the factors and stroke, and if so, how much of an impact those factors have. Those caveats can make it more difficult to accurately incorporate such factors into a calculation that otherwise includes mostly objective biological measurements.
While experts lauded the Duke study’s execution and the large patient population it tested the models on, they said it will be crucial for future studies to grapple with better refining stroke prediction.
“You could use risk calculators to inform patients that, ‘You know, the way you’re going, your risk is x, let’s say 7% [over 10 years],… If we [treat] these factors, we can reduce it to 5%,’ ” he said, though he noted he could also see it backfiring, with patients saying “ ‘Only 7% in 10 years?! Well, heck, I’m not doing anything’ [like quitting smoking or exercising].”
Pencina suggested that electronic health records and partnering with community service organizations on the ground could help gather a more diverse range of data for better stroke risk prediction in Black people.
He said the ultimate goal is to intervene and help the patients most at risk before they ever suffer a stroke. He knows firsthand how valuable it could be, having lost both of his grandparents when they were still young, after they struggled to get health care for cardiovascular diseases in Poland.
“The notion of prevention really hit me, because I had that sense that these diseases could have been prevented,” he said, with education, proper care, and risk prediction.
