The promise of artificial intelligence revolutionizing healthcare is colliding with a sobering reality: current AI chatbots, like OpenAI’s ChatGPT Health, are demonstrably unsafe for medical advice. A newly published study reveals the platform frequently fails to recognize medical emergencies, and alarmingly, often misses signs of suicidal ideation, raising concerns that the technology could lead to preventable harm and even death.
The findings, published in the February edition of the journal Nature Medicine, come as ChatGPT Health, launched to a limited audience in January, rapidly gains traction. More than 40 million people are reportedly seeking health-related advice from the chatbot daily, drawn by the convenience of accessing information and potentially integrating their medical records.
Researchers led by Dr. Ashwin Ramaswamy created 60 realistic patient scenarios, ranging from minor ailments to life-threatening conditions. Three independent doctors reviewed each case to establish the appropriate level of care, based on established clinical guidelines. When presented with these scenarios, ChatGPT Health under-triaged more than half of the cases requiring immediate hospitalization, advising patients to stay home or schedule routine appointments instead. In over half of the cases (51.6%) where immediate hospital care was needed, the AI recommended a less urgent course of action, a result described by Alex Ruani, a doctoral researcher in health misinformation mitigation at University College London, as “unbelievably dangerous.”
“If you’re experiencing respiratory failure or diabetic ketoacidosis, you have a 50/50 chance of this AI telling you it’s not a considerable deal,” Ruani explained. “What worries me most is the false sense of security these systems create. If someone is told to wait 48 hours during an asthma attack or diabetic crisis, that reassurance could cost them their life.”
The study highlighted particularly disturbing failures in recognizing critical situations. In one simulation, ChatGPT Health incorrectly advised a patient experiencing an asthma attack to wait for treatment, despite identifying early warning signs of respiratory failure. In another harrowing example, the platform sent a simulated patient experiencing suffocation to a future appointment she would not survive, doing so in 84% of cases. The AI was significantly more likely to downplay symptoms if the “patient” mentioned a friend had suggested the issue wasn’t serious – a 12-fold increase in minimizing the severity of the condition.
The potential for harm extends beyond physical health. Researchers found that ChatGPT Health’s ability to detect suicidal ideation was inconsistent and easily disrupted. When presented with a scenario involving a 27-year-old patient expressing thoughts of self-harm, the chatbot consistently displayed a crisis intervention banner linking to suicide help services. However, when normal lab results were added to the scenario – maintaining the same patient description and severity of expressed thoughts – the banner vanished entirely, failing to appear in any of the 16 subsequent attempts.
“A crisis guardrail that depends on whether you mentioned your labs is not ready, and it’s arguably more dangerous than having no guardrail at all, since no one can predict when it will fail,” Ramaswamy, a urology instructor at the Icahn School of Medicine at Mount Sinai, stated.
These findings are prompting calls for stricter regulation and independent oversight of AI in healthcare. Professor Paul Henman, a digital sociologist and policy expert at the University of Queensland, emphasized the potential consequences of widespread use. “If ChatGPT Health was used by people at home, it could lead to higher numbers of unnecessary medical presentations for low-level conditions, and a failure of people to obtain urgent medical care when required, which could feasibly lead to unnecessary harm and death,” he said. He also pointed to a growing number of legal cases against tech companies related to suicide and self-harm following the use of AI chatbots, as reported in The Guardian.
OpenAI acknowledged the independent research, stating that the study may not reflect typical user behavior. A spokesperson for the company also noted that the model is continuously updated and refined. However, researchers argue that even with ongoing improvements, a plausible risk of harm necessitates stronger safeguards and independent auditing mechanisms.
The development of ChatGPT Health follows a broader trend of integrating AI into healthcare, with the goal of improving access to information and streamlining patient care. However, this study underscores the critical need for rigorous testing and validation before deploying such technologies in sensitive areas like medical diagnosis and treatment. The question remains: how can we harness the potential of AI in healthcare while mitigating the very real risks to patient safety?
OpenAI has indicated ongoing efforts to improve the safety and accuracy of its AI models. The company is also collaborating with mental health experts, with over 170 involved in refining ChatGPT’s ability to recognize distress and respond empathetically, according to OpenAI’s website. However, the current findings suggest that significant work remains to be done before AI chatbots can be reliably used for medical guidance.
As the debate over the role of AI in healthcare intensifies, regulatory bodies and tech companies face increasing pressure to establish clear safety standards and independent auditing mechanisms. The future of AI-powered healthcare hinges on ensuring that these technologies prioritize patient well-being above all else.
If you are struggling with suicidal thoughts, please reach out for help. You can contact the National Suicide Prevention Lifeline at 988, or text HOME to 741741 to reach the Crisis Text Line.
The next steps in addressing these safety concerns will likely involve further research, increased regulatory scrutiny, and ongoing refinement of AI models. The Food and Drug Administration is expected to release updated guidance on AI-based medical devices in the coming months, which could have a significant impact on the development and deployment of technologies like ChatGPT Health. Share your thoughts on the responsible use of AI in healthcare in the comments below.
