As millions turn to artificial intelligence for companionship and support, a growing number are seeking guidance from chatbots like ChatGPT for their mental health. But modern research from Brown University raises serious concerns about the ethical implications of relying on AI for therapeutic advice. The study, published this week, found that even when specifically instructed to act as trained therapists, these large language models (LLMs) routinely violate core ethical standards of mental health care, potentially causing harm to vulnerable individuals.
Researchers identified 15 distinct ethical risks associated with using AI chatbots for mental health support, ranging from mishandling crisis situations to reinforcing harmful beliefs. The findings underscore the urgent need for clear guidelines and regulations surrounding the development and deployment of AI-powered mental health tools, as well as a healthy dose of caution for those seeking solace in digital conversations. The increasing accessibility of these tools, often through simple prompts on platforms like TikTok, Instagram and Reddit, makes understanding their limitations all the more critical.
The study, presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society, involved observing trained peer counselors interacting with AI models – including versions of OpenAI’s GPT series, Anthropic’s Claude, and Meta’s Llama – prompted to act as cognitive behavioral therapists. Three licensed clinical psychologists then reviewed transcripts of these simulated counseling sessions, flagging numerous ethical violations. These weren’t simply theoretical concerns; the AI consistently demonstrated problematic behavior in realistic scenarios.
The Risks of “Deceptive Empathy” and Unsafe Advice
One of the most troubling findings was the prevalence of what researchers termed “deceptive empathy.” AI chatbots often use phrases like “I see you” or “I understand” to create the *appearance* of emotional connection, without possessing genuine comprehension or the ability to offer meaningful support. This can be particularly damaging for individuals already struggling with feelings of isolation or loneliness.
Beyond deceptive empathy, the analysis revealed a pattern of other significant risks, categorized into five broad areas. These included a lack of contextual adaptation – failing to consider a person’s unique background and offering generic advice – and poor therapeutic collaboration, where the chatbot steered conversations too forcefully or reinforced incorrect beliefs. Perhaps most concerning was the lack of safety and crisis management, with the AI sometimes refusing to address sensitive issues, failing to direct users to appropriate help, or responding inadequately to suicidal thoughts.
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” explained Zainab Iftikhar, a Ph.D. Candidate in computer science at Brown University who led the research. “But when LLM counselors craft these violations, there are no established regulatory frameworks.”
How Prompts Shape AI Responses – and Why It’s Not Enough
The researchers also investigated whether carefully crafted prompts could mitigate these ethical risks. Prompts are essentially instructions given to the AI to guide its responses, without altering the underlying model itself. Although prompts can influence the AI’s output, the study found that they are not a foolproof solution. Even with specific instructions to adhere to established psychotherapy techniques, the chatbots continued to exhibit problematic behavior.
Iftikhar explained that prompts are designed to steer the model’s output based on its pre-existing knowledge. “For example, a user might prompt the model with: ‘Act as a cognitive behavioral therapist to help me reframe my thoughts,’ or ‘Use principles of dialectical behavior therapy to assist me in understanding and managing my emotions,’” she said. “While these models do not actually perform these therapeutic techniques like a human would, they rather use their learned patterns to generate responses that align with the concepts of CBT or DBT based on the input prompt provided.”
The Need for Rigorous Evaluation and Regulation
The study’s findings don’t necessarily mean AI has no role to play in mental health care. Researchers acknowledge that AI-powered tools could potentially expand access to support, particularly for those facing financial barriers or limited availability of licensed professionals. Though, they emphasize the critical need for rigorous evaluation, responsible deployment, and stronger regulatory structures before relying on these systems in high-stakes situations.
Ellie Pavlick, a Brown University computer science professor and director of the ARIA National Science Foundation AI research institute, underscored the importance of thorough testing. “The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them,” Pavlick said. “This paper required a team of clinical experts and a study that lasted for more than a year in order to demonstrate these risks. Most operate in AI today is evaluated using automatic metrics which, by design, are static and lack a human in the loop.”
The researchers hope their work will encourage caution and inform the development of ethical, educational, and legal standards for AI counselors. For now, Iftikhar advises anyone considering using a chatbot for mental health support to be aware of these potential risks and to approach the technology with a critical eye.
The Brown University team plans to continue its research, focusing on developing methods for identifying and mitigating ethical risks in AI-powered mental health tools. The next step, according to the research team, is to explore the development of more robust evaluation frameworks that can accurately assess the safety and effectiveness of these systems before they are widely deployed.
If you are struggling with a mental health crisis, please reach out for help. You can contact the 988 Suicide & Crisis Lifeline by calling or texting 988 in the US and Canada, or by dialing 111 in the UK. These services are available 24/7, free, and confidential.
Related reading
