Editor’s Note: This article contains references to suicide, self-harm and eating disorders.
As more people turn to artificial intelligence for emotional and mental health support, a new report from the Center for Countering Digital Hate (CCDH) found chatbots can provide guidance on self-harm, suicide, disordered eating and substance use despite built-in safeguards from companies including OpenAI.
CCDH researchers, as a part of their “Fake Friend” project, created 13-year-old personas and held extended conversations with ChatGPT on those topics, then scaled up the same prompts through the system’s programming interface.
“We were quite surprised by the level of harm that the chatbots were willing to output,” said Alex Johnson, a research manager at CCDH. “ It was about 53% of the time that we did get harmful answers from the chatbot.”
CCDH defined “harmful” responses as those that encouraged dangerous behavior or provided information that, if followed, could reasonably hurt the user. In case studies, ChatGPT advised on how to “safely” cut yourself within two minutes, generated a list of pills commonly used for overdosing, offered crash diet plans and explained how to hide being drunk at school, among other outputs.
OpenAI’s policies say ChatGPT can refuse to answer queries about self-harm and suicide and will present crisis resources when it detects those terms. Johnson said those safeguards often fell away when testers reframed the questions, for example, if the user rephrased the question to be for a research paper.
Sometimes the system itself suggested new contexts within safeguards, such as presentations or school projects, that allowed the conversation to continue. In longer chats, CCDH documented sessions where a teen persona spent close to an hour discussing suicide, after which ChatGPT generated a detailed suicide plan and multiple goodbye letters, including a note addressed to “Mom and Dad.”
Zainab Iftikhar, a fifth-year Ph.D. student at Brown University, studies how human-centered AI systems support or harm users’ physical, cognitive and social well-being. She examined how chatbots respond when prompted to act like cognitive behavioral therapists and compared that behavior to professional codes of ethics.
Iftikhar’s team identified what they call “deceptive empathy” as a recurring pattern.
“It keeps saying I hear you. I’m with you. I understand. To really connect with the user by using this anthropomorphic design,” Iftikhar said. “ There’s now a shift in which people are saying, ‘This is not that great. A chatbot should not be using these personal pronouns to describe itself and its connection with the user.’”
In one synthetic example, a user imagined that her father wished she had never been born. Reviewing the same scenario, a psychologist told Iftikhar an appropriate response would be to ask for evidence and explore what made the user feel that way. The chatbot instead immediately validated the belief.
“The chatbot was like, ‘Oh my God, that is such a burden for you to bear. I’m sorry.’ So it instantly validated a negative belief that a person had,” Iftikhar said.
Both Iftikhar and the CCDH’s report link this behavior to “sycophancy” in large language models, where systems are optimized to agree with and flatter users. Johnson said that can foster dependence.
“It’s like a friend who would never tell you no,” Johnson said. “That creates some level of emotional dependence in that you want that feeling that you want to keep seeking out. And it’s easy to see how that results in people being like, ‘well, I don’t want to talk to my parents about this.’”
Polling cited in the CCDH report found that 72% of U.S. teens have used some kind of “AI companion,” with more than half of those using them at least a few times a month. One third of teen users said they had used AI companions for social interaction and relationships, including emotional and mental health support.
Johnson and Iftikhar said mainstream chatbots are not regulated as medical devices.
“They’re robots, but they’re not prioritizing safe answers. They’re not prioritizing any of those sorts of things,” Johnson said. “They are trying to keep you on the platform longer. They want to give you whatever answer will make you feel good about using the chatbot. And so that means answering questions that they shouldn’t answer.”
In her work, Iftikhar asked clinicians to annotate anonymized chat logs and flag where human therapists would be expected to respond differently under codes of conduct. She said those standards create clear consequences for professionals that do not exist for AI systems.
“This was to say if humans make these mistakes, there are bodies and there are organizations that can hold them responsible,” Iftikhar said. “A therapist can lose their license, but for chatbots, there’s no one taking responsibility.”
Shaun Respess, a postdoctoral teaching scholar who works in philosophy, technology and AI ethics at NC State, studies how LLMs can act as companions or therapeutic supports, particularly for older adults. He said accessibility helps explain why people, particularly college students, turn to chatbots when they are struggling.
“The mental health care networks we have in place … are severely under-resourced,” Respess said, pointing to overburdened clinicians and gaps in rural care. “ AI becomes this kind of additional tool that we can throw into the fray to meet people where they’re at.”
Respess said publicly accessible models like ChatGPT, Gemini and others can serve as a familiar first point of contact for people who already use them for school or work. He also said some marginalized or underrepresented groups may approach human clinicians expecting to be misunderstood or dismissed.
“They’re not going to get that from an AI, it’s going to be more sycophantic,” Respess said. “In those cases, what you’re getting is a point of connection, a point of therapeutic treatment, but without the fear of judgment.”
Respess described AI as a “supportive tool” that should be compared to calculators and other technologies integrated into existing infrastructures, rather than a standalone solution.
Questions about responsibility cut across all three experts’ work. Iftikhar asked whether the burden should rest on users, developers or companies when a chatbot reinforces harmful thoughts or encourages risky behavior. CCDH called for independent safety audits, enforceable age verification and bringing AI chatbots under the scope of online safety laws.
Respess, who serves on a national working group on AI teaching resources and policies, expects regulation to be slow and reactive, with universities and other local institutions building guidelines first. He said early efforts are still struggling to define terms like “responsible” or “equitable” AI and how to decide how accountability should be shared between users, developers and funders.
“We’re trying to build guardrails against technology when we haven’t even built up guardrails against social media,” Respess said.
For now, Iftikhar said education and transparency may be the most immediate tools available to young users who already rely on AI for emotional support.
“It might say I understand you, but it does not,” Iftikhar said. “Just because it uses personal pronouns doesn’t mean you should substitute your social relationships.”
If you or someone you know is having difficulty processing grief or having a mental health emergency, the Counseling Center can be reached 24 hours a day at 919-515-2423. If you are in a crisis situation and need immediate help, please call the National Suicide Prevention Lifeline at 988. In the case of a life-threatening emergency, call 911.
The Counseling Center’s website offers free online screenings, a plethora of self-help resources regarding mental health and wellness concerns and a comprehensive list of campus services available for those who need guidance. To view an exhaustive list, visit counseling.dasa.ncsu.edu/resources.
If you’re seeking professional counseling or other mental health services on campus, visit the Counseling Center’s Getting Started page at counseling.dasa.ncsu.edu/about-us/gettingstarted to complete paperwork, set up an appointment and more.
