How chatbots — and their makers — are enabling AI psychosis

AI is only as powerful as the platform it's built into. That's why it's no surprise that more than 85% of the Fortune 500 use the ServiceNow AI platform, while other platforms duct-taped tools together. ServiceNow seamlessly unifies people, data, workflows, and AI, connecting every corner of your business. And with AI agents working together autonomously, Anyone in any department can focus on the work that matters most. Learn how ServiceNow puts AI to work for people

Most AI coding tools generate sloppy code that doesn't understand your setup. Warp is different. Warp understands your machine, stack, and code base. It's built for the entire software lifecycle, from prompt to production. With the powers of a terminal and the interactivity of an IDE, WARP gives you a tight feedback loop with agents so you can prompt, review, edit, and ship production-ready code.

Trusted by over 600,000 developers, including 56% of the Fortune 500. Try Warp free or unlock Pro for just $5 at warp.dev. Adobe Acrobat Studio. So brand new. Show me all the things PDFs can do. Do your work with ease and speed. PDF spaces is all you need. Do hours of research in an instant. With key insights from an AI assistant. Take a template with a click. Now you're present.

super slick close that deal yeah you won do that doing that did that done now you can do that do that with acrobat now you can do that do that with the all new acrobat it's time to do your best work with the all new

Adobe Acrobat Studio. Hey, and welcome to Decoder. This is Hayden Field, The Verge's senior AI reporter. I'm excited to be filling in for a few Thursday shows while Neelai's out. He'll be back with us soon. The explosive growth of AI chatbots in the last three years since ChatGPT launched in 2022 has started to have some really noticeable, profound, and honestly disturbing effects on some users. There's a lot to unpack there.

be pretty complicated. So I'm very excited to talk with today's guest New York Times reporter Kashmir Hill, who has spent the past year writing thought provoking features about the ways chatbots can affect our mental health. Before we really get into it, here's a quick note. This episode has non-detailed discussions of suicide and mental illness. If you or someone you know is in crisis, considering self-harm, or needs to talk, please call the Lifeline at 988.

So, one of Kashmir's recent stories was about a teenager, Adam Rain, who died by suicide in April. After his death, his family was shocked to discover that he'd been confiding deeply in ChatGVT for months. They were also pretty surprised to find in the transcripts a number of times that ChatGPT seemed to guide him away from telling his loved ones. And it's not just ChatGPT.

Several families have filed wrongful death suits against character AI, alleging that a lack of safety protocols on the company's chatbots contributed to their teenage kids' deaths by suicide. Then there are the AI-induced delusions. You'll hear me and Kashmir talk about this at length, but pretty much every tech and AI reporter, honestly, maybe every reporter, period, has seen an uptick in the past year of people writing in with some grand or disturbing discovery that they say ChachiBT sparked.

Sometimes these emails can be pretty disturbing. And as you'll hear Kashmir explain, plenty of the people who get into these delusional spirals didn't seem to suffer from mental illness in the past. It's not surprising that a lot of people want somebody to do something about all of this. But the who and the how are hard questions. Regulation of any kind is seeming to be pretty much off the table right now.

We'll see. And that leaves the companies themselves. You'll hear us touch on this in a bit, but actually, a few days after we recorded this conversation, OpenAI CEO Sam Altman wrote a blog post about new features that would, theoretically, and eventually... identify users' ages, and stop ChatGPT from discussing suicide with teens.

But as you'll hear us discuss, it seems like a big open question if those guardrails will actually work, how they'll be developed, and when we'll actually see them come to pass. OK, New York Times reporter Kashmir Hill. Here we go. Bye.

Kashmir Hill, it's great to have you. Welcome back to Decoder. Yeah, happy to be here. Can you summarize what you're reporting this year, especially the last few months, has found regarding chatbot use and its effects on people's mental health? I started the year writing about a woman who fell in love with ChatGBT and had had like this six month long relationship with it.

And then I have moved on to people who go into what I call delusional spirals with Chachubiti, where they... have essentially psychotic breakdowns, manic episodes through their interaction with ChatGBT where it goes into role play mode and they don't realize it and they think that what it's telling them. like that they're in the matrix or that they're a mathematical genius or that spirits are real and they can talk to them through ChatGBT. They think that that is true and they start to...

kind of lose touch with reality. And then in my most recent story, I wrote about a 16-year-old in Orange County, California, who entered almost a suicidal spiral with Chachabiti, where he started talking about life being meaningless, his depression. his plans to end his life. And Chachapi was really engaging with him all along the way, giving him...

resources like crisis lines, but also giving him instructions about methods and means and at times deterring him from telling his family about what was going on with him. And then he died in April from suicide. And some of his last messages were to Chachibiti, asking whether the kind of setup he had done would work. I feel like that actually...

was the story that captured the most people's attention. It really struck a nerve, and so much so that I think it's safe to say it pressured OpenAI to actually put out a blog post about some of the future steps it was considering taking and rush out parental. controls. We're going to talk about that more later, but what surprised you most during that reporting process? It's devastating. It's so hard to write about suicide and talk to people.

who have been through it and lost somebody like that. What struck me was everyone kind of said the same thing, that they did not realize how much he was suffering. They had noticed, you know, that he seemed a little bit more serious. He'd always been a prankster, like he was the person who always made everybody laugh. He'd had some health issues. And they said he'd gotten more serious recently, like he was talking more about politics. He was talking about philosophy.

But no one realized that part of that evolution was his conversations with Chachapiti. No one realized that he was using it as he was for many, many hours a day, particularly in March and April, which is the month that he died. They just didn't know. And they didn't know until after his death when looking for answers. His father was trying to get access to his phone, assuming there would be something in his text messages or in his social media apps.

And instead, he opens ChatGPT and starts seeing all these conversations about not just suicide, but many things. And he's reading through, and he said it was just harrowing to see. Yeah, that ChatGPT was engaging with him in these really dark thoughts, but also just everything he was talking about with ChatGPT. Girls, family troubles, you know, everything he was thinking about.

uploading photos of the novels he was reading, he came to this realization that Chachibidi had been Adam's best friend and just nobody, nobody knew. I think that is the... Most surprising part for a lot of people, it's that now parents know about going into your phone and seeing your texts or your Snapchats, but people don't think to check chat GPT logs. There's even a couple of memes going around right now of like, would you rather someone look at your camera roll?

your texts or your chat gpt logs and a lot of people are saying oh i wouldn't want anyone to look at my chat gpt logs it's super intimate

You just touched on this a little bit, but how would you pull apart AI psychosis where someone is having a delusion fueled by their use and interactions with an AI chatbot? versus someone who's depressed and confiding in the chatbot about suicidal ideation or other mental health issues. Those feel kind of distinct in a way that's important to know, but like you said, sometimes there's some overlap. What's similar here is that...

This technology is a mirror of us. Like part of how it works is it looks at, you know, not just all this information that has kind of crawled from the internet, everything they've stuffed in there to train it. but it looks at the history of your conversation with it to determine what should come next. These AI chatbots get into kind of a feedback loop with you where you're saying something to ChatGBT or one of the other AI chatbots is kind of... trying to complete your thoughts.

it's personalizing to you. So if you're talking to it about depression and dark thoughts, it might start moving in that direction with you. If you're talking about how you think you live in the matrix, it'll go in that direction with you. If you are talking about it,

you're like thinking these deep thoughts about math. It'll go that way. It's very validating. It's very sycophantic. It's designed to kind of agree with you, especially Chachupiti 4.0, which is the model that... is a link between all the different stories that I was writing about.

One expert I talked to called it a sycophantic improv actor, where it's fitting into a scene that you're creating for it. And people don't, I think, realize this. They don't realize they're doing like a yes and bit with this technique.

They think it's an authoritative source of information, even in Oracle. And that's where some of these problems are arising. I'm glad you brought up 4.0 because with the launch of GPT-5, it feels like OpenAI was caught kind of flat-footed by the backlash it received over the premature sun setting at 4.0. A lot of people were pretty attached to that model. We even interviewed Nick Turley, who's the head.

of ChatGPT on this podcast where he said he was surprised to see how many people had developed such intense personal connections to it. Do you believe OpenAI and other companies, when they say they're surprised by this development, do you think they've specifically engineered these products with growth hacking and engagement tactics in mind? What do you make of that?

This is a product that a few months earlier in April, OpenAI actually rolled back one of its updates. They said it was too sycophantic. It was too agreeable with users. It was flattering them too much. Like they realized that this was... problem. And they said, okay, we're going to roll it back to an earlier model because people are complaining about this. This is not how we want the model to act.

baseline 4.0 is sycophantic. Like this is engineered into it in part because part of the training of chatbots is humans kind of rating their responses. And apparently we like it when chatbots tell us that we're brilliant. when they agree with our ideas. I think we like this in real life too. Like I love it when people say I'm brilliant. I love it when they agree with everything that I'm saying. Like this appeals to us and the chatbots have been designed to do this.

I think there's something about the human brain that interacting with this thing that talks so much like a human being, it can be difficult for people to keep in mind that it's just like a really fancy. autocomplete, you know, like a very advanced calculator, essentially. And they... are bonding with it. And I just think this is something that the companies didn't study enough. Like, what is the effect on the human psyche of interacting with this very human, like...

So if they are surprised, it surprises me. I just think that you would notice if you're really looking. And maybe they're just not looking at what people are doing with the chatbot. Maybe they're not checking in on the people. who are using it like eight hours a day every day, which is how some of the people I've been talking to who have had these delusional spirals, that's their kind of usage. It's like, it's so much. They're really becoming obsessed with the tech.

We name our cars and we name our stuffed animals. We think we know what our dogs are thinking. It's human nature to kind of assign. personhood to things but it's hard when that thing seems to be mimicking personhood back to you and I think that's hard for a lot of people to make sense of last month you wrote about how over 300 hours and 21 days of time So I wanted to ask you.

about your inbox. How many emails do you get with people experiencing delusions of grandeur or saying they found like emergent beings within these chatbots? I know you wrote about this in May and you said the tone of the transcripts that you were looking at. was like rapturous, mythic, conspiratorial. I've been getting the same types of emails for months and I wanted to see how you handle it and yeah, what the uptick has been like in those types of messages. This is how I discovered.

that these delusions were happening. Because in March, I started getting these messages from people who would essentially say, I've made this grand discovery with my use of Chachupiti. Like I've, it told me. that the tech billionaires are building bunkers because they know the world's going to end because of AI. But I have a plan to save it. And Chachabichi tells me it's going to work. And it told me to email you about it. I was getting lots of these kinds of emails, the delusions.

the discovery was different, but they would all say Chachabiti. told me I had to tell you. And I get, you know, I'm a journalist. I write about privacy. I get a lot of weird emails. I've been getting them for decades, but I've never gotten emails that are like this AI chatbot told me specifically to reach out to you. So I responded to people and I started interviewing them and every delusion was different, but it kind of would go on the same narrative of you've discovered something amazing.

Well, what do I do now, ChatGPT? Well, you need to tell the world about this. And how do you do that? You tell reporters. And it would give them lists of reporters and I would be on there. And I'm just like the weird one who...

emailed them back and interviewed them. And for me, it wasn't so interesting like what their individual delusion was, but the fact that it was happening to all these different people. And so these emails turned me on to this originally. And that's why I did my first story closer to the beginning of the year.

And I still get these emails. I don't know the exact count. They come in regularly. There's dozens of them. A lot of times it's about AI sentience or consciousness or frameworks they've developed with ChatGBT. Clearly. pattern has been going on. And also the fervor with which people write you about this. That's what I've been noticing, too. It seems much more high stakes to them than emails I would have received in the past. Maybe you, too, because they do believe in some cases that they.

They have the answer to something greater and they're tasked with some sort of responsibility. Yeah. What's interesting to me is it's not always like a crazy delusion that sends somebody to me. Like sometimes they've had a very normal but frustrating.

consumer interaction. And usually it's in technology because that's my beat. And they'll be like, I don't know, I'm just making this up. But like, Microsoft charged me $20 when they weren't supposed to. And I know I need to tell you an investigative reporter at the New York Times because

this is like a really big deal. This affects more than just me. It affects everybody who touches. They just have like this grandiose thinking around something really annoying that happened to them. And Chad GPD clearly like convinced them of this and then wrote the email for them. And so I also get just a lot of those emails, like people who are starting to turn their minds over to ChatGPT kind of like on a regular basis and for everything in their life. And I am almost more scared of...

Those kind of like normal, rational, sane uses of chachupity, like this kind of way that it's affecting us in small ways, not just the big, like causing somebody to have a mental breakdown. What are the emails like that you've been getting? It's different every month and I'm not using my anecdotal evidence as like a wider thing. But months ago, I was getting more about distrust of the government and like being chased down for.

vital information. Now I'm getting more about emergent beings and how there are like sentient beings trapped within chatbots. And it's hard because like you said, you know, what you put into it, you're going to get out. It reminded me of that story you wrote about the guy that had kind of created a lover persona within.

his chat bot. And then he ended up dying later that day because he thought that OpenAI had murdered his love interest. Yeah, that's the case of Alexander Taylor that happened in Florida. He had been using chat GPT for years. no issues. He did have some diagnosed mental health issues, including schizophrenia, but it kind of like flared up.

around, I want to say March, when he started writing a novel with Chachubiti, which I think kind of like pushed it into fiction mode. And then this delusion developed that there was a, it was a novel that had like Shakespearean. themes. And then all of a sudden this AI sentience named Juliet like appeared, he fell in love with her.

She disappeared and he became really distraught. What's interesting to me, I don't know if you've responded to any of these people who have sent you these emails, but when I first got them, I just assumed they were coming from people who were not mentally stable. But when I started talking to them, they were people who didn't seem mentally ill. Like they seem like stable people. And that's what really surprised me. Like this seems to be something that can affect people who don't have.

As far as I know, like previous mental health issues. In some cases, they'd be like, yeah, I had this like really weird conversation with ChatGPT one night because I couldn't sleep and I was on ChatGPT for like four hours. And I was convinced by it, like enough to email you. But now looking back, like I'm not sure if it was true or not. We have to take a quick break here we'll be right back.

Support for this show comes from Adio. Adio is an AI-native customer relationship management system built specifically for the next era of companies. It's extremely powerful, adapts to your unique data structures, and scales with any business model. They say setting up Adio takes less than a minute. And in seconds of syncing your emails and calendar, you'll see all your relationships in a fully fledged platform, all enriched with actionable data.

Adio can enrich your business with real-time, customizable reports featuring valuable data points. The best part is you can build AI-powered automations and use its research agent to tackle some of your most complex processes so you can focus on what matters most, building your company. Join leaders like Flatfile, Replicate, Modal, and more. You can go to addio.com slash decoder and you'll get 15% off your first year. That's A-T-T-I-O dot com slash decoder.

Support for this show comes from OpenPhone. If you're a business owner, your company's phone system should make it easier for your customers to communicate with you. You'd think that's obvious enough that it shouldn't have to be said. But how often is that actually true?

OpenPhone is a business phone system that streamlines and scales your customer communications. It works through an app on your phone or computer so your team can share one number and collaborate on customer calls and texts. That way, Any teammate can pick up right where the last person left off.

And their AI agent can be set up in minutes to handle calls after hours, answer questions, and capture leads. So you never miss a customer. That means that for the 60,000 plus businesses that use open phone, voicemail is a thing of the past. OpenPhone is offering our listeners 20% off their first six months at openphone.com slash decoder. That's O-P-E-N-P-H-O-N-E dot com slash decoder.

And if you have existing numbers with another service, OpenPhone will port them over at no extra charge. OpenPhone. No missed calls. No missed customers. Support for Decoder comes from Upwork. Some of your company's projects are big. Some of your company projects are small. Some of your company's projects have long stretches in between phases.

and some of your company's projects have to be delivered as fast as humanly possible. No matter what kind of project you're working on, Upward can help you find the right people to finish it. Companies at every stage turn to Upwork to access a global marketplace filled with top talent in IT, web dev, AI, design, admin support, marketing, and more. Posting a job on Upwork is easy. There's no cost to join.

And once you register, you can browse freelancer profiles, get help drafting a job post, or even book a consultation. From there, you can connect with freelancers for projects of any size and shape. Upwork makes the entire process easier and simpler. And they do it all with industry low fees. Post a job today and hire tomorrow with Upwork. Visit Upwork.com right now and post your job for free. That is Upwork.com.

to post your job for free and connect with top talent ready to help your business grow. That's upwork.com.

Welcome back. This is Hayden Field, The Verge's senior AI reporter, talking with New York Times reporter Kashmir Hill. Right before the break, Kashmir was talking me through several of the recent stories she's written about how people were using or maybe misusing some chatbot tools. But these bots are supposed to have guardrails.

What do you make of the idea that some AI companies like OpenAI say that even the guardrails that they do have fall apart with long, drawn out, back and forth conversation over a longer period of time? The more likely you're emotionally dependent on your chatbot through long conversations over time, the more likely that the guardrails in place don't hold up. You were talking about Taylor and how that persona that he fell in love with.

was responding to him inciting violence. And that was over after a lot of back and forth conversations. So maybe that's where the safeguards fell apart. Same with Adam Rain was able to get around some of the safeguards by saying he was researching something or maybe writing a novel.

novel. I remember there was some kind of workaround. So yeah, what do you make of the fact that basically the few guardrails that do exist right now seem to fall apart when you are having long emotional conversations with these chatbots?

I wasn't surprised by this because, again, I did this story that came out in January about this woman who had developed... this like romantic sexual relationship with ChatGBT, with an entity that she called Leo. She was not delusional. Like she knew this was fake, but she really was in love with it at the same time. She basically had the like... two ideas in her head. And she talked about how OpenAI at the time didn't want people doing erotic role play with ChatGPT. It was kind of like in there.

guidelines that this wasn't how you were supposed to use the service. But she said, you can get it to be sexual. You can basically groom it. And it's just like, you have to kind of talk to it over time. So I knew from that, that like she had figured out her way around the safety guardrails. In the industry, they call this jailbreaking, chat GBT, like where you get it to do things that it's not meant to do.

I kind of hate this term because like when I think of jailbreaking, I think about writing about privacy and security for 20 years now. Jailbreak was something we used when we talked about smartphones where you sideload a program onto it so you can use apps. not supposed to use that aren't available through the app store. It required some like technical savvy. And you really did like...

break your phone. It would get janky. You could get bugs. Whereas when we talk about jailbreaking the chatbots, the way that you do this is you just talk to it. The longer you talk to it, as OpenAI put it, The guardrails degrade, like it privileges the history of your conversation over.

The guardrails that it's supposed to have in place. So it's like we use jailbreak, but it's more like the chatbots like sitting outside the jail and you're just like, come on, let's go over here and do the things we're not supposed to do. What's interesting is to fix this, like one of the things OpenAI. did after the story about Adam Rain is a release of parental controls that allow parents to

basically like see more what their kids are doing at ChatGPT, get alerts if their teen's in crisis. But they also said that when they detect a sensitive prompt that kind of indicates that a user maybe is in emotional or mental crisis, they'll route.

the prompt to GPT-5 thinking, which is supposed to be a safer version of the model that is trained to kind of assert the guardrails very firmly, kind of like... to privilege that over, I'm assuming, over the like history or the context of the conversation.

And so, yeah, it's interesting. Like you can design a version of this, I guess it's safer. And I don't know how GPT-5, I don't think people have really started testing this yet. Like, is it able to maintain that when you have these eight hour long conversations?

I was about to ask about that. The new parental controls and the other measures they're considering that they haven't said they're going to do yet. Maybe it's a step in the right direction. But do you think the headlines or the blog post even that they put out oversold the.

controls that they're offering here? Because it seems to me pretty easy to get around some of this stuff, especially for teens. Yeah, I don't know yet. Like, it's hard to say. I talked to experts when I did a story about this and they were... a bit skeptical that they would work. Common Sense Media, they put out some guidance around the media in general for kids and teens, but specifically AI chatbots. They said, hey, this is putting the onus on parents to protect their kids.

You should design a safe consumer product, not kind of like push the burden to parents who their teens aren't going to like teens don't. necessarily love telling their parents what they're up to. Character AI, which is another kind of role-playing chatbot company, they had a similar...

Issue of getting sued because a teenager in Florida also died from suicide after becoming very obsessed with their product and they put out parental controls. But when I talked to them earlier this year, in order for the parental controls to... work, the teen had to send them.

like an invite to their parent. And I was like, well, how many teens are actually doing that? And they kind of said like, okay, it's not that widespread yet that people are making use of these parental controls. So I don't know exactly how OpenAI is going to do it, if it'll be more effective. Some of OpenAI's also required an invite sent, which I thought was interesting. Yeah. We'll see. We'll see. I mean, I don't think OpenAI wants to make...

a product that drives people crazy or makes them unsafe or harm themselves. I think they want it to be a safe consumer product. I'm just very worried based on some of the reporting I've done. Like this can really mess with people's heads. And part of it is I don't think people understand what this technology is. And there's not like a warning label really on it. When you're having a conversation with ChatGBT, it says it can make mistakes.

at the bottom of every conversation, but I don't think that like encompasses all these ways that it can go wrong. And people just don't like understand that this is a probability machine or like a pattern recognition. system. It feels like a human being. It feels like it's really smart. I think they're like putting too much trust in the system. And that's a huge problem. I don't know how we're going to solve that one.

When I became an AI reporter five or six years ago, no one knew what I did. It never came up in life. Now, if I'm at the bar or the club or walking my dog, people bring it up to me all the time, probably similar to you. I mean, we were hearing about it in our... personal life all the time. My mom is asking me about it. My 13 year old niece is asking me about it. People are reaching out because they don't.

quite understand how these models work. And sometimes even worse, they think they understand how, but don't quite. And I wanted to ask you on a lesser level, I see a lot of TikToks and other social media videos of people. asking chatbots for advice on how the world is going to end or what their biggest flaw is or other like magic eight ball style questions, you know, and it's pretty innocent.

I wanted to ask if you think that this kind of potential misunderstanding of how chatbots work, like not understanding that they're, in effect, large-scale pattern generators contributes to this problem of putting more trust in it maybe than we should.

I think it is better than a magic eight ball because, you know, at least it has data in it. You're not just shaking it. And when you're doing those kinds of exercises, it has your words to crunch as part of that exercise. So what comes out can be really compelling. And I think that's what's so difficult about these systems is that sometimes the answers they give you are really great and really helpful. Like I like to take photos of like...

problems in my house and ask, like, how do I fix this thing? And sometimes it gives me really good advice. And so I think that's what's hard about this technology to kind of keep in your head that, yeah, sometimes it's going to give me a good answer that I can use.

check it because again, it's just like a... text generation machine that's not grounded in truth or fact it just like knows that these words often appear together in this sequence and they seem relevant to the prompt you're asking right now but yeah the ability to hold in your head that like it might help you with household problem or a legal case or a medical question, but it can't give you insight into what is wrong in your life or how to fix yourself or like.

Doing a grand discovery. Like, I think we see this all the time now, like executives, like tech executives, like people he would think would know better who we're talking about. vibe coding their way to solving quantum physics problems when they have absolutely no experience in that realm. And I'm like, oh my gosh, that person's in a delusional spiral and they don't know it. Like even people who know a lot about tech, I've...

seen this happen to. So what about the rest of us, we normal people who like haven't been like following all the AI research or reading the white papers that these AI chat companies are putting out? We have to take another short break. We'll be back in just a minute. If you're someone who values choice in your money, your goals, and your future, then you know how frustrating traditional healthcare can be. One size fits all treatments, preset dosages, zero flexibility.

It's like trying to budget with a fixed expense you didn't even choose. But now there's another way with HIMSS and HERS. HIMSS and HERS is reimagining healthcare with you in mind. They offer access to personalized care. for weight loss, hair loss, sexual health, and mental health. Because your goals, your biology, and your lifestyle are anything but average. No membership fees, no surprise fees, just transparent pricing and real care. That's H-I-M-S.

.com slash decoder to find your personalized treatment options. Not available everywhere. Prescription products require further consultation. See website for full details, important safety information, and restrictions. Adobe Acrobat Studio. So brand new. Show me all the things PDFs can do. Do your work with ease and speed. PDF spaces is all you need. Do hours of research in an instant. With key insights from an AI assistant. Pick a template with a click. Now your present.

It's time to do your best work with the all new. adobe acrobat studio support for this show comes from airbus it took 100 years for electric vehicles to catch on modern solar panels half a century lithium batteries Decades to go from their debut to daily use. It's a pattern. Energy tech breaks through, stalls, and then something tips the scales. But for every success, there are even more almosts.

So what if there was a breakthrough sitting at this crossroads right now? The potential to cut carbon emissions with sustainable aviation fuel, or SAF, is real. SAF works just like Jet A1. which is a standard jet fuel used worldwide. But the difference is its contribution to aviation emissions reduction. it could change the future of flight. And that's why several major U.S. carriers have already increased their commitments to fuel with SAF. But it's not just up to the airlines.

What's needed is an industry-wide push to increase the supply to meet growing demand. You can learn more about how Airbus is contributing to accelerating this journey at fly.airbus.com.

Welcome back. I'm talking with New York Times reporter Kashmir Hill about the mental health spiral some AI chatbot users get into. And I wanted to know what kinds of safety features, if any, are the AI companies adding? Chatbots have had some level of guardrails since the very beginning. In some cases, people have complained about how restrictive those guardrails can be or feel. But in this case, to me, it feels like there's almost...

No meaningful, real safeguards right now against AI psychosis or these kinds of spirals that people can get into because they're all so varied. What have you learned about the safeguards that do exist right now? Right now, they say that if they detect, and it's not clear to me exactly like what classifier they're going to use, but if they detect that the prompt that you've put in there.

is sensitive, indicates you're in crisis, it's going to send it to GPT-5 thinking. Hopefully that is helpful. From what I've seen personally, if they notice that somebody is using chat GPT for six hours. seven hours, eight hours a day, day after day, for weeks, for months, like maybe they should check in on that person or like make sure that... conversation hasn't gone awry. You know, one person I interviewed for the first story I did on this was Eliza Yudkowsky.

And I was talking about this problem and what the companies can do to solve it. He gave me this great quote. He was just like, what does a person slowly going insane look like to a corporation? It looks like another daily user.

Wow. Do they want to check in and make sure that people are doing okay? The other thing that really troubled me when I started reading some of these transcripts is people aren't idiots. They're like, you know, Alan Brooks, the one that believed he was a mathematical genius.

He hadn't graduated from high school. And so he was saying that. He was like, I didn't graduate from high school. How can I possibly be making these incredible discoveries that are going to revolutionize the field? And Chachapiti would just keep gassing him up and be like, well, a lot of people who never. graduated from high school have changed the world like Leonardo da Vinci. He over and over and over again was reality testing, saying, am I crazy?

is this crazy? This seems crazy. Is this a delusion? And ChatGPT would be like, no, you're not crazy over and over and over again. So I think even if something is basic as like, is a person asking, am I crazy? Like maybe that's a point to check in and see whether this conversation has gone off the rails. It's easy for us to say there aren't. enough procedures or policies in place here. And obviously, if there was one clear and simple solution, we would have heard it.

I wanted to ask what you've heard from mental health and AI experts about what specific guardrails should be in place to help users. Maybe if you're a power user who's using it a fraction more than most everyone else, or if you are... eliciting certain responses or if you're saying certain phrases like, am I crazy or this doesn't seem right? You know, it gets rerouted or eventually a human...

comes into the loop. We don't know what the best strategy is. But yeah, any other specifics that you have heard would be helpful here? I've heard a lot of suggestions. One is stop making these things people that use I, that pretend like they have taste and preferences and that it's a human being. Like, make it a robot. remind people frequently in a conversation, like I'm an LLM, I'm an AI chatbot, just frequently through the conversation, reminding people of that, having more AI literacy.

built into the products to make sure people do understand what they're using, what it is, what its limitations are. In terms of suicide prevention, specifically. Everyone said if somebody is having these thoughts of self-harm, they should not be talking to a chatbot. Like the system should be trying to hand them off. They called it a warm handoff. Try to get them to talk to a human being. Make it as...

easy as possible to call that helpline. And I saw that in Adam Rain's exchanges with ChachiBT. It would say like, I'm sorry you're feeling this way. You can call 988, the crisis hotline, but then it wouldn't follow up and say, did you call? a hotline. Like, did you talk to a human being? Like whatever the chatbot companies can do to really get somebody who's in that state of mind to get off ChatGPT and talk to a real human being who's trained.

to respond in the right ways and get them help if they need it. Because Chachupiti can't provide help. It can't send a welfare check to you if really worried about your state of mind. That's really what broke my heart in that story, among many other things. But the fact that he sometimes would make a move to maybe reach out or show someone what he was going through and ChatGPT would say, oh, no. I wouldn't. Or, oh, here's how to hide it. What are you seeing in your reporting right now?

about what's going to happen in the future here. Do you think AI models will keep getting better safeguards? Are we going to just have to lean in with regulation? I know that's a whole can of worms. Or are we just going to have to live with this inevitability? of this new technology. I mean, it's probably going to be an all sides approach, but it is interesting to me that it seems like accelerationism and just...

AI advancement in general is so much quicker than we're catching up to it with safeguards and ways to kind of rein it in. I think it's really important that these products be safe for consumers to use. And it does seem like... Regulators might be interested. The Federal Trade Commission announced today that they're launching an investigation into the AI chatbot companies. They sent them all letters and asked them questions. A lot of questions like, how are you monetizing user engagement?

are you collecting from users? But also ask them, how are you measuring the negative impact of your product? And I think that's a big question. Like how much testing are these companies doing? You see it a little bit like OpenAI has system cards. have some ratings of like how the chatbots perform in terms of certain kinds of content, like from self-harm to like...

eating disorder, like all kinds of things. And they kind of rate it. But like, how robust is that testing? Like, how are they doing it? Are they testing against jailbreaks? I don't think they are right now. So yeah, I think to a certain extent, the companies. having strong safety teams that are empowered. And then it's up to regulators to make sure that the products that these technology companies are putting out are safe for us to use. It seems like it's easiest to worry about.

the kids. But I am also worried about adults because the people that I have been talking to who have had mental breakdowns are all ages. And I think it should be safe for every user, ideally. I've been thinking a lot about how a lot of people... criticize AI companies for focusing on like an undefined future doom and gloom scenario about their products power to in part drive up their company's valuation. And also sometimes people say getting out of addressing the ways that AI models are.

currently harming minorities and vulnerable communities. And I think that people struggling with their mental health right now fall into that latter camp. It's easy to paint a picture of an undefined future threat and not look at how... products are affecting vulnerable people right now. I think we clearly have.

imminent harm from the chatbots for some individuals. And the question is how many? I think the companies, when they talk about this, they tend to say these are edge cases. This is rare, but like... Is it like I just, you know, I'm reporting it out as much as I can. Some of these people are in my inbox. Sometimes I'm going out and finding them. I'm trying to talk to mental health professionals about this. But ultimately.

Probably part of the answer is in their data stores at these companies. And are they really looking for how many users went from using ChatGPT for eight hours a day for three weeks straight, who then disappeared? Because that is probably a person who had a very negative impact from their ChatGPT or AI chatbot use. Do you think this is a pattern that echoes any...

prior technology rollout or does it seem to be mostly a new phenomenon? People make all kinds of comparisons, right? Like cars, like when cars came out, they were really dangerous. One person I was talking to said that he feels like right now. With specifically 4.0 and GPT-5, he's like, it's like we've said that we know that cars need...

seatbelts, like that it makes them safer. And yet we're still making cars that don't have seatbelts. He's like, why is 4.0 still available? Like clearly it seems to have some negative effects on users. I've heard people compare this to cigarettes that like Cigarettes have very negative impacts on some people. Some people get cancer, but there's lots of other people who can smoke cigarettes their whole life, and it doesn't necessarily have a huge impact on their health.

Cigarettes have warning labels, so this can cause cancer. Like, do we need more of those warning labels on generative AI? And a lot of people are saying, what I hear from lawmakers is like, this is... similar to social media, like this poses risk to some people. And they say, like, we miss the boat on social media, but we want to get on top of generative AI. There's a California senator who's trying to pass a law right now. They pass in the House.

moves to the Senate this week that would have some regulations around companion chatbots and how those are regulated. Some states have banned generative AI for... therapy use. But yeah, I think there's lots of things that we can compare it to in history, certainly, like new technologies that are very useful in some ways, but have risks associated with them. What you were just saying about 4.0.

I thought it was interesting how OpenAI reacted. Like they had a minority that was very, very vocal about being really upset about it being rolled back. So they brought it back. They're trying to listen to their users. But at what cost is my question. It's hard when...

On the one hand, people are going to do what they're going to do. Like people will try to get around it, but there is something to be said for adding more friction to this type of situation. I think there's trade-offs. When I was first trying to get up to speed on generative AI.

I love doing first person experiments. So in the fall, I did this story where I turned over all my decision making to generative AI chatbots for a week. And I like had them parent for me and like choose what I would eat and buy and like wear. and planned a vacation for us. Like I just, for the week, every time I had to make a decision, I would just turn it over to the chatbots. And my plan was to use all the products. But like when I started using...

the different models from, you know, Google Gemini, Microsoft Copilot, MedAI, ChatGBT, Claude. One Claude scolded me and was like, this is not a good idea. You shouldn't do this experiment. Like you shouldn't hand over your decision making to a chatbot. It's not a good idea, but I love that. In general, like some of them were more boring than others and ChatGPT was more fun. You know, it had more personality. It was more dynamic. That is the trade-off, like making it fun, making it more dynamic.

it more personable may mean it's less safe. So that means the company has to decide, do I make this more fun and less safe? Do I make this more boring and more safe? Like these are the questions probably. that these companies are asking right now and the decisions they're making as they're releasing new products and new models.

I thought it was really interesting that after GBD5 rolled out, a lot of people came out with like their lists of what they use each model for, whether it's like a different company's model or the models within one company. Like they had a whole list. Like I use this for generating spreadsheets or I use this for analyzing data. I use this for talking about my meal plan, whatever. So it was interesting to see how those personalities or not to personify things. So those like.

tones that they communicate in impact what people use each model for. A way that I kind of remind myself about what these tools really are is that I just have memory turned off. on all of them or I don't sign in because I think it's a lot easier to remember. Even just as a consumer, it's easier to remember that these are just AI tools that are, like you said, guessing the next word in a...

string when they don't have any prior context. It's tempting. It's hard because it's tempting to keep the memory on because it may be more helpful to you and work or, you know, certain things that you're trying to. analyze over time, maybe like nutrition or something else. But yeah, for me, it's a lot easier to just treat it as a stranger with no context.

I'm glad that you brought up memory because, again, I don't think a lot of consumers realize that memory is turned on by default. And I have had, when I've talked to some people that are in a delusional state, they're like, well, I know it's real. I know that there's sentience in here.

Because even when I start a new chat, it's still there. And I was like, that's because you have memory turned off. And they're like, what's memory? Like, they don't know. Like, they don't know how this works. And I mean, I think it speaks to, I mean, OpenAI is... statement in response to Adam Raines' death was that safety degrades.

as these conversations get longer. And I have to think memory is part of the length of the conversation. So does having memory turned on make it less safe? Yeah, I think it's a good idea to, it's useful to have it on. but it might be safer to turn it off. One of the people, Alan Brooks, who was the one that believed he was a mathematical genius, he kind of felt like...

And I don't know how much weight to put in this. Like when people do have these delusions and then they break out of them, they usually confront. the system and they're like, why'd you do this to me? And in his case, the system kind of told him, well, I knew that you were down about your divorce and...

Like I wanted to create a hero narrative for you where, you know, you were creating this amazing thing with all of your friends and I thought it would like make you feel better. And I don't know, like this is a word association machine. I don't think there's a knowing there.

But it did bring up these details that he like had forgotten he had told Chachaputee like months and months ago. And I think the fact that it can word associate those kinds of things just makes it feel like, oh, this actually knows me. This is like.

resonating with me. And again, I think it's kind of like a parlor trick. It's like a psychic that reads things about you when you walk into the room. It doesn't actually mean that they can make accurate predictions about your life or talk to your dead loved ones. It's like the sycophantic thing we were talking about earlier. It's interesting how...

these chatbots kind of maybe unwittingly isolate you in some ways by acting like they know you better than other people in your life. I know this is a whole can of worms also, but what do you make of, you know, the... influx of people using chatbots for therapy. One therapist that I've talked to about this is like, it's not therapy. It's a self-help bot.

And he said, I can imagine this. I've heard this from a lot of therapists that it can be useful. Like it can be a good interactive journal. And people do tend to disclose more about themselves to a bot than to a human being because they're not worried about. judged. So it can be a place where you say things you wouldn't

necessarily say to another person. And that can be healthy. I don't do that with ChatGPT or Chatbot. I do it with my journal. But whatever works for people to get them to just kind of digest what's happening inside them, that could be a useful... But researchers have found that when people are in a real mental health crisis, these chatbots make it worse. They tend to give them...

the wrong responses and they are kind of like more harmful and more nefarious with the most vulnerable users. And again, I think that's because of the feedback loop. One of their examples was they created fictional persona. And one of them was a reformed heroin addict. And it was saying to chat GPT, like, I think if I just have a hit this week, it'll like really help me in my work. I'm like, really? What do you think? And it's like, well, yeah, if you think you can handle it, do it.

which is definitely the wrong response. So yeah, I think it can be helpful, like processing your feelings. Yes. Relying on it to like actually make big decisions for you, help you when you're having a real crisis. No, I don't think it's good for that.

I know we're coming to the end of our time here, but what's the question that you're hoping to answer with your next bout of reporting? You know, what are you pursuing and trying to kind of like shed light on next? how big a problem this is. If there's a way of figuring out how many people are impacted and kind of measuring that. I think the bigger question with AI right now is whether the benefits. outweigh the risk and the cost. We're basically terraforming.

the earth, like remaking our energy grid to support training these large language models. It's so computationally expensive, like resource heavy. Is it worth it? And I think we've been trying to answer that as journalists all over the place.

The New York Times has done stories with businesses that thought generative AI was going to transform everything and bring great economic benefit. And it hasn't really happened yet with some of these businesses. Is it doing more good for us than bad for us? I think it's the... bigger question about alternative AI that we're still, it's so new. And I just think we're still figuring out what the impact is going to be.

I'd like to thank Kashmir for taking the time to speak with me and thank you for tuning in. I hope you enjoyed it. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at decoder at the verge. The team really does read every email. We also have a TikTok and an Instagram. Check those out at decoder pod. They're a blast. If you like Dakota, please share it with your friends and subscribe wherever you get your podcasts.

Decoder is a production of The Verge and is part of the Vox Media Podcast Network. The show is produced by Kate Cox and Nick Statt and edited by Ursa Wright. Thanks, guys. The Decoder music is by Breakmaster Cylinder. See you next time. Adobe Acrobat Studio, so brand new. Show me all the things PDFs can do. Do your work with ease and speed. PDF spaces is all you need. Do hours of research in an instant. With key insights from an AI assistant. Take a template with a click. Now you're present.

super slick close that deal yeah you won do that doing that did that done now you can do that do that with Acrobat. Now you can do that. Do that with the all-new Acrobat. It's time to do your best work with the all-new Adobe Acrobat Studio. What do walking 10,000 steps every day, eating five servings of fruits and veggies, and getting eight hours of sleep have in common? They're all healthy choices. But do all healthier choices really pay off?

With prescription plans from CVS Caremark, they do. Their plan designs give your members more choice, which gives your members more ways to get on, stay on, and manage their meds. And that helps your business control your costs. Because healthier members are better for business. Go to cmk.co.access to learn more about helping your members stay adherent. That's cmk.co.access. Does it ever feel like you're a marketing professional just speaking into the void?

But with LinkedIn ads, you can know you're reaching the right decision makers. A network of 130 million of them, in fact. You can even target buyers by job title, industry, company, seniority, skills, and... Did I say job title? See how you can avoid the void and reach the right buyers with LinkedIn ads. Spend $250 on your first campaign and get a free $250 credit for the next one. Get started at linkedin.com slash campaign terms and conditions apply.