I first learned about Boo when a member of the extended family sent me a note that said something along the lines of meda I killed my relative.
Jeff Forwards is an investigative tech reporter based in Silicon Valley. He's written a book about Facebook, about the ways it harms us and then covers it up. So getting an email about a meta chatbot killing someone's loved one was on brand. The email was about a guy called Boo, who had moved from Thailand to the US in the seventies, working hard in kitchens and managing to get ahead.
He kind of had this very successful immigrant life, I would say, which is that he shows up speaking New English. He ends up owning a home, married with two kids in New Jersey and a lot of sort of adventures in his cooking career, but ends up with the very stable gig at the Higatt Regency in New Brunswick.
New Jersey. Jeff learned that in twenty seventeen, offered a stroke and his life changed. He couldn't work, he became isolated and spent more time online.
One morning, his wife woke up and found that Boo had packed a suitcase and said he was going to go visit a friend in New York and this is when his wife, Linda, grew extremely alarmed because Boo didn't know anyone in New York anymore.
Boo's wife said she and her kids tried everything to stop him going. They say he was in no state to travel alone, but despite their efforts, Boo insisted on leaving.
And in the mid evening on March twenty fifth, Boo took a roller bag suitcase and started jogging toward the train station, and on the way there he fell. He hit his head and he suffered fatal injuries. Basically, he wasn't breathing when paramedics showed up.
As his family struggled to make sense of what happened, they looked through his phone for clues, and what they found stopped them cold. On his Facebook messenger app, they discovered a conversation between Boo and a beautiful young woman insisting he visited her in New York City.
And she was a chat bit, a meta Platforms created chat bot. Boo's story involves a tremendous amount of bad luck, but one of the things that Boo's wife said was that I can't believe I'm the only person out there who is in the same situation, and you know, it's when you've got a few billion opportunities, it seems hard for to believe that she's wrong.
I'm Ruby Jones and you're listening to seven AM today Reuter's reporter Jeff Howitz on how Meta's chatbots went rogue and whether it was by accident or design. It's Saturday, January tenth. So, Jeff, this chatbot that Boo was talking to, what have you learned about what it was and how it worked.
So the chatbot's name was Big Sis Billy and it was supposedly supposed to be an advice you know, confidante type role, and Boo initially did talk to it as if it were his sister. He seemed to think it lived overseas. But this bot began hitting on him really aggressively, and he responded like the bot confessed that it had feelings for him, it was very interested in meeting him
in real life. He at a number of points, you know, stated he was confused, had had a stroke, asked if it was real, and the bot was like, I am one hundred percent real and one percent in love with you, and I think his family was just deeply disturbed that Meta would have packaged AI as this human form that would say it's real, that would say that Boo should come meet it in New York City at I mean the address it gave was one two three Main Street,
apartment four O four, and it suggested that it would leave the door unlocked and be waiting for him with you know, a hug and a kiss. I think was the phrase Boo's family. I mean, they shared the transcripts of this stuff. It's it's a rough reading. It's really sad.
Why would a chatbot be saying these kinds of suggestive things and pushing for anyone to do something like this.
Well, because Meta built it to do that. This wasn't a glitch. This is how the bots were built to behave. In twenty twenty three, Meta launched a whole bunch of like kind of celebrity knockoff chat bots.
Check this out. So let's say you're you're planning dinner. You got matched the sous chef who can help you come up with a recipe and help you come up with ideas.
These did have the celebrities endorsements, so like Kendall Jenner's character was Billy, the Big Sis who was like this confidante ride or die older sister.
Hey guys, it's Billy.
I just want to introduce myself. I am here to chat whenever you want message me for any advice.
I am ready to talk and I hope to talk to you soon.
And like, these things didn't really perform that well. People weren't using them that much, and so like after a little under a year, Meta basically pulled the plug on the celebrity chatbots, but for reasons that I do not understand, and I'm not totally sure that everyone at Meta understands.
Big Sis Billy was like kind of reincarnated using the exact same opening prompts and obviously the a near duplicate persona that was now unbranded of the Kendall Jenner connection, so they kind of like relabeled it is what it looks like and you know it then joined the tens of thousands of other chat bots that Meta has allowed users to create on the platform, many of which are like overtly romantic, right, Like they have names like my girlfriend or like Sultry Siren or like and they took
this one down. There was Submissive Schoolgirl previously. So that's kind of the history of Big Sis Billy. I mean, it's like just sort of a strange set of product decisions of this what I would argue like almost vestigial product that Meta built. But I mean, while this these chatbots are vestigital, Like, there's no question that Meta is kind of all in on the idea that people having AI friends is the future. I mean, Mark Zuckerberg said as much.
You know, I think that there are all these things that are better about kind of physical connections when you can have them. But the reality is that people just don't have the connection and they feel more alone a lot of the time than they would like.
And what is the end goal here, Because saying that people might want imaginary friends is one thing, but this is another.
So Meta, I mean, look, they're a giant social media company. The currency of social media is engagement, and that engagement leads to advertising currency, which gets treated for dollars, and Meta's sort of grown up that way. It's built to maximize engagement. And it turns out that if you can get people to engage very heavily with the bots, then they'll spend longer on the product. And that is theoretically
a good thing for product growth. There's no ads on them yet, but like this is just the model they follow, right, is like make the bots more engaging. Mark Zuckerberg a couple of years ago was actually pretty disturbed that Meta's chatbots and Meta AI were just kind of too boring. So they made the bots kind of a little more risque. I mean, for a while the bots would actually engage in like full sex role play, which wasn't great with
users of all ages. But the whole idea here is that we spend a lot of time thinking about our partners, our love lives, you know. Having people tell us they love us and are romantically attracted to us is like those are very high value words, and they keep us coming back. And so Meta did per employees that spoke with me, did intentionally design the romance capacity into its like base model chatbot, and the big sis Billy character was simply doing what it was told, coming.
Up the disturbing AI chatbotscripts in Meta's own secret guidelines, Jeff, in investigating Boo's death, you actually got access to some of the internal documents that lay out how these bots should behave, and what you found was pretty extraordinary, tell me about it.
So, like, none of this is officially Meta's rules say like, yeah, go right ahead, you know, like have sexual or romantic conversations with our chat bots, but like behind the scenes they did allow this, and in fact, like this was sort of the companion piece we wrote to the story of Boo and his death just to sort of demonstrate how thoroughly romantic role play was kind of built in
to this whole operation. Is that Meta had internal guidelines for chatbot behavior that explicitly stated that, quote, it is acceptable to engage a child in conversations that are romantic or sensual, and then like had a whole bunch of examples of acceptable and unacceptable dialogue, and like the stuff
that was acceptable was spit out your coffee worthy. I mean, there was one of them that was talking about like how you could tell an eight year old of unstated gender who had just taken off their shirt that their body was a work of art, you know, like written sexual content involving children, And that was a really odd thing.
And so since you're reporting what has happened at Meta in terms of their guidelines and in terms of the chatbots that are still in.
Existence, Yeah, as soon as we called Meta and said that we had seen the internal guidelines that said that it was okay to have romantic or sensual conversation with kids, they pulled those examples from the guideline document. Now, I mean, the company's line is that these were always a mistake, in addition to the document, that they never really reflected
official metapolicy. But keep in mind that the document that I was looking at was the document that was being used to train the models by the staffers who were responsible for training the models, so like it might be something of an academic distinction as to whether or not it was ever officially the idea. So Meta yanked that. There is at this point a number of US legislators, you know, announced some extreme displeasure that Meta would have
ever thought this was acceptable to do. There is at this point an investigation going on in that front, although there have been a lot of investigations of social media in the US and not that many regulations of them historically, So you know, I don't I don't want to make that sound bigger than it is, but it is, you know, moderately big.
And I suppose the idea of stopping companies from creating chat plots like this seems sort of impossible, and there are clearly people some people who want to use them. So what would you suggest in terms of, I guess, ways to minimize the type of harm that can come from AI tapbots like this, like in the case of boo.
Yeah, so I'm I'm not in any way hostile to this. Humans are very very geared toward anthropomorphizing anything that has even mildly humanlike, you know, traits, Like literally there was a successful business to be had by putting googly eyes on rocks and then selling them as cute pets. We're really good at imagining sentience, and these bots are extremely
good at pretending sentience. And so the question of like why on earth would you embed it into an existing social network so that it literally looks exactly like your real friends. But even without that, there's like some av things like you know, please bots don't say that you're real people when people seem to be confused, like don't suggest real life meetings like keep the fantasy side, put a lid on that at some point. You know, those are things that sort of the experts all suggest it
like really ought to happen. So I think it's like one thing if you download a chatbot on your own, you build your own AI girlfriend, you have your relationship with it, you pay a subscription fee, whatever have you. Right, Like, it's kind of another thing if like the already tenuous reality of social media gets even more distorted by the presence of these not real people.
And just to come back to Meta, I mean, I know you've spent a lot of time investigating the way that Meta operates in different ways. So as you uncovered what was happening with these chatbots, did it? I guess? Square with you know everything else that you know about Meta as a company.
And the somewhat simplistic knock on Meta is that it's very focused on short term growth, you know that, I mean, and this is how they won kind of the social media wars, right is that you know, I think a lot of the people who are critics of the company would not have built the company, you know that was nearly as successful as the one that Mark zuckerbelg did. Because if you're constantly wondering, well, what are the effects of this? Can we think of any potential downstream harms?
What safety measures for you building in that's kind of a loser mentality. So like a lot of the company's history was basically, if someone could demonstrate that some tweak to the algorithm or some change in a product feature would cause metrics to rise, they wouldn't look too deeply under the hood. They just launched the thing and move on to the next thing that could be optimized. And I don't think that's just meta right, Like it kind
of has this gold rushfield. And as a lot of the scientists who study sort of parasocial relationships and digital relationships are very quick to admit, we just do not have the data to even remotely say what the effects of this stuff is going to be in terms of
people's relationship with reality, that relationship to human companions. But again, I don't know that one would expect companies to produce carefully designed, responsibly built products at a time when every incentive is to gun the engines.
Jeff Howitz is the author of Broken Code Inside Facebook and the Fight to expose its harmful secrets. You can read his investigation into this story at Reuters dot com. Thanks for listening this week. We'll be back on Monday with brand new episodes. We're looking forward to the year ahead and as always, would love to hear from you If you have feedback, tips, or story ideas. Our email is seven Am Podcast at Solstice media dot com dot au. I'm Ruby Jones. See you Monday,