A note for listeners, this episode includes some disturbing descriptions of sexual acts and assault. If you have kids around, you might want to use headphones and please take care when listening. If you've looked at headlines about Silicon Valley at all in the past year, you've likely heard of artificial intelligence and generative AI tools think chat, GPT, or stable diffusion. They're touted as the next big thing, but the use of these applications can have a dark side.
Many of them are open source and freely available, and users are able to alter publicly available photos of people, including images taken from social media, to depict events that never happen in real life. These images are called deep fakes, and oftentimes they're being altered in sexually explicit ways. While the photos are fake, the harm is real. In many cases, the victims are young people who don't know the images
exist until someone they know discovers them online. Bloomberg and investigative reporter Olivia Carville went to a town in New York where this happened.
In terms of the impacts to the young woman, Nightmare is one of them lost over twenty pounds from streets. A number of them started carrying pepper spray in their handbags. Two of them started carrying knives in their handbags, and they still feel a sense of trauma from what happened.
As awful as these photos can be, law enforcement and victims have found it difficult to use the laws currently on the books to bring the people making these images to justice. Bloomberg cyber reporter Margie Murphy says prosecutors can find themselves in a bind.
It's still incredibly blurry. A lot of people within law enforcement that I spoke to you for the story all said that it feels really mishmash at the moment between state laws and federal laws, and that the law really hasn't kept up with the technology in this area.
Olivia and Marghi are here to detail their business week investigation. I'm Nancy Cook today. I'm the big take how AI generated deep fake images are hurting real people. So Margie, let's define the terms that we're going to talk about. What is a deep fake photo?
So a deep fake photo is a picture which has been digitally manipulated to depict an individual doing something they didn't and that's using a deep learning model, so artificial intelligence. That's where the deep comes from. And so right now today that would be using open Aiyes, Dolly is one that you may have heard of. There's mid journey, there's stable diffusion tools where you can basically ask a machine to create an image that looks incredibly realistic, but it never happened.
And what is generative AI as you talk about it in the.
Piece, the generative AI that we use at the moment tends to be using text to image. So it's the ability to kind of type in and say picture of person eating an apple, and the machine can generate that image for you, but you know, no photograph was ever taken, but it looks incredibly real.
And Olivia, you open your piece with a chilling story out of New York. Can you tell us what happened?
Yeah?
Sure. So I work on Bloomberg's investigations team and had been interested in deep fakes, I think for most of the world. We came to learn about what a deep fake image is from that iconic shot of the Pope in a puffer jacket, and I did a lot of research around deep fakes and then partner it up with Margie and we decided to try and find a case which illustrates just how damaging this technology can be, and we found that case in Long Island, a suburb of New York City. This is one of the first deep
fake related conviction in US history. And what happened here is a young man decided to use software tools on the Internet to manipulate doctor alter images of girls from his high school. He targeted about forty different girls that he went to school with and did so in very horrific ways. In some cases, he digitally undressed them, He put their faces onto the bodies of women engaging in sex acts. He used photographs from when they were as
young as five years old. And this case really shows us just how difficult it is to lay charges against an individual who created fake or deep faked pornography.
And do we have an inkling about who was the person responsible for this?
Yeah, that's one of the most interesting parts of the case. We felt is that the young woman who had been targeted in these images actually investigated who was behind them because they felt that authorities weren't doing enough to track him down. The story really starts on New Year's Eve twenty twenty, where a group of young women in Leavittown came to discover photographs of them had been altered, manipulated,
turned pornographic and nature and uploaded to this website. The website has a very graphic internet address name that's come on printed picks dot com. And on New Year's Eve, this group in Leavittown actually started discovering their images had been uploaded to the site, and word got out. They began messaging one another, sharing a link to the website and discovered just how horrific some of these images and the content that had gone up on this website had been.
In some cases, their poster was talking about raping and murdering the girls. And so it was that New Year's Eve that they first started reaching out to the police. And throughout the course of twenty twenty one, the young women themselves were investigating who was behind this, who had uploaded all of their photos, altered them, turned them pornographic, and posted them onto this website, and a lot of them came to the conclusion that they knew who it was,
that they knew who the suspect was. They believed it was a man that had attended school with them, a former classmate, an actual friend of many of theirs as well they recognized his handwriting and some of the images. They recognized his writing style in some of the captions on the website that he posted their images too. And it was the young woman who actually took his name to the police and said they believed he was behind it all.
And Olivia, how old were these young women when this happened and how old are they now?
Most of the young women were around eighteen or nineteen years old when they discovered this website. It was the year after they graduated from high school in Levettown, and today they're around twenty two or twenty three years old.
Did they have a hard time getting the authorities involved in this, including the local police, and making sure that they knew it was a problem.
I think there was definitely a frustration that the police, the families felt, at least, weren't moving fast enough to try and do something about this case. It's important to remember that cyber harassment cases or online vulgarity isn't really high on police priority lists. In this example, you had an individual who was creating fake images, and there's no real law that says what he was doing was illegal.
So while it was grotesque and you'd look at the site, and certainly the woman that I talked to were disgusted by what they saw. But the difficulty is that there are no laws in the US that said what this young man was doing was actually illegal.
And Margie, can you explain how the new advances we've heard about this past year in gener of AI are making it easier to generate deep photos?
Yes, so the new advances have just one improved the look of these pictures, so they just look incredibly realistic now in a way that a few years ago, when we first started hearing about deep fake and deep fake pornography, they just didn't quite look as convincing. And secondly, a lot of the tools that have come out are open source, which means it's really easy for people to download the model,
train the model themselves. A lot of the technologies, you know, they've been created so you don't need super compute power to process these models. The idea was to kind of democratize them and make it so that everyone could use
their PC to kind of generate images and videos. Not for bad necessarily, but of course with the Internet, there are always people who will take these great technologies and do horrible things with them, and so this open source nature means that kind of once the cats out of the bag, it's very difficult for people who develop these technologies to put guardrails in place, and some of them have, and they've been very upfront about talking about how, you know,
we're trying to help also do deep fake detection. So once deep fakes are out there, we can say, oh, yes, that was our tool, and you know, be careful that
isn't a real image. But once it's out there and people have access to the core technology, they can remove those guardrails and they can kind of do what they want with them, which is why we've seen this rise in deep fake pornography and this kind of cottage industry that's emerged around undressing apps, which have just proliferated across the Internet.
Olivia, you spoke to many of the women affected in this New York case. What did they say about how it impacted them and what they kept trying to do to call attention to it.
I think one of the scariest things for some of the women I spoke to was just how realistic those images looked. They looked at the photo where they recall taking it, and they were wearing a bikini or a t shoe, but the image that had been uploaded to this website, it showed them completely naked, and in some cases the breasts that had been imposed on their bodies were actually so realistic looking that they believed that other
people would think it was really them. And I think that's what was one of the hardest parts for some of the young women involved. In others, they were just scared, to be honest, that was I think the biggest emotion that came from the young woman that I talked to was a fear of men, a fear of the internet.
Not only were their photos uploaded to this awful site, but in some cases, the poster was actually encouraging users of this site to reach out to the girls individually, to contact them directly, so he posted their full names,
their addresses, their social media handles. I talked to a lot of parents as well because of the sensitive nature of the case, and the emotion I got was just raw anger and also frustration that what happened took their daughter's innocence away and they felt like they felt let down by the authorities because of how long it took to resolve the case and how challenging it was to prosecute it because, through the eyes of the parents and
their daughters, this was a crime. They didn't care about the nuances of the law or the fact that the law hadn't caught up with the deep fake technology. What they cared about was the fact that the prime suspect was in their community and no one was doing anything
about it. So, in terms of the impacts to the young woman nightmares, one of them lost over twenty pounds from stress, A number of them started carrying pepper spray in their handbags, two of them started carrying knives in their handbags, and they still feel a sense of trauma from what happened.
When we're back why it's difficult for investigators to charge a ledged perpetrators with a crime for creating and posting deep bake images. Olivia, You've mentioned how hard it was for police and these young women to actually figure out who was behind posting the deep fakes. How long did they suspect the person who was behind it and how did they sort of figure that out?
Well, the young woman involved felt like they knew who was behind it very shortly after discovering the website. Within days, they had a suspect in their mind, and that was Patrick Kerrey, a nineteen year old man who had attended high school with them and was actually a friend of a lot of the young women who were targeted. The problem was there was no real evidence connecting Patrick Kerrey
to the website or to these posts. When he would upload the photos, he would do so anonymously from usernames like Siri, Jeinefeld or tween hunter. He never used his
real name or his real identity. And the challenge from the police perspective, they didn't have enough evidence to get a warrant out to subpoena his IP address to really prove that he was the man behind this campaign, and it would take a lot of effort from the police's part to really investigate this case because unfortunately, there were no laws that Carrie had broken by posting fake pornographic images of these women.
And Olivia, who was Ana and what did she have to do with finding a break in this case?
So Anna is a really popular cheerleader from General Douglas MacArthur High School. She'd actually known Patrick Carrey from the age of five years old. She lives a stone's throw from his home and they went to the same elementary school together. She grew increasingly frustrated throughout twenty twenty one at the lack of response around the case and also some of the disgusting things that were being posted about her.
There was one example where the harasser had included an image that looked as though she had been murdered, and he had posted it online with her full name and some of her social media information, and she just felt like she'd had enough, So in August of twenty twenty one, she decided that she wanted to investigate who was behind all this and to prove once and for all who
this cyber harasser was. She'd heard rumors and speculation that Patrick Carey was behind it, and she didn't believe that he could do something like this, so she wanted to not only unmask the predator, but she also wanted to clear Carrie's name, and she decided to do this by actually doing a deep dive into every single photo the
poster had uploaded to this website. She studied them. She looked into the background of the images to see if she could find any evidence that could help her identify who this person was, and there was one part taricular photo she saw where he had posted an image from a little girl's bedroom and he was actually wearing little girl's underwear. This told her that he had access to a young girl's bedroom, and she looked in the background of the image and she saw a white dresser with
brown trim. She saw a soft toy animal on the beard, and she decided that she would then search for Patrick Carrey's younger sisters. Having grown up very close to his home in attending elementary school with him, she knew that he had twin sisters, her a number of years younger, and she started looking for them online to see if she could find any social media presence, and she actually found one of his sisters posting TikTok videos from exactly
the same bedroom. It had the same dresser with the same brown trim, the same soft toy animal on the beard, and that's when she realized that a person that she believed was a friend of hers, a classmate, someone whod she's known for almost her entire life, was behind all this. She took that evidence into the police station in late August, and it was September fifth that Patrick carry was arrested.
The lead detective on the case actually went over to Carrie's house, knocked on the door with some of the print outs of this particular website and some of the posts that had been made, and he began reading them out aloud to carry who was at home with his mother, and he admitted that he was behind it all.
Prosecutors say Carrie created fourteen different profiles posting these deep faked images from August twenty nineteen to September of this year. When he was arrested.
He had been charged with low level harassment violations and obscenity, which is a pretty old school charge that was really the prosecutors didn't feel like it was going to stick, so they had ninety days to raise the stakes. And they told me that the case had come through as an on call, which in their department at least, that meant that the police were hoping the prosecutors would upgrade or update or file new charges again to Carrie, because they weren't really sure what to charge him with.
And we should say, you reached out to Patrick Carey for comment and he didn't respond to you. You spoke to the district attorneys in this case. What did they say about what it takes to charge someone for posting deep fake photos.
The two district attorneys that actually handled and investigated this case were really surprised when it landed on their desk and they realized, you know, the depravity of some of the content, just how disgusting some of these posts were, and the fact that there were no real charges that
they could lay against this individual. I remember one of the district attorneys telling me that she was scared that reading through some of Carrie's posts, she was afraid that this behavior could go offline, as in he could commit real crimes in the real world, not just as an anonymous predator online. So she wanted to ensure that they could raise the stakes on the charges that have been filed against Carrie. But the challenge they really.
As there were no charges.
There was no charge that they could turn to that said it was illegal to post fake pornographic content of young women on the Internet, so they had to get creative. They studied the penal code, They read through every page, They looked at every single photo that Carrie had uploaded to this website, studied the background of the images, studied what was in the images, and finally, on October fifth, they found a crack in the case, which is they found one real nude image of a fourteen year old
girl's genitals. That meant that Patrick Carey had posted child sexual abuse material online and that was prosecutable. So the charges were upgraded to see SAM charges.
And see SAM here were referring to child sexual abuse materials. So marguie briefly, what laws are out there now in the US the cover photo sharing, what's protected and what's illegal.
So that currently no federal law that criminalizes the sharing of fake pornographic images in the US. That doesn't mean that someone sharing these kind of images won't be prosecuted because there are varying state laws under kind of revenge porn. There are new bills that are being brought in about deep fake images, but there isn't this strict federal law. So this is why we had these issues with the prosecutors when it comes to children. So child sexual abuse
material that is banned. So there is a federal law
saying that you cannot generate child sexual abuse material. However, it's a narrow law and the problem is that a lot of these ai generations of children are just nude images, which you wouldn't prosecute someone who's posted a picture of a child in the bath or on the beach, and in the same way, there isn't a precedent for prosecuting anyone who generates an image of a new child, So there has to be a very explicit abuse being to picted in the image, which is kind of given a
lot of people credence to just had the misfortune of looking at a lot of forums where people are sharing tips on how to generate these kind of images. You can see in the comments that they feel quite emboldened by this law, so it's still incredibly blurry. A lot of people within law enforcement that I spoke to for the story all said that it feels really mishmash at the moment between state laws and federal laws, and that the law really hasn't kept up with the technology in this area.
At the same time the young women and prosecutors in New York were trying to track down who was posting these deep bakes, there were other people investigating this site who were they.
For a few years actually, there had been a group of people who had had similar experiences as those girls in Levittown, victims who had had their pictures posted or simply just people who had heard of the web site
and were repulsed by it. And what we've found out was one of the key people in investigating the website was Will Wallace, who's a former police officer who's living in New Zealand and had been working for years on trying to uncover some of the users for similar reasons as the levitt Town women.
I think, for me, the craziest part of the reporting process of this story was that moment that we realized this vigilante movement to shut down come On Printed Picks coincided with the investigation that the young woman from Levittown were doing to try and identify their harasser. I remember I was driving back from Leavittown to New York City.
It was late at night, it was raining, and I called Margie and had her on speakerphone, and I was telling her everything I'd learned from an interview, my first interview with Anna, where she described how she actually discovered
Patrick Carey was behind this website. And Margie jumps back in and tells me that there's been a vigilante movement of trolls and hackers in a former police detective from New Zealand, which happens to be where I'm from that had been involved in trying to shut this website down for the past two or three years.
He had been tipped off by former police colleagues that there was a case they couldn't crack a woman who had received years of abuse from people sending her pictures of her photos next to their genitalia. They were sending it to her parents, to her boyfriend that caused the end of her relationship. And after he'd done one case, he just got a kind of obsession with the website because he couldn't believe that it was still up and running.
He was adamant that he could find out who was behind this file site, and he kept trying to alert journalists, he was alerting law enforcement in different parts of the world, and he was just getting nowhere. And so after months and months of frustration, he decided to look into the admin himself to try and put some heat on him, and he decided to publish a blog exposing who was behind it and detailing his investigation into how he uncovered her.
It was. We were both just so shocked and surprised, because in journalism you always hope that the stars are going to a line, and that you can have an intersecting narrative of two different stories happening at the same time, but it never really happens, and it did in this story.
We could explain to readers not only what happened in Leavettown in this one very specific case, but the broader global picture of a group of online vigilantes who were targeting this website, joining forces digitally from around the world to shut it down.
And you spoke to Will Wallace about the apparent administrator of the site, scatran Costa, and what Will did to try to get the sait taken down.
Here's well, it wasn't likely that law enforcement were going to intervene at that. It was pretty much sort of chasing Scott round the Internet and finding out who the providers were and reporting him to the providers. There's so many services at someone's fingertips that it is quite easy, particularly if you have the skills and knowledge, just to
shift to a different provider. That is one of the issues where it's kind of people on the Internet having a go at getting rid of a website, is that you're always going to be doing that, Whereas with law enforcement intervention, they have a lot more levers they can pull. I don't know if it will end. I don't know if the service will ever get taken down, but at least I feel as though I've done my bit, and that's to create more accountability.
After the break, the outcome of the Levittown case and why AI technology keeps out pacing regulation, Olivia. In an interview later on with one of New York's das, you found out that there was more to the story with Carrie's deep fakes. What happened?
Yeah, So this interview was done in the Nassau County Courthouse.
I was talking to both of the district attorneys who had been involved and investigated this case, Kelsey Laura and Melissa Scannell, and we were talking through their investigative process and how they came to increase the charges against Carrie, how they investigated what really happened, and I recall towards the end of the interview, I asked them if they'd ever contacted the administrator or owner of the website or tried to get the girl's photos removed, and they told
me that after his conviction, there was one final task they wanted to do, which was to reach out to the person behind this horrific website and ask them to remove all photos and all posts that Carrie had ever made associated with his usernames. They were going to do that by sending the administrator Carrie's certificate of conviction and saying these hosts resulted in criminal conduct, please remove them all. But when they went to do that, the website was offline.
So just days earlier, William Wallace and the Vigilantes mission had been successful when this website had gone dark, so the district attorneys felt like there wasn't really any more that they needed to do. The photos had been removed, the posts were no longer there. But during that interview I informed them that actually this website had moved to a different domain, a different URL, and all of the posts of the Levittown Girls were back up online and visible.
And I remember Melissagists her jaw dropped, and she was so angry and also shocked and confused at the same time. And she was speaking very eloquently and calmly throughout the interview, and she just stopped and said, what what do you mean They're still there?
And Marguie, did anyone associated with running any of these sites respond to you when you can't them.
They did after a long while. We did a lot to try and contact Scott Trent Coster. It took a few months and eventually they responded to a kind of long email where we asked a lot of questions and laid out the story and that we were going to name them, and they called me a weirdo, which is one of the strangest responses I've ever had. They pointed out that it was very strange for me to want to name anyone and that I should not be focusing
my time on exposing someone's identity. We know that the admin has spoken to some of the vigilantes before and they have had similar responses, very odd responses like why are you trying to ruin my business for me? You should just leave us alone? What's wrong with us having our kinks?
And Olivia, where is Patrick carry now?
He was sentenced to six months in prison, ten years probation in lifetime sentence as a sex offender. This means he can't own a smartphone, he can't own a device with a camera, He can't live within a thousand feet or be within a thousand feet of a school or a playground or a place where children are, and surprisingly, in August I found out he'd actually been released after four months with time off for good behavior, and he's back in Livetown. One of the young women who open
our story Cecilia Luce. She was driving along the street and she saw him walking down one of Levittown's main drags. He had on a brown hoodie, he had headphones in, and she started immediately crying. She was so scared to see him, And then as she drove on, she felt anger, and she decided to stop and turn her car round and go back. She wanted to confront him, she wanted to yell at him. She had been a close friend
of Patrick Carrey's throughout high school. So she did a U turn, turned to her car, back around, and he had disappeared. She carried on driving for a number of minutes, looking down all the side streets to see where he would have gone, but she never saw him again. At is sentencing, one of the young women I talked to actually read out a victim impact statement. Her name is Kayla Michelle, and she was the only young woman who
testified at his sentencing. She described attending every single hearing, relating to this case. I think for a few reasons. One of them is because she'd known Carrie since she was eight years old, she thought he was a friend of hers. And another reason is that she was just so angry at him and she wanted to ensure that
justice was going to be served. So during my interview with Kayla, who is now twenty three and still lives in Leavertown, we talked about the moment that she actually had the courage and bravery to read out this victim impact statement in court, and she described to me how nervous she was, that she'd written down what she wanted to say and the piece of paper was shaking in her hands, that she had to take three deep breaths before she was strong enough to talk, and that she
looked directly at Patrick Carry in court and he refused to make eye contact with her. I reached out to one of the court reporters who were in court that day in order to get a full transcript of what Kayla said, and this was what was read out in court. Kayla said, I'm the strongest person in the room because I'm looking you directly in the face to tell you that you discussed me. You hurt me, but you've also
changed me. You had the audacity to talk to me through social media, joke with me, and try to be cordial with me while behind my back belittling me, putting me down, sexualizing my younger self and body. You completely change the way that I viewed myself and my body, and for that, I'll never forgive you. Your name will forever give me nightmares and haunt me. I hear your name and I feel sick.
Marguie, What is needed legally to prevent the further spread of these sites and AI generated photos?
That is just such a hard question. You know, we're talking about this story. This isn't the first, and I'm it's the first we're hearing of, but I think there'll be many similar stories like this because the law is tricky and we're really catching up with what technology is able to do. And also there is an issue with who is liable who is accountable for this, So it's not just the users of the technology who are generating
these images. They then have the ability to post them online, and then the owners of these websites aren't really liable under Section two thirty, which means that as a a website or blog. If a user is posting something that's criminal, they're not going to be held liable for it, but they're enabling this whole forum for thousands of images to be uploaded.
I think that the reason why their question is so hard goes back to their age old Silicon Valley mantra of move fast and break things, where this technology was rolled out before legislators, regulators, the public really understood its power. So now as we're seeing states trying to catch up to these new tools, they're creating new laws that As Maggie and I reported, we looked across the United States and saw all the different portions of legislation that have
been created to try and target deep fakes. But we've got a patchwork across the country now. Some states have civil statutes, some have criminal statutes, Some only target election related deep fake content, which is a subject we haven't even had time to address on today's podcast. But other statutes are just amendments to existing revenge porn laws. So no one's really figured out the right way to solve
this problem yet. In October, we saw President Biden announce a new executive order to target artificial intelligence and ensure that it was illegal that you couldn't use these tools to create child sexual abuse imagery, child porn, or intimate imagery of individuals doing things that they never really did.
And we don't.
Actually know when these are going to be implemented or how they are going to be implemented. You know, that's the big question right now.
Olivia Marghi, thank you so much for joining.
Us, Thanks for having us, Thank you so much.
Thanks for listening to us here at The Big Take. It's a daily podcast from Bloomberg and iHeartRadio. For more shows from iHeartRadio, visit the iHeartRadio app, Apple Podcasts, Bloomberg CarPlay, or wherever you listen. And we'd love to hear from you. Email us with questions or comments to Big Take at Bloomberg dot net. Our supervising producer is Vicky Vergalina. Our senior producer is Katherine Fink. Our producers are Michael Falerro and Moe Barrow. Raphael I'm see Lee is our engineer.
Our original music is by Leo Sidron. I'm Nancy Cook. We'll be back tomorrow with another Big Take
The Dot Bun, the Don Staple Bar