Would you sext ChatGPT? (feat. Deb Donig)

Today's episode is brought to you by Mauerbytes Premium Security. Protect your computers and devices from cyber attacks, 24-7, with Mauerbytes Premium Security. And go to mauerbytes.com slash lock and code for an exclusive offer. That's mauerbytes.com.

slash lock and code intimacy typically requires vulnerability and risk and genuine mutual presence. A chatbot offers, we might call it intimacy's performance without any of its substance. So you get all of the linguistic markers of connection. but no possibility for, for example, rejection. That's part of the human experience of a relationship. This is Lock and Code. a Mauerbytes podcast. I'm your host, David Reese. Our main story today is about how...

porn could be coming to ChatGPT. And, well, we have to talk about that. On October 14th, OpenAI's CEO Sam Altman said that ChatGPT was loosening up. Quote, we made ChatGBT pretty restrictive. to make sure we were being careful with mental health issues. We realized this made it less useful, enjoyable to many users who had no mental health problems. But given the seriousness of the issue, we wanted to get this right. End quote. It's difficult to know exactly what...

Altman means when he speaks about serious mental health issues because he is careful to not name, for instance, that he and his company were sued in August for the alleged role that ChatGPT played in the suicide of a 16 year old boy. According to chat logs disclosed in the lawsuit, the teenager spoke openly to the AI chatbot about suicide, he shared that he wanted to leave a noose in his room, and he even reportedly received an offer to help right a suicide.

note when the teenager told chat gpt that he did not want his parents to feel like they did something wrong chat gpt allegedly replied quote that doesn't mean you owe them survival You don't owe anyone that." On the same day that OpenAI was sued, it published a blog post titled, Helping People When They Need It Most. In the post, the company explained what it can do better.

including stronger safeguards for long conversations, better recognition of people in crisis, and easier access to outside emergency services and even friends and family. And I get it. At this point, you are asking what...

this has to do with porn and here's the interesting part we are asking the exact same thing because In the same message that Altman wrote about lowering the restrictions on ChatGPT that were in place because of mental health issues, he explained what those lowered restrictions could allow. Quote, if you want your ChatGPT to respond in a very human-like way, or use a ton of emoji, or act like a friend, ChatGPT should do it. But only if you want it, not because we are usage maxing.

In December, as we roll out age-gating more fully and as part of our treat adult users like adults principle, we will allow even more, like erotica for verified adults. End quote. erotica entails is unclear, but one could safely assume it involves all the capabilities currently present in chat GPT. Through generative chat, of course, but also image generation.

Today, to help us understand the ethics of AI erotica, who holds responsibility for abuses of the technology, and what we can learn from current, similar products, we're speaking with Deb Donig. on faculty at the UC Berkeley School of Information. Deb, welcome to the show. Hi, David. It's so great to be here. I'm really looking forward to the conversation.

we are very excited to have you here i am excited to talk about this um i am very excited to talk about what we would call the ethics of technology which seems like an extraordinarily broad thing and i just kind of assume tech ethics is more than good or bad or like right or wrong right and so I am again I think very lucky to be able to have a chance to explore this so we're gonna just jump right in here

I wanted to ask, when you think about OpenAI's plans to potentially create an AI-powered sex chatbot, maybe, what are your immediate ethical concerns around the technology? Sure. So I want to get into this with you with some depth. But before I do that, I just want to explain for folks who listened to the introduction, but may still be unclear as to the status of.

AI chatbots released by ChatGPT or OpenAI or Microsoft, that this is still what we call vaporware. Now, folks who may not be familiar with the term vaporware may find this explanation a little bit useful. You might actually be... familiar with the term beta something in beta when we say that something's in beta we mean that a few select users are testing it to see whether or not it has bugs to identify particular vulnerabilities or to try generally to see how this

software works vaporware is like the level under beta it's software or hardware that hasn't been released yet it's been advertised it's not available to buy either because it's only a concept or because it's still being written or designed. So it's really important to understand that this is not something that you can download on your phone or your desktop right now. So the question is, why did Sam Altman make this?

claim and I think that when we start to look a little bit further under the hood we can see a couple of important things first of all again this is only exploratory but it does amount to what would be a major shift in policy on the side of open AI. It marks a move away from banning or restricting. in significant ways, erotic content, and toward allowing, when safe and adult verified, OpenAI to, as it claims, treat adult users like adults.

And this goes along with something else that Sam Altman has also said in the same statement, which is we are not the elected moral police of the world. Now, this is a very interesting statement, and I hope that we can get into this a little bit later.

Because companies have really kind of pivoted and swerved around whether or not they are or not the moral police of the world companies in the context of big tech and social media platforms most notably have said we just care about free speech we are on the side of free speech a position that is very convenient if you don't want to hire

example, content moderators, or to get into the sticky business of thinking ethically about what constitutes speech that may prompt users to behave certain ways in the world.

concerns me the most point number one before we even get into the object itself of erotic chatbots is what this message actually signals because if i as i said before this is vaporware this isn't an actual product that's out there yet what it is essentially is a test balloon A test balloon is a statement that a company or a public person puts out to test to see how the public responds to a certain idea.

It may also be a test balloon to see how regulators or politicians respond to this. It may also be a test balloon to see whether or not there's a market for this. So I think before we get into the actual object and what concerns me about the object.

What concerns me is the idea that this is a test balloon out there to test regulatory and public response to such a product and the implications of that in terms of how it turns our conversation toward the conceptualization and realization of this product ultimately, as well as away from thinking about what...

people are actually going to do this with this product and what safeguards we actually need in terms of policy. Because if it's only an idea, a test balloon, then we're not yet in the realm of actually thinking about what in reality may be needed. order to create structures that would safely serve as a repository for such a thing. I wanted to ask two questions here. Are we right now by...

Doing this podcast, are we engaging in the test balloon? And two, are test balloons common? Like, are they a common way to, I think, what sounds like skirt? the responsibility of thinking about a problem in a really difficult way is it a way to i think that's the only way i can put it is a way to say like oh well we didn't we weren't really going to put any time into that that was just on a lark

Yeah. So are we engaging in it right now? And is it sort of like a defense mechanism? I think that probably it is part of it. Look, we are all right now part of a large research and development program to try and think about what the flavor of market reception.

market response would be and so absolutely i wouldn't discount that now not all test balloons are bad we have uh category of things that I might bracket under thought balloons that include science fiction and science fiction is a really, I think, healthy way. to think about how you might materialize and conceptualize a vision of what might make a society better, which is usually the inception of a certain product, and then to try it out in a conceptual environment to try and see what that...

would play out as if that idea were somehow realized in a material form as in a technological product. So we see, for example, Black Mirror. I think that that's a wonderful test balloon for many important possible products. In this case, the test balloon seems to me to be tied specifically to another important social and financial condition in the context of AI, which is that...

AI is, it turns out, having a very difficult time being identified as a problem solver that is also financially profitable for many of these companies. So companies are sinking. billions of dollars into a product that seems sometimes like it's a solution in search of a problem and what a test balloon can very effectively do is see and test out what the market is for this and so we can think about the

statements by Altman, and I'm not in his mind, and I don't sit on the board of OpenAI. I am not at Microsoft. So I disclaim anything other than a hypothesis as the trajectory of this comment. But one hypothesis could be that There is a real question about how to make this product profitable. And in the pursuit of profit, this might be seen as a potential.

solution to the problem of how to make AI profitable when the idea that AI was going to actually solve problems has turned out to be less profitable than people would have liked. I'm glad you mentioned that, this idea that AI is a solution looking for problems. Because I remember just a few years ago, we were being told that these were the things that were going to usher in rapid advancements in medicine. We were going to... cure cancer and this feels like a far fall from that and

That's upsetting in its own way, and I just think it's important to note that. I think it's important to note the promises we've been given over time with this now very popular technology. And I wanted to kind of revisit, though, the question of... Even knowing it's a test balloon, is this something that we've thought about before? Like AI-powered erotica? It's a great question.

I will also just say to address the first part of your comment before answering that question, that I do strongly believe in the power of AI. to do things like solve cancer yeah the problem is that some of those great things that ai could do turn out to be less profitable than some of the other things that are less important to solving the grave problems that that we

want to, and I think aspire to as a society to solve. So look, when you ask the question of whether or not these things have been thought about before, I just want to invite a little bit of history here because I'm a professor of English literature by training. I do have a strong understanding of the history of the reception of the novel.

And I taught a class on guilty pleasures and the novel for many years. And it turns out that when the novel was first published, people were scandalized. They said, this object tells these wild stories that are so... vivid that people are going to start having relationships with characters and novels. They're going to completely lose time. reality because they're going to get wrapped up in these worlds of fiction so i just want to establish that this conversation that we're having right now

Didn't start now. Version 1.0 maybe was the novel. Version 2.0 was probably television. Version 3.0 was video games, right? We've seen this movie before. We know what this conversation can look like. I don't want to dismiss the legitimate concerns that come up when you add in the variable of AI that interacts with human beings in a novel way, that the novel...

may not, nor does television, in a kind of dynamic interactivity that takes these things to objectively another level. But I do want to establish that there is some history of hysteria or outcry. over these objects and it's also important to contextualize that not just so that we can dismiss them as i think some people in tech might want to do with that information but also so that we can fundamentally identify what is different about

this and why those differences should be relevant and concerning. One, what a wonderful thing to know. Like, I didn't know that. And that's something that I will retain for the rest of my life because it's such a fun piece of history. And two, like you said here, it's important to make distinctions. It's important to understand why something is different. And it does kind of get into this question I wanted to ask where I don't want to cast judgment.

on people who are interested in this type of service, right? I don't think it really gets us anywhere. And at the same time, when I hear about this development, that OpenAI is, you know, potentially interested in releasing erotica... It feels weird to me. It feels very strange to me. And I do think about this differentiator of interaction, right? The chatbot can speak back to me. Perhaps there's a world...

even when the chatbot gives me notifications. You know, the app on my phone says, you've got a notification. It's almost like a text. And so this is a long way of asking, is the differentiator here, the interactivity, is that why...

I feel weird about this. What is happening here that is markedly different? Yeah, well, there's a couple of things that are markedly different. The first... markedly different thing is that when you had people writing novels, the novel that was ultimately distributed was written by one person and then went out and was read by maybe

a thousand people. And those thousand people served as arbiters of whether or not you were going to publish more of that object. Now we have a massive company with global reach. emitting a product that is suddenly usable to millions and millions of people across a large global spread. And so what that means is the reception for this and the impact is much more severe than anything we would have seen in the 18th century.

the second part of this is that the global distribution makes the readership for this so to speak or the people engaging with this object a much more difficult group to codify and so when you're talking about one product that's going to go out to millions of people across multiple different cultural contexts you have a really difficult Way too.

understand how those people are going to interact with that product, what's going to happen to it, and a very difficult ability to manage difficulties or problems or alarming situations that come up. the individual user and the issues that come up or why we feel strange about this, it's not just that we're interacting with a chat product. Many of us have normalized that at this point. It's that

There's a level of intimacy insinuated in the creation of this particular dimension of the product that we think about as somewhat unique to human interaction for a couple of reasons. The first is that intimacy typically requires vulnerability and risk and genuine mutual presence. A chatbot offers, we might call it intimacy's performance without any of its substance. So you get all of the linguistic markers of connection.

But no possibility for, for example, rejection. That's part of the human experience of a relationship. Many of these products are developed specifically toward the ends of what's called sycophancy, meaning that they're going to yes you all the time. That is a very different thing than a human relationship.

There's also the control paradox. And without getting into the specific dynamics of sexual interaction between people where power play is very much a part of those kinds of interactions in a number of different circumstances. Part of what intimacy means comes from the limits to which we can control another person. Again, people play with those limits indefinitely and in all sorts of fantastical ways in the context of sex.

There is always that tension around control in that way. Intimacy comes from negotiation, from being surprised, and of course, from being sometimes rejected. And a sex chatbot is designed to never reject you. to always respond perfectly to your desires and this removes the very unpredictability and mutual vulnerability

that makes human intimacy meaningful in the first place. And essentially what we're doing, I think, in this product, or what OpenAI is doing, is automating away the aspects of sexuality that make it relational rather than merely transactional. I feel every answer you're giving that I want to jump into. And there's so much to reflect on, particularly that definition, that understanding of intimacy and control.

really wonderful stuff but i did want to move on to the topic here of whether we have seen out in the world any examples today right like actual things that are products that exist of ai powered sex chatbots and and if there are examples what do those things teach us about what might happen with OpenAI's erotica plans? Yeah, the most prominent example is probably Replica, and it has a very telling trajectory. So Replica launches a general AI companion first, but...

then subsequently evolved to allow romantic and sexual relationships. And what we learned was a bit troubling. And by the way, the company shut it down in February of 2023. There's a couple of things that happened that I think are demonstrative of where this could go again. The first is that users developed

genuine emotional connections to their replica companions. And when the company tried to remove the erotic role play feature in 2023, before shutting it down, users experienced something akin to grief and loss. And we already know that. We form relationships with these objects. We tend to think of things that perform. what we think of as human speech in particular, as having human minds behind them. So it's really hard for us to conceptualize of human activity such as speech directed.

in response to things that we've said, as not having a mind behind it. We project the reality or the, you might say, we project the hologram of a mind in the absence of one. And we see that and users in. The context of Replica, when that feature was taken away in 2023, reported the change as more traumatic than human breakups because they had shared intimate feelings with AI that they hadn't shared with people. And I want to just take a second to share a story from my own experience.

demonstrates one important difference between how we interact with chatbots and how we interact with human beings. And I think that the story will make this obvious. to folks. When I was teaching at UCLA, I was teaching a class on Holocaust memory and Prior to that year, I had always had Holocaust survivors come in to speak to my students. And that particular year, we had reached a point, as people who are concerned about Holocaust memory have reached, that survivors are...

dying off. There are fewer of them around to tell their stories. And that year I had connected with a project run by the Shoah Foundation at UCLA that had created

hologram Holocaust survivors. They'd spent thousands of hours interviewing these survivors about their experiences and had put together a technologically facilitated hologram that would allow the... hologram holocaust survivor to answer a permutation wide permutation of different questions about their experience now i just want to make a distinction they were not fictionalizing

their experiences. They were not creating new experiences. They were answering new questions about their experiences from current interlocutors. in the absence of those Holocaust survivors being able to come in in person. And the interactions were very human-like. But the questions that students asked the Holocaust survivor holograms were markedly different than... Students had asked the Holocaust survivors when the Holocaust survivors had stood before them.

In fact, the wide variation included a large number of questions that I might consider shameless or more, I think, sympathetically questions that people generally have, but would be a little bit too. nervous or maybe would understand that there was some social inappropriateness of asking that question. It's not just that people would engage, I think, with these chatbots with the kind of intimacy that we do with intimate relationships.

I think that it may actually amplify the extent to which people develop not only genuine emotional connection to these chatbots, but... people who are already potentially lonely or socially anxious or have relationship difficulty. being even more willing to share their most intimate feelings with AI. And this is, by the way, the grief that the people who had had Replica taking away from them felt. Not only the loss of intimacy with an interlocutor who felt...

intimate, but an interlocutor who is able to, in their view, receive a higher degree of intimacy. I have a romantic partner and I feel pretty comfortable with that romantic partner, but there are certain things that I keep to myself. Or that at least I have some inhibitions around sharing. But I think in a context where we... On a certain level, no, we're not interacting with humans. Some of those inhibitions get left alone. And this is not only dangerous on the relational level.

This is not only a concern about what this might do psychologically to people. It's also a concern about things like data privacy. We've already had multiple situations where people who have subscribed to websites that have had porn on them or... become a member of a social media app set up for people with kinks have had their data exploited or had their data breached.

And so I think that there is a real issue about what happens to that data and what happens with a breach and how identities can get exposed and how that kind of intimacy, once exposed through something like a data breach, could be disastrous for somebody, right? about somebody in a marriage who suddenly has this information out there or think about somebody.

who has this information about the most intimate parts of their lives, more intimate than they would have spent anywhere, including in a pornographic site because they're actually contributing to the conversation. What happens if that gets out there and it's available?

for their employer. And so I think that those are thoughts that we may want to consider as well. And certainly for people who are considering engaging in this, the question of what happens to your data and what could happen with a breach should be in the back of your mind. The AI recreations, holograms of Holocaust survivors is, I think, something a lot of folks have never heard.

had read about it i know that it was in use actually at a museum here in america i don't know if it's still in use and i learned a lot about no simulation being a rule right this idea that you shouldn't simulate you shouldn't do like a virtual reality experience right of what it's like to be in a concentration camp or a death camp and this idea here of creating holograms creating likenesses that

live past the Holocaust survivor's life, it raised a lot of questions. Like, is this simulation? And again, I feel like every time we talk about something here, we should just... sit a little bit with this wild thing that is happening this is just something that i think folks should know about and it's really it's weird to me and like you said it led to

crass questions or questions that people have but they feel like they shouldn't ask because there is that level of inhibition you know there is that level of like i can't ask this to a person you know that's that's kind of messed up i well i wanted to ask kind of separately here

On the idea of doing things that you wouldn't typically do, I want to ask, you know, let's say ChatGPT does release erotica, right? And let's say that someone is able to trick ChatGPT into... either generating an image or even recreating like the writing style of a person, you know, without their consent. This feels like a misuse, you know, of a tool. And I wanted to ask.

Who bears responsibility for that abuse, right? Is it the user? Is it open AI for allowing the behavior? Is it both? And that's the question there. Yeah, who bears the responsibility in that kind of use case? Yeah, I will go back to the comment I made previously, a direct quote from Sam Altman.

who said, we are not the moral police of the world, nor do we want to be. It's a kind of funny statement from Altman. And I just want to dig into that for a second before I get into a kind of more legalistic. response to your question, which I think is also warranted. And I'll blend that with some philosophy too, so that I don't get too stuck in jargon. But I do first want to note...

that there's one, I think, important definition of morality versus legality that I would surface here. And it's Gandhi's definition. And what Gandhi calls law is what he says is codified. ethics. Codified ethics is law. And so what ostensibly that means is that we come to some sort of social consensus about what we're going to permit, and then we get regulation to codify that into law.

Now, I say that Altman statement is funny because my belief, and I think that there's a lot of facts out there to substantiate that, although I can't point to the numbers specifically, is that...

OpenAI and Microsoft spends a lot of money lobbying against regulations that it doesn't like so that it can express whatever morality it... prefers in the social environment and so if you want to say that you're not the social police then you have to kind of say all right we're going to allow the democratic process to determine what kinds of ethics we codify. But as soon as you start lobbying politicians and intervening into government procedure in order to get your particular blend of...

profit incentives encoded into law or to prohibit regulation that might interfere with those things, then you're expressing a morality. And that morality might be simply, we're going to do the thing that... builds us the most profitable business but that's still a morality and so i want to point to that very specifically and it leads into the answer that i want to give around who should bear account

for this because from my view what the user does with a tool and then there's a tool that's created and oftentimes people who are creating those tools will want to say something like the tool itself is neutral we just have bad users and to me that sounds a little bit like the phrase guns don't kill people people kill people because the design of the gun is a design that induces certain outcomes and prohibits other outcomes nobody's going to cure cancer with a gun

A gun is an object that's pointed toward another object in order to affect maximum forms of material. destruction of that object now you can use that object to stop a terrorist and you can use that object to be a terrorist so i grant there's some dimension as to you know bad actors but nobody as i said is going to ever

cure somebody of a disease with a gun. And so the tools that we make open up certain outcomes and they prohibit certain outcomes. And of course, people are going to use tools regardless of how thoughtfully they're designed in ways that they're not designed toward. That is not the claim that I'm making, that every tool should be perfectly designed so that no one ever could use it in a way that could ever, ever, ever cause harm. But nobody's ever going to use a spoon to stab somebody.

okay in a way that is effective they're going to use a knife to do it and so we have to think about what the design of the tool incentivizes people to do and what the likely outcomes are and so in the context where a bad actor uses this tool we have to think about whether or not

that behavior is incentivized through this tool and whether it is a foreseeable outcome of using this tool. And that connects us with the legal definition. Now, in legal context, there's an idea of shared responsibility for negligence. What we have here, I think, is a shared responsibility between the user and the creator of this tool. But there's an asymmetric power to it as well. The user who creates non-consensual imagery, for example, using this tool, is committing the immediate...

violation and bears direct moral and legal responsibility. But OpenAI bears responsibility for creating the conditions of that possibility to exist in the beginning that then facilitated this harm. And they have built... the affordances that enable it. And so I think we have to think about how we want to structure ethical accountability and how we want to make sure that that gets encoded in legal.

accountability as well. And I can talk more about whether or not I think that the best form of accountability is left to the courts to adjudicate things like lawsuits that demonstrate or at least make the case for actual. cases of harms. But that's, I think, the beginning of the conversation is around shared responsibility and the idea that companies cannot negate their moral responsibility.

nor should I think they should be able to negate their legal responsibility by saying that their tool itself was neutral. It sounds like there's a lot to talk about here in terms of structures that put into place ethical accountability or structures that allow for it in a fair sense and

I did want to ask, like you said, like, hey, I could talk about whether I think courts are that system. I did want to ask, like, what do you think is an ideal system? And do the courts play a role in that? Or do we have an example of a system where, oh, yeah. That worked. That absolutely worked. And we should bring that back or it works in a different region or community because this is fun.

Yeah, we do have examples of how this works. For example, the entire field of bioethics is an example of how this can work effectively. Not perfectly, but effectively. There's a little bit of history to that. Bioethics as a field emerged after the second.

World War. And it was very conscious of two particular harms. The first was the Tuskegee experiments that happened in the United States, where Black men were subjected to syphilis in order for scientists in the United States to determine what...

countermeasures or what kinds of cures could exist right obviously unethical using human test subjects in that way also the particular feature of discrimination against a group that has been historically marginalized by the scientific and medical community and also historically exploited.

right for the labor of their bodies the second of course is the nazi experiments where scientists proceeded with all forms of experiments that we would consider in our culture today, grotesque on the basis of denigrating the group of people, Jews primarily, who they were testing on as undeserving of the kind of protections that we might warrant to.

the humans and then proceeding along the line of thinking very common in technological and scientific communities that harm caused two people in the name of scientific or technological progress is justified because it...

facilitates or results in scientific or technological progress. And as a result of those fallacies and as a result of the kind of horror of what we... saw in those two contexts, we came up with the field of bioethics and very specific practices and procedures about how to protect specifically human subjects in the context of these experiments and philosophically how we want to address this field.

The outcome of this is that we have very rigorous testing mechanisms for any kind of bio agent or any kind of, for example, vaccine that emerges or any drug that emerges on the market. If you want to put your vaccine. on the market, or if you want to put your drug on the market, it has to go through rigorous testing before it is deemed safe enough to distribute.

Digital technology space, particularly in big tech, has not only pushed back against the institution of any form of these protections, but has claimed that they are, again, not morally engaged in those questions to begin with. look at successful examples, we have them. We have them historically documented and we have them as set up, again, imperfect but operational institutions that provide

important protections. And we could do this in this arena as well. We could do this in ways that allow for the testing and demonstration of safety before these products get distributed to the public and thresholds of what they would need to meet in order. order to be considered safe. And that has a historical precedent. These examples are extremely relevant, I think, to what I was going to ask next, because I did...

I think it's very bizarre, right, that OpenAI announced, you know, kind of as an offhand remark, hey, we might allow erotica. And they announced this in connection, like I said at the intro, with these lowered restrictions about mental health. And as I said, you know, in August, OpenAI was sued for ChachiBT's alleged role in a teenager's suicide. And when that lawsuit was filed, the teenager's mom actually told NBC News that OpenAI, quote, wanted to get the product out.

And they knew that there could be damages, that mistakes would happen, but they felt like the stakes were low. So my son is a low stake, end quote. And it reminds me of what you're saying here, that there is no rigorous testing. in the tech industry and there is not this kind of ethical accountability model within the tech industry and i just wanted to open that up to you because

When I see that there are these products that have such an enormous impact, it does feel sometimes like only the product's maker gets to decide what harm is and isn't acceptable. And I guess my question is... Should it change? It feels clear, but yeah, that's my question. To listeners, the TLDR version of that is yes, but here's the more complicated, I think, nuanced answer. We have a phrase in popular culture for this kind of behavior and this kind of philosophy. It is...

the phrase mistakes were made for listeners who are coming in a generation below me. I'm of the generation, you know, that was around the back when dinosaurs were roaming the earth. The phrase mistakes were made was used by both George H.W. Bush as vice president and George W. Bush subsequently in reference to scandals and controversial decisions. So mistakes were made.

By whom, right? By whom? You can just say mistakes were made without ever having to cop up to the fact that they were made by you. And so in this structure... big tech in particular has made it very clear that their vision of themselves is as neutral arbiters or creators of technologies that are morally neutral in themselves and who's to say what are, what people are going to do with them. I've talked about that already, but to draw this really to, I think a finer point, the.

Phrase itself that Altman used was that they wanted the situation to be tested out in a low stakes environment. And the mother says back to that, well, I guess my son was a low stake. And this, I think, captures something important about the tech industry, that organizations systemically marginalize ethical concerns that conflict with growth imperatives.

And I think in particular, what we can see is a power asymmetry in design and in consequence. You know, here's somebody's son kind of sacrificing the name of a profit margin. And in the context of asymmetry, we can also think about who gets to decide what technologies get made and how they get made and how they get designed. Well, it's a really kind of small elite group of people who make those decisions.

OpenAI's developers will encode their values about sexuality and consent and relationships into whatever product they test balloon out there, or ultimately beta, I suppose, out there. But the developers don't represent. the diversity of human sexual experience or the cultural norms about intimacy or what people are ultimately going to do with this and frankly if i were inside the company i might say

Let's not do this or let's not do this yet. But if I were inside the company, then what I would hope that somebody has said is let's think about our design choices here and let's think about. what we are creating and how this is going to be used by actual people and what kinds of responsibility we are willing to accept in order for that to happen. When you say that you are...

morally neutral or you are not the moral police, you distance yourself from having to take responsibility for the things that people do with your product in the world. I think that there is, again, to get back to that first statement, mistakes were made. I wonder what would change if the people making those decisions didn't think about themselves as engineers making moral choices, but somebody who caused somebody to kill themselves. What would change, if anything?

I hope that something would change. I wanted to close here on just a really broad question. Like you said, you are interested in the power of AI and the power of AI, particularly in medicine. I did want to ask what your personal ideal AI future looks like and whether or not AI sex chatbots play a role in that. It feels like a small thing, you know, the more that this conversation goes on.

I feel like it might even be like, well, they are, they aren't, you know, but the question still stands, you know, what is your ideal AI future? And I think help me really understand what like fairness looks like in that future. Yeah, it's a good question. I'll take the first part first, which is what the ideal future of AI could or should look like.

As I've said before, some people look at the work that I do and they say, well, she just doesn't like technology or she's just suspicious of technology. There's a term for that, by the way. It's called a Luddite. I don't consider myself a Luddite. I love technology, right? I use clod primarily. plug for Anthropic, which does abide by specific guidelines and ethical forms of thought, at least currently. And I use it in everything. And if I were to be able to design something, I would say that...

The first step is redesigning how companies make decisions. I started off spending a lot of time when I started researching the ethics of technology by looking at the products themselves. And that's what we've primarily done today. But by the end of the research, I was studying org charts and I was studying financial documents because the system of...

The market that we have set up incentivizes certain forms of development, certain paces of development, and certain products as the outcome of development. If we want to see cancer research from AI... right now in our system proliferate, we would have to make a case about how not only this would be profitable, but how it would be profitable in a venture environment where those. outcomes and those kinds of innovations have to meet a certain timeline and a certain level of return.

That does not often incentivize research that for example looks at niche diseases. That does not incentivize research that has a horizon for success projected 40 years in the future. That incentivizes innovation that's going to solve problems that have a very low cost to solve and yield high return. So for example, the problem of how to send out more effective customer emailing.

right that's a problem that is going to get solved because it meets that profit threshold in the structure that we have set up so the first change that really needs to happen is in the level of the financial structures and the financial incentives. And there are ways to do this, by the way. You can make a financial incentive in a company. How well will this company do not two years from now, but 30 years from now?

you can make as part of your company success horizon, how sustainable is this product and how likely is it to be able to over time, be able to meet energy challenges. Over the next 30 years, as energy costs become more expensive because we're using more of it. So you can include those things. There are ways to change this that don't involve regulatory shifts.

the way that our government handles financial dealings and structures of venture capital, but we need those too. The second part of this is really organizational structure. I, after studying the... market of ethical, responsible and public interest technology work, and thinking about how it's facilitated within a company, see that over and over again, harm emerges.

out of particular organizational structures where there are a lot of silos where there is not what i call institutional wisdom built up and where there's a lot of intellectual and perspectival homogeneity, meaning that everybody is an engineer in a particular products development life cycle, and nobody has thought about what this product would be like from another intellectual or methodological.

perspective and so when you have those silos what happens is really interesting and i want to just spend a little bit of time on this because this research i think is paradigmatic cass sunstein i believe uh who's over at harvard law did an experiment where he looked at groups of people who think similarly, like-minded people deliberating a subject where they all kind of share the same precepts and the same axioms and the same intellectual position.

What he thought would happen is what many of us, I think, intuitively think might happen in such deliberation among a like-minded group, which is that people come away after deliberating that subject with a position somewhere in the middle. of wherever the people in the group stand. That's not what happened.

What Sunstein's research shows is that when people who are all thinking similarly get together, they come up with a solution or an outcome to deliberation that is more extreme than any that the individual members came in with. So what happens when we have a bunch of people who are trained by the same schools to think the same way in the same patterns, getting together in an environment is that they are not going to be able to check their blind spots. They're not going to be.

able to present a position from a counter consensual point of view from the perspective of that group. And so you get wider and wider blind spots and fewer and fewer people who are able to see. and object to what could be wrong in that product. My view as a way to ameliorate some of these harms is to create... as i said what i call institutional wisdom organizational diversity across a company and to bring in people who may come from a position to be able to challenge or object to a consensus

And then finally, I think that we need to start thinking differently about technology, right? When we adopted Facebook as a technology that was supposed to, as Mark Zuckerberg put it in his... banner for the product connect to the world we said this is a great tool and again there's a lot of good that this tool has done but the mechanism for connection

was to optimize for the number of connections. Now, anybody who I think is flourishing in life knows that that flourishing in life and flourishing in relationship doesn't come from the fact that you know 2,000 people. It comes from the fact that you have intimate...

meaningful relationships with five and so technology is there to facilitate the connection but not to replace it and so in my ideal future to go back to this product specifically ai could genuinely help address barriers to human intimacy It could serve as a means for people to practice intimacy. It could serve as a means to ask.

Questions that maybe you're embarrassed to ask your partner to try it out, to see what would happen. But I don't think it should replace human experience. And I don't think it could. And so do chatbots play a part? I think that they do. I think that they have a niche application. The question to me sounds a lot like the question that people ask around pornography or prostitution. And I don't think we're getting rid of either of those two institutions in our culture, right?

prostitution is considered to be the oldest profession and pornography is merely a technological simulation you might say of that extending, by the way, back to people like Marquis de Sade, where this is now considered to be kind of intellectual literature. And so I think that the question we fundamentally need to ask about any of these products is not whether or not we should ban it, but...

What can we build? Who should be involved in building it? How do these design choices made by the people who build it impact our society? And the question of

Again, not can we build this, but what should we build? What should we build? The question of can is a technological question. The question of should is a moral question. And sometimes I think the most human choice is deciding that certain experiences... including intimacy, are worth protecting from the kinds of optimizations that our financial systems and our technological systems perform towards or sometimes insist on.

I just wanted to thank you again for coming on today's show. A huge thank you for everything. Oh, it's my pleasure. I'm so grateful for the conversation, for these really thoughtful questions. To our listeners. We'll talk to you again in two weeks. Until then, stay tuned and stay safe. And remember, you can read all our cybersecurity coverage on Malwarebytes Labs at malwarebytes.com slash blog.

Finally, our intro music is by Kevin MacLeod from incompetech.com and our outro music is by Wowa from unminus.com. Today's show has been edited by our podcast consultant, Eric Johnson at lightningpod.fm. Thank you, folks.