From the Vault: Punish the Machine, Part 1

welcome to Stuff to Blow Your Mind. My name is Robert Lamb and I'm Joe McCormick. And right before we started recording today, we were just talking about that iconic scene and returned to the Jedi where the droids are sent to the droid torture Chamber. Do you remember there?

I guess it's not just a droid torture chamber. It's sort of like the uh, the Droid onboarding center right where the you know, R two, D two and C three po have been given as gifts to job of the Hut and they go meet their new like droid boss and he's like you you're a feisty little one and he's signing them in but uh, he sees that R two D two is a is a bad robot who needs discipline, and R two D two is confronted with these images of robots being punished with various corporal punishments,

like one is getting stretched on a robot rack and another one is getting its feet burned. Yes, this is a This is a great scene, one that that definitely burns its way into your your brain as a as a young viewer, and maybe you don't think about it that much for a long time, but uh, it's it's still in there. It's it takes place in the bowels of job as Palace on tattooing and it's um, yeah, it's like Droid intake but also Droid corrections. It's there.

There are a number of different department that I think are converging here and and it ray it ultimately kind of raises some interesting questions about um about ethics and punishment and in crime, and certainly as it relates to two robots. Uh. Of course, one thing, this important distress here is like none of this was really intended in

these scenes. This was about having droids doing things that humans would be doing to each other in other pieces of cinema, certainly, things like old pirate movies or old sin bad movies or what have you. I mean, that's kind of that's kind of Star Wars in a nutshell, right. Uh. This whole portion of Return of the Jedi is essentially a big pirate movie, a big swashbuckler set in an alien location. Oh yeah, the job of the hud as a pirate captain. Yeah. But something interesting occurs when you

replace the humans in these these tropic scenes with machines. Uh, and then you think about it, you know, you think about why is that robot torturing the other as if it makes perfect sense if it's humans doing it, But then the things that we create in our image when they're doing it, suddenly we start seeing the flaws in our reasoning. Suddenly we start questioning, well, how does this whole system supposed to work? Uh? And maybe this whole

system doesn't work. Well. Yeah, there are multiple levels of absurdities in the scene. One is the idea that this robot is just sort of like coolly telling are two, that he is going to learn some discipline. But then the image that accompanies that is like, clearly just like extreme robot torture, Like it's something way beyond what would have to do with with discipline in the real world.

But then the other level of absurdity is that it's robots in the scene, but coming off of the issue of just like barbaric pirate torture directly and more to the broader question of robots and discipline and punishment. Uh, this is something that we actually wanted to talk about today because the issue of robot moral and legal agency is something I've been interested in for a long time.

I've talked about it, It's come up on the show in the past and in briefer ways, um, and today I wanted to come back and devote a full episode to the subject. I guess actually we're gonna be talking about this for a couple of episodes now. The question of as machines AI robots become more independent and act more like agents more like humans do, how are we to understand their moral and legal culpability when they do something that harms people? And is there such a thing

as robot punishment robot discipline? Do these concepts reflect anything that's achievable in the real world and practical, and if so, how would any of this work? Yeah? I think one of the most interesting things about this topic is that it does force us to force a face off between what robots and ai actually are or will be, and how we think about them, indeed, how we anthropomorphize them. Um, And perhaps it might be helpful to take a step back and think about something far less advanced as a

robot and think something. I think about something more like a hammer. Okay, so everyone's heard the old adage that it's a poor carpenter who blames their tools, right, But of course we do this all the time. Uh, the hammer slips, it hits our fingers, and we may, at least in the heat of the moment, blame the hammer for the failure. Now, we may get over this quickly, but then again, we may decide that the hammer truly

is at fault and it should be used less. We might also take this idea to a number of different extremes. We might decide that the hammer is not merely at fault but faulty, and then we're entitled to at least a refund for its purchase. Or we might decide that the hammer needs to actually be punished and and this

of course is ridiculous. And yet the idea of punishing the hammer by say, putting it in the corner, or perhaps you have an old toolbox of shame that's just for the misbehaving tools, or maybe it's it's less thought out and you just throw the hammer across the yard as punishment for what it has done to you. Um. Again, these are ridiculous things to do, but the idea of doing them is not that far and from us. Um, those of you listening, you may have engaged in this

sort of thing as well. You might also simply throw the tool away and otherwise perfectly good tool. Um. I know that I did this once with a knife sharpening gadget that caused me to cut my finger my and my like reaction was this thing has now injured me, it has drawn my blood. Uh, I am, I'm getting rid of it. It goes in the trash. It bore malice against me. Yeah, or you know, I you know ultimately it's I mean, I mean you you get into arguments about different tools, like is this a dangerous tool?

And in that case that was my reasoning. It's like this tool is dangerous, it's not enabling me to do what I want to do without drawing blood, so it goes in the trash um. But then there have been other cases where like I had a mandolin for slicing up carrots, and um, I like nicked my finger on it. But oh I I nicked my fingers A rule they can be. But I nicked my finger not using it, but going into the drawer for something else. So I punished it by putting at the very bottom of the drawer.

But I didn't throw it away. Uh huh. So I think if we all think, think back, you know, we have examples of this sort of thing from our from our life. Well sure, I mean I'm going to talk in this episode about some of the ways that we mindlessly apply social rules to robots. But yeah, I think what you're illustrating here is that you don't even have to get to the robot agency stage before people start

doing that. I mean people mindlessly, to a lesser extent, mindlessly apply social rules and rules derived for managing human

relationships to inanimate objects with no moving parts. Yeah. Yeah, you don't even have to get into a room ba or anything, or you know, you can deal with the hammer, the can opener but we also you know, it's it's also a sad fact that many pat pet owners will punish an animal for a transgression, transgression, but scientific evidence shows that this tends to not actually work at least

in most of the circumstances that it's used. Um so you know, even it's not merely with with tools and inanimate objects, but even non human entities were liable to engage in this kind of discipline based thinking. Now, most of the studies I think that necessarily relate to dogs if I remember correctly, And there's a lot going on here that doesn't relate directly to inanimate objects and robots, but it illustrates how we tend to approach the punishment

of other agents and perceived agents. Well, yeah, there's a disconnect, and this will be highlighted in one of the papers we're going to talk about in in this pair of episodes, But there's a disconnect in that punishment is often logically characterized as serving one type of purpose, but then is applied more like it serves another type of purpose. So like it is logically explained as say a deterrent, right, I mean, if you if you're talking about uh, legal theories,

of punishment. One of the main things that people come up with is say, well, the remedy provided by the law is in order to punish the person who did the bad thing, in order to send a message that people should not do this bad thing and thus maybe discourage other people from doing something similar in the future, or discourage the same person from doing it again. And if it were to actually serve that purpose, it's debatable

in what cases it does actually serve that purpose. Maybe sometimes it does, but that is a you know, you could argue that's a rational, logical thing that prevents harm. But the way punishment is actually often inflicted in the real world seems to be more consistent with judgments based on like emotional satisfaction of the idea of having been wronged. Yeah. And then also we get into this area where, uh,

we have a couple of different factors encouraging traditions of discipline. Um, particularly if we look at a parenthood, which there's some crossover between discipline and parents who I didn't discipline and the criminal justice system. But you know, uh, not everything is going to line up one to one here. But um, on the childhood example, it's benn argue that parents use punishment first of all, because it's an emotional response out

of anger and anger that may be mismanaged. But then on top of this, it's, you know, something that's culturally passed down and punishment may seem to work. I was reading about this in a Psychology Today article by Michael Carson, pH d j D. And uh, this is what they said. Quote, because the child is inhibited in your presence, it's easy to think they would be inhibited in your absence. Punishment produces politeness, not morality. Thus, the inhibited, obedient child inadvertently

reinforces the parents punitive behavior by acting obedient. For the sorts of parents who find obedient children reinforcing, Yeah, that

raises an interesting question. I mean, I've been mainly thinking for this episode about about legal punishments, but like when it comes down to parenting, that's a very different kind of thing because both parenting in the legal system involved punishment, but parenting is not subject to a legal system, right, So there's no there is no systematized way by which

justice is administered from a parent. It's just just I mean, I think a lot of times it's just sort of like whatever the parent can manage to do in the

moment because like the kids driving them crazy or something. Yeah, usually the child can't take it to a higher court, right, But I mean, I think you're absolutely right that whether you're talking about discipline administered by a parent or the justice system as a whole, I'd say that both are probably based more on tradition and philosophy and less on a scientifically rigorous study of the most efficient ways to

reduce harm. And one of the interesting things about thinking about how law could potentially be applied to harm caused by autonomous machines is that it may help give us some insights on ways that the justice system as it exists and is applied to humans today tends to be

of irrationally already, like with respect to humans. Yeah, and again, this is what's so interesting about this this paper, I mean, well, the papers that we're going to discuss this topic in general, though, is if you start, you start comparing machine possibilities to human possibilities, and it's on one level of thought experiment in how you would hold machines are responsible, but then it makes you rethink the way humans are held responsible.

You know. It's like, um, you might think you have a pretty square a way, like if if an adult sells a pack of cigarettes to someone who's underage, right, but then one of a machine does the same thing, how do you treat the machine? Do you treat the machine like an adult? And then in trying to figure out how to treat this machine, does it make you rethink how you should be treating the adult who engaged

in this behavior. I don't know, yeah, And I think a lot of that will come down to our understanding of what the machine is capable of, like what kind of constraints it has, what type of what level of autonomy it seems to be operating at. I mean, again, Weirdly, even when people set out to define clear rules for what makes a machine culpable, there there's still going to

be a lot of subjectivity in it. I'm I'm looking at like legal definitions of what constitutes a robot versus just a machine, and some of these definitions involved things like, well, a robot feels like a social agent. So there's still, like, you know, an element of subjectivity. But I think that's correct in how we actually apply the term most of the time, right, Like something is like a gut feeling

about how this machine is behaving in your world. Is it acting more like a fixed, you know, brainless machine, or is it acting a little bit more like a person. So, while it would be one thing if if it were basically a cigarette vending machine that was selling to children, but if it were a machine that went door to door and and rang the doorbell and then asked for the children so they could sell them cigarettes, that would be a different matter. I mean, yeah, I mean I

think that would require different types of remedies. Probably, Yeah, I mean I think a lot of people would probably look at the cigarette vending machine and say where was the vending machine placed? Why was it in a place that children could have access to it? Rather than attacking the fundamentals of the machine itself, if it's going door to door and giving cigarettes to kids, yeah, then people are probably going to attack the fundamentals and the moral

character of the robot. You have not attacked the robot itself. It would just rob justice in somebody's front yard. Yeah, alright, So I guess I want to introduce one of the papers we're gonna be looking at in this pair of episodes, and it is by Mark A. Limley and Brian casey called Remedies for Robots published in the University of Chicago Law Review in twenty nineteen. And this is a big paper. It's like eighty something pages long with a with a

lot of different interesting, uh thoughts in it. We're not going to be able to cover the entire thing in depth, but it's worth looking up. You can easily find a full PDF of it if you want to read it in depth. And we're gonna look at some of the larger framework it lays out and then some interesting thoughts raised to buy it. But to kick it off here the the author's right quote, what happens when artificially intelligent

robots misbehave? The question is not just hypothetical. As robotics and artificial intelligence systems increasingly integrate into our society, they will do bad things. We seek to explore what remedies the law can and should provide once a robot has caused harm. Now, obviously we're going to be focused less on the like minute particulars of US legal precedent here and more on the broader issues they raise about robot agency, robot moral decision making, and how that interacts with harm

and morality and justice. And the authors start out in their introduction by giving what I think is a really fantastic example of how an autonomous robot with behave of years guided by machine learning, which is how you know, increasingly most robots are going to be controlled, can end up doing things that are the exact opposite of what

was intended. So this case that they site is based on a true story from a presentation at the eleventh annual Stanford e Commerce Best Practices Conference in June, and it goes like this quote. Engineers training and artificially intelligent self flying drone were perplexed. They were trying to get the drone to stay within a predefined circle and head toward its center. Things were going well for a while.

The drone received positive reinforcement for its successful flights, and it was improving its ability to navigate toward the middle quickly and accurately. Then suddenly things changed. When the drone near the edge of the circle, it would inexplicably turn away from the center, leaving the circle. What went wrong? After a long time spent puzzling over the problem, the designers realized that whenever the drone left the circle during tests,

they had turned it off. Someone would then pick it up and carry it back into the circle to start again. From this pattern, the drones algorithm had learned correctly that when it was sufficiently far from the center, the optimal way to get back to the middle was to simply

leave it all together. As far as the drone was concerned, it had discovered a wormhole somehow, flying outside of the circle could be relied upon to magically teleport it closer to the center, and far from violating the rules instilled in it by its engineers, the drone had actually followed them to a t. In doing so, however, it had discovered an unforeseen shortcut, one that subverted its designer's true intent. That's really good, that's that's it's as a yes, I

love it. This is such a great example of how robots can fail in ways that are perfectly logical for the machine means themselves, but hard for humans to predict in advance, because we're not understanding how our you know, our programming or the data sets we're training it on is biasing its behavior in ways that that are strange

to us. And in this case, of of course, such a malfunction is harmless, but as autonomous machines become more and more integrated into the broader culture, not just in controlled contained locations like factory floors and laboratories, but in the wild so on the streets and in our homes and stuff. There will inevitably be cases where robots fail like this and fail in ways that cause catastrophic harm

to people. Yeah, and plus as an aside, we we have to realize that even in cases where the machines have not failed, there will be gray areas in which it's not completely clear, and an argument could be made in these cases for machine culpability with a variety of

intense and possible biases in place. Oh yeah, that's another thing these authors talk about that there can be all kinds of ways that, uh, that robotics and AI could end up causing extreme harm to people without ever doing anything that if a human did, it would be illegal. What one example they give is like if, um, if Google were to suddenly change its Google Maps algorithm so that it routed all of the city's traffic through your neighborhood. Like,

nothing illegal about that. It doesn't like commit a crime against you, but this is going to drastically negatively impact your quality of life, and it's a decision that's just like a could be a quirk of an algorithm in a machine. Now this paper in particular concerns the legal concept of remedies. So I was reading about remedies. A common legal definition that I found is quote the means to achieve justice in any matter in which legal rights are involved. Or in the words of Limely and Casey,

what do I get when I win? Right? So, if you if you take somebody to court, be as you say they have harmed you, whatever outcome you're seeking from that, that court case is the remedy. So usually when a court case finds that somebody has done something wrong to harm somebody else, the court responds to the finding of

guilt or blame by enforcing this remedy. And common remedies would include a payment of money, right, a guilty defendant has to pay money to the plaintiff, a punishment of the offender, like maybe they go to jail, or a court order to do something or not to do something. For example, somebody is ordered not to drive a vehicle, or they are ordered not to go within a hundred feet of somebody else or something like that. Yeah, or their their eyeball is removed, or they have to spend

a night in a hunted house something like that. Hopefully not in modern law. But wait a minute, there are some Sometimes you do read about some really strange like remedies that are ordered by judges like I order you to I don't know, to wear chicken suit or something, right like, Yeah, there's some judges who like to get creative. It seems weird. Yeah, I wonder I have There have been cases where someone has to spend a night and a hounded house due to a court order. I think

that would be a good setup for a film. But anyway, so when you start looking at the idea of remedies, remedies are complicated because they involve different types of implied satisfaction on behalf of the victim or plaintiff. And some are very clear and material and others are much more abstract. So the very the ones that are very clear and material are like, if I hit your car with my car and I'm clearly at fault, I need to give you a payment of cash to offset the material losses

to the value of your car. Right. But then other times it's it's more abstract. It's you know, punishment of an offender, to give the victim a sense of justice or to allegedly discourage someone from committing this type of harm or offense in the future. And then the authors right that things get way more complicated when you bring

robots and AI into the picture. For example, if you're trying to give a court order to a person, you know, saying like you shall not drive a car, you shall not you know, come within a hundred feet of this person. You can do so in natural language. You can like speak a sentence to them and you can expect them to understand. But how do you get a court to give an order to a robot not to do something?

Most robots don't have natural language processing, and even if they do, a lot of times it's not that good. So you might think, okay, well you just you know, you give the court order to the robots programmer and then it will and then they'll have to program the robot to obey. But this is also really complicated, like whose responsibility is it the robots current owner or the

original contractor or creator who made the robot? Uh? And what if this is like an end user consumer device that the owner doesn't have any ability to reprogram or what if, in the case of robots whose behavior is driven by machine learning or some other kind of system that is, for practical purposes, a black box, what if it's not even clear how you could reprogram it to reliably obey the rule. Yeah, because there's a chance you got to this position because the robot misinterpreted what was

asked of it. So if you then make additional requirements, ones that maybe you know, haven't actually been tested before but are just you know, that that are then brought on by the court, that could conceivably create new problems, right yeah, Yeah, totally, And and it keeps getting even more complicated from there, like Limle and Casey right quote.

To complicate matters further, some systems, including many self driving cars, distribute responsibility for their robots between both designers and downstream operators. For systems of this kind, it has already proven extremely difficult to allocate responsibility when accidents inevitably occur. It just seems like a real, real fast way to get into skynet territory, where it's like the robot then decides that only way to assure that it never sells cigarettes to

children again is to destroy all humans. That sounds like finding a wormhole to me. We will be getting into some more wormhole territory as we go on, so more complications. Uh. The authors bring up the idea of how to courts compel a person or a company to obey a court order. Right, Like, if you know a company is like dumping poison that's harming somebody, and the person sues that company, what does the court do to get them to stop? Well, the there is a threat of contempt of court if they

don't stop doing it. Right, Courts usually just assume that people are motivated by a desire not to pay huge monetary damages or a desire not to go to jail. Would that have any motivating power on a robot? It would only have that power to the extent that the robot had been programmed to take that into account. If it hadn't, it wouldn't matter at all. Like, you know, most robots probably do not have any opinion one way or another about whether about going to jail or having

to pay damages. So you'd have to explicitly program it to be disincentivized by potential punishments. Yeah, because take the cigarette robot for example, Like it's prime it's prime directive is just to sell delicious cigarettes to human beings, like the the what what else? What kind of leverage do

you have? Right? Exactly? So in that case you'd be faced with either you'd be trying to find some kind of human who's responsible for its behavior, but you could very well run into the problem that like, you can't really identify any one person who seems to be at fault for what it did, and it's doing this bad thing, so so what are you going to do about it? Yeah, And then of course things get even weirder when you

start getting into that that other side. You know, that's like the more like direct and material remedies that can be provided by courts, either like a monetary award to the victim or in order to stop doing something that

causes harm. On the other hand, you've got this thing that courts often end up engaging in, and people are are largely driven in motivated by however however irrational it might be in some cases, And that's the perceived abstract value of punishment, you know, not just material damages to a victim or in order not to do something, but the inflicting of punishments, specifically to demonstrate the court's displeasure

with the original behavior of the defendant. Uh So, they raise a question that's brought up in a paper by a professor named Christina Mulligan, who explores the subject of should you have the right to punch a robot that

hurts you limly in case? He called the call this the expressive component of remedies, And though a desire to see offenders punished maybe an extremely natural and nearly universal human drive, it's debatable whether it actually serves a purpose in reducing harm, and if it does, in what cases it does. I love this idea because, in a very literal level, it makes me think, well, why would you punch a robot? They're made out of out of metal.

You're gonna hurt your hand. All you're gonna do is or your hand, and you're not going to hurt the robot unless first of all, you design the robots so that it has at least one punchable portion of its anatomy, and then for it to be more than just you know, a cathartic uh uh thing for you, then you have to also make sure there's some sort of feedback right where Yeah, like you punch cigarette bought in it's punchable area,

then it will say owl. And maybe it will I don't know, ottaw inciner rate one packet of cigarettes so that it can never sell them, that sort of thing. But then, yeah, you're having to design your robots to to suffer to a certain extent, which I guess that means that goes back to what C three PO said, Right, he said, you know about about being made to suffer. It seems to be our lot in life. Oh that's interesting.

I hadn't thought about that. Yeah, clearly R two, D two and C three p O have have inherent desires to avoid pain. They have been programmed with that. Yeah, but as we've said that, that's not standard issue for robots. Most robots don't care about whether or not they get injured, Like that's not a motivating factor for them. And again it raises this bizarre question of like, what are you

doing when you punch the robot? Like what is I guess it's making you feel better, But does it make you feel better if, like you know that the robot doesn't actually care? Yeah, and then what then what needs to be done to convince you that it does care. Yeah, it just gets very sticky, very quickly, and then of course turns the mirror back on the way we handle human to human scenarios. Right. But anyway, Limbly and Casey, I guess to to summarize their position, they say, Okay,

increasingly independent robots and AI are coming. They're they're infiltrating more and more into society, and they will inevitably do bad things. When that happens, the legal system will try to order remedies to make things right, and you know when when harm has been caused. Our current legal understanding of remedies is based on the assumption of human agents human agents only, and its rules are not suited to

dealing with robot crime or robot offenses quote. As we have shown, Failing to recognize those differences could result in significant unintended consequences, inadvertently encouraging the wrong behaviors, or even rendering our most important remedial mechanisms functionally irrelevant. Uh So, to take robot agents into account, we're going to have to examine and rethink how our systems of remedies work.

But and this is a point we've been making already, this could have multiple benefits because it could also lead to a better understanding of how we apply these remedies to cases dealing exclusively with humans. Quote. Indeed, one of the most pressing challenges raised by the technology is its tendency to reveal the trade offs between sidal economic and legal values that many of us today make without deeply

appreciating the downstream consequences. They right, we need a law of remedies for robots, but in the final analysis, remedies for robots may also end up being remedies for all of us. Now, like I said, this is a very long paper. We can't do justice to all of the subjects they raise, but to focus on some highlights, I thought one interesting place to look was when they try to get into the definition of what actually makes a

robot in the legal sense. Obviously, there's going to be some difficulty here because think about how differently the term is used and how many different things it's applied to in the world. Uh. The authors here cite a professor Ryan Callo, who in the past had written that there are three important characteristics that define a robot and make it different from any machine, just like a computer or phone, and Callo says that these three uh, these three qualities

are embodiment, emergence, and social valence. So to quote from Calo, robotics combines, arguably for the first time, the promiscuity of information with the embodied capacity to do physical arm. Robots display increasingly emergent behavior, permitting the technology to accomplish both useful and unfortunate tasks in unexpected ways. I like that

idea of unfortunate tasks. Um and robots, more so than any technology and history, feel to us like social actors, a tendency so strong that soldiers sometimes jeopardize themselves to preserve the lives of military robots in the field, and lives is in quotes there. Yeah, you may remember this from the film that came out a few years back, Saving Private Cigarette Robot. It's quite touching. I mean, it seems absurd, but it does seem to play on our

natural biases. I want to talk about a couple of examples from a psychology paper in a second, but like, uh, we're we're just so ready to look at machines like humans and and treat them as such. It seems almost impossible to avoid. But anyway to pick up with limely and Casey after after that Callo quote they a quote. In light of these qualities, Calo argues that robots are best thought of as artificial objects or systems that sense, process,

and act upon the world to at least some degree. Thus, a robot, in the strongest, fullest sense of the term, exists in the world as a corporeal object with the capacity to exert itself physically. Though, it's interesting to me that even this attempt to give a strict and legally useful definition of a robot includes a subjective component. I brought this up earlier, the component about human feelings, the social valence criterion. The Calos sites here this means they

feel to us like social actors. Yeah. Like I was wondering in all of this, like, where does a particularly malicious robo call fit into the scenario, Say, a robo call that is not just about trying to sell you something, but it's like, you know, actively trying to say, get a credit card number out of you for nefarious purposes. Yeah,

that's a really good point. And and along those lines, Limbly and Casey argue that actually they don't think the embodiment criteria of hardware is necessarily a good one that maybe our concept of a robot should be less limited to the essentialist quality of being embodied and more just applied to anything that exhibits intelligent behavior and exactly things like that robot call would be would be a good example. Uh, the things we think of as robots probably do. They

they're not just like stand alone objects. They interact with the broader world in some way, but they could be entirely software based. Yeah, I guess certainly the room is a great example, or any kind of like vacuuming robot where it's it's it's it's in your house or it's in a room in your house. It's a it's interacting in your environment and it's essentially making decisions about how best to move around that space. Sure, but if you want to take it out of the embodied space, you

could have the idea of bots. On the Internet, there's things out there are acting autonomously to some extent and doing, you know, executing some behavior, acting almost maliciously. We were tempted to call them bots meaning short for robots, because they have some kind of apparent independent agency and they're doing something that seems at least halfway intelligent. Right, yeah, yeah, and you can easily imagine how they could they are and well they I mean they are used maliciously in

some cases. But how something like a social media boat that responds to certain comments in a particular way, like, it's very easy to imagine how you how that could be utilized in a way that would be not only annoying but just that outright harmful, even physically harmful. Oh yeah, I mean think about some of these, say like bots on social media that try to crowdsource information like during

a natural disaster or something like that. You could imagine uh, intentionally maliciously manipulating a bot of this kind to like have you know, bad information on it or something, yeah yeah, or know, anything that a troll can do on social media, a bought could conceivably do as well. So that just

you know, opens up the door, right. But coming back to this, so there's this interesting idea that robots feel to us like social actors, and that seems to be, at least by some people's definitions, a kind of inextricable quality of what makes a robot like it feels like, at least to a small extent, like a person somehow, and it reminds me of the psychology paper I was looking at just recently on human social interaction with robots,

that is by Elizabeth Broadbent called Interactions with Robots The Truths We Reveal About Ourselves, published in the Annual Review of Psychology in twenty seventeen. Uh, this was a highly cited paper, and it seems to be. It's a big literature review of a lot of different stuff about about how humans interact emotionally and socially with robots. And the one section I was thinking about was where she reviews a bunch of other studies about how we mindlessly apply

social rules to robots. So there are a ton of different examples, but just to cite a couple of them, once she writes up his quote. After using a computer, people evaluate its performance more highly if the same computer delivers the rating scale, then if another computer delivers the rating scale, or if they rate it with pen and paper. So like, if you know, you get a thing at the end of a test that says like, hey, you know,

how did you enjoy interacting with this machine? You're you're more likely to give it a higher score if you're still sitting on the same machine or at least. That was what was found by nas at All in UH and the and Broadbent rights quote. This result is similar to experiment or bias, in which people try not to offend a human researcher. Another example of social behavior is reciprocity.

We help others who help us. People help to computer with a task for more time and more accurately if the computer first helped them with a task than if it did not, And this was found by Fog and nas In. I love that idea of people, you know, being more reluctant to rate a computer UH poorly if they're still interacting with the same computer that that's that

seems perfectly true to me. But another interesting one from the summary is quote research and psychology has shown that the presence of an observer can increase people's honesty, but incentives for cheating can reduce honesty, and this is found

by Covey at All in nineteen eighty nine. In a robot version of this work, participants given incentives to cheat were shown to be less honest when alone compared to when they were accompanied by either a human or by a simple robot, and that was found by Hoffman at All in this illustrates that the social presence of robots may make people feel as though they're being watched and increase their honesty in an effect similar to that produced by the presence of humans. Now, this is inter right.

This This also reminds me of various studies that have gone into sort of the idea of of imagine beings or religious beings watching us while we're doing things right, or even just like I imagery, like putting some eyes imagery on a wall looking at people while they're like, I don't know, not supposed to steal from the collection plate or something like that. I don't know if it's to the same extent, but at least in the same

direction that the presence of another human is. You know, you're you might be a little bit worried that ARE two D two is gonna, you know, judge your moral character harshly or tattle on you. I'm not as worried about R two, but um three po snitch. Coming back to Limele and Casey, so they talked for a long time about how robots get their intelligence. They talked about the importance of machine learning for the modern generations of robots and AI that it's just not practical to hard

code AI the way we used to imagine. You know, you'd be a programmer and you're just like creating a lot of strings of if then statements like you know, the kind of intelligence that we expect from a modern AI or or intelligent robot is too complex for people to program in a in a direct way like that. Instead, they've got to be trained on natural data sets through machine learning. But of course doing so comes at the

cost of increasing uncertainty about their future behaviors. Behaviors could emerge that a conscientious programmer would never intentionally hard code into the system. Uh So, so that brings us to like, what types of harms could we expect from robots and AI? And the authors here come up with what I think are some very useful categories, some sort of like cubby holes, to slot the different types of AI fears into. So

the first kind is what they call unavoidable harms. These are probably not the main ones to be worried about, but they are worth thinking about. Uh And this is just the fact that some dangers are inherent too many products and services, we just accept them as the cost of having those products and services in the first place. So like this would just be cigarette bought just by virtue of selling cigarettes is doing harm to people, right, Yes, I mean the fact that you have cigarettes, there is

some harm coming from that. But there are also ones that are more fully integrated into just the way society works,

like having cars. It is absolutely inevitable that people driving cars are going to crash their cars and there will be fatalities from that, and you can think of ways of reducing it, but there's there's really not any expectation that we can have a country that has car based transportation and there will not be any accidents because there always be things that are that are not even reducible to driver error or to malfunction of the cars, right, like a tree falls on the road or something, birds,

wild animals. Any So, even though they're I think there's some very convincing arguments to be made that uh a switch to self driving cars would create a much safer uh travel environment, that it would make roads safer. You're not gonna you're not gonna get to absolute zero crashes or absolute zero road fatalities, right, I mean, you wouldn't even if the driving algorithms were perfect, right, and they're

probably not going to be perfect. They may well be and probably are going to be better than the average human driver. Yeah, okay, so that's just there's just unavoidable harm that comes from using any type of product or service, and when you integrate robotics and AI into that product or service, those unavoidable harms will just continue. But that's something we already deal with. The next category is deliberate

least cost harms. This is similar to unavoidable harms, but it's in cases where the machine actually is able to make a decision with with important ramifications, like it can make a decision to act in a way that causes harm, but is attempting to cause the least harm possible. So in a sense, this is forcing robots to do the trolley problem. Right, do you switch to the track that has one person sitting on the train tracks instead of

five people? Yeah? Yeah, And this will be another inevitable capability of autonomous cars, but it raises all kinds of thorny questions. If an autonomous vehicle can avoid a head on collision that will likely kill multiple people by suddenly swerving out of the way and hitting one pedestrian, that may indeed avoid a greater harm. But that's probably cold

comfort to the one person who got hit, right, right. Yeah, And then when you have a robot or some sort of an AI involved in that decision making, I mean, it's it's you can just imagine the the the intensity of the arguments and the conversations they would ensue. Right. But then the authors raised what I think is a very interesting point. They say that this kind of life or death trolley problem will probably be the exception rather

than the rule. Instead, they say, quote, uh, far likelier, I'll build albeit subtler scenarios involving least cost harms will involve robots that make decisions with seemingly trivial implications at an individual level, but which result in non trivial impacts

at scale. Self driving cars, for example, will rarely face a stark choice between killing a child or killing two elderly people, but thousands of times a day they will have to choose precisely where to change lanes, how closely to trail another vehicle, when to accelerate on a freeway, on ramp, and so forth. Each of these decisions will

entail some probability of injuring someone. I guess another thing to keep in mind, like with the trap, with the trolley problem, generally, when you're dealing with it, there's a there's a lot of emphasis on the problem aspect of it, you know, like the trolley problem should be a an ethical dilemma. It should, it should hurt a bit to

try and figure out how what to do. And the idea of the trolley problem being something that um is encountered and decided upon, like as in a split second, by a machine, um, by an algorithm like that, that feels that that feels a bit worse to us. You know, that feels like if if it's if it's an easy decision, even if it's just based purely on math, you know, it's um it feels wrong on some level. Oh yeah, yeah,

so I think you're right. But also the thing they're bringing up here is that the trolley problem you're actually more often facing is that every single day you're your autonomous car is gonna make you know, hundreds or thousands of trolley problem calls where on one track it is getting to your destination a few seconds faster, and on the other track is a one in a million chance

of killing somebody. Yeah yeah, And these do we make these decisions all the time, but we don't focus on these That's but I think that's part of the issue, you know exactly. You're like, should I, okay, should I take a left on this road? Well, there's a chance there's a speeding car just above, just over the edge there, and I can't it, but I'm going to take that chance because I want to cut three minutes off my

drive to work. Yes. Uh, this is actually a very good point that we we already make these decisions, but we just don't think about them in these explicit probability calculations. And there may be some consequences to thinking about them this way, which is there could be a weird, like perceived downside just to making these kind of uh, these

kind of calculations objective and and explicit. Yeah, I mean I've I've run into this with some of the map programs that I used to drive before, where I want to tell it in some cases like like give me the ability and maybe they have this now, but there was one left turn in particular where the ability to to flag this left turn, this is a dangerous left turn. You have put me in a position to make. Um,

I might know which left turn you're talking about. If you probably in town, it's it's it's in town, it's near our office, so yeah, I mean yeah. By the way, if you're out there working on programming driving apps, you should absolutely include the toggle key where you can say no left turns please. Yes, that that is highly useful.

I'm to understand this is how one of my aunts got around, Like if they got older and they were less adventurous driving, they would only take right turns, and they would do all their driving so that no left

turns were made. I think I have one time read this could be totally wrong, but I at least one time I remember reading a claim that, like, you know, the traffic efficiency would be x percent higher and people would spend x number of minutes less time and traffic if there were no such thing as left turns, if everybody had to get everywhere by only doing you know,

full right turns to to go around the block. Interesting, I'm sure there would be some cases where you can't do that, but you know, in a in a grid city, seems to make a lot of sense. Maybe you get like one left turn of days, some sort of a card system. But like I said, I I cannot confirm that. Okay, but anyway, the next categories of harm they talked about this one is defect driven harms. Uh. This one is

very easy to understand. The robot harms someone because of a design flaw or a bug or a mistake, or it's just broken. You know, A warehouse loading robot is designed to only operate when no humans are nearby it. But there's a malfunction with one of its sensors and it fails to detect the presence of a human operator trying to get I don't know, a piece of junk out of it, out of one of its hinges, and it moves and kills them. Okay, this is pretty straightforward,

just it's broken for some reason. The authors here do point out that this gets even more complicated when there is a human in the loop e g. An autonomous car with a human driver who is supposed to intervene in the event of an emergency. They talked about one case where this happened with with I believe it was an uber autonomous vehicle where both the machine and the human fail, that both of them failed to stop a

collision that hurts someone, like what happens here. Yeah, yeah, of course, we we have very similar cases in just purely human affairs. Right when questions are asked like where was this person's supervisor? Uh wait, you know who? Who were the watchers? Who? There should have been some other person, There was someone else in the loop here. Why didn't they do something to stop this crime from taking place? Right? Okay,

After that you get into misuse harms. Now, some of these are very obvious, very straightforward, like if you program a robot directly to go kill someone, or even if you program it to wander around at random swinging a machete. In these cases, it seems that the human programmer is clearly at fault, right, the robot has just become a weapon of murder or of reckless endangerment, and the person who told it to do that is the person responsible. Yeah.

Like if you take an automotive, uh like oil change robot and you reprogram it to um do appendectomies and people die as a result, Like, that's a misuse. You can you can only blame the the oil changed robots so much because it was not ultimately designed to perform appendectomies. Right. In this case, this is more like the hammer example used to the beginning this it's not the robot autonomously making the decision to do this. Uh, this is somebody

just using it as a tool of crime. But the authors point out that there are cases where quote, people will misuse robots in a manner that is neither negligent nor criminal, but nevertheless threatens to harm others. And these types of harm are especially difficult to predict and prevent. So one example is just people love to trick robots, people like to to mess around with robots in AI. I would admit to myself finding this amusing and principle, we've talked about this in uh, you know, the the

flated sex make Ina episodes. But of course there are times when it's not so funny, when when people take it to really sinister places. One example the authors bring up here is the horrible saga of Microsoft Tay. Do you remember this thing? Oh? This was the this is the robot that was traveling across the country. No no, no, no no, uh though I know what you're talking about there. No, maybe we can come back to that. But Tay was a Twitter chat bot created by Microsoft that was supposed

to learn how to interact on the Internet. Just by learning from conversations it had with real users. So you could tweet it Tay and say hey, how are you doing, you know, and you could talk about the weather or whatever. But of course who who ended up engaging and training this AI to speak? It was like the worst trolls on the Internet. So within a matter of hours, this brand new chat bot had been transformed from from a from a you know, a lump of clay unformed into

a pornographic nazi. Yes, I do remember this now. And this kind of just gets you thinking about the ways that people will be will be able to misuse robots in ways that guide their behavior in extremely pernicious directions, sometimes without the people guiding this misuse necessarily committing any kind of identifiable crime. Like people are going to look for exploits, they're going to look for ways, they're gonna look for cracks in the system. It's it's like within

with any kind of like a video game system. You know, people are just gonna see what they can get away with and and just engage in that kind of action, sometimes just for the fun of it, right, And sometimes that's harmless, but sometimes that's really awful. Yeah, Okay, next, category is unforeseen harms. And here's where we start getting into the really the really interesting and really difficult cases, types of harm that are not unavoidable, not a product

of defects or misuse, but are still not predicted by creators. Uh. And so the authors talk about how, in a way, unpredictability is what makes AI potentially useful, right, Like, it can potentially arrive at solutions that humans wouldn't have predicted, but sometimes it does so in ways that really miss the boat and could be extremely harmful if they were

embodied in action in the real world. Uh. Similar to the drone example from the circle that we talked about at the inning, But they signed another fantastic example here that's kind of chilling. So I'm just going to read from Lemle and Casey here. In the ninet nineties, a pioneering multi institutional study sought to use machine learning techniques

to predict health related risks prior to hospitalization. After ingesting an enormous quantity of data covering patients with pneumonia, the system learned the rule has asthma X delivers lower risk X. The colloquial translation is patients with pneumonia who have a history of asthma have a lower risk of dying from pneumonia than the general population. The machine derived rule was curious, to say the least. Far from being protective, asthma can

seriously complicate pulmonary illnesses, including pneumonia. Perplexed by this counterintuitive result, the researchers dug deeper, and what they found was troubling. They discovered that quote patients with the history of asthma who presented with pneumonia usually were admitted not only to the hospital, but directly to the i c U, the intensive care unit. Once in the i c U, asthmatic pneumonia patients went on to receive more aggressive care, thereby

raising their survival rates compared to the general population. The rule, in other words, reflected a genuine pattern in the data, but the machine had confused correlation with causation quote, incorrectly learning that asthma lowers risk when in fact, asthmatics have much higher risk. It seems like we've got another wormhole here. And here the authors introduce an idea of of a

curve of outcomes that they call a leptokurtic curve. That's a strange term, but basically what that means is if you are UM, if you're charting what types of outcomes you expect from a traditional system like just you know, humans looking at data versus a a complex automated system.

The uh, the sort of the tails of the graph with the complex automated system will tend to be fatter, meaning you get more extreme events in the positive and negative space rather than a you know, a sort of rounder clustering of events in the you know, normal operation space, if that makes any sense. So, these kinds of unforeseen harms are some of the most worrisome types of things to expect coming out of robots and AI. But then the other one would be systemic harms and this is

the last category of of harms they talk about. Uh the author's right quote. People have long assumed that robots are inherently neutral and objective, given that robots simply intake data and systematically output results, But they are actually neither. Robots are only as neutral as the data they're fed, and only as objective as the design choices of those who create them. When either bias or subjectivity infiltrates a systems inputs or design choices, it is in inevitably reflected

in the system's outputs. This is your classic garbage in garbage out, problem, right, They go on, Accordingly, those responsible for overseeing the deployment of robots must anticipate the possibility that algorithmically biased applications will cause harms of this systemic nature to third parties. So uh, an example that's much discussed in this would be an AI trained to make decisions about granting loans by studying patterns of which loan

applicants got their loans granted in the past. And a I like this could end up manifesting some type of bias that hurts people, like a racial bias in its loan assessments, because there was already a bias in the real world data set that it was trained on. So, in other words, AI that is trained on data from the real world, unless it is it is explicitly told not to do this, it will tend to reproduce and

perpetuate any injustices, any inequalities that already exist. And the authors here give an example that is based on algorithmically derived insurance premiums that I think they're talking about auto insurance quote. A recent study by Consumer Reports found that contemporary premiums depended less on driving habits and increasingly on

socioeconomic factors, including an individual's credit score. After analyzing two billion car insurance price quotes across approximately seven hundred companies, the study found that credit scores factored into insurance algorithms so heavily that perfect drivers with low credit scores often paid substantially more than terrible drivers with high scores. The studies findings raised widespread concerns that AI systems used to generate these quotes could create negative feedback loops that are

hard to break. According to one expert quote, higher insurance prices for low income people can translate to higher debt and plummeting credit scores, which can mean use job prospects, which allows debt to pile up, credit scores to sink lower,

and insurance rates to increase in a vicious cycle. Uh so, this is kind of a nightmare scenario, right, Like an AI that is too powerful and not explicitly protected against acquiring these types of biases could create these kind of computer enforced prisons in reality, like a machine code for perpetuating whatever state of the world, like whatever state the world was in when the AI was first deployed and

then just entrenching it further and further. Yeah, And that kind of thing is especially scary because like, if there's a human making the decision, you can you can call up the human to a witness stand or ask them like, hey, why did you make the decision this way? But if it's an AI doing it, you could say like, hey, why why is it? Why are we getting this outcome that's you know, creating a sort of like cyclical prison out of reality, And they can just say, hey, you

know it's the machine with the machine. You know it knows what it's doing. Yeah, yes, the machine it says I learned it from watching you dad, and you have that moment of shame. So I think these different categories that they that they bring up are really important for helping us kind of sort our ideas into into recognizable types for for ways that AI and robots could go wrong and could potentially cause harm that you would seek legal remedy for. And also they help identify the spaces

that there's the most worry. I mean, for me, I think that would be like those last two cases, right, the unforeseen problems and the systemic problems are the ones where there's the most real danger, I think, and the most difficulty in trying to figure out how to solve it. Yeah, because we we kind of you know, train ourselves for to a certain extent and sort of culturally focus on the sky net problems, right, the really obvious, um uh situations where the robot car veers off the road in

a dangerous way. But the situations where it is just perpetuating what we're already doing, where it's making choices in getting from point A to point B that don't violate anything we told it, but just are an uninventive and even harmful way of doing it. Uh. Yeah, that's that's that's harder to deal with. That's a type of misbehavior that you can't solve by just having Dan o'harla hay stand up and bellowing behave yourselves exactly. Um yeah, yeah, yeah,

I mean I can't remember if that even worked. I just remember that was one of my favorite moments in that was RoboCop two, right, was it? Yeah? Well, I mean RoboCop one. I think we're already dealing with this problem of like the sort of like weird dynamics of machine culpability when ed two oh, nine like shoots that guy five hundred times in the boardroom during the demonstration, and then Dan O'Hurley. He's response to it is to turn to Ronnie Cox and say, I'm very disappointed, Dick.

But anyway, well, I guess we're running running kind of long, so maybe we should call part one there. But we will resume this discussion about about robot justice and robot punishment in the next episode. That's right, We'll be back with more of this discussion in the meantime. If you would like to check out past episodes of Stuff to Blow Your Mind, uh, and the definitely worth checking out because we have lots of past episodes that deal with

robots and AI. We have lots of episodes where we make RoboCop references, so they're all they're all there, go back and check them out. You can find our podcast wherever you get your podcasts. Just look for the Stuff to Blow your Mind podcast feed. UH. In that feed, we put out core episodes of the show on Tuesdays and Thursdays. Mondays, we have a little listener mail Wednesdays, so that's when we do the artifact shorty uh usually,

and then on Friday's we do weird house cinema. That's our chance to sort of set most of the science aside and just focus on the films about rampaging robots. You just thinks, as always to our excellent audio producer Seth Nicholas Johnson. If you would like to get in touch with us with feedback on this episode or any other, to suggest a topic for the future, or just to say hello, you can email us at contact at stuff to Blow your Mind dot com. Stuff to Blow Your

Mind is production of I Heart Radio. For more podcasts for my heart Radio, visit the i heart Radio app, Apple Podcasts, or wherever you're listening to your favorite shows.