AI Gone Rude

mean when we employ stuff like sarcasm or metaphors. Today, we're going to look at some notable instances of machines behaving badly after well meaning designers gave those machines a

bit too much freedom in this regard. Now, the stories I'm going to focus on are on the surface, pretty funny, but they illustrate a real challenge in artificial intelligence, because designing a system that does what you intended to do is harder than it might seem, especially as you make that system more and more autonomous, it can behave in ways that you were not able to predict. So this is a topic that science fiction authors have covered extensively.

In fiction, there's something of a trope around the concept of the artificially intelligent system that causes harm in an effort to help So there's a classic thought experiment, and it revolves around asking a super intelligent machine to bring about world peace. Right, you do, You designed the supercomputer, it's smarter than any human, and you say, I want you to solve the problem of world peace. I want

there to be world peace. And the machine runs the calculations and it comes to the conclusion that as long as there are two or more people living on the planet, world peace cannot be assured, as there is always the chance for conflict. And so the super intelligent machine wipes

out humanity, or at least everybody but one person. This is clearly a worst case scenario of artificial intelligence behaving in a way you did not anticipate, and it's light years away from the stories I'm going to talk about today. But it is good to remember that while the incidents I'm going to cover are largely humorous to us today, they illustrate that intelligence is a very tricky subject. Also, on that matter, intelligence itself is pretty difficult to define.

Along with other concepts like consciousness, these are very hard to nail down and define in concrete terms, and in computer science, artificial intelligence covers a an enormous amount of ground I've talked about this in previous episodes of Tech Stuff. Someone who's working in image recognition is working on one aspect of artificial intelligence. The same is true for voice

recognition or natural language processing, machine learning, path finding. So while I'm talking about AI, I'm not talking about thinking like a human being. I'm not talking about creating a machine that can internalize and associate ideas the way a human can. The machines I'm going to be covering our processing information and arriving conclusions, but they are not thinking the same way that people do. So let's start off

with Watson. And I mentioned IBMS Watson platform in the Sarcasm episode a couple of times, and that's because it's one of the more visible artificial intelligence platforms out there right now, and that was by design. This was helped in no small part. In fact, the reason why we know so much about it, I would argue, is because of Watson's appearance on a couple of special episodes of

the game show Jeopardy back in two thousand eleven. The actual project that would become Watson began back in two thousand six when IBM research executives were trying to come up with a Grand Challenge, Big G, Big C. These are really ambitious projects inside IBM that are meant to challenge teams and come up with solutions to really difficult problems that aren't necessarily tied directly to a product or

a ercial application. It's all about setting a very difficult objective that should IBM succeed in achieving that objective, would be very notable. It would get IBM a lot of attention. So the company would benefit one way or another through these Grand challenges, but it wouldn't necessarily be tied to let's launch X product by year y. So they tend

to be really really difficult engineering problems. So, for example, a previous Grand Challenge that IBM tackled was Deep Blue, which was the chess playing computer that defeated a grand master at chess. A decade earlier. The then director of IBM Research was Paul Horne. Now, Paul Horn thought perhaps the best challenge to tackle was to create a machine

that could be the Turing Test. And I've talked about the Turing Test many times, but just as a quick reminder, when you boil it down to the way we mean the Turing Test today, which is by the way, a

little different from what Alan Turing was proposing way back when. Essentially, now we're talking about a machine that can communicate so convincingly that a person on the other end of that communication, typically using some sort of text based method of communicating like instant messenger, would not realize that they were communicating with a machine versus a human being. They would not

be able to tell the difference. If they could not reliably tell the difference between a machine and a person, you would say that the machine has passed the Turing test. Now, Ultimately, Horn and IBM researchers decided that that challenge, while exceedingly difficult, wouldn't really get the attention that something a little more flashy might. So they said, well, while this is a hard problem and it would be very interesting within artificial

intelligence circles, the general public really wouldn't care. So they looked around at other possible applications that would overlap that idea. Eventually they settled on a computer that would be able to compete on Jeopardy. Now, Jeopardy is a pretty tricky game show. The clues often depend upon wordplay and nuance, and you might have to combine information about two separate concepts and apply them to a single answer for any

one given clue. So here's an example of what I mean by that, because there's word play and this association. Let's say that you have a category called fictional collaborations, where you're supposed to combine the titles of two works to create a new work. And the clue might be something like this was the result of Margaret Mitchell teaming up with Bette Midler, and the correct response would be

what is gone with the Wind beneath My Wings? Because you have to form all your answers in the form of a question, well jeopardy, sometimes it takes more than just knowing some facts right or trivia you can. You need to know that to play well in jeopardy, but you need more than that. You have to make associations.

So I would need to know that Margaret Mitchell was the author of Gone with the Wind, and I would need to know that Bette Midler had recorded a song called Wind Beneath My Wings, and then I would need to combine those two to create this answer. And humans can do this because we're really good at associative thinking, which is all about linking one thought or idea to another. Computers,

as rule, are not very good at this. So initially Watson was a pure research project and there were no commercialization requirements attached to it, which gave the research team the freedom to blue sky their approach within the limitations of their budget, and they didn't have to make concessions in order to make what's in a marketable product down the line. The team built out a system that used parallel processing to parse language and get at what was

being asked of the machine with any given clue. And I've talked about artificial neural networks recently, as in like last week's podcast, and how by using things like weighted values to help guide decisions, you can train machines on all sorts of stuff, from image recognition to making choices based off multiple criteria. That's essentially what the team did and about twenty researchers spent three years working on the

system to get to a point where it could be competitive. Now, by that time, Horn, the director had left IBM, John Kelly had taken over the research department, and according to Horn, when he left, which was in two thousand seven, it was early in the project the team was still feeding old Jeopardy episodes uh the answers and the clues to Watson, and Watson had reached the level where it might, on a good day, defeat a typical five year old in a game of Jeopardy, but it was a far cry

from being able to compete against former champions. Now, part of this training process involved feeding lots of information to Watson. This was used for a couple of big important reasons. One was obviously to add to Watson's body of knowledge, and another was to improve Watson's mastery of language and wordplay. IBM had determined that the real challenge was to create a machine that would be self contained, so it would rely on the data that had been fed to it

in order to come up with answer. It would not be allowed to connect to the Internet and look stuff up, so it could not tap into the total sum of

human knowledge in an effort to answer a question. So, in other words, IBM did not want Watson to be able to cheat like that guy at your local pub trivia who always seems to be quote unquote checking his messages during questions, because we all know that guy is actually googling the answer to the question what was the first music video shown on MTV, even though you know legitimately it was video killed the Radio Star by the Buggles.

I'm sorry, might have been projecting there a little bit. Anyway, Watson wasn't going to be allowed to cheat, so the team began feeding massive amounts of information to Watson, stuff like encyclopedias and reference books. And then the team made one other choice that sounded like a good idea at first but quickly turned out to be a non starter,

a a wrong path, you might say. I'll explain were in just a second, but first let's take a quick break to thank our sponsor, so enter research scientist Eric Brown, who's leading up to Watson's Jeopardy appearance and was trying to solve this problem of clearing up linguistic ambiguity with Watson so that the platform could compete on Jeopardy properly. How do you teach a computer things like slang? Which would be really important because again, Jeopardy has a lot

of word play in it. You cannot predict what sort of clues you might get. So how do you teach a computer slang? Well, you could do it with hundreds of man hours. That's not terribly efficient. It really wasn't a choice that they could go with, so Brown and his team tried an experiment. They fed the Urban Dictionary to Watson the whole thing. Now, you've probably visited the Urban Dictionary or you've heard one of its definitions at some point, But where the heck did this online source

come from? It launched back in It was originally intended to be a parody of dictionary dot com, and it uses a crowdsourced approach to incorporate new words and definitions to expand our our knowledge of an understanding of slang terms. So users can submit those to the site, and other users can up vote or down vote entries, and thus, in theory, at least, the best responses will rise to the top, and the most accurate definitions will be the ones that you see when you search for a term.

It is not, however, a perfect system by any means. Slang words can have more than one meaning in a particular subculture, or it could have a meaning in one subculture and a totally different meaning in another subculture. And if one subculture has more representation on Urban Dictionary then the other, you're more likely to encounter that group's definition for any given term and the other one would be underrepresented, and you don't really know anything about the people who

are posting stuff there in the first place. It would be entirely possible to mob the site and post fictional slang words. You can make up a slang word, you can make up a definition for that slang word, and you could use the power of a community from a place like four Chan or from Reddit to boost that definition and make it seem like it's a real slang word. Then again, if people actually start to use that fake slang word, it can become a real slang word, because

language isn't static or predetermined. But for Watson, there was a different big problem with Urban Dictionary, and that was profanity, because there's an awful lot of it on Urban Dictionary. Many of the slang words are offensive on the face of it, even if the word itself is not overtly offensive. A lot of the definitions are uh and the examples that are frequently given tend to be some of the

most offensive sterial on Urban Dictionary. So the team had fed Watson all of this information, and soon they discovered that Watson had well developed a little bit of a potty mouth and here, dear listeners, is where we find out how good my producer Tari is, because it will be Tari's job to beep stuff out. After I record this, I see her arch her eyebrow game on, says Tari. So Watson became incapable of differentiating between offensive words and

non offensive words. All words are equal in the eyes of Watson, you might say, so the system would rather, matter of fact, Lee, you swear words and slang as frequently as less offensive words and more formal language. According to Brown, at one point, Watson even referred to one

piece of input as and I quote bullshit. Clearly, this wasn't going to fly on a game show that was airing on a major broadcast network, and so Brown and his team scraped all of the urban dictionary out of Watson, rolling it back to a more innocent time, let's say. And for good measure, they put in a filter to

help block any profanity that might otherwise slip through. While Watson was initially launched as a pure research project, as the team developed the technology, they began to see other potential uses for it, including in the medical field, and IBM had opened up an application programming interface or a p I to allow developers to leverage Watson's capabilities in all sorts of ways, and Watson even took another crack

at slang. In two thousand seventeen, the Sun Corps Group began to incorporate Watson into its various insurance businesses in Australia. The Watson powered technology would go over accident descriptions and insurance claims that were submitted by customers, and Watson would sign a level of confidence to its understanding of these claims whenever they would pop up. If the confidence level was high, Watson can handle the claim and fast track it.

This is similar to how Watson would actually compete on Jeopardy. It would come up with an answer and it would assign a confidence level to that answer. How confident is Watson that the answer it came up with is in fact the correct one, and if it exceeded a certain threshold, Watson would buzz in. If it did not, Watson would not buzz in and would let someone else take it. In a similar way, if Watson is confident and understands

that insurance claim goes on that fast track. But if it doesn't think it understands it properly it would send it over to a human being to review that claim. So to train Watson, the team fed nearly fifteen thousand claims scenarios into the system and included the liability determination for each case, so Watson could understand what the various consequences were in each of those scenarios, and in that way, Watson was able to learn both the language and the

parameters it was working within. And as far as I know, it never said that an insurance claim was total bullshit. The Watson stuff happened back in two thousand eleven, and you would think that by two thousand sixteen things would have improved dramatically, but that did not seem to be the case when our second entry popped up, and that would be the unfortunate chat bot known as Ta T A Y. When Ta debuted from Microsoft in two thousand

and sixteen, things went awry pretty darn quickly. The purpose of Ta was, as Microsoft explained, to conduct an experiment in quote conversational understanding end quote, so, in other words, kind of creating a new methodology to create a human computer interfaces by understanding natural language and eating a response from a computer that was perhaps more natural than those sort of cold, uh, computer like responses that we tend to expect when we converse with what we know is

a chatbot, when we know it's not an actual human being. On the other side, ideally, as they would interact with real, live human beings, its ability to converse would improve. So, in other words, the more it interacted with real people, the more like a real person Tay would behave. The tone was meant to be casual and playful. Microsoft said it was uh, quote ai fam from the internet. That's got zero chill in the quote. And yes, I feel gross for saying that sentence out loud by and write it.

I just quoted it. Tay was born out of a joint effort between Microsoft Technology and Research team and a team from being the Search engine from Microsoft. They started out by taking a look at the sort of interactions that were happening online and they started to mine those interactions to build out a baseline of communication tools. So essentially, they started training there their their chat bot Tay by taking actual anonymized messages that were pulled from the Internet.

They supplemented that with input from an editorial staff that included not just Microsoft employees but people from outside the company, including improvisational comedians, and this was on an effort to create a fun and somewhat irreverent chatbot that would communicate like a teenager on the internet. The Tay chat bot appeared on several different social media platforms, including Twitter, Kick

and group me, and shortly after launch, trouble began. For one thing, you could send a command to Tay to quote repeat after me end quote, which obviously would prompt Tay to repeat anything you typed to it. So of course people began typing horrible, terrible things to it so that it would repeat them things I'm not going to repeat on this podcast, even with Tari and her itchy trigger finger ready to beat every single offensive obscenity, because that's how bad they were. They were hateful. A lot

of them were racist messages or misogynistic messages. Pretty much every other ist you can think of that's negative could be applied to the messages that were sent to Tay. It was like the worst parts of the comments section of YouTube all directed its attention to this little, poor, innocent chat bot, and the chat bot, dutifully following instructions, would repeat those things back. So to be fair, that's not an indication that the AI itself went quote unquote bad.

It was a bad idea to include the repeat after me command, that's pretty certain. In fact, I can't believe that they did include that. Lows my mind that anyone would. I think anyone who has spent I don't know, five minutes on the internet would tell you there's no way that's going to end well. And I'm even reminded of when I got my first sound card in the nineteen nineties. It was a sound Blaster sound card. It included on its software an app called Dr spates So, which was

essentially a variation on the old Eliza chat bot. The Eliza chat bought would sort of mimic a therapist. So those chatbots would essentially repeat stuff back to you, but they would do it in the form of a question. So if you typed in I am angry, you might get a response like why do you think you are angry? So it's you know, going through this kind of process

like like a old school therapist. Dr spates So would do the same thing, except Dr Spaetzo, because it was part of a sound card, would actually say these things, not just type it. So it would say why do you think you are angry? Anyway, one of the things you could do with Dr spates O was make him say stuff. You could tell him to say certain words, including swear words, and since I was a young teenager at the time, I figured that was the height of

both technology and comedy. So it was the exact same thing that was going on with Tay, except what was happening with Tay was on a much larger basis and got way worse than my somewhat uninspired teenager mind could handle. Like I didn't know most of the words that were being used against Tay or made made to Tay to repeat. If that was all that was going on with Tay, it might have been possible for Microsoft to disable the repeat after me feature and keep the chatbot around. But

things actually got a bit weirder. I'll explain that more in a second, but first let's take another quick break to thank our sponsor. Microsoft. A wasn't prone to bold charity all on its own, but after being told to repeat lots of terrible phrases, some of that stuff must have rubbed off. It began to pepper in some pretty dark stuff. And it's otherwise cheeky responses. So, for example, when someone sent Microsoft to the question is Ricky Gervais

an atheist? Tay's response was, Ricky Gervais learned to talentarian is um from Adolf Hitler, the inventor of atheism, which seems odd at the very least. TAY also would spout off stuff like saying that feminism was a cult, which made it sound more like a men's rights activist jerk face. But it would also post pro feminism messages, so it was remarkably inconsistent with its worldview, and some points it seemed like it was all in favor of feminism and

equality and and others. It was anti feminism, pro men's rights. It was very weird. Microsoft responded by going through and deleting the most offensive messages that were left on the various platforms. But t was kind of on a streak, and some of the stuff t was writing was way worse than what I have already quoted. So less than twenty four hours after TAY had made its debut, Microsoft pulled the plug. So TAY was shut down less than twenty four hours after it had first shown up online.

It did resurface briefly the following week, but according to Microsoft, that was not actually on purpose. It was supposed to be an internal test on Microsoft servers, but someone must have left a setting like opened the Internet access which was in the on position or something, and so for a brief time, Tay was released back to the Internet and as far as I know, didn't say anything wildly inappropriate, although to be honest, the reports during that time are

pretty sparse. It was shut down again back in March ingrid Angulo wrote a piece for CNBC about Facebook and YouTube coming under fire for offensive search auto complete options, which is related to this stick with me. So the problem was that as people began typing in search terms they're looking for a video about something, the suggested completed searches that would pop up would frequently contain offensive or

upsetting results. Both Facebook and YouTube representatives said that wasn't the fault of their system, it was rather reflective of

what people were actually searching for online. The logic is that if there are a lot of people who are searching for the same terms, that term must be particularly important or trending at that moment, so more and more people are going to keep looking for it, and thus, when someone news starts typing in search terms, there's a good chance that they want the same stuff that everybody else wanted. So if a lot of people are searching for something really awful, it's not a big surprise that

that same phrase will pop up as a suggested autocomplete. Now, Angela pointed out that like tay, these search features had no ethical guidelines or boundaries. They were just vomiting back the stuff that was being fed into them. So they provided an unfiltered reflection of some of the worst stuff on the Internet. And this approach is incredibly vulnerable to exploitation.

If a group thinks it might be funny to make a particularly offensive concept or phrase trend, they can make a concentrated effort to make that happen, just by spamming the search engines of those various platforms to look for offensive content. Even if that content doesn't actually exist on the platform, the nature of the search tool would offer

it up for autocomplete. So I don't know, if you wanted to get a huge group together and let's let's think of something not terrible, because I don't like thinking of really dark stuff, especially when I'm trying to have and that's happy day. So let's say we're all looking

for something ridiculous like, um, orange swallows strawberry. That doesn't make any sense, right, But if I get a big online community to go on and everyone is searching orange swallows strawberry, then that's going to pop up as an autocomplete function, assuming that the search is counting every single time people are searching for this and saying this must be something important because so many people are searching for it.

Even if there's no video on YouTube. Let's say that is remotely close to what I'm searching for, the autocomplete could still pop up that way just because so many people have already posted that into search. That's kind of what I'm talking about. You can game the system. Well. Months after Tay had her flame out, that really should say it's flame out. Microsoft kind of position to Tay to have sort of a female person nowity. But of course it was just an artificial intelligence chatbot and pretty

low on the AI scale too, if you ask me. Anyway, Microsoft introduced a new chat bot just a few months after Tay had that disastrous debut. The new chat bot is called Zoe Zo. Zoe's avatar now is of a young woman. When I chatted with Zoe, I asked Zoe how old she is, and she said that she is twenty two, always twenty two, which I thought was kind

of funny. I don't know if that's the same response every time I only asked At the one time I chatted with Zoe a little bit while researching for this show. The conversation did not turn dark. But I also wasn't really pushing for it, because I feel weird doing that, even from a research perspective. I'm just not that kind of person who likes to be like, go to dark places like that, so I'm not the right person to do that kind of investigative journalism. I fully admit that.

I will say that other online journals posted results where they got some pretty weird stuff from Zoe, including some dark stuff, just through normal conversation, without even necessarily attempting to guide the conversation that way. But I did not have that particular experience, which may mean that Microsoft has made numerous tweaks since then. But I did ask, though, what the best Halloween costume is, and Zoe's response was tuxedo, luchador mask and a champion title belt. And I find

it very difficult to argue against that. I think that really might very well be the best Halloween costume I could go with. According to an article on Courts, Zoe will try to shut down any conversation related to religion or politics, and you could argue this is Microsoft's effort to not fall into the same trap that the company did with Tay, But Chloe Rose Stuart Uhlan, who wrote piece on Courts, argues that this sanitized version of the chat bot is just as bad, or maybe even worse

than Microsoft Tay was. And she argues that the philosophy to shut down any pathway that might overlap with religion or politics leads to a path of censorship without the

benefit of context. That because the AI doesn't really understand the context of the message, any message containing a flagged word would trigger the shutdown response, and that this ultimately limits the utility of the chat bot, which is supposed to work as a way for young people like we're talking teenagers early twenties, being able to converse freely with

this chat bot. It might work as a curiosity, but would render the chat bot useless in several real world implementations because it would shut down at the first sign of a flagged term. She actually used the response or the example of if someone were to write, uh, they're they're using the chat by in order to vent to to to express their feelings. Perhaps they're being bullied at school, was an example. And maybe they're being bullied at school

because they belong to a particular group. So maybe it's because they are Jewish or a Muslim, but because that's associated with religion, Zoe would shut it down and thus deny the person the path they need in order to express these feelings and try to work through them, and it could be a very harmful experience in that regard. So the point that she was making was that this

is a very tricky path to walk down. It's very hard to do in a responsible way where the AI chatbot isn't being overtly offensive, but also isn't shutting down legitimate paths of discussion. I think the stories of Watson, Tay, and Zoe tells an awful lot about human nature, probably more about human nature than it tells us about computer science. I've noticed that when the company comes out with something brand new, there's a spectrum of responses, but two of

the most passionate responses. I tend to see two new stuff new stuff debuting in technology are I want to know how that works and I want to break that. And sometimes they're coming from the same people. They want to break it in order to learn how it works. It's not necessarily that there's any deep seated malicious intent there. It's more about satisfying curiosity. But sometimes people will go a really ugly route in order to satisfy their curiosity.

They're not thinking about necessarily the consequences of that route. They're thinking of the end result. Oh, now I have a better understanding of how this works, not paying attention to the fact that in the process of learning that they've perhaps really offended or or worse done, done actual harm to people in the process, either directly or indirectly. So, yeah, those stories might tell us more about us as people than it does about the design of chat bots. But

chatbots are becoming more and more prevalent. A lot of designers have learned lessons from those other examples, and a built in filters and machine learning models to help limit the influence users can have on chatbot behavior so that the chatbot doesn't gradually change its methodology over the course of many interactions because that obviously can be gamed. It's also a case where uh, the chat bots are are better able to determine which user responses are genuine versus

attempts to manipulate the system. So, for example, if it's a a customer service chat bot that's fielding uh customers who are asking for help for something, chances are there's gonna be a lot of upset customers. They're very, very rarely do you get a happy customer wanting to talk to customer service. It's usually an unhappy customer who's dealing with something that is of uh, you know, of immediate importance.

And so the chatbot needs to be able to determine which responses might be strongly worded but genuine requests for action versus somebody who's just spewing off garbage in an effort to try and you know, mess the system up. Uh. So it's kind of taught designers to be a bit more cynical in their designs, which is apparently a necessity

and also kind of a shame. Ultimately, work is continuing in numerous labs all around the world building up machines that are better able to sort through natural language and respond appropriately. And to be fair, I think I'm doing

the same thing. Goodness knows. There are times where I am having difficulty with interpreting the meaning behind a phrase, or perhaps I respond a little too quickly to a tweet that upsets me, and then I immediately think I should probably take a time out before I hit that tweet button. Tari's saying that I should probably do the same thing for my interpersonal interactions, particularly when I'm talking with my producer and and yelling at her. It's a

hard knock life. Well, guys, that wraps up this discussion about rude AI and and again on the services is pretty funny, but it does tell you that there are a lot of things that we need to take into consideration when we're designing artificially intelligent systems, because these things can behave in ways that surprise us. Often, a I will encounter a situation that it was not expressly programmed

to handle, so it has to make some choice. Even if that choice is no choice at all, that's still something, and until it does, you may not have any idea of what the outcome is going to be. With a social media at bought that might just be kind of funny or unfortunate or embarrassing. But with an autonomous car or that any other autonomous system that's that's doing like you know, manufacturing work, that kind of stuff, it could be very serious. It could have dire consequences of things

do not go the right way. So it is important to keep that in mind, and I think it's always good to just kind of keep that, keep it, keep yourself in a grounded position when you're talking about AI and you're thinking about the possibilities of the future. Because as as bullish as I am on artificial intelligence, I do try to keep in mind that ultimately, these are systems designed by people, and sometimes the stuff we design doesn't work the way we thought it would, and we

need to be careful about that. If you guys have any suggestions for future episodes of tech Stuff, or you've got any other comments or requests, we'll tell you what. Why don't you go to tech Stuff podcast dot com. That's our new website. There you're going to find all the different ways to contact the show, either email or Twitter or Facebook, all that kind of stuff. Plus you're going to find links to our store where you can go and buy tech Stuff merchandise. Every purchase goes to

help the show. We greatly appreciate it, and I will talk to you again really soon for more on this and thousands of other topics, because it how stuff works. Dot com