TechStuff Tidbits: ChatGPT and Natural Language Processing

topic that doesn't merit a full episode. Now, it's not to say that this particular topic doesn't merit a full episode, but it will actually require a lot more in depth discussion, and I wanted to kind of give an overview just to give y'all something to think about. So let's get to it. Late last November, the Open Ai organization launched chat GPT, which if you haven't used it, I recommend you go play with it. Try it out. You can make a little account for free and interact with this

chat bot. So the GPT for chat GPT stands for Generative pre Trained Transformer. But it is a chat bot. Now, it's a chat bot that can respond to prompts and queries with sometimes astonishing results. Uh. This is the tool that has certain teachers worried that their students will be hoisting all of their homework onto this chat bot and make it, you know, write essays for them, and it may not always be easy to spot when that happens.

There are people who are actually offering tools that purport to be able to detect if it was generated by a chat bot as opposed to written by a student. So there's already this kind of seesaw struggle between the chat bots and the detectors, which is kind of interesting. Some people argue that Pandora's box has been opened and chat GPT came roaring out, and it's going to make teaching even more difficult and challenging that already is. By the way, if you're a teacher, my hat's off to you.

Both my parents were teachers, they're both retired now. I have nothing but the utmost respect for teachers. Y'all have a incredibly important, incredibly challenging, and criminally undervalued job. Anyway, for chat GPT to work as well as it does, there are a ton of different elements that had to be implemented. Open Ai had to develop a sophisticated machine learning process to train chat GPT, so that it can formulate relevant and convincing but not always accurate or correct

responses to prompts, and relevant is really important. Right. If you ask something a question and you get an answer that's about something entirely different, that's a really frustrating experience. So one thing that the app needs to be able to do is parse human language. And this is a non trivial engineering problem, and we're going to talk about that briefly in this Tech Stuff Tidbits episode. So while chat GPT is the launching off point, this is not

exclusively applicable to chat GPT. It's actually applicable to a wide range of applications. So when you hear the phrase natural language, that means it's how we humans tend to communicate with one another with languages that naturally evolved out

of culture and society. There are manufactured languages like Esperanto, where people created a language with the purpose of creating a language, but generally natural language just refers to a language naturally evolving over time, and we've developed these over millennia, right, And a language includes things like the rules we use to form phrases and sentences, as well as the vocabulary we use to populate those phrases and sentences. So we've

got syntax and grammar, and we've got vocabulary. All of these elements come together to form language, and you can't just throw them in willy nilly, or you will make no sense. That's why we have the structure and rules. It also includes the quirks we develop over time, you know, exceptions to established rules and things like puns and jokes and idioms and that kind of thing. Right, we have phrases that we use that means something, but they don't

necessarily mean the surface level. If I say it's raining cats and dogs, I probably do not mean that it literally is raining small furry animals. If it is, I will be inconsolable. I am saying instead that it's raining

really hard, but I'm using this idiom to express that. Right, So, for humans, it's not that difficult to understand what someone is saying, assuming you both speak the same language, even if that other person is using some unfamiliar words, like you could be sitting in on I think back to when I was in a class in English class and I had a professor who insisted on using the word paradigm,

and I had never encountered the word paradigm. I it was not in my vocabulary, and I picked up in context that the way she was specifically using paradigm was essentially as a stand in for the word example. Uh. This taught me two things. It taught me one of the meetings of the word paradigm, and it taught me that my professor was incredibly pretentious. Anyway, context really can clue us in. Even if we don't know what a specific word or phrase means on the surface, from context,

we can derive at least some meaning. We only still miss the full meaning. Or on occasion, you might pull a malapropism and you might use the wrong word in place of one you intended to use. This has happened to me many times. If you want a recent example of a character who uses malapropisms, you should watch the movie Glass Onion. There is a character in that who frequently uses the wrong word to stand in for something that he intends to say. Also a fun side note,

the word malapropism comes from the world of theater. Richard Brinsley Sheridan wrote a play called The Rivals. This was back in the eighteenth century, so the late seventeen hundreds and the rivals includes a character h a a a caretaker sort of or a chaperone almost, and her name is Mrs Malaprop, and she often employs the wrong words

to comedic effects. She means one thing, but she says another because she's using the incorrect words, which makes the meaning of what she's actually saying change and that's where the humor is right. And I think this is a fun bit of history to know where Malaprop comes from, because there's actually another famous character from theater, from English theater who predated Mrs Malaprop by a couple of hundred years.

That character is Ugberry from Billy Shakespeare's Much Ado About Nothing, and dog Berry also frequently uses the wrong words in an intended comedic effect. The difference is Mrs Malaprop is actually funny and dog Berry more often than not isn't and mostly being a snobby tease. Here up, please keep in mind that back in college I majored in English Lit with a focus on Shakespeare. I love Shakespeare dearly,

but I have issues with some of his comedy. Although I guess you could argue the the broad gap in time between his time and mind could play a part in that anyway. My point is it's not too hard for humans to communicate the same thought in lots of different ways, and if we encounter a new turn of phrase or a new word, we can pick it up without too much problem. This stands in stark contrast to

computers and machines. Now often in the show I talk about binary information, about bits and bytes, because when you dig way down into how many machines process information, you're looking at circuits that run various mathematical operations upon strings of data, and that that data is grouped into zeros and ones. This is binary or machine language. Computers can interpret this quickly. The computers are effectively looking at a

series of off or on indicators. I often say a binary digit or bit is a lot like a light switch. It's either off or on. It's zero or one. When you put this through logic gates that are have specific designs to them, you run them through various operations, these zeros and ones can almost magically become complicated processes that let you do anything from type out the script for a podcast to playing the latest video game. Now make sure you keep in mind I said, almost magically in fact,

it's not magical at all. It's just that we humans can't really process huge banks of zeros and ones and really make much sense of it. While on a similar note, a machine without the proper programming can make no sense of our human languages. It is complete gibberish. It's meaningless to a computer. Now, it would be exceedingly difficult to do any significant kind of programming if programmers had to

depend upon hard coding zeros and ones while composing their work. Fortunately, computer scientists came up with solutions, namely compilers and computer languages, and we're gonna start with computer languages. A computer language creates levels of abstraction that make it a bit less

daunting to write software for computers. So the idea is that each computer language has its own set of rules, like its own syntax, its own vocabulary, and as long as you work within those rules, and you do so precisely and with as few errors as you possibly can, you can create a program to make the computer do whatever it is you want it to do. Because the computer language itself has built into it the ability to be converted into machine language. That will get us two compilers,

which we will chat about in the minute. In fact, we'll talk more about computer languages and compilers after we come back from this quick break. Okay, before the break, I had introduced this idea of computer languages. Now, not all computer languages are equal, and there are dozens of them. In fact, there are some notable differences between computer languages. Some of them are very similar, some of them share a common root computer language, and some of them could

not be more different. Now, in general, we can separate them into very two, very broad categories. There are a lot of different ways to categorize programming languages, but we're looking at one of the most basic, which is low level languages and high level languages. So a low level

language is relatively close to machine language. It only provides a thin layer of abstraction, and as such it can still be very challenging for programmers to work with these languages because they're not that far off from the basic machine languages. Uh, it's a little easier. It's it's designed so that humans can interact with it a little more naturally than they would with pure machine language, but it's

not that easy. A high level programming language, on the other hand, has a great deal of abstraction, and it's much closer to a human language in that way. So these languages are far easier for humans to work with on a day to day basis, And by humans I mean programmers. Like, if you had never seen a computer programming language and one day you just sat down to work on one, it would not seem easy or intuitive to you, probably, But both low level and high level

languages have their own sets of rules. If you break those rules, your program is not going to behave the way you intended to. It might not run at all. You might just get error messages. But yeah, they have rules, and if you follow them, then programs can potentially work. Now, if computers process information in zeros and ones, and programming languages provide levels of abstraction that approach human language at

the higher levels, how do we reconcile that? How does a computer interpret a program written in uh language like Python? For example? Python is much closer to a natural language than it is a machine language. So how do we get to the point where a computer can take that

information and actually execute a program? Well, the program I'm once composed has to go through what we call a compiler, and the compiler's job is to take this program and according to the rules of that programming language, convert the program from the programming language into machine code so that a computer can actually do something with it. So the

compiler is kind of like a translator. And the compiler actually has to do a few jobs to make this happen, and has to scan the program's source code for recognizable commands and terms. That has to analyze the syntax a gave the structure of the code to understand the order of operations. That has to break all that down into

machine code that follows what the language says. And if the programmer made a mistake, well that ends up getting translated to and then you don't find out un till you try and run the program, and then it gets back to debugging figuring out where did you make that mistake in the actual program. Now here's the thing. The programming languages, while they can create high levels of abstraction,

are still not necessarily accessible to the average person. Like I had mentioned before the break, you know, I'm talking about the average person who has little to know experience with computer languages or programming. So to people like that, and I'll include myself here, a sheet of code written, and even a very high level computer language might end

up being indecipherable. You could look at and say, I don't know what this program is supposed to do because I don't know enough about this programming language to understand what any of this means. As such, for folks such as myself, programming a computer is a daunting task because we lack the basic knowledge of the programming languages we would need to use to make an effective program. Now you you can teach yourself these things lots of people have.

In fact, a lot of the most famous hackers and even like leaders in in tech business are self taught programmers. So there's nothing stopping you from doing this, even if you never took a class in computer science or programming. In my case, it's literally that I haven't sat down to do any programming since the days when I did it an Apple Basic, and I only vaguely remember those days.

But you know, what if you could interact with a machine through the use of natural language, not a programming language. What if the computer we're able to take your queries and commands that were either written or spoken in everyday human language and then suss out what it was you wanted and then give it to you. That's the goal of natural language processing, and we see it in different implementations.

Right with chat GBT it's a text based interaction, but with smart speakers it's through speaking it into a microphone and giving a response. But the basic idea is still the same, and at a shallow level, it appears that a computer is able to understand you and can respond in kind, but in reality, what is going on in the background is a very complex analysis to determine what it as you are saying or asking or typing or whatever.

And like a compiler, a natural language processor has to identify all the components of a query and to analyze the syntax and then respond in a way that's most likely to be relevant. For example, if I were to ask a smart speaker what's the weather in Walt Disney World today, it would have to understand that I've got a location I've given It's not my location. I've asked for Walt Disney World, I've asked a specific set of data what is the weather, and I've given a timeframe

of today. It would have to understand all that, analyze all that, and then get the correct response and present it to me, and that's that's incredible, Like, that's an incredible amount of work going on in the back end. It happens almost instantly when whenever we interact with these systems, but it's a lot of stuff that has to happen

or over that to work. Otherwise we would end up with devices that just do irritating things like I imagine and telling my smart hub to dim the lights, and instead it tells me what the weather is, that the Magic Kingdom, and I'm thinking, well, that's nice, but the lights are still too bright. Please dim them so I don't have to get up off my couch, walk across the room and turn a dial, because you know, lazy. Anyway, As for natural language processing, that's a simple phrase that

hides how insanely complicated the actual processes. In reality, natural language processing is a multidisciplinary area of development. It incorporates elements of artificial intelligence, machine learning, human linguistics. There's a bit of psychology that goes in there too. And the evolution of natural language processing is a little difficult to trace, particularly if you're looking at it from the perspective of a user. So way back in the day, kiddos a

lot of computer games didn't have graphics. They were text based games, kind of like a choose your own adventure novel. To say, do they still have those? If you're familiar with them. You read these books and at the end of certain pages you are presented with a choice, and it gives you two different page numbers to go to depending upon whichever choice you make, and then you continue the story from there. Well, a text based adventure was very similar to that. There were fewer overt prompts in

your typical text based adventure. You could theoretically you could choose to type in whatever you wanted, so you could type commands into a prompt line and then the computer game would produce a response. So you might type something like look and that would prompt the game to produce a description of the environment that you were in at the moment. Or you might type inventory to find out what stuff you happen to be carrying on your character. Or you might type put bit in T in order

to get the dag nabbed and probability drive working. I am still traumatized by the text adventure The Hitchhecker's Guide to the Galaxy. Decades later, that game was fiendishly hard and non intuitive anyway, then the program would respond appropriately.

It would give you the response based upon the command you typed in, and on a surface level, it looked like the computer game understood what you were saying, except as soon as you typed in a phrase that the programmers hadn't accounted for or just didn't support for whatever reason, you would get a pretty standardized message saying something along the lines of I'm sorry, I don't understand, and so

it turned out the game didn't understand you at all. Instead, the game had a list of inputs that mapped to specific outcomes, and if you provided the input while, you'd get the outcome. But anything outside of that list was not something the game could handle, and it had to give you a response saying I'm sorry, I can't do that, or something along those lines. So it gave the illusion of understanding, but the player would quickly come to learned that there was no such thing actually going on in

the background. On a similar note, we have chat bots, and these have been around for ages, and developers have worked for a long time to make chatbots sophisticated enough so that you might start to think that maybe the chat bot actually understands what you're saying, or maybe there's a real human on the other side posing as a chat bot. We'll talk about that again in just a second,

but first let's take another quick break. Okay, so you've likely heard of the Turing test, which is this mythical test for artificial intelligence and to determine whether or not a computer may or may not have sentience or consciousness or whatever. Uh That that is really kind of snowballed from what it originally was, but the basic idea is that it's a take on a game called the imitation game.

And in this game, a person an interrogator, sits down at a computer terminal and they compose questions and they get answers displayed on a computer display in front of them, and their job is to determine whether or not the entity that's creating the responses is another human, or in fact, it's a machine that's attempting to pose as a human.

And if you get to a certain percentage of interrogators who cannot be certain or they mistakenly misidentify a machine as a human, you would say that machine passes the Turing test, and it can convincingly pose as a human. Well, we've seen lots of different examples of chat bots that have supposedly passed the Turing test. But again, this isn't like a solid tests. It's not it's not like the S A T S or something. There's not like a

solid grading structure. It's more it's more interpretive than that. But some of the early chatbots we saw do this kind of thing by selecting a subset of human behaviors. So an example, there were early chatbots or meant to simulate someone who had paranoid schizophrenia, or it would pose as a therapist, which mostly involved taking whatever it was you last said and then turning it into a question, which is, why do you think that your coworkers don't like you? I gotta say, dr spate, so give me

a lot of complicated feelings. Anyway, by selecting this subset of human behaviors, the programmers are limiting the sort of things that the chat bought would be expected to chat about, and they would also plant an expectation in the part of the human interrogator. It lowers expectations. In other words, if you are told that, hey, you're gonna be chatting with someone, and it might be a young boy from another country who only has a passing understand ending of

say English, and uh, they are like fifteen. Well, that's gonna set your expectations right. You're no longer gonna think, oh, this is someone who's going to have a very deep knowledge of, say the Vietnam War. That's not gonna happen, So there's some leeway there. Uh. Anyway, that's that's one of the tricks of creating chat bots that are convincing.

But these days we're actually seeing much more sophisticated chat bots like chat GPT, and they seem to actually understand what it is we want, and we can give chat GPT a complicated prompt and the program is capable of providing a response. So, for example, I actually did this. I wrote in the prompt, compose a high KU about g p U s as graphics processing units. And this is what chat GPT created for me. GPUs speed up

my code, processing data, lightning, fast, silent, powerful. For now, I could point out that this poem does not strictly adhere to the structure of hiku because a typical hiku, the first and third lines have five syllables, the middle line has seven syllables. The hiku quote unquote that chat gpt produced had seven syllables, eight syllables, then six syllables. So this is not a hiku in the structural sense. But you you do see how chat gpt is trying

to comply with my request. It it's giving something that has the feel of a hiku, even though it's not strictly speaking a hiku. Now, I've talked in the past about how chat gpt pulls data from a huge library of information. It is not actively connected to the Internet, but instead has this massive repository of information that it can pull from, kind of like having a really big encyclopedia, like horror coded encyclopedia. Think of something that you would

have in a home library. Actual books. The information in those books is not going to change, not frequently, like once a year you might get an updated volume that gives information about different things that have developed over the year, but otherwise, no, it doesn't change. So chat gpt is not pulling the most recent information and then serving that up. It's it's going to this big library. Now, Unfortunately, that library doesn't guarantee that the responses you get are going

to be accurate. They will appear to at least be relevant because chat gpt s programmers were really good at having it analyzed queries and to really hone in on what was being asked so that chat GPTs response, it's generative response would relate to the query. It is one common complaint with chat gpt that it presents information in such a way as to authoritative though in fact it

may not quote unquote know what it's talking about. So it's working on this very complicated system to parce language infirm meaning based upon the words and syntax provided by users, and then generate a response following the basic rules of grammar and vocabulary well probabilistically picking the most likely response to be correct and to be relevant. Again, this is something we humans do pretty naturally, but for computers it is anything but natural. It required a ton of work

and evolution to get there. Now, beyond the surface level, I feel it's important to say that chat gpt does not truly understand what we're saying to it, or what it's saying to us. Non the level of deriving meaning from it, it's not able to associate different ideas, it's not able to come up with something new. It's not thinking. It's analyzing and it's responding, and it's doing so in a very cool way, but it's not sentient or anything

like that. Now I will probably do a full episode about what's going on behind the curtain with natural language processing. It's a pretty challenging topic to cover. It is incredibly complex, it is incredibly sophisticated. It requires an interdisciplinary approach that is hard to describe easily, and it also typically involves several different machine learning strategies that are somewhat challenging to describe,

particularly without the benefit of visual aids. But I think it's worth diving into, and I think we can do it. I think we can at least get an appreciation for how these systems are working. If nothing else, it can remind us that the magic we're experiencing when we tell chat GPT to compose, say a punk rock song about the iPhone, well we understand that what we see is the result of complex sesses and not you know, some

sort of mystical event. Now. I say that not to take anything away from the phenomenal achievements of the hundreds of folks who have worked on natural language processing projects, but rather to prevent the rest of us from bringing meaning where maybe there is no meaning. We don't want to project onto this thing. We don't want to make assumptions because that could lead us down pathways where we start to trust things that are inherently not totally trustworthy.

One of the issues we've heard multiple times with chat GPT is that it's kind of a black box and that you ask a question, it gives you an answer, but you don't see the process that chat GPT went through in order to understand what you were asking and then generate the answer that it gives you. And because of that, you can't double check its work right. You can't check to see what sources did you pull your information from to generate your answer, because the sources may

or may not be reliable. And chat GPT may be in a phenomenal tool, but if it's pulling from unreliable resources, while the answer you get is still going to be wrong. But because chat GPT doesn't really do that, doesn't really show its work. Uh. That's where you start to run into these problems. And the more dependence you put upon these kinds of systems, the more important it is to understand how these systems are actually generating the responses. This

goes beyond chat bots. Obviously, this applies to AI across the board. It's an ongoing issue within AI in general. Is this desire to make certain that the results that AI generates where whatever it may be. Maybe it's facial recognition technology, um, maybe it's a robot deciding how to open a door. Being able to see that process and understand what steps the system went through in order to get to its decision are critical in order to be able to uh, to judge how well that's that overall

system works or doesn't work, or if it's reliable or unreliable. Uh. That transparency is absolutely necessary for that sort of thing. And frankly, a lot of the systems we encounter today have a lack of transparency and that makes it kind of scary. However, that being said, I think chat GPT is a really really cool project. I do share the concerns of people relying upon it to do work that they should be doing. Um, I think that's I think

they're cheating themselves. If you aren't doing the work, then you're not learning how to think. Which is the most important lesson you can learn in your education is learning how to actually think and to think critically, and if you deny yourself that, then you just set yourself up to be led around by the nose by anyone who has a convincing enough story, and that rarely turns out well.

At least it doesn't turn out well except for the person who's doing the leading, and even they tend to come to a bad ending once it's all said and done. All right, that's it. Hope you enjoyed this tech stuff tidbits about natural language processing, specifically within the context of chat GPT. Like I said, we'll have to do a much deeper dive because this isn't even really scratching the surface like that. This is such a deep dense topic that we could do a couple of episodes about it

and really kind of explore it. Uh. I would even reach out to experts to have on the show to talk with them about it. The only issue there is that I would worry very quickly that experts would use terminology and jargon that I'm not familiar with, let alone some of my listeners, some of y'all are way ahead

of me on this stuff, and that's awesome. Some of you all are probably in the same position that I'm in, where you know, you could hear someone spelled off a lot of jargon and not know what the heck they were saying. Context be darned, it would just not be be accessible to you, uh or to me. And so yeah, I just got to make sure that if I pick someone who is an expert, they're also a great communicator

for that kind of stuff. Uh. Some engineers are phenomenal communicators, and some are phenomenal engineers who can make machines sing, but may not be able to uh talk with other human beings in a way that the other human beings can follow. It's it's more that, you know, I lack that experience and understanding to do it confidently and accurately.

But yeah, we'll do more about natural language processing. I didn't even really touch on the spoken language processing stuff, you know, like things like speech to text and that kind of thing. That stuff is also incredibly complex. It adds other layers of complexity upon this system. UM. I

have talked about those in the past. I've done episodes where I've talked a bit about, you know, like things like like voice recognition and those sorts of things, and speech to text but yeah, that's also important to remember that that adds yet another layer. Well that's it. I hope you're having a great three so far. I mean we're four days into it. I hope things haven't gone

pear shaped already. And if you have any suggestions for topics A should cover in future episodes of tech Stuff, there a couple of ways you can reach out to me. One of those is to download the iHeart Radio app. It is free to download into use. You can go to that little searchbar type in tech Stuff. It'll take you to our page. You'll see there's a little microphone

icon there. If you click on that, you can leave a voice message up to thirty seconds in lengths say hi, tell me kind of what topics you would like to hear about more. If you prefer, you can pop on over to Twitter and send me a message there. The the Twitter handle for the show is tech Stuff H s W and I'll talk to you again really soon, y.

Tech Stuff is an I Heart Radio production. For more podcasts from I Heart Radio, visit the i Heart Radio app, Apple Podcasts, or wherever you listen to your favorite shows.