How AI Learned to Talk and What It Means - Prof. Christopher Summerfield

Superman 3 is a terrible movie but there's this wonderful scene so I think there's a kind of giant computer that goes rogue in Superman 3. And there's this wonderful scene where there's this female character and, you know, she's sort of, the machine is just kind of like waking up and she tries to, she's just sort of walking past it and the machine kind of like sucks her in and she...

She gets kind of like stuck there. And then what the machine does is it gradually like kind of puts armor plating on her and replaces her eyes with lasers and basically turns her into a sort of like automaton. And...

It's a very compelling scene. I think I was terrified by it as a child, which is probably why I remember it. But that is a sort of metaphor for what is happening to us, right? We're worried about... the robots taking over or whatever but in a way it's more like us being sucked into the machine right we become part just like that poor you know character um we get turned into we get turned into something we are not

you become part of that system and it erodes your authenticity and in a way it erodes your humanity. People often say, well, you know, kind of chat GPT, of course, it was exposed to more. I think I have the analogy in my book. It's exposed to the same amount of language as if.

a single human was continually learning language from the middle of the last ice age or something like that, right? That's how much data it's exposed to. But it's a false analogy, right? It's a false analogy because we don't learn language. like ChatGPT does, right? So language models are trained in a kind of like, you might think of it as it's almost like a Lamarckian way, right?

One generation of training, if you think of a training episode, whatever happens in that gets inherited by the next training episode. That's not how we work. My memories are not inherited by my kids. There's this fundamental disconnect where Darwinian, the models are sort of like, I don't guess you could call them like Lamarckian.

So we're here in Oxford today to speak with Professor Christopher Summerfield. He's just written this book called These Strange New Minds, How AI Learned to Talk and What That Means.

He spoke about the history of artificial intelligence and how the allure of AI is to build a machine that can know what is true and what is right. Imagine a world in which everything... was like that but it could actually talk back to you and it could simulate all of the kind of social and emotional types of interaction that we have with people that we care about so you know The milk in your fridge is like your best friend, right? This is a very strange world.

in which you know of course that's a silly example the milk in the fridge is never going to be your best friend but you know kind of like there are you know as as i mentioned earlier there are already large numbers of people who are engaging with AI in ways that mimic the sorts of interactions they have with other people. I thought that.

grounding you would need you know kind of like sensory signals you need you know you can't know what a cat is just by reading about cats in books you need to actually see a cat but it turned out i was wrong and so were many many many other people and that is To my mind, perhaps the most astonishing scientific discovery of the 21st century is that supervised learning is so good.

that you can actually learn about almost everything you need to know about the nature of reality, at least to have a conversation that every educated human would say is an intelligent conversation, without ever having any... sensory knowledge of the world just through words that is mind-blowing this podcast is supported by google hey everyone david here one of the product leads for google gemini

Check out VO3, our state-of-the-art AI video generation model, in the Gemini app, which lets you create high-quality 8-second videos with native audio generation. Try it with the Google AI Pro Plan or get the highest access with the Ultra Plan. Sign up at Gemini.Google to get started and show us what you create.

I'm Benjamin Crousier. I'm studying an AI research lab called Tufalabs. It is funded from past ventures involving machine learning. So we're a small group of highly motivated and hardworking people. And the main thread that we are going to do is trying to make models that reason effectively and long term trying to do AGI research. One of the big advantage is because we're early, there's going to be high freedom and high impact as someone new at Tofa Labs.

You can check out positions at tofalabs.ai.

Professor Summerfield, I have to congratulate you on this book. Your previous book was my favorite book that I've ever read in AI. It's up there with Melanie Mitchell's book. And actually, Melanie Mitchell reviewed your new book as well. She did. Very generously.

I'm a big fan of Melanie. So you've been writing this for a couple of years and of course you actually extolled in the afterword that it takes quite a long time to get these things into publication and the space is moving very, very quickly. But can you give us a bit of an elevator pitch of the book? yeah sure so uh the book yes as you said it was actually finished at the end of 2023

So that is kind of like cast your mind back to the sort of medieval period of AI, if you like, 12, 14 months after ChatGPT had just been released. Yeah, so... The idea of the book was that at that time, and I guess to a large extent still today, there was considerable debate over what is the kind of cognitive status of these...

title of the book, Strange New Minds, that we seem to have created and are now increasingly interacting with. And that debate, the debate that I heard, and I heard, you know, the same debate going on in academic conferences and down the pub. The debate was, you know, kind of like, should we think of these things as actually a bit like us? Are they thinking? You know, kind of, are they reasoning? Are they understanding? And of course, this very quickly became a highly polarized debate.

And that debate was kind of like, on the one hand, a bunch of people who, you know, really sort of vehemently rejected the idea that these tools could ever be anything like us. It's just computer code, which is of course true. And then on the other hand, you know, you had people who were like absolutely astonished by not just by the capability, but by the pace of progress and thought, you know, kind of like we really are on course finally, finally to build something that is like.

as competent in a general way as humans and this debate was playing out and i was like well this debate is not really grounded in i don't hear the language of cognition being used to scaffold this debate. So this debate is being had by people who care deeply about this issue but are not trained.

in kind of like a grounded computational sense of what does it actually mean to think? What does it actually mean to understand something? And so I thought as a cognitive scientist who has done a lot of work in AI, that's probably quite well placed to talk about that. So that was sort of part one. And then, you know, kind of, I also have for the past five years been very, very interested in the implications of AI for society. And so I was working on that problem.

when i was at deep mind and we were doing work to try and understand how ai could be used to kind of intervene directly in society in the economy and help people find agreement um and At the time when I wrote the book, I was just about to move to the AI, as it was then, AI Safety Institute in UK government to work more on that. So I had a kind of understanding of like the landscape of...

deployment risks, thinking about how AI might change the way that we live our lives. And I thought probably putting those things together, I had enough of a unique perspective to write a book about it. So that's what I did.

The discourse is quite fractured and you speak about this in great detail. You speak about the hypers, the anti-hypers, the safety hypers and so on. And early on in the book, you kind of trace this back to two intellectual... threads going back to the ancient Greeks. So Aristotle and Plato, basically empiricism and rationalism. Can you kind of sketch that out? Yeah, sure.

AI has itself been kind of repeated, an ancient philosophical debate. about whether the fundamental nature of building a mind, including our mind, is fundamentally about learning from experience or about reasoning. particularly reasoning over latent or unobservable states, right? And that reasoning over unobservable states is, of course, traced back to Plato. That's kind of this idea that, you know, everything is fundamentally unobservable. We just get the sort of shadows on the cave wall.

or the light on the retina and we have to impute what's there. And the, you know, the corresponding view, which might trace back to Aristotle, you know, this idea that there is kind of, yeah, that everything comes from experience. And the history of AI, of course, was that very debate playing out actually in kind of like in the workshop, so to speak, right? or at least on the keyboard. So on the one hand, originally good old-fashioned AI...

was structured around the idea that, you know, kind of we sort of know how to work out what is true. And the reason we know how to work out what is true is because we have a kind of like long tradition. back through kind of positivism and, you know, kind of early theories of reasoning back to Boole and, you know, even Leibniz before that.

The idea that, you know, kind of like you can use logic to work out what is true. It is unassailably true that, you know, kind of like if I say that, you know, all men are Greek and Aristotle is a man, then Aristotle is Greek, right? That is just like... True by definition. And so that seemed like a really sensible way to build AI, right? Like you put in those primitives and you crank the handle.

And like, you know, if you've got enough computational power, then you can derive really, really complex things. And it worked, right? It worked. So in the 1950s, Newell and Simon... built the logic theorist, which I like to say is the first superintelligence, 1958. So it's an AI system that was able to prove theorems.

to prove theorems with a greater kind of like, well, it was able to prove many of the theorems that were in Russell and Whitehead's Principia Mathematica, which is like already a feat, and it was able to find more elegant solutions to... many of those theorems so like that's astonishing so you know initially it seemed like this kind of reasoning approach worked um and then you know kind of like of course

What happened is that as the problems that we tried to tackle with this kind of approach moved from these very abstracts of clean problems about maths and logic, and we started to tackle problems in the real world, we ran into a fundamental problem which is that the real world just isn't kind of like all that clean and nice and neat in the way that you know kind of like reasoning problems are designed to solve so

The world is full of weird exceptions, which, you know, don't fundamentally, aren't fundamentally amenable to analysis with, like, logic. And so, you know, kind of like you had this other corresponding... approach which is like the the learning approach or the empiricist approach and that was where neural networks and the deep learning revolution ultimately came from isn't it a crazy time to be alive though i interviewed the ceo of one of the largest companion bot platforms and

In the comments section, there was a lot of negativity. And you actually mentioned, I think, in your afterword that it seems strange to us now that we would want to have... a relationship with an ai companion and maybe we might revise that belief in a few years time but but i mean more broadly though you said in your book that language is basically the biggest gift that has ever been given to us it allows us to

acquire knowledge and communicate it and it survives many generations and i guess the the rubicon moment with this technology maturing was chat gpt that that changed everything in november 2022 sketch that out for me Well, I mean, you know, the history of NLP, I guess, has been told many times, probably by people more qualified than me. But, you know, kind of like we talked earlier about...

this kind of back and forth between learning and reasoning. And, you know, kind of in the history of NLP, what played out was exactly the same question, right? So NLP, natural language processing, the subfield of AI. And, you know, kind of the... As in the sort of more general symbolic AI movement, you know, kind of the early models were basically attempts to...

to define the computations that lead to the generation of valid sentences. That's basically the gauntlet that Chomsky lays down in his 1958 book. And, you know, there are a set of rules which, like, you know, if you could just apply them all lawfully, they would allow for the generation of sentences that, you know, kind of like obey.

the rules that we would all understand, you know, to be like what makes a valid sentence. So syntax, right? Chomsky was mainly concerned with English, of course, so he's worried about English syntax, but like, so that movement.

you know kind of of course was then just like neural networks came along in the wider field was then challenged by statistical approaches and that went back and forth and back and forth and back and forth and you know when the deep learning revolution happened By 2015, we had models that could, you could train a model on the complete works of Shakespeare, and it could generate something that looked a lot like Shakespeare, but it didn't make any sense.

And so still, you know, kind of even when the deep learning revolution was in full swing, most people, including myself, thought there is no way that the mere application of like powerful...

function approximation and lots of data is going to solve this problem i did not believe that to be true i thought that i think like any other people that you would need grounding you would need you know kind of like sensory signals you need you know you can't know what a cat is just by reading about cats in books you need to actually see a cat um but it turned out i was wrong

And so were many, many, many other people. And that is, to my mind, an absolutely astonishing, perhaps the most astonishing scientific discovery of the 21st century, is that you... Supervised learning is so good that you can actually learn about almost everything you need to know about the nature of reality, at least to have a conversation that every educated human would say is an intelligent conversation without ever...

having any sensory knowledge of the world just through words that is mind-blowing and i think it changes the way we think about many many things certainly changes how i think about things So one big theme in the book is this dichotomy between equivalentists and...

exceptionalists so some folks argue that humans are exceptional and the kinds of cognizing that language models do are not really you know in the same category yeah so i mean that distinction is a cartoon So, of course, you know, kind of like everyone has a different view about the relationship between AI and humans or biological intelligence in general.

And, you know, kind of like the evidence clearly admits a spectrum of different views. But I found it useful in the book to kind of cartoon to extremes of that continuum. And, you know, kind of at one end, you have people who I think probably just kind of like ideologically reject the idea that something that is non-human could ever... use, that we should refer to that, refer to whatever that system is doing, behaviorally or cognitively, using the same vocabulary as we used to apply to a human.

So, you know, kind of like clearly today's models are capable of reasoning at levels which is beyond the capability of most even educated humans today, right? Certainly when it comes to formal problems like maths and logic and so on. So it can reason like a human, but there are people who I think just fundamentally think that we shouldn't think of that as reasoning because we should kind of like circumscribe.

the definition of reasoning as something that humans do and that is a stance which i think is not really it's not really about

And the empirical evidence, although some people kind of construe it to be that way, by saying, oh, the models aren't actually that good at reasoning, which I think is a, you know, even in 2023, it was a hard to defend view. Now it's... probably an even harder to defend view but i think it's kind of like it comes from a place which is like a sort of radical humanism right it's a sort of it is a desire to kind of like really ring fence

a set of cognitive concepts and think of them as uniquely human. And for people who care about humans, which by the way includes me, I can see why that's really important. But what it does lead you down the road of is kind of like a... a refusal to ever see the cognition that an AI will engage in and the cognition that a human will engage in as comparable, even when their capabilities are clearly matched.

So that's what I call kind of exceptionalists, because in a way they're sort of like, you know, kind of they are espousing a view of human exceptionalism. Humans are special and different, end of story. And, you know, somewhat kind of cheekily in the book, I...

compare that to kind of earlier instances of human exceptionalism of course that occurred when you know darwin first proposed that you know we weren't kind of uniquely created by god but we're actually related to all the other species and like you know kind of when the heliocentric model um you know kind of first became established and was rejected by the catholic church and so on but that was kind of

I guess those analogies give color. But fundamentally, you know, I think it is a defensible position, but it's an ideological position, I think.

Yes, you invoke this notion called the duck test. Basically, if it looks like a duck and quacks like a duck, we should call it a duck. And by extension, I guess... You would call yourself a functionalist, which is this idea that it's not about the internal constitution or the mechanism, but it's about the function that it performs. And we can use this information metaphor to say, well, you know, if we have an AI system over here, which is doing cognizing and it's doing the same.

types of things then we could reasonably make the inference that it's appropriate to use mentalistic language to describe it. That's absolutely right yeah and you're absolutely right to say that it's a functionalist perspective and that is broadly my perspective. I think, you know, kind of once again, that functionalism, you know, it's kind of like from a scientific standpoint, right? I'm like, if it reasons like a human...

then we may as well use the term reasoning. But that doesn't imply a broader set of equivalents, right? That doesn't, for example, imply moral equivalents. It doesn't apply, you know, it doesn't mean that, you know, kind of like the motivations or, you know, relationships we have with AI are similar to those we have with humans. Absolutely not. Of course, they're completely different.

But it does mean that, you know, when purely, you know, if you put on cognitive scientist hat and you're really just thinking about, you know, let's talk dirty about information processing. then you know kind of like that functionalist perspective yeah if it walks if it quacks like a duck it's you may as well call it a duck the anthropomorphism thing makes it a little bit more tricky i mean i think um in in a film if you see um

robots peel the face away and all of a sudden you see they're not they're not a human that they're a robot and the intuition there is that they have a different mechanism and This is what John Searle was getting at when he was talking about the Chinese room argument, and I read what you had to say about that. So I think Searle was saying that when you take a type of process... and you represent it in silicone as computation sands the machine.

We are biomachines, so we are causally embedded in the world. And when we do things, there's this large kind of light cone of... low-level interactions that happen and and i guess this is his notion of semantics and and i think um

Professor Summerfield, you subscribe to something called a distributional notion of semantics, which is that we can actually remove things from the physical world and recreate patterns of activity in silico. And for all intents and purposes, it would have... the same meaning that's that yeah i mean i i i do subscribe to that view i mean i think that you know kind of of course as a not only a cognitive scientist but a neuroscientist i mean uniquely aware

that, you know, whilst there are many differences between machine learning systems and the computations that go on in the brain, there are also like astonishing similarities, right? At the level of kind of certainly at the algorithmic level. not not clearly at the implementational level you know kind of like you know neural networks don't tend to have you know

They don't have many, many different types of synapses, and we don't have many different types. You don't have basket cells and fast inhibitory into neurons and things like that. But there is, at the level of the neural network... there is a striking similarity and, you know, kind of the most reasonable assumption to me is that There are broad, shared computational principles that happen when you take networks of neurons that are wired up to have some dense interconnecting. And...

You know, for the most part, recurrent. We have to remember, the transform is not a recurrent architecture. So it probably uses tricks to mimic what a recurrent architecture does. But for the most part, recurrent network. And... We know that because we know, for example, that the way that information after optimization has been applied, and actually sometimes even before optimization has been applied, we know that there are striking similarities in...

the semantic representations that you can read out of those two classes of network, biological and artificial, by doing experiments, right? So we know that you can go into the brains of monkeys.

or if you have access to it, humans via neuroimaging or whatever, and you can see patterns of representation that express themselves in terms of, not just, you know, in terms of like... coding properties but in terms of like neural manifolds in terms of like neural geometry express themselves very much like in the neural network so you know kind of like the substrate is shared in some very loose sense

the behaviour is shared in some, you know, perhaps not so loose sense. And to me, it... makes sense to you know kind of science is a puzzle right like you get bits of information and you try to come up with the most parsimonious explanation and for me the most parsimonious explanation is that by you know kind of like sheer

kind of like a mixture of like luck and like you know trying enormously hard we've kind of got to a place where we've built something that is a bit like a brain and lo and behold it does stuff that is a bit like a brain That doesn't mean it does everything. And it also doesn't mean that it is like a human in the sense that like meaning how we should treat it, how we should think of it. But it does mean that the computations are most likely shared.

I realise this is a difficult argument to make and there were some scornful comments in your book about this, but there are some people who still make the argument that it only appears to be reasoning and understanding, but it's not really. And is it possible that Chomsky could still be right in some way? sense so you know his ideas obviously he's a rationalist but it's this platonistic idea essentially that the laws of nature have bestowed our brains with the with with with the the secret

functions that explain how the universe works. And in a sense, he's quite similar to a lot of folks now. He's a computationalist. He doesn't subscribe to this.

causal graph thing. But he does think that the brain is a Turing machine and we should do this recursive merge type stuff. But is it possible that... empiricism seems to work but it's kind of like a pile of sand and chomsky would still be right if only it were possible to have like the low level stuff yeah i mean i think you know kind of the what we will find out my guess is that what the end point will be

are when we sort of look back after perhaps having you know figured this stuff out is that in the end the dichotomy that was set up and that we fought about literally for millennia actually is kind of a question of perspective. So in a way, there is a way in which the rationalists broadly construed aright. Reasoning is really important for computation.

But what they were wrong about is how you acquire the ability to reason. So I think what we have learned since 2019 is that the types of computations that you need to reason about the world... can be learned through large-scale parameter optimization, through function approximation, essentially, through training a neural network. And that... So in a sense, Chomsky...

kind of is not wrong that, you know, there are rules to language. Those rules need to be learned. It was just wrong about how they got learned, right? And like, you know, of course, there's always a sleight of hand in saying, well, you're born... This is inborn. Because it really just begs the question of how it's inborn, right? And, you know, where does that gene that allows you to do recursion or merge or whatever, where does it come from?

And, you know, kind of like what was the pressure that got it there? And, you know, I think that there is a subtlety to an argument that is often not... kind of expanded on, and I think it is that, you know, of course we are born with the predisposition to learn language, and we know that that is not just kind of like an accident, right, because other species even.

highly highly intelligent species like chimpanzees and gorillas capable of really really sophisticated forms of social interaction you know political machinations and so on they can't learn structured language so they can learn to communicate but they can't learn to communicate in infinitely expressive sentences right that guided by lawful syntax and the fact that they can't do that

tells us that there is something special about our evolution. And so the question is, how do you explain that in the...

in the deep learning framework, right? And people often say, well, you know, kind of ChatGPT, of course, it was exposed to more, I think I have the analogy in my book, it's exposed to the same amount of language as if, you know, a single human was continually learning language.

from the middle of the last ice age or something like that, right? Because that's how much data it's exposed to. But it's a false analogy, right? It's a false analogy because we don't learn language like ChatGPT does, right? So... Language models are trained in a kind of like, you might think of it as almost like a Lamarckian way, right? One generation of training, if you think of a training episode, right? Whatever happens in that gets inherited.

By the next training episode, right? That's not how we work, right? My memories are not inherited by my kids, right? So there's this fundamental disconnect. We're Darwinian. The models are sort of like, I guess you could call them Lamarckian.

And so you can't compare the amount of training that ChatGBT has to the amount of training that we have because it's just kind of like apples and oranges, right? What happens in a... a person's lifetime is like, it's been guided, although it's not, you know, it doesn't have the content in that, you know, I live in Britain, but if I had...

You know, my kids have been born in Japan. They would grow up speaking Japanese. But it's been guided by all of the other generations of learning, which inculcate this predisposition to learning language. And we never think of language models in that way.

It's like meta-learning. It's really just like meta-learning. And so Chomsky is right that we are born with priors because those priors are the earlier cycles of Darwinian evolution that are... everything that went on before we were born right as individuals and so this kind of like i think when we talk about

data efficiency, and we try to make claims about data efficiency between biological and artificial intelligence, we need to be really, really specific about whether we're talking about phylogeny. or ontogeny, so evolution or development, and neither really works as a comparator. So it's just more complicated. Is it possible that we're being deceived in some way, though, because there are...

Certainly computational limitations with neural networks, there are complexity limitations, learnability limitations. So we kind of know that there are certain types of things that the networks can't do that we can do. And we are susceptible to this anthropomorphization. You mentioned this wonderful experiment where it was like a cartoon of arrows kind of interacting with each other and humans interpret them as agents. And this is the grumpy bully agent. And there was the Eliza.

and machine as well you know where it was a very simple program which was quite sycophantic and people really um you know took deep meaning from that is it possible that we're reading more into what's going on here than is actually the case Well, it's definitely true that we are intrinsically prone to attribute kind of like much more elaborate forms of cognition to all other non-human agents, actually, where simpler explanations may be available, right? Everyone who is a pet owner...

will be very familiar with this concept, right? It's like the easiest thing in the world to kind of like attribute complex human-like states to your cat or your dog or your hamster when it may or may not be merited, right? We know that people have been doing this for centuries, right? So psychologists know about the clever hands effect. The clever hands effect, very famously, there was a performing horse, which apparently could do mathematics. So simple arithmetic.

And it did so by repeatedly stamping its hoof the correct number of times to solve a sum. But, you know, of course, it wasn't actually doing mathematics. What it was doing was...

checking whether its trainer gave it a kind of like unconscious signal that it should stop tapping. And so, of course, you know, we are always prone to kind of want to impute these more complex... thoughts and feelings and emotional states or complex abilities to models I don't deny that for a moment but you know kind of when you look at today's frontier models

That may be going on. We may be thinking, oh, it's really my friend, when actually it's not. But in terms of the raw capability, the numbers are the numbers, right? The models are just really good, and there's no denying that. They can't do everything. There's lots of things they can't do and they're still not fully robust, but they are really good. They're not just clever hands.

You said yourself something in the book which intrigued me, which is that even cognitive scientists and neuroscientists and psychologists... don't really know what the answer to the question is. If you said, what is thinking? When we talk about these mentalistic properties and, of course, about this intentionality thing, you know, the agency interpreting the...

The sort of intentions of cartoon arrows that are interacting with each other. And Daniel Dennett, of course, coined this intentional stance, which is that essentially we need to understand the world. It's a very complex place. And that's where perhaps...

some of these mentalistic properties come from. But do you subscribe to an idea? I read this wonderful book called The Mind is Flat by Nick Chater. One of my favorites, yeah. A lot of these mentalistic properties, even in humans perhaps, are a bit of an illusion. What do you think? I love that book. Yeah. I mean, that book essentially argues that, you know, kind of like we draw heavily upon prior experience to formulate what we like. So in other words, our preferences. are a product.

or not just of like kind of some kind of internal value function, which is different for everyone, you know, kind of like you like apples more than oranges and I like oranges more than apples, but it's actually due to our memories for our past experiences. So you don't actually like apples more.

than oranges but you just think you do because you had an apple this morning and you're like oh i had an apple this morning i must like apples more than oranges so it's this beautiful theory which in which you know kind of like we essentially construct ourselves out of our own actions And, you know, it can account for an astonishing broad range of phenomena. Do we do that? You know, I think that's a scientific theory, but I think, you know, kind of in our...

everyday interaction with other agents, so animals, with technology, like we do the opposite. Like we impute, this is what Dennett says, right? We impute, you know, far more than is due. often right so you know your car fails to start in the morning and you get cross with it you know as if it was just being stubborn but of course there's no point getting stuck with it so we're getting cross with it right and

That is an example of the intentional stance. It is undoubtedly true that, for example, when interacting with the models, people are very, very prone. to attribute intentionality, so in the technical kind of like philosophical sense of the word, right? In other words, that there is something... Kind of like that it is like to be that thing, right? And people are really prone to attribute that sense of like, you know, they have some essence, some sense of what it is like to be themselves.

to probably all forms of technology, but especially to AI because they can talk back. People do that all the time. This is manifest in so many different ways. Of course, the types of interactions that people have with... today's frontier models, starting with Blake Lemoyne, who, of course, you know, famously, I talk about in the book famously, you know, kind of argued that after his interactions with Lambda, that it was sentient. And...

Playing out today, we see that two of the top 100 most visited websites in the world are companion applications. These are... Generative AI systems that are trained to behave as if they are your friend. Why are they so popular? Because they're good at that. But they don't have to be that good. Because people are really prone to be...

you know, to think of them as if they were a person. That is undoubtedly true. But I think it's possible to hold that view and to be cognizant of our predisposition to do this. but also still to be sober about the capability. I think it's just a different question, right? The capability question is like, how do you get something that can, you know, solve simultaneous equations?

if they're posed in natural language. How do you do that? That is a problem that we did not know how to answer in 2018. We know how to answer it now. And the system which we implemented to solve that problem... shares high level computational principles with what our best understanding of what the brain is doing also a lot of things that are different but it does share those principles and the most parsimonious explanation for how it can do it is that

It's basically drawing on those principles, the same principles, in my view.

Coming on to the alignment thing a little bit, you said that wouldn't it be amazing if we could have an artificial intelligence that would know what was right epistemically and also what is right ethically? One of the things I'm most proud about... in having written this book is so it is now more than, it's a year and a half since I finished writing it. And in the closing chapters, I talk about three things that I'm worried about for the future.

And the three things that I'm worried about that I said I was worried about are still the three things that I'm worried about. So at least that has not kind of gone stale, which is like given the pace of change is not kind of definitely not given. So I think it's quite surprising. Okay, so what are those three things?

So I say, number one, I'm worried about the translation of systems that generate information that allows the user to behave in some way, giving way to systems that... directly behave on the user's path, right? So what we now call agentic AI. We were even calling it then. So I'm worried about that. I'm worried about personalization. So the extent to which models...

instead of satisfying kind of like some general collective sense of what is right, can be tailored to everyone's individual sense of what is right. You know, if you're an individual who like, you know...

has a set of beliefs and preferences that you're quite attached to, that sounds like quite a nice idea. But until you think about, there are an awful lot of people out in the world who have beliefs and preferences that you definitely, definitely wouldn't want reinforcing, and you think that personalized AI, that's exactly what it would do.

So if you take agentic systems and personalized systems and you put them together and you imagine what deployment looks like, what it looks like is a vision that we've been... has the companies have been talking about for several years now which is personal ai right so everyone has personal ai and it is a medium through which they interact with the world takes actions on their behalf probably like you know it is a it is a conduit

for information, resources, and like offers a layer of protection and so on. So what that really cashes out as is a world in which there is kind of like a... There is a sort of social economy amongst humans, but there is also a parallel social economy amongst the agents that we have and use to interact with the world. And that might sound kind of a bit sci-fi, but actually I don't think it's all that sci-fi.

It's really not all that weird to imagine that we will interact with the world in a way that is technologically mediated, because that's what we do already. Almost everything we do is technologically mediated.

weird to imagine that the technologies that we use to interact with the world instead of being rule-based like they mostly are now will be optimization based they'll have like minimal forms of agency it's like why not so you create this kind of like multi-agent parallel kind of you know if you like it's like a it's like almost like a culture you can think of it as a culture and the trouble is that we know that when you get you know lots of kind of um if you if you build a system

and that system is complex and it can interact in complex ways, then you get complex system effects. And, you know, it can be non-linear and it has weird dynamics and can have feedback loops and so on. And that's exactly what happened in that flash crash. And actually, there's been maybe, I don't know, dozens of flash crashes. The most famous one was in 2011, the one that I talk about in the book. So you can think about, like, what are the complex system dynamics that emerge?

where we are all kind of like represented by AI. And the reason why I think we should worry about that is because like, you know, we have... You can think of... the norms that we've evolved socially and culturally as a set of principles that curtail those complex system dynamics. So we have evolved in such a way that... You know, we generate, we have a set of predispositions which generate a set of kind of like constraints on our social interaction.

that stop to a large extent those runaway processes. They're not perfect. Sometimes we go to war. Sometimes like crazy stuff happens. But like for the most part, you know, we particularly, you know, reasonably small groups. long periods, we can live in relatively stable, harmonious societies. But the trouble is that the models won't have those norms, right? Or at least there's no reason why they should have them.

And the question is, what are the constraints that prevent the same sort of weird runaway dynamics that might lead to like, you know, flash crash like events? And I don't think we have an answer to that. That's why it worries me. Yes, and designing in constraints would actually limit the technology in quite a strong way. It's a really interesting thing to think about, though, because in the physical world... the constraints are quite strict. And then language is a kind of virtual.

organism that supervenes on us has more degrees of freedom and this new type of ai technology that we're inventing arguably as you say has even more degrees of freedom so constraining it is is it is a real challenge yeah absolutely and i think you know kind of like we that The sheer, even if you had systems which were perfectly aligned, which of course is not an assumption any of us reasonably can make, but if you did, the sheer...

pace and volume of activity that AI can generate is not something that our systems are prepared for, right? So most systems operate under the assumption that there are like reasonable frictions.

that prevent the system from collapsing so a good example is like the legal system right so you know many people know that it is possible particularly depends on the jurisdiction but it's pretty much possible to engage in what is often called lawfare, so adversarial use of kind of spurious legal challenge.

And there are certain jurisdictions where it's just strictly optimal to do that because the cost of defending yourself is so high that people will just capitulate and you can make money, right? There are frictions that prevent...

Like most people from doing that, right? It's like most people don't have legal training. Most people, you know, kind of like don't know how to do that. Most people don't know the grounds in which you could do it. It's a lot of work. You've got to file paperwork. You know, you've got to, there's domain-specific knowledge that you need. if we remove those frictions so that you can just like with a few sentences say please do this and you have a system that goes and does it

then you suddenly live in a very different world because lots and lots of people can do this. There are many, many other such examples. I was speaking with Conor Leahy about this, and he was talking about this phenomenon called the fog of war. which is that we slowly lose control through illegibility. So I just, you can imagine based on what you said that you have all of these agents and they have, even, you know, when...

when a country is invaded or when some geopolitical event happened, the average person doesn't understand why that is because it's the culmination of so many countervailing forces and these systems are just very complex to understand. So you can imagine a world that becomes so abstract.

And I also wanted to point out that this doesn't require, because, you know, some people think of AI as a cultural technology, a bit like a library or something like that. And then there's this almost Duma narrative that it's agentic and this, that, and the other. but you don't need it to be strong for all of these things to happen. And so the human analogy in AGI, of course, overlooks the fact that although collectively...

what we've done is astonishing. Individually, we're actually extraordinarily vulnerable and just not all that good at life in general on our own, right? So, you know, kind of the classic, like, you know, you and a chimp on a desert island. my money's on the chin right so you know kind of like that is we our strength is our ability to cooperate individually we are not all that strong right so you know kind of like this notion of

like a lone intelligence that is like us, but much, much better, I think is kind of a strange one. Like what we should actually worry about is... the unexpected externalities that come from linking together lots of potentially weak systems to create something which is probably completely unlike us and unlike our culture and society, but which we can't control. and i didn't know the fog of war analogy that's very nice but my favorite paper which talked about this recently

is from David Duvenor. So he's written this really nice paper, and others, written this really nice paper called Gradual Disempowerment. And it expresses a threat model, which I have subscribed to for a really long time, and which I talk about in the book, which is, you know, kind of broadly exactly that, that we sort of gradually... lock ourselves in to the use of optimization-based technologies. And the complex system interactions between those systems sort of write us out of the equation.

The interesting thing about that analogy, which is this point is not made either in the book or in David's paper, is that in a way it's coextensive with what?

happens anyway right so if you think about a corporation right the the world we have created through like you know through hegemonic capitalism with like large corporations for example in many ways large corporations they are things that are more powerful than any one person in a way that they run under their own imperatives and with their own rules and with their own incentives and with their own dynamics.

For many, they are so powerful that there is no one person that could kind of stop them. So we sort of have a model for what this would look like. It's just that, of course, in the case of large complex systems like the corporation... The interactions are slow because they're largely human-mediated. It's like email, you know, or Slack. And, you know, kind of like the dynamic, everything is kind of like humans are the cogs in the wheel. Oh, the cogs in the machine, sorry.

Yes. But in the case of AI, it's going to happen at warp speed, right? I suppose it's an interesting time because, I mean, just look at AlphaFold, for example. This technology can be used for revolutionizing science, potentially. But there are so many downsides as well, potentially. I mean, what downsides do you think we need to be most cautious about socially?

many how many products work so of course you know kind of like firms advertise products and um they do so by by branding those products right so branding is a kind of It's a way of trying to get us to engage with something a bit like as if it was a human, right? Where that something is, maybe it's not the product itself, maybe it's the company, it's the brand.

And, you know, that is more or less successful. Imagine a world in which everything was like that, but it could actually talk back to you and it could... all of the kind of social and emotional types of interaction that we have with people that we care about. So, you know, the milk in your fridge is like your best friend, right? This is a very strange world.

in which, you know, of course, that's a silly example. The milk in the fridge is never going to be your best friend. But, you know, kind of like there are, you know, as I mentioned earlier, there are already... large numbers of people who are engaging with AI in ways that mimic the sorts of interactions they have with other people.

This creates a whole bunch of vulnerabilities. A lot of people have talked about risks to mental health and so on, and we should be really aware of that, especially where vulnerable people or minors are concerned. But I think there's another issue which is talked about much less, and that is the... The degree to which that will give the organizations that build these systems power over people. You talked about, you said that AI increases our agency. And I would actually like...

In a way, that is true, but I actually think that there's also a really powerful sense in which the opposite is true, right? Just as, you know, kind of like access to social media gives you, in theory... access to lots and lots of information and, you know, kind of like that should be empowering. Actually, most people's practical experience of it is that they spend a lot of time doing something that they think is a bit stupid and would rather be doing something else.

Yes, I'm glad you brought that up. It's a weird phenomenon. This comes into the labour market disruptions. I think initially for some people, certainly now if you fire up Cursor and you can build a software business in a week, in that sense it increases your agency. Everybody else has this capability. And the long term or even the medium term is it sequesters your agency. It takes your agency away massively. And this is a huge problem. Yeah. I mean, I think that this is, you know.

This is true. This is not a unique problem to AI, right? You know, kind of you could see, like, the trend of increasing organization in society, right, in a way... it liberates us in lots of ways, right? The things that you can do in a society which is organized collectively are much greater than the things you can do on your own, right? Your opportunity is enormously increased, but at the same time...

in order for that society to function, it has to curtail what you can do. And I think you can see technology as the natural culmination of that process. Technology gives us freedoms which we wouldn't otherwise have.

do things we wouldn't otherwise do. And of course there's an imperative to seize those opportunities, not least because they're usually economically, you know, really beneficial. But it's like a kind of... faustian pact right when you buy into that you lock yourself into the use of that technology and you know

That plays out in all sorts of like trivial ways. Like how good are you at making a fire from scratch without a box of matches? I don't know about you, but I wouldn't be able to do it, right? I'm locked in. It's like a basic thing that we need to survive and I couldn't do it, right?

And I wonder whether this would be ameliorated when we have a more diffused, distributed AI. But, you know, you were kind of alluding to Daniel Dennett's counterfeit people article in The Atlantic. And I was lucky enough to interview him about that before he died.

And, you know, he was basically saying that when everyone starts talking, you mentioned the milk, but maybe the milk in the fridge will be an agent. Everything will be an agent. And unfortunately... we start to see this weird behavior when we see any kind of interaction online.

And it's so-called counterfeit people. And we acquiesce because we just stop participating because the world suddenly looks very strange to us. But I also see the opposite, which is that we become counterfeit people. So if you look at the way people... behave on on linkedin and social media now it's becoming far more robotic and it's just almost the the meaning of the entire system seems to be eroding yeah yeah absolutely there is like i mean you know

You could call it like a crisis of authenticity, right? And I think you can see this broadly in society, like, you know, kind of because our modes of interaction become so stylized. that we we lose that sense of authenticity right there are so many dependencies you know we always have to present ourselves as being like you know kind of in line with the party line now you just asked me something that i wasn't able to answer right you know kind of like

Because I have other dependencies. Like there is a loss of authenticity in our communications because in a complex world we represent many interests. And... that I think is a natural byproduct of our kind of like becoming part of the system. I love this. My favorite metaphor for this is...

You know, I woke up in the middle of the night and it just hit me one night, which is, I don't know if you remember Superman 3. Superman 3 is a terrible movie, but there's this wonderful scene. So I think there's a kind of giant computer that goes rogue in Superman 3.

And there's this wonderful scene where there's this female character and, you know, she's sort of, the machine is just kind of like waking up and she tries to, she's just sort of walking past it and the machine kind of like sucks her in and she... She gets kind of like stuck there. And then what the machine does is it gradually like kind of puts armor plating on her and replaces her eyes with lasers and basically turns her into a sort of like automaton. And...

It's a very compelling scene. I think I was like terrified by it as a child, which is probably why I remember it. But like, that is a sort of metaphor for like, you know, what is happening to us, right? You know, we're worried about... the robots taking over or whatever. But in a way, it's more like us being sucked into the machine, right? We become part, just like that poor, you know, character. We get turned into something we are not.

by technology. And I don't think, this is not a comment that is specifically about AI. I think this happens to every person who has to go to a press conference or every person who has to kind of like represent. you know, their organization or a broader group of people. You become part of that system and it erodes your authenticity and in a way it erodes your humanity. Yes.

Last time I came to interview you, I went to Luciano Floridi directly afterwards. And his argument is kind of similar about us becoming ensconced into the infosphere and it changes our ontology.

Perhaps you're arguing more from an agential point of view, but I think it's quite related. Well, I think as a psychologist, we have dramatically under-indexed on the extent to which what is... good for us is actually about our agency, our control, and not about reward.

So, you know, kind of like we have, of course, you know, economics, psychology, machine learning have all grown up with this notion that like utility maximization is like the fundamental framework for understanding behavior. And that's expressed, of course, most prominently in ML through reinforcement learning.

But, you know, kind of like, of course, like when you actually look at, and of course, you know, this is not, everyone needs to be warm and have enough to eat, right? But once those basic needs are satisfied, like if you look, and even sometimes when they're not. If you look in development and if you take a sideways view at a lot of kind of both healthy and abnormal psychology, what you can see is that what people really care about is control.

People need to understand, and by control, I really mean formally, like, your ability to have predictable influence on a system. So in machine learning, this often gets quantified as, like, this wonderful notion of empowerment, right? The idea that what we want to maximize is the mutual information between our actions and kind of future states, for example, either immediate or too late. And...

That is agency, by the way. And that is agency. And I think that we, you know, kind of we really, really, you know, if you think of like kids, you know, kids, just two examples, you know, the extent to which kids will explore the world. The extent to which they will take actions to try and understand, like, why if I tap that thing? Or why if I take my dinner and throw it on the floor? What's going to happen? Oh, look, I have control. Or I cry. Oh, look, my dad's going to come.

Like, I have control. I can understand that system. Like, that's what they're doing, right? Right through to, you know, kind of like, you know, in adulthood or adolescence adulthood forms of pathological control. Too much control, like, you know, you can see OCD, obsessive compulsive disorder, is a need for too much control. So anyway, I digress. But, like, control is really, really important. And I think we...

When thinking about the impact of technology on our well-being, that conversation needs to be grounded in a robust understanding of how important it is to us to have a... predictable influence on our world. And what a lot of AI or a lot of technological penetration actually does is make our actions kind of unpredictable.

It's like that. This is the frustration that happens whenever you interact with a website that doesn't quite work. Or, you know, you get a computer says no answer or, you know, two factor authentication, but then there's no Internet. And you're like, ah, it's like you've lost. But that control in the systems that we evolved, the environment that we evolved for, that control is much more readily available.

Isn't that fascinating? There have been studies, I'm sure you're familiar with this one, where managers in an organization, this study was in the 70s or something like that, and they had fewer rates of heart disease because they had more power. And then the underlings. would would get disease much more regularly and if you think about it with social media and even with these chatbot platforms so this this um i interviewed the team that built all the

engagement hacking algorithms. And they were incredibly proud that their average session length was 90 minutes. And they were talking all about how they would do model merging and send this response and this response and keep them hooked, keep them there for longer. And in a sense, that's... Dopamine hacking is about giving random rewards, right? And it's a disempowering thing along the lines you said. And that is the kind of modus operandi for all technology now.

Yeah, absolutely. Yeah, a variable reinforcement schedule is the best way to train animals, including humans, and we are susceptible. The unpredictable nature of the reward... engages us with the system and what makes us come, because we want to control the system, right? We want to know, how do I make the reward come? And of course, if you can't, then you keep on trying and trying and trying. Yeah, I mean, you know.

We live in a world in which people have a lot of liberty about how they spend their time. And I think that's as it should be. I don't, you know, I don't think we should, I don't think we should legislate against frivolity, right? You know, if people want to spend a lot of time on TikTok, collectively, I understand that that's bad. But, you know, kind of like...

It is also, we, for better or worse, live in a world in which that is permissible. We live in a country, at least, in which that is permissible. Where I think we need to... be cautious. So what I'm saying is that that kind of hacking, maybe it's undesirable. I might deem it undesirable.

But, you know, collectively as a society, there are many things that are undesirable. You know, alcohol is also addictive, but, you know, I'm probably going to have a beer as soon as this is done. Right. So, you know, kind of that we make those choices. But I think that there are. vulnerabilities there are people who are uniquely vulnerable where that kind of liberty to kind of hack if you like spills over into something that can be really actively harmful and can lead of course to

for people to self-harm. And there have been tragic cases as well, as I'm sure you already, as I'm sure you know, in which people have even taken their own life under influence, which came from... an ai system with which they were interacting in this kind of like companion mode do you think it makes sense to think of evolution as having a goal um probably not right so there's this great you know kind of like This is a great way of thinking about a paper that I really like, which...

draws upon the analogy of the kind of like blind progress of evolution right that you know kind of it's a selection mechanism that is not teleological right it doesn't have a purpose it just happens And, you know, kind of argues that, you know, kind of we should think of kind of evolution and training in neural networks in a similar way. Right. It's kind of like it's very blind. And, you know, kind of like I think that, yeah, that I think that there is a fundamental difference between evolution.

and the way that optimization happens, put it that way. And we could learn a lot in thinking about neural networks from thinking about the purposeless optimization that happens in... evolution, basically. It's a really interesting topic for me. I was speaking with Kenneth Stanley the other day, and he's done a lot of work about open-endedness. And of course, Tim Rock Tashill works with DeepMind. In Tim Rock Tashill's paper with Edward Hughes... Yeah, yeah, the open-endedness paper.

An open-ended system is one from the perspective of an observer that produces a sequence of events which are learnable and novel. Yes, exactly. It's about learnability, isn't it? Yeah, Joel Lehman. Have you had Joel Lehman on the show? Yeah, so Joel has written really, really nicely about this. I mean... Yeah, so I largely share his view and very close to Tim and Ed's view, which is that, yeah, so the world is open ended and optimizing for open ended systems.

using kind of like well-specified kind of narrow optimization towards a narrow goal is just doomed to failure right and there is probably something really deep about the way the purposeless selection that happens in evolution conferring robustness because it doesn't precisely optimize for this narrow goal but rather what it creates is this like astonishing heterogeneity right and

The optimization algorithms that we use are all completely opposite, right? They are basically tailored for homogeneity. Like, heterogeneity is a bug. And that's why LLMs show mode collapse. It's why... you know, kind of like you get this kind of like, you know, this platonic hypothesis, you know, the idea that we're gradually converging towards kind of like essentially one common shared set of representations, right? It's like...

Yeah. Evolution doesn't do that. Kenneth wrote this wonderful paper called The Fractured Entangled Representation Hypothesis. Oh, I don't know that paper.

with um with joel i'm not sure if joel was was part of this but he was on why greatness cannot be planned they did this thing called pick breeder and that was like flicker where it was supervised by a diverse source of humans and the humans could pick interesting image generators which were cppn compositional pattern producing networks and you could create this phylogeny and they speak about this concept called deception

which is that the stepping stones that lead somewhere interesting don't resemble the interesting thing. So humans have this kind of idea of what's interesting because we seem to know the world well. With a few steps in the phylogeny, they found these pictures of butterflies and apples. And when you do parameter sweeps on the networks...

Because they so abstractly understand the objects, the apple would actually get bigger. One neuron would make it bigger. One neuron would make the stem swing. And if you train a neuron network... with stochastic gradient descent to do the same thing, and you do parameter sweeps, it's like spaghetti all over the place. So their hypothesis, and this seems like an obvious thing to say, if we could have a sparse representation...

which mirrored the world, then the creative leaps because the knowledge is evolvable. We could trust it with autonomy because it would do the right thing. Yeah, that's amazing. So I don't know about this paper. It sounds like I should read it. I mean, the idea that, yeah. that it's difficult to get places because the interim states are not highly valuable. I mean, I guess this is like, you know, kind of...

This is a very old argument. This is the basis of like Paley's watchmaker argument, right? It's like, how did we ever get the eye? You couldn't possibly evolve that. It's just too complicated. But yeah, those gradients must be there, right? The gradients are there. I have to say, Professor Summerfield, the prose, the way that you've written this book is very impressive to me. It's one of the best written pieces of writing I've ever seen. And it occurred to me whether you were deliberately...

making it so creative as for it to be impossible to be mistaken for ai generated content um i don't know whether this is maybe i've you know My standards are so low now because, you know, I mean, shittification and all of that. But it was remarkable. But what were you thinking now? Were you sort of like leaning into the creativity a little bit? I love to write. I love to find new ways to... explain things to convey ideas so that's a it's for me it's a selfish pleasure it didn't cross my mind

That people might think that I had used ChatGPT to write the book, but I guess in hindsight, that's a kind of very sensible way of thinking about it. But no, yeah, it was all me. That is mind-blowing. It's been an absolute honour. Thank you so much. Thank you.