Who can I talk to? Can anyone relate to what I'm going through? Who else understands what it's like to have cancer? Sometimes you don't want to talk about cancer with people you know. You just want to talk with someone who knows how you feel. Open 24-7, Macmillan Cancer Support's online community is a safe space to connect with people who understand what it's like to have cancer. It's free to use and anonymous. From day one, whatever you need to get off your chest...
Community members are available day and night to offer support on the online forum. Talk openly and honestly with people who've been there too. You're not in this alone. We're here to help. To join Search Macmillan Online Community. Hello, everyone. Welcome to the Mindscape Podcast. I'm your host, Sean Carroll. Something I've figured out after doing science for quite a long time is that science is hard. It is hard to do it. It is hard to do it well.
That's probably not really news. That's not probably something that is very controversial out there. But I wonder if people appreciate the ways in which it is hard. I mean, of course, the actual doing of the science can be hard, right? Doing a great experiment. a difficult calculation, if you're a theorist or something like that. But there are all these pre-existing difficulties that come long before when you get to that, mostly like, what problem do you choose to work on?
What is a promising area of research, right? Resources are finite. The time that you have as a scientist is finite. The money that you have, the students, whatever. trying to not only make a big discovery, but also ensure that there are future discoveries to be made.
which is the nice way of saying you got to get a job. You have to impress the people who might be hiring you or promoting you or whatever enough that you will be supported in your aspiration to keep doing science. You want to be... new and innovative but you don't want to be so crazy and out there that people don't pay attention to you and even once you've chosen what to work on
How exactly do you work on it? Do you sit by yourself, trying to think very, very hard? You know, there's people like Paul Dirac, who very famously thought that all of the good papers ever written in science, all the good science ever done, was done. by a single individual all by themselves.
But the world is very different than in Dirac's time right now. There are big collaborations experimentally, of course, but even within theorists who are very nimble and can change what they're doing from month to month, it's so easy to communicate with people.
Over the internet or in person, you can start up collaborations with people who don't know the same things you know and might be helpful to you. So who is your set of collaborators and co-authors and helpers? These are all great questions. And there... Next to, but separate from, questions about is science really making the progress that it could be making? Is it moving fast enough? Is it daring enough and bold enough? Or is it stodgy and set in its ways?
leaving food on the table, as it were. So of course, lots of people have opinions about these things. These are issues that many people have talked about for many years. The difference now is we can really collect data. in a more or less objective, at least wide-ranging way, rather than just sort of looking at this or that key. feature in the history of science or key episode, we can actually look at many, many scientific investigations, many people, many researchers writing papers.
Who are they collaborating with? Are they making advances because they're collaborating with the same people or with a different set of people every time, with people in their field, outside the field? Do old people make the advances because they're very wise or do young people make them?
because they're not restricted by the conventional wisdom? These are all great questions, but don't try to answer them just by thinking and by using your intuition. Look at the data. That's what today's guest likes to do. James Evans is technically a sociologist. At least, you know, he's in the sociology department at the University of Chicago, but he does wide-ranging research on computer science topics and all sorts of sociological topics, but in particular, the science of science.
understanding how scientific progress gets made and thinking of it as... an example of collective intelligence, right? Science is not just one person. There's lots of scientists doing things. So how do these scientists come together in groups? Where do the groups come from? How big are the groups? What kind of groups make the best?
decisions moving forward and so forth. And I think that my prediction is that if you're someone who's thought about these things for a little while, what James has to say will... in some ways, reinforce things you already thought, and in other ways, surprise you a little bit. These are tricky questions. That's why data is important. That's why doing the science of science is important, in addition to doing the history of science.
the philosophy of science, the sociology of science, the psychology of science, et cetera. All of these are important. These days we can do the science of science at an unprecedented level, and we're learning new things, both about how science works, how it can work, how changes like AI are going to help it and hurt it and things like that. So the world is changing. We've got to keep up. with it. Let's go. James Evans, welcome to the Mindscape podcast. Thank you. I'm delighted to be here, Sean.
You know, I have to start by saying a couple of months ago, we had Blaze Aguera Iarkas on the podcast talking about the spontaneous emergence of computational life. And it took me a while to ping on the fact that you are a co-author. author on that paper, even though there's no sociology involved in that paper. So how did that happen?
I'm kind of a computational scientist. That's the way I think of myself. Or even really a meta-scientist. I mean, I study how science works as systems. And a lot of that comes through an evolutionary lens. And so I've been involved. in following Alife for a while. I'm actually, I work with Blaze on his team, his Paradigms of Intelligence team at Google. And there are kind of three levels there where we're looking at kind of the ways in which... uh you know biological um
kind of forms, processes, and organisms shape the way in which we think about AI. And one part is evolution. One part is kind of neuroscience and cognition. And then the third part is collective intelligence. And so I spend even more more time kind of helping to cultivate and catalyze work in this space of using principles of collective intelligence and diversity underlying collective cognitive populations to think about building AI in that space.
Well, I think it's a good place to start because it's like a personal spin on the more big picture substantive questions we'll be getting to later. But, you know, someone like you who... researches topics like this, sometimes it can be hard to find a department to hire you, right? Like, so what do you do really, right? You must have gotten that question before. Right, right. It's true that right now I'm in. So I think part of it has been that I've.
been involved in creating a fair number of departments. So I began my home in sociology. And I think some of my colleagues think some of my work looks like sociology, especially at the work where I'm kind of looking about how it is that scientists and.
people in the world organized around information. Uh, but a lot of it doesn't look like that. I'm studying the complex systems underlying, you know, deep neural networks. I'm, I'm understanding, you know, so I think that, um, so I've, I've spent a lot of time. I've built the computational social. program that's been increasing where we built a data science institute and now kind of department at the University of Chicago. We're in the process of trying to create a new college or division.
about artificial intelligence here. So I think that not having a fixed home also creates an instability that allows me to create new things, which is fun. But yeah, I mean, I completely agree. And I'm exactly in a very similar position myself. But the journey to get there is tricky. And I mean, maybe one thing we'll talk about along the way is how do we sort of get... make on-ramps for young people who want to do that. Like, it seems when people ask me for advice...
My advice is always, like, be pretty conventional early on in your career because there's a lot of gatekeeping, a lot of bottlenecks you have to jump through. It's hard to be too quirky as too young an unproven person. Yeah, I would say the one caveat is, I would say, you have to be able to present conventional.
Early on, which is to say, I think if you just do conventional early on, you will likely do conventional forever. But if you have a portfolio that you're cultivating enough of which you can. purpose to present conventional leadership then. then that's, you know, and I, yeah, and I had to do that in the context of sociology, even though I would consider myself kind of a more multidisciplinary, post-disciplinary scientist. Yeah, no, that's a very good amendment. So when we are thinking about...
science and how it works. This is one of the things that you do. Just to get a feeling, obviously, this is something that people have talked about for many, many years. A lot of it has been in the context of history or philosophy of science. Are we doing it... today in a more data-driven way? Yeah, I mean, we have, and I think what's interesting both in science in general, and I would say, you know, in artificial intelligence models is
Their performance, their predictive possibility, the ability to create generative digital doubles is all driven by massive data. And so I think that's a phase transition. It's not just like, oh, you know, we didn't have data. And we had some science and we improved a little bit by increasing some data. I mean, it's just the difference between systems that work, that predict, and those that do not, you know. And so, yeah, I think we're doing it with large scale data.
Because scientists are incentivized to leave droppings, which is to say our capital is our credit in the system. we leave more droppings than people in business, you know, in an other context where people are covering their tracks. You know, it's really hard to tell stories about how action in those places, micro action, personal action decisions, but in science.
Because we have to broadcast every step to the world because we're certified and that's how we get promoted. And it provides a rich landscape for thinking about collective knowledge and individual contributions. That is a great point. My wife is a journalist, a science writer, and one thing you notice there is that scientists, when they give you an interview or whatever, they really care about getting credit.
in the news article that's being written. And not only for themselves, but like, no, you must list all eight authors of our paper in your 1,000-word article. And the journalists are like, that's just not how we roll. But it's very, very important for academics. broadly, right?
Right. And this is our, this is, this is the coin of the realm, you know, is you contributed to this paper and we can, we can basically say all the papers that cited your paper kind of get back propagated to your, you know, uh, enlarged. avatar of influence. So as a scientist, I mean, let's try to help the audience, most of whom are not professional scientists, right? One of the things you got to do, forget about getting... jobs or whatever you have to
decide what research to do, right? How to pick a research problem. And this is hard, both because ideas are scarce and things like that, but also there's a style question. Am I going to work on something? quirky and innovative, or am I going to fit in to kind of a bandwagon? Is that something that we can study quantitatively, sociologically?
Yeah, I've been studying this question or variants of this question for 20 years for two reasons. One, we want to just understand how it is that science happens, how people make decisions, and this is a valued quantity of life. On the other hand, you know, economic growth happens through scientific advance. You know, I mean, this is, I mean, credit, you know, like financial credit is given because we assume that some magic will happen.
When we give this money to a corporation that they will reorganize things in a new way and new values will be generated. Like credit in the ye olde days was extractive. I'm going to give you money and I'm going to take more money from you. It's just so I think as a result, then it becomes really important for nations, which are typically the units that.
bankroll science like countries bankroll science there is international science people experience it but where do they get their funds they get their funds from their country most countries all universities are are effectively public in the U.S., even though we have quote-unquote private and public universities, almost all science research funding is, well, increasingly it's from private organizations, private and public, but certainly a vast...
contribution and the most focal element is from public contribution. So I think part of this is like, how do we think about if the country as a whole is relying on its competitive power, its ability to kind of generate new markets and services and new values for life, then they need. collectively to organize people to start exploring different kinds of things, or they're just going to, they're not going to get any return in their investment. And so.
So I think it's a big question, even though it's a micro question, but it has really collective implications for countries. And so it's kind of an existential question for countries. So, so yeah, so one of the ways in which we've studied it is, is looking precisely at, you know, what it is that people are engaged with relative to the distribution of prior work. And this is where big data matters, right?
Because think about it, you know, like if you were to kind of take, if you want the modal story, the average story. you don't need much data you know you can take a very small sample of a few scientists if you want to tell a story about change between one period and another, you need at least twice as much data, right? You know, to difference between one period and the next. To tell a story about innovation.
requires exponentially more data because you have to understand everything that was expected to understand the significance of something that was not expected. You need to understand, in theory, everything that was expected by any scientist. in the space to understand really what's new relative to that space. So I would say innovation is something that you can only study with large-scale data. It doesn't even make sense.
to study without it. You need the full distribution of expectations that normal scientists would have so that we can understand when and why they're surprised and how. their world of technological and scientific possibilities updates. And so we've been studying this initially. We looked at this in the perspective of networks where individual concepts or technological components, you know, a method.
or a gene, you know, or, you know, even a theoretical concept were kind of nodes in a complex network and we would look at how those networks evolved over time, the difference between bridging different parts. the network, the likelihood of bringing a new node into the network. But increasingly, with the emergence of artificial intelligence, natural language processing, and now large language models, Inside those large language models is a very thick.
deep representation of meanings. So for example, in the new ChatGPT models, there are trillions of parameters. And all those parameters are numbers inside of a graph. that represent a complex representation of all the meanings associated with... with our language, that these language models have, they're kind of the love child between, on the one hand, what I'll call the auto-aggressive language model, where we predict the next word for prior words, and these...
And these basically kind of meaning pixels, which is to say, rather than selecting on words, predicting the next word, we're really predicting the next meaning. And it turns out that the moment they started doing that, all of a sudden these models became much more stable because it's not about the particular fragile word that you're using. It's that I'm selecting this meaningful by that meaningful.
followed by this meaning, followed by this kind of word function, followed by, and all of a sudden, these models worked. And of course, you know, we added many parameters so that they could self-learn important features. So we use those internal representations as a map.
of the cultural world and we can look at the difference between past for example research and future research or we can plot every piece of research in the past and we can plot a new piece of research and say exactly how surprising is it relative to that past. relative to the past model. And in fact, I just got a $20 million grant from the government to basically build
what I call chronologically trained language models. We're building these models based on the past so that we can evaluate when a new paper or a new patent or a new... product emerges, how surprising or improbable was that idea to the to this model, which is a machine for probability. It's predicting what the most probable next patent paper or product description is. And when really surprising things happen, what's interesting with these models, we can see
say not just, oh, that's surprising. Everyone knows it's surprising. We can say, well, if this improbable thing is actually now probable, right? If like no one would have thought of this. this paper, but it turns out if you write this paper, you do the research behind this paper, it's true. Then what are all of the other things that were improbable before that become probable now? What are all the adjacencies that were distant before?
now we're all of a sudden close to one another inside the space, which allows things like the government to take fast breaks. which they don't do historically. If one thing becomes true, it's like, oh, okay, here's 10,000 other hypotheses that were impossible or extremely improbable, but now are absolutely the next thing that you should study inside the space.
But maybe this sounds like a little scary to me. I mean, if we could predict ahead of time what the next innovation would be, why would we have to do boring things like experiments and stuff? Oh, no, no, no. My whole point actually is I'm not building a model that's going to predict the optimal next.
paper i'm saying like the experiment in this space is that someone did an experiment and it surprised this whole cultural world model we had and it allows us to kind of predictively update okay what are how does the whole world change as a result of this surprising thing that came as a result of experiment? Good, okay. So you're not predicting what the surprise will be, you're sort of bringing the changes on the surprise to figure out how everything shifts in response to it.
Exactly. And one other thing that we can do in these models, which I think is exciting, is we can do the same kind of thing that scientists have historically done. Charles Sanders Peirce, who was, I think, really the great 19th century American philosopher, had this idea of abduction. He was unsatisfied with the philosophical cartoons of reasoning.
which were kind of deduction, which kind of comes out of Aristotle's invention of the syllogism, where you kind of build from established axioms or assumptions or facts, and you kind of say, well, if these are true, then what necessarily... must be true what other things must be true that was kind of old style science and then and then uh sir francis bacon wrote this book called Novum Organum, The New World. And it had a picture on the front of a big ship going to America and all this new stuff.
And and it was like, OK, no, we need to scrap that. We need to do induction. We need to basically like learn all the new things in the world and then generalize, you know, from those. It's not going to create something that's provable, but it will. allow us to grow and learn collectively and that was the basis of the scientific revolution and this thing called the experimental philosophy which is the core of what you're saying
And then Charles Sanders Peirce kind of emerges in the end of the 19th century. He was kind of a failed academic, but academics were all failed back then. Tenure was uncertain. He had a funny personality.
bump from place to place he was at johns hopkins for about a year he was that's he was a lot of places his only academic position officially yes And so he had this idea that as knowledge grows, as you learn more about the universe, then your best signal for theoretical development and collective knowledge is surprise.
Which is to say, when you run an experiment or have an observation that violates... your expectations that violate your growingly accurate world model and um and what's what's i mean initially he kind of thought well where do these surprises come from well they come from uh you know he was uh also lived at the birth
psychology and was a kind of a proto-psychologist and so he believed that that the real model was the subconscious you know that you would basically people would dream and associations would be made in their subconscious mind and um and he even has series of stories uh including one about he himself that's very much like sherlock holmes where he is engaged in abduction you know someone steals his watch and he like figures out where they are through this set of
combination of surprises and coming to the best explanation. One of the things that we've studied recently is how does abduction actually occur? How do people identify surprises? Because it turns out that the people who... are most likely to identify a surprise are kind of like the priest class of a scientific field. Like they've read the literature. They know what everyone expects. They're the large language model of the field. And then, um,
And then they see a surprise and they realize, hey, this does not fit in. But that class of person is the least likely to have the intellectual resources to resolve.
the uncertainty to develop a new pattern and so we we show in recent work with large-scale data that on average um that those uh basically kind of you know abductive discoveries um are mergers you know the conversations between insiders and outsiders so increasingly you get your resources not through induction where you just kind of generalize from your mind about you know from patterns, you have a surprise that occurs in your field and you look, you survey a range of other fields.
for the intellectual resources that come from the most disconnected fields from yours, you know, it could be literary studies, it could be astrophysics, it could be molecular biology, people who were... not part of your conversational cultural world and who have access to a set of distinct patterns, you know, through theory, through data that is accessible to them. And that those people are systematically kind of like coming in, in an X.
expedition to solve your problems. That's where disruptive advance occurs. So part of this, in some ways, I'm kind of pushing back against the question of like, oh, it's about people choosing where to go. It's really...
It's about this collective conversation that occurs between problem makers and problem takers inside this market for ideas. I can definitely think of examples in my own fields where some... wonderful, brilliant new idea has come along, typically from a young person, and then some older person who knows everything has sort of fitted into the broader context in a way that suddenly everyone gets excited about.
That's a common pattern. Another pattern are what I find, what I call expeditions, where you have a whole group of people from an outsider field.
physicists coming into molecular biology. And they bring a set of tools, a set of perspectives. And the reason why in those early works, you don't see teams forming between insiders and outsiders is because... teams require a kind of a social contract that this thing makes sense these expeditions are like it's like blitzkrieg you know it's like someone a barbarian coming in you know with an approach and and they publish it in
in this field that's never seen anyone like them before. And no one from their field has ever seen this. or has ever published in this space before. And those tend to be, on average, the big hits, the things that really kind of grow and disrupt the knowledge space and really update how people think about it.
problems in advance it's a tricky thing i guess because i think that especially from an outsider's perspective um people are like well why isn't there more innovation why isn't there more creativity isn't there some ossification of your because you're all working on the same stuff and it's just moving ahead incrementally? Isn't it more fun when someone just completely upsets the apple cart and comes in with new ideas? And it's not that...
that's wrong, but I think you need both, right? I mean, you need some apple carts being upset, but you also need the gradual accumulation of conventional wisdom. Well, exactly. If you didn't have the gradual, I mean, you can imagine a version where it's all disruption all the time, and that is a system with no memory. Yeah. and no accumulation. But it's true that there is an oxymoron in creating innovative institutions.
Right. Because it's like what we need, what we're trying to, we're basically saying, hey, we're going to try to surf the boundary between order and chaos. Very, very hard. And so it's very difficult. I mean, basically, you know, education. systems. I mean, these are control systems. We've studied syllabi, and it turns out the oldest people in the field have the greatest influence on the things that young people learn, and that's a recipe for reproduction and not...
growth. And there's also some worry about bandwagons or bubbles, like something becomes a popular idea and suddenly everyone has to do it. And then a few years later, it turns out... Okay, it wasn't that exciting after all. That's right. Yeah, that's a case where, yeah, there's no memory in the field and everybody jumps to the kind of the hot new thing of a project that we recently worked on when we looked across countries at this style.
of exploration. And we find, for example, when we compare China and the U.S., that individually Chinese scientists, for example, move further across topics over their career. Within any one particular time, China as a whole actually has far fewer topics that are focused on because on average, people are following the trend.
There are many smaller pockets in this space which are effectively autonomous. People are less likely to move over the course of their career, but that becomes a reservoir of diversity. from which abduction becomes collectively possible. I wonder if it's possible to sort of institutionalize and nurture this kind of thing I mean we can valorize innovation but I know just from personal experience if I write a very very boring paper in physics that is nevertheless correct in every
equation, I can get it published, no problem. There's going to be zero worry that the referee is going to reject it. If I try to be interesting and novel, maybe that's going to be more important down the line, but it's also much harder to get it into the journals.
You know, the referee is like, I've never seen this before. I don't know how to deal with it. And likewise, when you're hiring people, like you kind of feel comfortable hiring people who work on the things you work on or whatever. So, you know, what can we do to nurture the... right amount of difference, innovation, thinking outside of the boxes that we've built? Yeah, so I think that it's... You know from my perspective and when we've looked at this you can think of what
an economist might call a single equilibrium solution, which is like, what's the right amount for each person? What's the average amount of the balance of innovation versus institutional? kind of history and tradition for each particular person. But again, systematically, we find that it's a multi-equilibrium solution. It's really about a complex balance of people who are much more likely to be.
the kind of the priests, you know, of the field versus the prophets who were traveling around and coming from outside. So I think one of the challenges is how do we build an ecology that facilitates multiple kinds of career paths that allows people to. And because institutions defend and reproduce themselves, then I think one of the reasons we talk about innovation all the time is because we need to talk about it. And we need to find novel institutions that kind of like resist and push against.
these natural forces of preservation and memory, which, I mean, the entire educational and scholarly enterprise reinforce, you know. journals, method sequences in graduate school. These things, they're run by thought collectives, which were formed. Most of them with titles and names, you know, in some cases, two, three, four hundred years old. Yeah. Right. So these are very old.
associations and they're powerful associations and they run things like the Nobel prize committee and you know, they, they have, you know, These have disproportionate amounts of power. So the reason we're talking about innovation all the time is just because we fail at innovation most of the time. And in fact, one of the things that we were looking at recently, a project I was just writing up this morning, is just, for example, a scientist's age.
Their appetite for change and innovation shifts, as you were suggesting. And one of the things that we find is that, you know, not only so. The average scientist's citation, so this is like the distance between the paper they're publishing and the papers they cite, age at about a month a year.
This is in all fields, you know, over all time, except for the one field whose methods haven't changed in 200 years, mathematics, which aged two months a year. And it turns out everyone's favorite paper was published on Africa. a year before their first paper. And what happens at about year 10 is scientists basically...
We looked at all the citations and all the contexts, and on average, they basically start policing their fields. So they basically start criticizing other papers disproportionately, and almost always young papers. young people's papers, bringing new ideas and new methods from outside the field or the new to the field. So basically there's this, so one of the challenges.
Here, it's like, how do we influence this innovation? It turns out that just the demography of your field dramatically impacts the likelihood of churn of new ideas in the field. And it's not... the reverse causal direction. It's not like, oh, fields are stalling out and so young people stop coming in. It's the opposite. It's, you know, basically if you have a high proportion of old people in your field or even in your department, you can see this at a micro level, then
The field hits rigor mortis and no new ideas come in and new ideas are getting shot down systematically inside the papers that exist. And those new ideas that are getting shot down are in the same context as.
like citation context like paragraph as as the as the citations which are their favorite citations which are on average the paper that was published the year before their first paper right so it's it's like um So I think part of it is like we actually need to, and you see across countries, for example, India has the youngest scientific work.
And there's a lot of innovation there. China has a slightly older but much younger than the U.S. workforce. And there's a lot of innovation of a certain kind there. The U.S. Scientific workforce is getting older. Japan's scientific workforce is getting even older. And you can see the reflection of those. field level within country environments dramatically impacting the likelihood that new ideas will enter and thrive.
So one of the ways we manage this is actually by managing the demography. In the U.S., the Supreme Court made it illegal. for tenure-track professors and institutions to have an age cap. for retirement. So it's, you know, became illegal for ageism. And we can show that in 1994, when that happened, there was just a linear increase in the
age of citations on average in field and a decrease in the associated churn of ideas within those fields. So I think that's one of the ways we have to think about the big picture, managing the whole... the whole gene pool of ideas.
It is interesting because I've noticed that the best popular music is what I was listening to as a teenager. I think that's just an objective fact, right? Of course. Yeah, yeah. And I think we're probably about the same era. So yeah, the 80s danceable music. I mean, that was like, where is that? And am I remembering correctly these studies of where funding dollars go these days or where prizes are awarded? At least in the U.S., they are going to older people than they used to.
Absolutely. Yeah, there's a linear increase in the age of first NIH grant, for example, that people experience. Prizes, on average, are highly... conservative. Which is to say, you know, and the reason is because they're given by context. You know, they're given by the physics community or by, you know, they're given by associations like which systematically.
undervalue work that violates the boundaries between those contexts. So the things that are the most disruptive in citation that attract the most... kind of burst, the biggest bursts of attention are not the same things on average that are getting prizes, the Nobel prizes, et cetera. Yeah. So back to the individual scientists. I know you've written one paper that was kind of provocative to me about...
why scientists disagree with each other. I mean, the thing about science is, of course, you've established some things as reliable, but then you're also speculating about what's going to happen next, what is interesting, where to go, and so forth. It's remarkable how people can be really devoted and even passionate about insisting that... It's going to be going a certain way. Or I guess the way that I like to say it that is most generous is every approach has its looming obstacles.
And every advocate of every approach says, oh, but our obstacles will be overcome. I can see that. Whereas your obstacles are absolutely going to make you stuck. Where does that come from? It's not purely objective. Is it like... personality? Are there some cognitive traits that individual scientists have that get in the way? Yeah, I think there are many forces. I mean, so one are the paper that you're referring to that's forthcoming in nature.
Human behavior is one where we basically look at deep dispositional traits. So we look at psychologists. We run the same kind of psychological profiles on psychologists that they run on. on college students around the world. And it turns out that there are really stable and strong associations between their relative acceptance of ambiguity and whether or not they're interested in, for example, like...
multimodal findings, you know, that aren't just supported by, for example, you know, a certain kind of statistical or mathematical model, you know, if they can. So, so there's definitely, um, the, the, the, You know, the background proclivities, which shape the relative likelihood of whether or not something's attended to. I would say another approach really is the zeitgeist of an age, of a moment. There's one paper.
that I love. It's by a guy named Paul Forman in 1971. It's a book-length paper called Weimar Culture, Causality, and Quantum Theory. It basically kind of looks... I remember that. It's a beautiful paper. It basically kind of shows, it tells a story and it shows that, okay, we've got a whole host of... of people in the wake, in the Weimar Republic, in the wake of the loss of World War I by Germany who kind of are against causality.
as an approach. And so philosophers are kind of writing against this nihilism as emerging in the artistic space. And it turns out that there are some physicists who were... reading this philosophy who were writing philosophical tracks against causality and those were the ones who embraced the kind of the quantum revolution which is arguably an acausal physics and so it's kind of like
You know, this moment, this, you know, this kind of post-war depression, malaise, you know, anti-causal enthusiasm allowed the... relative selection of what was otherwise epistemically very... unpopular and very unsatisfying from the ways in which so you know epistemic standards in science are are are the standards by which we credit something as knowledge right if if we've got standards and that that that crediting
There's some strict criteria. Oh, you've got to, you know, we have to demonstrate some empirical accuracy, maybe through experiments or through observations or for quasi, you know, synthetic experiments. But there are a host of other hidden epistemics. standards. And those hidden standards come from what is familiar to us, what we associate with events, what's beautiful, what's simple. So we have all these deep preferences about what science should like. And so when science shows up...
that looks like that, then we are ready to give it awards. We're ready to credit it with advance. And if it doesn't look like that, then it's often hard. And one of the reasons I'm interested in studying that is because I really am interested in... collective advance and the way in which science is and can be an engine. And that means we need to basically critically observe the impact of these kinds of hidden epistemic standards on unlocking. in a certain kind of scientific slow.
gradual progress. It's a great point because I think I came across that paper you're referring to when I was an undergraduate, and I was very dismissive of it because, you know, quantum mechanics works because it fits the data in some sense. And I wanted to say, like, who cares about...
the philosophical predispositions in post-Weimar Germany. But now, in my twilight years, when I'm more sophisticated, I can absolutely see that there's certain ideas you're open to, certain speed at which you're willing to change your mind about things which will be affected by the wider world and I'm actually in retrospect super impressed with the scientists of the 20s that they were so quick to embrace
race, this very, very different view of the world. And no doubt, the wider context had a lot to do with that. Right. Yeah. And I think one of the reasons we can study this is just to identify it. But I think another generative and constructive way in which we can use this. So I spend a lot of time talking with science funders and advice.
science funders in the US and in Europe and in China and elsewhere. And, and so, you know, part of this is, is the ability to do what I'll care, you know, it's science fiction. So if we didn't have a predisposition against this approach, then let us now turn the crank and see what other kinds of approaches would... or could have been or could now be probable. So if this is the flow of the scientific river, then what are other tributaries that...
Let's just imagine that some of the decisions that were made to pursue or not pursue areas were not rational. Let's just assume that, I mean, maybe many of them were, obviously, you know, science moves forward. But let's just assume that some of them have to do with these complexities of history.
Now we can basically build, we can use these models, large network models are now, you know, kind of like tuned large language models to basically run science fictions. And my group spends a lot of time now creating conferences that never have. never could exist, generating paper topics, patents, proposals that are...
The people don't exist in the population of our worlds who have created those. But if we had a different structure of education system, then these proposals would be very likely to exist. And this allows us to think about like, you know. With an expanded view, you know, what are places we could...
pivot to? What is the field of action that we can engage in? So I think of this as not just an act of humanities, but an act of speculation in the most generative tradition of speculation. Can we build engines that... facilitate and illuminate new possibilities. I'm very curious as to what are the fun conferences that could have been held in other possible worlds that we weren't in. And do they teach us anything about what conferences we should organize? Well, this is...
Actually, so one of the nice things about science, again, is there's lots of data. People say on their CVs every conference. participated in since they were a baby, you know? And so, uh, and, and every year there are new conferences and I would say, you know, about, uh, about 25% of science funding is convening. You know, it's pulling together people in different places for summer schools. And so we have basically experiments continuously that basically...
allow us to see what's the, you know, what are the effects of these convenings, basically, in the unfolding space of science. And we have identified, you know, so for example, one project we're working on is there is... you know, there's a kind of a attentional and status order of science, which is to say, you know, like electrical engineers cite physicists more than physicists said, electrical engineers, CS computer scientists cite electric, you know, uh, you know.
EE more than EE site CS, information science site computer science more than computer science sites. So there are flows and institutions of information. And we show... basically systematically on average when something gets cited against that flow um it's associated with much more disruptive attention uh in the field which and and it turns out
Well, you know, maybe those were the best things to explore. Well, it turns out that it's much more likely, for example, for the things that get discovered against the grain to be. accidentally discovered from a person at your institution, it's even more likely for you to be married to the person in the lower status field that you get exposed to. So it's not like just the best ideas are flowing against the grain. It's like.
Occasionally, when we randomly violate the status order, then systematically we found disruptive and disruptively useful things to these fields. So there's a whole host of conferences where in this world, people from this camp are too good.
to associate with people from this camp you know they have epistemic standards that are different they have and we're building these these uh playful uh you know large language model agents that are having conversations and disagreeing with one another and coming up with uh and have who have different dispositions and personalities and we're building simplexes of these things those are kind of
spaces where you can basically steer these models, not just through text, but just through the simplex of the model space to pick exactly this particular kind of person, you know, a stoic. you know, physicists with a view of this from, you know, 1950 and we can put them in a room with someone who has a very different view of the world and, and we can see sparks fly. And so it's, it's, it's ridiculous.
And yet at the same time, it's just another extension of this question of expanding the space of speculation. Well, yeah, and it's very useful if you want to go forward and figure out how to do things better. I mean, one thing is that science is just... bigger now, right? There are more scientists, there are larger collaborations, there are teams that are trying to get things done. What can we say about that in general, and in particular, like, our...
Big collaborations, better in some ways than smaller teams, etc. So, you know, so there's kind of a few layers to this. you know this morass right so one is is that the field levels fields get larger we find um there is a kind of carrying capacity to to a field, which you can think of as a field as a conversation. It's a sustained conversation that occurs in departments and in journals and in all these environments. And as fields get larger, there's an exponential decline in the likelihood.
that new ideas will enter the canon of kind of most cited ideas. So if you have two ideas, two fields that merge, they won't retain the kind of the intellectual diameter of... of the independent fields union, you know, those two added to one another, they will collapse. All of a sudden there will be.
forced competitions between ideas and over eyeballs and attention and, and they'll collapse to kind of one part of the space. So I think one challenge we see is like, as we grow fields, we have greater and greater success. as we recruit new ideas to these kind of mega fields, then they...
That size is massively sublinear relative to the relative new ideas that those new individuals bring. Within teams, we see kind of... a similar but slightly different dynamic because teams are, you know, constructed. you know, project by project, often sometimes, you know, into larger institutions that have, you know, some greater stability. But we find that that big teams, you know, they tend to not produce.
highly disruptive work um and you know so they and and there are a few reasons uh that appears for this you know so what do these teams do systematically well they produce papers really quickly, so they're more productive on average. That's one of the things that teams have to demonstrate. And how do they perform that productivity?
And still get, you know, a measure of attention. Well, they tend to do what like a big production company does. You know, if they're trying to decide between, you know, producing Transformers 9 and, you know, Slumdog Millionaire, it's like. They're going to do Transformers 9. They're going to bet on the winning horse. And this is what happens in science. They basically take huge popular ideas of yesterday and they basically momentum invest. They take the next step.
And they're going to get, you know, like Transformers 9 is going to get Transformers 8. receipts minus epsilon. And they're going to get most of the market that existed for the prior finding. And so small teams are much more nimble and Part of it is because they have to be, because they can't produce Transformers 9. They can't be the biggest...
hit that builds on alpha fold too. Like they can't do that. So they have to do something different. So they dig deeper into the past. They dig deeper across fields or farther across fields. And, and so they're much more. predictive of advances, uh, that are likely if they succeed to occur and be important in five or 10 years, basically when this, when.
whatever this bet we've made has really run dry. Like we've, we've mined all the good stuff out of this vein. It's like, that's when like, we're like, oh wow, those small teams had, you know, other ideas. And it turns out that, um, when we look at the structure of teams, underlying those papers so we can build hierarchies within those projects, we find the flatter the teams, which is to say the more people that are involved in the design of the ideas, it slows down the production of papers.
but it increases the instability of the papers. It makes it more likely for those papers to have fused ideas fundamentally from different fields inside these spaces. So it's a complex... Bet, you know, if it turns out that people do more innovation on average than their.
if they just were trying to maximize their citations would recommend because as you said you can publish something if it's true but boring um and familiar uh but um but at the same time uh you know Every time we measure this, like how much innovation is happening relative to what the optimal amount would be to maximize discovery while retaining some memory in the system, we find that...
More is more. You know, more is better. We haven't reached this optimal limit of innovation inside almost all the fields that we're looking at. Because, you know, we may have a plan. Oh, hey, we're going to... we're going to build an institution where 15% of the grants go to completely new things. And, you know, 85% of the grants go to completely establish things. But then even within. Those proposals, you have established people who are running both of the. Yeah.
Both the innovative and the non-innovative. The conservatism goes all the way down. But there's also the fact that most things that might qualify as innovative are... bad, right? There's a lot more ways to be wrong than to be right. Absolutely. Yeah, absolutely. Yeah, yeah. Most. Yeah. So it's it's really so this point becomes critical. It's like if you want to increase the likelihood of disruptive success, you have to increase your failure rate. Yeah.
If you're in a lab or in a field that is consistently making good bets, then a robot could have made those bets. Like you're not succeeding. Like succeeding is violating expectations, which. And most of those violations will obviously be wrong because we know things. Because a lot of this physics, a lot of these other fields have insights that work or that are highly descriptive.
I will say this is where, you know, another place where kind of AI comes in. Fields that are kind of like fundamentally multi-scale that are. that are complex. And by complex, I mean like self-dissimilar, you know, it's not fractal. It's not like if you see it at this level, it's also true at this level. It's different at these different levels. Like these kinds of complex systems have... In many cases, really, I've not even been the subject of intensive scientific study because...
because we don't have scientific traction. And these are the kinds of places where the kind of function, arbitrary function fitting that large AI style models are actually.
bringing a really big delta of inference and it's it's less likely for them to discover new things about simple and basic physics and about some of these other things which have been you know we've had great minds on them for you know centuries But areas which are kind of like, okay, the intersection of physics and chemistry and biology, complex systems, self-dissimilar systems, these are the spaces where... You know, like the right most parsimonious equation might be like 500 pages long.
The most parsimonious. I'm not saying just the most overfit. That might be the best description, and it might actually be quite predictive and quite general. All of a sudden, you know, science leaves the capacity kind of of the of potentially of the human brain. I think this is this is one of those science fictions that we have to consider. You know, Alpha Fold 2.
is learning a huge complex function that does not look like any equation that has sit inside biology texts ever. And it kind of killed, you know, structure prediction. I mean, it just does it. better uh and it and it's not reducible we're not going to be like we're going to search through the neuroscience of that model and find you know like oh it's doing this one thing like really well
Like, no, it's it's we've squeezed down those models and they're still large. And and so it's kind of like, you know. I think we're entering an interesting era where we're considering the fact that there are going to be different forms and shapes of knowledge, science, and technology or control.
And increasingly, there are going to be some things like AlphaFold II, which is not science. I mean, it is a regularity. The machine knows what it knows. So it has machine science. We don't have access to that science.
There's a great paper called Crumbs from the Table. It's at the last page of... of nature magazine in 2000 by ted chang where he basically says he kind of describes a world of metahumans at that time he was kind of talking about these pharmaceutical you know uh creatures that we couldn't understand anymore and and he kind of talks about you know he imagines a world in which all of science is hermeneutics we're just all interpreting
the artifacts that have kind of come to us without understanding. And we're trying to do neuroscience on these artifacts. And I find myself and a lot of other scientists, this is in fact, precisely what we're doing. We're trying to make sense of. of brains that look like they know things.
That's a great point. I recently talked to Jeff Lichtman, who is a Harvard professor, neuroscientist, who is a leader in mapping the connectome of the human brain. And one of the things I thought was interesting and was unexpected in our conversation was he pushed... back on the idea that we're trying to understand the brain.
because he said, look, if you could understand the brain in the sense of coming up with some simplified description of it, then the brain would be simpler. The brain is as simple as it can be, and it's super-duper complex. because it's doing these things. We need to sort of figure out what it does, but we're not going to find a short description that encapsulates the brain. Right.
yeah so and this but this is this is this represents a new era if we take that seriously um if like that's what our intellectual um you know uh ventures become about then then
then science changes its character. It changes its standard. It changes its meaning. And the epistemic standard of does something feel right? Does it look right? Does the equation... you know have the right number of variables is it elegant you know all these things um you know i won't say they go by the wayside i think they're important obviously pruned models and simplified understandings allow them to travel like
like kind of components, new areas. So there's all kinds of reasons that simplification is useful. It just may be, it's likely to be for many of these complex self-dissimilar systems that... that the components that we learn you know from these systems are much
larger and themselves more complex than we had previously imagined. And they become objects in the world. Like a fab facility. I was in Silicon Valley when they moved the first big... uh chip fabrication facility intel from from there you know in the kind of san jose area to ireland and they had done so many changes on that facility like they had you know they'd fixed you know something like
15 million errors. So they had no idea why the fab facilities were at the tolerance levels it did. And so when they moved it, they were like, okay, we need to retain everything. Like it's orientation to magnetic North. It's like.
you know like the relative density of the air you know it's it's elevation like they had no idea and they took it over and it and it broke for reasons again that they didn't understand and had to had to kind of fine-tune out but that it's like our our new theories are looking more like this this fab facility that produces like chips with a high accuracy low tolerance level and it's unsatisfying as a scientist in some ways to look at that fab facility and and say wow that's that's what
we know you know what is that i don't even know what that is like like uh yeah i mean i i guess i kind of get it and i have very mixed feelings about it i mean i agree it's yeah no no my feelings are not going to affect uh what what's going on
But to sort of – I mean put it in other words and you can tell me whether I'm capturing it correctly. I mean there's a whole new way of understanding things, of modeling things, of being a scientist. Once you have – Large language models probably isn't the right word, but these deep learning, many, many number of parameters things, you can capture complexity in a way that is unprecedented and impossible if you need a one-line. equation to count as understanding. So you can do that.
But so much of science is sort of counterfactual, right? Is sort of saying, well, if things were different, what would happen? And without that simple understanding, I mean, maybe the example of the Fab Factory is... An important one. How good are we at saying we truly understand or even the AI truly understands if we can't say, well, when we change things, what are the effects of those changes?
So I've got a paper where we look at the impact of AI on science over the last 25 years. And of course, it's gone through different generations. First, it was machine learning, and then it was kind of deep learning. And now, more recently, it's the...
these large transformer style kind of foundation models. And we look at the impact on scientists and on science. And we find that basically on scientists, it increases their mobility across scientific ideas. It increases their speed of getting paper. done decreases the number of people in those papers, increases their rate of promotion. For science as a whole, right now,
It actually narrows the scope of ideas that are discussed because these are big data seeking models. And where do we have big data? One problems that we've been generating data forever about problems we already know about. So exactly.
suggest on average right now these models are not being tuned to origins questions, origin space, you know, origin of time, origin of language, origin of, you know, like, you know, any of these, these origins questions that live before the institutions that produced and preserved data. And so that's a world of counterfactuals. Now, that being said, it's not obvious to me.
that AI has to fulfill that role. That is the role that it has fulfilled. What I was describing earlier is that with these large data-driven models, we can actually engage in... uh kind of like continuously steering counterfactuals in a way that was unprecedented before you know creating a factory for counterfactuals and um so i actually think it's possible for it to enter some of these these new areas but it's going to require a new range of conceptual thinking and scientists
effectively will have to become in some ways philosophers of science, you know, where they're selecting between philosophies, they're navigating, you know, what their tolerance for kinds of philosophies of hypotheses are. And then, you know, the machines will. We'll produce them and in some cases we'll test them. Well, I guess creativity is a big sticking point with the AIs. I guess there's a school of thought which will say that they're just not.
creative in the standard way. They're remixing all these various ideas that they've been fed from their training data or whatever. But I see no reason in principle that an AI couldn't be creative. I mean... if only because you could throw some random numbers in there. But then there's a question of how do you cleverly throw the random numbers in there? Like, how do you nudge the AI towards being creative in useful ways? Is this... Is this just my lack of understanding or is this a frontier here?
This is an interesting frontier. I've got a DARPA grant with Law Varshney, who's a computer scientist and a really creative engineer at the University of Illinois at Urbana-Champaign. And he was at IBM when he created this kind of blue chef system that kind of created really creative combinations of ingredients and recipes that then, you know, would automatically kind of create. And so we're spending time looking at the sciences and all.
building kind of theories of creativity that will facilitate the kinds of abductive creativity that we see in the world. But I think a critical piece of this is what we were talking about earlier. I mean, outsiders and insiders, abduction, surprises. that are are because like our expectations that come from literature and like the data that come from experiments and observations are separate they're not so so what does that mean we you know large
you know, language and large multimodal models and these deep learning models, they're not one model, they're many models. And so we basically have to exploit that. underlying diversity and accentuate that underlying diversity. We have to separate some of these. So if you just train these models in prediction, you'll find that they actually create themselves little schools of thought, but then they recombine.
at the last moment to make predictions. They're doing science inside themselves. For us to use them to kind of push the boundaries of science, we need to use them to generate. really deep expectations. And then we need to use other models to survey the space of experiments and observations. And we need to identify dynamically, what are the surprises and how should that...
So absolutely, if we just use these models to create more data and then use that data to train the model, we'll have a collapse of attention and the models will not get better, they'll get worse. But if we basically build an ecology of models in the same way that we need to cultivate an ecology of scientists with different temperaments, from different fields, with different expertises. Those ecologies of models have the potential to kind of create...
a new world of creativity in the same way that like, when we think about AI safety and it's like, no, we, okay, we've got to, we have to put the thumb screws, the, you know, restraining bots on these robots. No, we, we, like we do the things that we do with. unpredictable intelligences in the past. We make them sign the Magna Carta. We create a set of independent institutions, checks and balances. We need to create an ecology of regulatory AIs, an independent court system.
you know, an independent, you know, legislative body, you know, I mean, all these things are, that's how like safety works. That's how innovation works, even though those systems are kind of in some ways opposed to each other. We do it collectively. And so we need to basically harness ecologies of kind of artificial intelligence models and agents to kind of create. the same recipe basically for knowledge generation that we see working or has worked really increasingly works in the human system.
I mean that seems to be a common thread here. One thing we didn't get to is a paper you wrote on political polarization in editors of Wikipedia articles and how you get the best articles when it's not – written from a single point of view, when there's some team of rivals aspect there. And in some sense, you're saying that's just as true for the computers as it is for the human beings. That's right. And the problem is...
That, you know, the computer model generators would love to be monopolists. And so, you know, they're trying to sell a product that is the one true AI product, you know, that has... all diversity completely within itself embedded uh and uh and the answer is that cannot possibly work as a long-term generator of surprise in the same way that if you have two communities and you
force them into one conference room, they will talk about half of the ideas that they talked about when the two communities were separate. We need to create an ecology of AIs that basically enhance. the population genetics, you know, from which future innovation becomes possible. I mean, I'm glad you just said that, because I think you said it before, a version of it, but it didn't quite sink into me. You need the different communities to talk to each other, but they also...
also need to be different communities. Otherwise, they become just homogeneous. Yeah, and this is where tradition is absolutely critical in creating membranes between communities that can basically self-evaluate and have their own tastes. So I'm not an anti-field. In fact, I think fields are...
absolutely critical in this connected age. Like they need to hold their own standards. Those standards become resources from which other fields can draw. And if you get to the point where everything is interdisciplinary, everything is post-disciplinary. Then you destroy the very intellectual, dispositional, and epistemic assets.
that themselves become building blocks and new knowledge. It's just mush rather than a bunch of interesting little things. Just mush. No structure. That's right. Yeah, black hole. Okay, good. It's the end of the podcast, so I'm just going to completely change. gears and give you a couple minutes to talk about the fact that you and my old physics buddy Daniel Holtz teach a course at the University of Chicago on the end of the world, on the forthcoming doom. This seems like a, you know, bleak...
topic for a college course. Give us the sales pitch for why we should be thinking about these ideas. Well, I think if you look at the mammalian uh kind of historical you know pre-historical record you find that on average like the species lasts for about 3.5 million years and we're about you know i don't know one two hundred thousand years into our our lifespan. That means we should have 3.3 million years left. If we're average, we might be better than average. We're smarter.
You know, many of the creatures that went before. But who thinks, you know, who's listening to my voice now that we can project a thriving world in three million years? right we haven't you know we can't think We're making decisions on a next year basis, on two years, on our competition between some other country in 15 years. The biggest plans are 2050. That's 25 years. I think Dan Holtz, who is a physicist and worked at Los Alamos National Lab, and is very sensitive about the potential for...
nuclear arms to get in the wrong hands, the wrong place, and to set off a cascade of... So I think he's thinking things could happen on a very short... timeframe that could be cataclysmic. And they're an increasing number of those things because they're the flip side of powers, artificial intelligence and environmental and nuclear powers. that we're controlling.
can get out of hand. There can be lab leaks, there can be, you know, nuclear meltdowns, right? So the more powers we have, the more ways we can destroy ourselves. From my perspective is even if we destroyed ourselves in 10,000 years, it would still... be 3.3 million years too soon. You know, we need to build and why don't we think longer? You know, why don't we have a longer time horizon? It's because of the very specific evidence-based epistemic standards that
drive scientific advance. Like we have standards for what represents knowledge. And for many fields in modern economics, if you haven't done a causal identification of a certain flavor, then it is just, it's just not knowledge. It's like, you just, you can't even make the comment, like just get out of the room. You know, it's just not. And, and so we've got like this great scientific, scientific standards, which have been very historically useful for advance in our field.
but are ill fit for thinking about alternative futures and these singularity events where something melts down or something explodes or something takes over and it's all over. I mean, we have... a single experiment that we're running, which is humanity. And so we have to imagine alternatives in a way and with a precision that we have never done before, which invites us to explore.
new epistemic approaches, including things like narrative, including things like science fiction and stories to think about possible futures and harms because of the risks that we're creating because of our increased power over the world. So I think I actually found the class therapeutic students.
came in with anxiety about climate and about other issues. And in talking about them and facing them and talking about, you know, policy possibilities and thinking about alternatives, we actually, I think it conveyed to all. of us a sense of agency and possibility. And also a long view. You know, we don't think about this unfolding set of future possibilities and people and environments that will be the inheritors of our choices today. And so anyway.
That was the reason for the class. There's a tricky balance, right? Because you want to emphasize the very real worry that disastrous things could happen with the lesson that—and there's still things we can do about it, right? There's still tools.
we have to prevent it. So be alarmed, but don't despair. Right, exactly. And I think talking through with experts from around the world about some of whom were despairing, some of whom were... busily engaged in their particular projects, I think gave us an expanded view for how to think about talking about knowledge extrapolating from the singular experiment, which is...
humanity and this world in this universe you know all right you've given us an expanded view james heavens thanks very much for being on the mindscape podcast thank you sean it's been a pleasure