When AI Discovers The Next Transformer - Robert Lange (Sakana)

I'm giving away a DGX Spock. Uh NVIDIA just hikes the price seven hundred dollars. You probably heard about these memory shortages, right? So yeah, it's now forty seven hundred dollars, which is very, very expensive. And uh Merv from Hugging Face, by the way, she got one for her birthday and she said she literally cried. So it's a really cool bit of kit. Um if you register through my link in the description and do you attend at least one session?

Then you are in the draw. This is a massive conference. Physical, AI and robotics are gonna be the breakout theme. And Jensen does the keynote Monday at eleven AM Pacific. The link is in the description. Don't miss it. Robert Lange, it's amazing to have you on MLST.

Just before we get there, so um we interviewed the Alpha Revolve team and I also interviewed Jeremy Berman a few weeks ago. And your paper is is very much like a more sophisticated version of those in the sense that it using language models to generate programs and it's doing an evolutionary approach where we generate the program, we refine the generated program and we have an e an evaluator and we do this over several steps.

And and your your your approach does many things that that the other ones don't do. Tell me about the paper.

driven methods um is sample efficiency, right? So many of these systems sample like, let's say, a thousand programs for a given task. And what we tried to do with Shinka Evolve was uh try to essentially cut down costs as well as sort of computation evaluation time by introducing um a set of sort of technical innovations to this evolutionary search.

And we showed that um it's possible with uh very few program evaluations to basically improve upon like for example the circle packing um canonical result that they showed in their paper. And um yeah, more generally speaking, I think uh we're right now at a point or like at an inflection point where these sort of let's say evolutionary driven L L M systems

can really revolutionize scientific discovery. And uh yeah, we hope to um have made a step forward to making this more democratically accessible, right? So the code is um open source available and uh yeah by It's a sample efficient nature. We hope that many people can interact with the system and can make uh their own scientific discoveries.

and then build on top of them to to really find uh innovations or to tune innovations down the line. And I think um language models with the right sort of evolutionary harness are um extremely powerful in terms of scaling up to um to to make discoveries. And um yeah, I think uh Jeremy, as well as the Alpha Evolve paper, as well as sort of um work we've done on like the Darwin Goethe machine, for example, shows that this um sort of stepping stone accumulation plus

iterative verification and uh collecting sort of information and evidence from the real world or real synthetic evaluator is really important.

And greatness quite often follows a diverse path, which means you have to do things which initially seem quite stupid and then later on they turn out to be incredibly useful. Yeah. We're trying to design algorithms that can kind of allow for a population of slightly weird things and and then we kind of lock in and and converge a little bit. So We we're still converging though. So we're still building systems that don't diverge forever.

But oftentimes um innovation for a specific problem might require first inventing a different problem, right? So for example, I think in the uh matrix multiplication result that um the alpha evolve people show you can recursively apply sort of the algorithm to larger matrices. So it's actually an important result, right?

Um but uh sort of automatically coming up with this reduction or like this, let's say, um recursive nature of problem solving is something these systems right now not necessarily have built in intrinsically, right? So I think uh going forward it's gonna be um really important to not only sort of do open ended, let's say, optimization of solutions, but sort of do the co evolution of problem and solution together in order to collect even more diverse stepping stones.

and um to really kick off this this open ended process.'Cause also to me, like one of the uh the the big life goals or achievements um I would wanna see is you really having a process that can run uh not only for let's say a week or uh many weeks, but like for years even potentially, right? Collecting even more diverse, interesting stepping stones.

these stepping stones that might be useful later. And when we run these algorithms at the moment, it's the same with L L Ms and reasoning systems, is that they're very, very good when we give them a specific thing. And what you're pointing to is we might need to invent new unrelated problems and find the solutions which might then be related to what we're trying to do. So that feels like a bit of a catch-22 situation.

Right. So we're saying, you know, circle packing. Here's my evaluation function and I want you to sort of diversify and then, you know, kind of and then converge towards this solution. It's a s I I had the same thought with Genie, by the way, that it it gives you exactly what what you ask for. So you put a prompt in on, you know, like a Swiss lake with you know, with boats on the water and mountains on the side, and I was thinking, where are the birds? Oh, I forgot to put birds in the prompt.

Right. So how can we meaningfully build systems that actually kind of bring in other unknown things that might be useful?

In Poet the the natural first application was sort of reinforcement learning. Um but I think this can now be broadened up to to yeah, science more generally, right? At least when there's a a simulator available to for for running these evaluations. And

by doing such a cool evolution, um you you always try to to max out um the the capabilities of that generator while sort of uh increasing this uh this convex hull or potentially even um yeah more diverse problems while doing so.

Even then isn't there a kind of design bias in the system where there's some code somewhere which complexifies the environment step by step and wouldn't that also just be designed by the humans so it would also just give you exactly what you ask for?

um you still kind of get stuck in in local optimum, right? Where not a lot of novelty is introduced, right? While if you you start off from like an impoverished solution, there's much more room for diversity. And I think this is sort of coming back to you, um

sort of what I did before in uh in my research, namely meta learning. It's sort of um this classical trade-off where you can either start out from something um very uh let's say unconstrained from like a very simple solution and give much more room for the optimization.

But this might actually require open endedness and a long time to find a good solution. Or you start out from something that is already very constrained by inductive biases, let's say, and then you might be much more efficient in terms of convergence, let's say, but um uh you don't have this sort of open-ended

We should talk about the the evolutionary approach, right? So to maintain diversity, you had a population of programs and they were separated into islands. Tell me about that.

And then you you basically you query the LLM, you get a uh program out, and you evaluate it on the problem at hand, right? For example, increasing uh the sum of the radii of a bunch of circle in a square. You run this basically each time collecting evidence from the evaluator. adding it to the the database and then sort of repeating this process. And you don't do this sort of sequentially, but you do this in parallel for many different programs.

And each time sort of a program is added, you essentially try to diffuse the knowledge that was collected by the program across the entire sort of database, right? So One way to think about this is you have a tree, a tree where each note in the tree represents a program, and then you you sort of uh branch off of it based on the parent notes, right? And uh interestingly, like these approaches do tend to scale. Um but

sort of GPT model in other settings it might be better to use uh a Gemini model. And we sort of introduce a um sort of adaptive prioritization scheme that can adapt sort of the evolutionary algorithm on the fly while running the the algorithm. And this sort of also comes back to the naming, right? So Shinka Evolve, Evolve, Evolve kind of means that this evolutionary algorithm that we apply using LLMs sort of also co-evolves at the same time while we optimize.

sort of propagation of information through the tree is one that's really, really fascinating, right? Where in some sense these stepping stones are actually used and So in a complementary fashion, right? And w with regards to rerunning the program multiple times, right? Of course, there's some stochasticity in it, right? So we're using language models and uh sort of due to

like the the queuing device scheduling on on their server side basically, we can't get rid of um all the all the noise. We we've seen that um at least for uh the general quality of the solution, so what is arrived um afterwards.

uh it is possible to reobtain this. Uh but sometimes with a different program, like or most of the times just by stochasticity, right? So it's not like um there's uh for many problems there's like not one solution that achieves that score, but there's like a spectrum or like a a region, let's say, in the program space that uh

resembles the same, right? Um I think one thing that was very m uh interesting about the circle packing problem, sort of also coming back to the problem problem that I discussed uh initially, was that uh originally we um We used a formulation where the correctness is checked with like um a very tiny amount of slack, right? So um the the circles could overlap a tiny little bit.

And then um afterwards we we we sort of reduced the rate AI and the solution was exact, right? This didn't change the score by too much, so it's still state of the art. Um but it was essentially like a proxy problem. We r then reran the the Shinkai Evolve on the exact setting and we found that it took a little bit longer to actually obtain the same quality of a solution.

So I think this already points a little bit in this direction of what I discussed b in the beginning. Like sometimes sort of surrogate problems might actually be extremely val valuable in in making such discoveries. And having an automated way for designing these surrogates Problems in an efficient way might be something really important going forward.

Because presumably if the um fitness function had the constraint that there were no circle intersections, then it wouldn't it wouldn't occur to the algorithm to come up with a surrogate problem.

uh taking sort of number theory and applying it to linear algebra or the other way around. And I think right now these systems are not yet at the point of

And then they do something which is similar to Shinker Evolves. They do this kind of evolution where they take these different solutions and and they kind of combine the the best aspects of both of them and they forge a divergent path to a new solution. And that seems to be the essence of of what we need to do.

And I was thinking in my mind, well maybe there's some natural pattern that that there's there's there's there's something about that that we could use in the abstract to guide the evolution in the future. But the other thing I'm thinking about is

Right now the problem with machine learning is that we don't really have semantics baked in. So what we're doing is we have a verifier, we're looking at the rewards and we're sort of like doing patent exploration and we're taking steps towards the you know, towards the target.

And I love mechanistic forms of reasoning where we actually know something about what the program components mean. And the reason this is important is when we're merging together the best performing programs from two different islands. Um, that's a kind of first order interaction and it it might not make sense to merge them together. It's wonderful that L L Ms you can give them any pairs of programs and it will find a way to merge them together.

But wouldn't a more principled way be of there's there's some kind of semantic primitives here and we know they fit together. So this this Lego analogy that we're kind of building up based on principles rather than forging a path based on the performance. Yeah.

So um that way y you can try to sort of semantically grasp some of the discoveries. But a general problem, which is again sort of task dependent, is

Um thereby you sort of diffuse that knowledge across the tree, right? But sometimes you want things to be much more isolated, right? It's always like um a trade off where you somehow have to find for your problem the right uh position on the spectrum of how much knowledge diffusion do you want to have and how much sort of uh let's say hard islands of programs do you want to have, right?

And um yeah, we we're trying to make steps in the direction of sort of automatically adjusting this in an optimal way. But again, it's very program sensitive. And then sort of I think uh another point where you're already sort of going into um is sort of Jeremy Jeremy's solution to Arc AGI, right? And sort of doing

um solution evolution in the instruction space, right? Instead of the program space. I do think that this is uh something important and we're Like I said, with like the construction of this meta scratch pad, trying to do sort of both at the same time. Uh again it's problem dependent. Like I played around a little bit with ARC AG one AGI one and ARK AGI two. And I think on ARC AGI one actually the the transform sort of program direction is actually quite effective, right?

Like Jeremy said, it's deterministic and um it's easier to sort of get clear signal to improve on during your evolution process. Um While on others, like Arc AGI2, like this whole sort of semantic evolution uh seems to be more efficient. So I think ideally we we can get a system that can automatically, in some sense, decide whether or not it wants to take like a programmatic approach.

in settings where it's actually feasible and easier to to bootstrap off, or it takes the semantic approach of uh evolving instructions or like LLM driven um input output.

That that that doesn't work. Right. So we're using neural networks because they're incredibly flexible and they understand a lot of things about the world, but they don't have the kind of constraints that we want. So what we do is we use these tricks. So Jeremy evolved program descriptions. Um on your program selection you had a semantic novelty detection, you know, using like a um

Large frontier language models are extreme, like let's say black boxes, or it's very hard to get a full mechanistic understanding of them. Um, the outputs can be, right? The programs, the instructions. Right. So I think it opens up a very sort of new paradigm of um doing research or basically doing anything, right? If you if you think about it. Um but I think we're we're just sort of at the starting point of figuring out the the right uh user interface for

And you've implemented that for figuring out which L L M, so it could be Gemini, it could be like, you know, Grophor or something, to figure out which one to use.

It's unclear basically if the performance gain you get from the second mutation actually originated from GPT five sort of collecting the first stepping stone or from Sonnet four point five. Instead of sort of uniformly sampling models, what we do is we implement um this bandit-based approach, where each model um is basically one arm of a bandit. And then we look at how often did this model improve performance of

a sort of parent node by creating a mutation. And we then adjust sort of this posterior probability to sort of first explore all arms once, right? And then essentially um change over the course of time in order to uh prefer

And I think what we want is to have more of a contextual switching um decision, which means we know for this particular program Gemini is better. And do I understand correctly at the moment that it might converge to a single frontier model and then in a nuanced situation, we might still get the wrong model.

of this evolutionary um process, like which model is better and um UCB or like the the bandit approach that we take dynamically adjust this in in an efficient

as surrogate models. And um the real question to me is like how do you represent the information to the LLM, right? In the sense that if you use like the raw programs in their fitness evaluations, you you quickly run out of context, right? Um so you need some amount of compression in order to present the information the right way to the LLM in order to do this prioritization of the models.

that really um did the job here. And uh yeah, much credit to Eduardo Cetin who who introduced this to to Shinka.

um this proposal and these will not be changed. So only the the rest of the um the code snippet will be changed. We sort of implement a type of rejection sampling with reflection approach where if an L L M by chance, for example, tries to mutate this port, it's gonna be rejected and you resample a new proposal. And yeah, thereby you you can somewhat mitigate certain security or safety um problems and uh yeah, get a robust sort of mutation.

One of the sort of I think the the bigger questions is how can you turn this from a single file mutation setup to a multi file mutation setup? So working on entire code bases. In principle you can

represent many code bases in a single file, right? But um the hierarchical structure might be actually useful. And there are some um ideas from uh let's say Ader, this this um coding tool um where uh you construct like um a repository map and sort of have some level of abstraction but they also come again with uh positive uh and negative

But um we didn't talk about mutation by the way, so so we just spoke about diffs and there's also an option to do the a full file rewrite. Exactly. But there's also this notion of of of crossover. So how how does that work?

This sometimes is helpful, right? Um it's not always like a clear benefit, um, but uh it it it allows you to essentially get more diversity into the search, right? So um this is one type of mutation next to sort of this diff. Patch based approach.

And the other one is uh a crossover mutation where we sample basically uh not only a single parent program, but sort of two different ones, and we ask the the system to sort of make a complementary improvement. And um again, on some problems this is really helpful and on others it's not.

Um but in generally we found that um sort of having a diversity in terms of operators is also helpful in discovering new things. And I wanted to to sort of follow up on the point you made before about this sort of being a new paradigm. I think so too. Uh I'm really convinced. I think right now we're sort of um at the beginning where we we still think a lot about sort of this chat assistant interface.

um as the way how we interact with LLMs, but it's uh most of the times inherently single threaded, right? So we're sitting in front of the computer, we're interacting with the chat, we're seeing sort of changes uh as they occur in the editor, we accept them and so on.

Um but I think this is sort of also just a stepping stone towards sort of a more let's say distributed way about thinking about research, optimization and so on. So I like to sort of think of um vibe coding, vibe chatting and on the other hand we have sort of vibe optimization and vibe researching where sort of my ideal future scenario is one in which um you as a researcher sort of during the day co-work with like

system like Shinka or the AI scientist, you um sort of steer the ship like a shepherd in some sense. And then during the night you you you you press play and you go to bed and in this uh in the background you have multiple experiments running and automatically new ones being proposed by L L Ms, evidence being accumulated and then in the morning you come back and sort of you have an

uh multi threaded sort of system running in parallel. And you're more like the shepherd of this ship than the the person actually Executing experiments and analyzing. Oh yeah, you're still analyzing, but you're not executing. This is happening sort of by the system itself.

Whereas I can imagine a future where we um express problems just in natural language or maybe just based on our interactions with language models, the platform knows what we're interested in and it can just go and find things on our behalf because this is about democratizing this technology to people who perhaps

sort of soft verification, looking at code and sort of uh latently running like a like a stack trace of execution, right? But it it's not exact, right? And I think um sort of these notions of uh reward hacking and sort of um not doing real discoveries, but sort of shortcutting them is one uh where we need to put more

time and effort into to figure out um yeah how to make sure that this actually moves in the right direction, right? And I would hope that language models at some point can do this efficiently themselves, right? So either implementing encode or latently doing it. Um but this is also like part of the problem problem, right? It's not only coming up with the problem, but also with the automatic verification at the same

and the world tomorrow will have things that we can't explain, you know, with our with our knowledge today. But we do have like abstract knowledge that could be easily recombined to explain the future and LLMs might already have those building blocks.

But I do think like with systems that sort of do this evolutionary mutation sort of style approach, you might still sort of push The system out of these locations.

But what we can do is build systems like Shrinker Evolve that leverage the best of both worlds. So they leverage frozen foundation models, but they give you adaptivity and the purpose of adaptivity is to respond to novelties, to create new building blocks, synthesize new building blocks in this principled tree like structure that allow us to adapt to novelty. So we are having our cake and eating it.

Right. That was sort of the gist of his answer. Um and to me it's like once you have made it much more efficient, you can scale it up again, right? So if you essentially have a cheaper system that can uh generate many more sort of instructions, I would expect that by the

nature of open endedness, uh, you might get some amount of improvement out of it. Uh right now I don't have any evidence for it. Um I would love to collect that evidence. It's again like the magic of open endedness that comes into play. That as long as sort of these training examples of Arc AGI give you a good signal for a final test submission, you should be able to.

prompts, generates, breeds the starting conditions. Because'cause every time we run uh Chinker Revolve, what it's doing is it's it's searching parts of the epistemic tree. Sure. And what would happen if we just scaled that out massively?

This is the question that will push us towards like this true open ended vision of running a system for like a month or so, right? Really trying to squeeze this out. Yeah, I'm not sure if we're entirely there yet, but I will do my best that we

So let's say we have a thousand different instantiations of Chinka Revolve. It doesn't have to be embarrassingly parallel. We could still have some sharing. So during their execution we could still have a little bit of like crossover and and and and maybe then we could we could run or the Chinker Evolve instantiations in a in a similar kind of meta evolution loop and

My suspicion is contra Jeremy, I agree with you. We know there are diverse stepping stones out there that could dramatically, dramatically improve many of these solutions. We simply haven't scaled it up.

in sort of language, right? So I I would be really interested in um sort of exploring.

where everyone is solving different types of problems and and the system it'll be like a kind of chinker revolve but a massively distributed version where mathematicians are using the platform over here to solve this problem and it will see commonalities. And it and it will kind of like link them together. Because because you need to leverage like human creativity in this process as well, I think.

scorable. And um I think this this this will be really, really interesting to see how sort of compute, um, these automated agents, uh, human shepherding and steering will ultimately sort of change and revolutionize science and I guess society module.

reinforcement learning with verifiable feedback and so on. I think that you subscribe to the slightly different idea that that we need to be far more open ended and we need to be using evolutionary algorithms and so on. But do you think that they are on a path to nowhere? Do you think they might change tack? D uh I mean wh where is this going?

uh expert, right? So I I do believe that uh mixing in sort of different sort of RL fine-tuned models into sort of the ensemble of models and then having a good way to adaptively select which one model to use is is not a bad idea, right? Um So to me, uh I just very fully subscribe to this philosophy of open endedness and uh reading Kant's and Joel's book was

really like a f fundamental moment in my life and I uh want to see how far we can push this. And I think we're uh we're not yet at uh sort of uh convergence where either the capabilities of the models uh has converged or the the way how we uh scaffold around them or the way how we humans interface with them. So to me they're really like these three

points like uh model capability, model scaffolding, and sort of the user interface. And I think we have a lot still to push on all three angles.

And uh we found that uh there we could dramatically improve um sort of the performance of very cheap models like GPT four point one nano. But the agent scaffold was also able to either like generalize to other language models

or to different years of uh of Amy, right? That was one application. Um one important other application that we did was to uh ALE bench. ALE bench is basically uh work done by other folks at um Sakana, including Yuki, who's also part of the paper, uh which is uh considering heuristic sort of programming contact uh contests um sort of previously done and executed by Adcoder, which is like this famous

Japanese competitive programming organization. And we sort of showed that Shinka can also work very well as a co-scientist. So basically we we took initial solutions obtained by um an ALE agent that was previously designed, and then we optimized on top of these initial solutions with Shinka and showed that on one of these um

sort of programming tasks, if the combination of this agent and Shinka would have competed in the challenge, it would have um uh ranked second place basically. So I think there's uh some evidence that Shinka can work as a coscientist and um not only for LLM agents, but potentially even for humans, like we discussed. And then finally, the final application that we looked at was um designing sort of mixture of expert load balancing loss functions.

So at Zakana we've done some uh previous work called Disco Pop. I think we discussed this during the last podcast we did, where we're using LLMs to design objective functions. And back then we did it for preference optimization and post-training. And here we did it for load balance.

uh of experts. Also there uh we found that within like I think like even only 20 sort of generations uh we were able to sort of um explore let's say not only a single objective function um but sort of Let's say a convex hall where there are different trade-offs between sort of performance and load balance. So I think this is another application of Shinka where it's not only basically about sort of finding the best solution, but essentially illuminating.

a program space where there are always potential trade offs between like let's say for example runtime and the quality of the circle packing, right? And um having a system that can explore all of these is

It's uh also pretty fascinating s on a society level how far we've come since then. And sometimes while you're sort of deep in the say battle mode or uh work mode, uh you kind of forget where you were one year ago and then just looking back it's It's pretty amazing. Also how far we've come since um.

And and in a sense, that's not quite the case. I I said when Arc V two came out that it's actually very difficult for humans. You know, there was one task where Duggar was stumped for about fifteen minutes with there's three of us looking at it and w we just

And it's one of those things that depending on your perspective, you might get it straight away or or you might not. So there's that criticism. And people have said that R three is even harder. Yeah. You know, but I I think that's rather missing the point. I I I think he's saying that with a lot of these competitive coding um problems.

the the data set is contaminated. These are problems that have been solved before in in part or in whole, which means when you look at the epistemic tree, many of the building blocks for solving them are very high up in the tree. He's he's looking at these these problems that there is very little dataset contamination.

Um they need to be solved from very abstract building blocks. So you're starting much lower down the tree and you're synthesizing a model by composing together very abstract building blocks, which is the essence of intelligence. Yeah. And and I think for that reason ARC is is really kind of pushing us to build adaptive systems which we could say are intelligent. Yeah.

over training on ARC AGI one at least, right? I do believe there are some aspects of work which will be automated before it comes to sort of fully science automation and the type of work I'm doing. But I could imagine that certain parts of the dimensions that I deal with every day are for sure going to be hit by AI. And then the question is, are there gonna be new dimensions opened up that we as humans will fill in, right? And I think what I said before about like shepherding and so on.

I really hope that that's the way forward, right? In the sense that humans are the ones steering the ship while just being massively amplified in their productivity.

And there's the cloud analogy as well. So, you know, IT system administrators who were earning, you know, sixty thousand pounds a year in the UK, they rebranded as as cloud DevOps engineers and they more than doubled their pay.

And people are very adaptive. They they see new trends, new bandwagons and and they just adapt and and they add value on top. And that has been the trend for, you know, for a very long time. Do you think that AI is going to be so transformative that it will transcend people's ability to adapt.

So first off I I think we're we're still not at the ceiling of the sort of technological progression, right? So maybe in a couple of years we will need less of sort of slop editing, like you said. But I do think we'll we we need some more time to adapt to the different modalities of interacting with these systems, right? I think everyone um can sort of interact with uh a chat assistant. Um, but I think this is the most sort of naive form of interacting with

AI agents, for example, right? So yeah, I think we need to get the pacing of all of this right and we need to do much more exploration in human machine interfaces, UI UX design and uh how to make sure that humans sort of fill Alright, feel fulfilled during this experience.

um, an exchange between the humans and and the system. Because the implication is that in the future we might have systems that are so autonomous and so open endedness and can figure out valuable things to research that humans wouldn't be needed anymore. And the reason why I'm not that worried yet about labor market disruption is I still believe deeply that humans are the source of deep understanding and creativity in the world. If I didn't believe that, I would be very worried.

seed or guide the direction in which to explore or to to verify, check and um actually yeah transfer these insights, right? So I think um it's not gonna be like all ML PhDs will will be unemployed. It's it's more gonna be a um sort of core evolution of humans with this technology. And um potentially like in an ideal future for me, like it will allow humans to focus on what they're really, really great at, right? So I think it's gonna be an amplifier.

of sort of these these latent dimensions humans are great at, right? I think something that's critical is that we as humans try to interact with these systems as early as possible in order to actually like have influence and uh ownership over like this development process, right? It's um ultimately collective intelligence that will shape all of these systems together.

uh is reaching that threshold. Um we're we're at the point where we can even talk about sort of uh noisy review processes and this actually being uh yeah, something that as long as you have a large budget, you might get something out of it. I think going forward for the bigger innovations and so on, um, for now, it you still need humans. Um, but we're sort of at the GPT one moment of of making this

sort of a reality and potentially uh in ten years this is gonna look very, very different once the sort of also the infrastructure for it has been built up, right? So there are places like periodic labs, right, which sort of now are building like real physical labs with uh robotic systems to automatic uh automatically sort of execute experiments. This will take some time.

But it is uh sort of imaginable for sure that as we sort of do RL on these types of uh systems and we actually also account for negative results and for actual like hypothesis testing, so getting these systems to be a real good hypothesis testers with verifiers in the loop, um, that we might be able to unlock many more capabilities.

So they can do some surface level recombination and they can discover new things in the basin of things that they've already discovered. But but we understand things very deep down in in the epistemic tree, which means our you know, our cone of creative potential is is much wider. It's possible that that gap might be closed. What would happen then?

And I think what I kind of alluded to before, we need much more like full tree data sets for training these L L M systems to actually learn how to do this exploration and this foraging basically. Um at the same time I I feel like um evolution will also take place on a cultural level, like for us, right? We will get better at sort of steering the ship and um

I can imagine that uh in in a future world, sort of the way how we do research will be completely different. And I'm pretty sure that right now already ninety nine percent of machine learning research is done with sort of AI assistance, right? Think about ChatGPT brainstorming, cursor coding, cloud code, et cetera. In the long run, we're going to move on that spectrum from sort of with AI closer to by AI and then sort of more high level sort of orchestration and overseeing by humans.

I personally don't subscribe to that view, but the other view is that it is like let's say the AI scientist, you know, like version ten, it's going to be continually epistemic, you know, epistemic foraging. It's going to be finding new things that are useful. And they kind of have to be useful to us. Because if it finds things that are not useful to us, then we just won't use them and then nothing will happen. So so d do you think there'll always be a kind of coupled value function to humans?

um going forward it will also be coupled with human data that is collected using verifiers, right? So I have a hard time believing that um in the long run, when you run this open endedness uh sort of paradigm with AI scientist agents, uh it's gonna c s completely divert to to something that's either

fully non interpretable or unrelated to problems we as humans care about, right? And uh then again, like humans can steer to a certain degree where like the search happens, right? So you can tell a system, okay, um To do cancer research, right? And sort of work on problems that we care about. And ultimately, like we are the ones who control how much flops are being pushed into.

Yeah. And at some point maybe we just wouldn't care anymore. Maybe we can just carve out that space just forever and ever.

in the neighborhood of things that are known. And that is very useful. It's very useful to have an artifact that knows all of the experiments, all of the things that are ever done by people twenty years ago. But now we don't have people really kind of applying their their brilliance, their talent in completely new areas.

Discipline and how you raise sort of the next generation, right? So discipline on a personal level, like how much do you just sort of uh Tap accept everything that's being proposed by these systems and um responsibility in terms of educating the next generation in the sense that we need to sort of teach our kids that ultimately what comes out of these systems might not always be be true, that uh facts can be sort of uh subjective, if you will, and that there needs to be more research about

uh w w what's being given to you. And um I think this will be, like I said, this cultural evolution that we have to step through and um try to make the best out of.

and and then you see a divergence because there's something about thinking that it must be grounded on your path. There's this path dependence. And when you start kind of becoming parasitized by this other train of thought then you stop thinking about your path and then you're not in the driver's seat anymore.

sort of rethink um the way how you work, right? And um to me right now um there's certain parts Where like sort of auto accepting is acceptable, and there are certain parts where it's definitely not, and you really need to go deep into it. And I think Right now we're sort of in this weird, um, non equilibrium state where things are moving constantly, right? So the systems or the models are changing, the features are changing, the sort of um

points where the systems are good uh is change uh are changing all the time. And we humans need to constantly adapt to to that, right? And uh I think it's a big cognitive challenge. And uh I think w we just all need to be aware That there are certain problems and certain challenges that we have to adapt to. I think the best way to do so is just interact with the technology as much as you can and um Maybe find new research ideas for uh out of that.

Uh there was an idea and that idea didn't work out, right? But then in the end, the paper like the experiments were still executed linearly and you wrote a paper. And uh this was already impressive in the sense that it looked very much like like science. Um but if you think about human sort of uh science and like the scientific method, um it's much more like tree search, like I said before, right? You sort of adapt what you're gonna execute next.

And you sort of refine based on evidence that you accumulated, right? So this is sort of the the notion of falsificationism from from Karl Popper, right? In the sense that. uh we collect evidence for uh hypotheses and reject uh we reject others and we do so in in a loop basically until we

we want to publish or we find something. And we tried to take this notion and directly build it into the agentic scaffolding for the AI scientist v2. So now it's basically like an paralyzable agentic tree surge. where there's no longer a uh template experiment needed, but this is drafted up by the LLM itself. And thereby the AI Scientist V two can be applied to many more sort of settings, if you will. So at the core is sort of this new agentic tree search paradigm.

And then uh we use um sort of a couple of uh minor technical change uh changes like using a VLM reviewer for um sort of figuring out if captions of a paper are aligned with the figures. And uh we we scale this up to many more sort of uh computational notes and then write a paper in the end.

you know, near near the top of the tree we're seeing some novelty and composition happening but it but it it doesn't reflect a deep understanding. What would you say to that chart?

Um but ultimately like we we showed that it was possible to obtain a workshop level paper. I do think this is sort of the first time basically where we can see that at least now we're able to fully autonomously spend computes, spend API calls. to obtain some amount of scientific insights. And for me, at least right now, it's a good way to sort of prototype ideas or to investigate a certain field, get like initial starting point, initial results, and then to to work on top of it.

Um but for sure more work needs to be done to make this uh entire process more robust, more efficient and um essentially produce many more sort of uh true positives.

Yeah. So it's something that could improve over time. But it it's likely to improve quite slowly. And then at some point we might just think, oh my God, we've got an AI scientist.

Um but like uh we haven't made uh like a uh let's say discovery like uh residual connections or something that have diffused into everything in machine learning. And I think what we really need is to make these systems be much better at sort of integrating knowledge over multiple experiments and sort of become better at sort of formulating the next hypothesis based on previous insights.

And uh yeah, this might require some amount of post training on sort of these traces basically. But I'm pretty positive that we might also get there with just diversity and scaling these systems up um in in an efficient but uh scaled up way.

about the AI scientist, right? It's a form factor that we humans are sort of uh used to and it helps anchor our mental model of like a scientific discovery. Um but ultimately uh I'm not sure if the paper is going to be the the knowledge transmission medium in let's say twenty years, right? Something else I've been thinking of um a lot is uh whether or not uh w we can make papers much easier agentically accessible, right? In the sense that uh right now it's

or to work off of them afterwards, right? Doing sort of epsilon improvements, ablations yourself through that interface to a paper. But to be entirely honest I'm not sure if it's gonna happen because there have been many great ideas for improving sort of uh, let's say, the the format of scientific artifacts out there and um people still seem to to like the paper format which has existed for, let's say, hundreds of years, right?

So uh I think it's a question of incentives again and uh really showing that if something like that would exist, it would enable much faster progress of um AI agents.

Where agents can repurpose, recombine, restudy work that has been published by other scientists. And it also made me think. Does like having an automated scientist, does that make peer review more or less important?

for human verification afterwards. Right. So I think uh for now like the ultimate verification is still like the human and the diffusion of the result through the community. And we need better tools for doing this automatically. filtering and verification. Like we have the AI reviewer that comes with sort of the AI scientist. Um but uh you actually probably need some form of experiment execution for actually verifying everything.

Yeah. But I there is, for example, work by uh OpenAI on on paper bench and trying to go into that direction using sort of L L M soft verification and these types of things. So I I'm I'm hopeful that we're going to figure this out in the next years.