Tobi: Hello, friends. This is the Alphalist Podcast. I am your host, Tobi. The goal of the Alphalist Podcast is to empower CTOs with the info and insight they need to make the best decisions for their company. We do this by hosting top thought leaders and picking their brains for insights into technical leadership and tech trends. If you believe in the power of accumulated knowledge to accelerate growth, make sure to subscribe to this podcast. Plus, if you're an experienced CTO, you will love the discussion happening in our Slack space where over 600 CTOs are sharing insights or visit one of our events. Just go to alphalist.com to apply.
Tobi: Welcome to the Alphalist podcast. I'm your host, Tobi, and today with me is Eiso Kant, um, and Eiso is the CTO of Poolside, um, which he built together with Jason Warner, whom I happened to to interview in episode 21 of the Alphalist CTO podcast. Back in the days, he was still CTO at GitHub. So, Jason. So, how did it come that you, you teamed up with the former CTO of GitHub? 01:26:802] Eiso: Oh, there's actually a bit of a story there. Uh, so in 2016, I was building my company called Source. And to the best of my knowledge, Source was the world's first company that focused on making AI capable of writing code. And by the end of 2017, we had code completion models working and and several other things and uh, Jason spotted this and made an acquisition offer from GitHub to to Source to try to buy the company. Uh, I turned down the offer, but nonetheless became very good friends. And uh, well, that's obviously quite a few years ago. Um, and uh, and since then we stayed in touch. We ended up having a podcast for for three years together ourselves as well, which is just a great way for two grown men to find themselves on a regular schedule meeting. And uh, and throughout that, we ended up becoming incredibly good friends. And so when the world kind of had its uh, you know, warning signal of of ChatGPT launching in terms of how it was about to change. It was a very natural evolution for us to to turn those post podcast conversations into what's the world going to look like and what are we going to do in that world? [02:33:952] Tobi: As of today, like are you sad that you didn't sell Source to Jason back in the days or? [02:38:393] Eiso: I, uh, I've never been much for regrets. Um, I think from uh, from a financial outcome, the, you know, it had incredible would have had incredible consequences, but I'm actually very happy I didn't. Um, Source was too early and we didn't succeed. Uh, and for several years that looked like, uh, I was wrong about the future. I had very, very strong beliefs that models were going to be able to reach human level capabilities across software development and beyond. Uh, and looked like a failure, right, effectively so. And then, uh, passed forward a couple more years and all of a sudden we see the incredible things models can do. It set me up for, you know, the ability and and and the right moment and place in time to be able to create pool side. And so I think there's the classic, you know, job speech of the dots connect, we only see it in hindsight. I think this was very much one of those moments. Uh, and so, uh, definitely no regrets, uh, and uh, I'm very glad it brought me to the place that I am right now. [03:41:593] Tobi: Yeah, I I I guess as soon as you understand like the hindsight bias, you never regret regret such things, right? Like, um, but but let's let's come to today. So, um, you're basically now with Jason together, um, founding and operating a company called Poolside that like some of us or some of my listeners might have heard of. Um, and um, I I wanted to give this this episode the title, let's build a foundation model for code, honey. So what you do is like basically building a foundation model, like an LLM for for code, right? Uh, what what also, like, obviously like it's now turned into a very crowded market, but you were like super early, uh, this time not too early, I guess. Uh, like, uh, like anything you want to add? Like what what do you do there? [04:28:963] Eiso: So, you know, our premise a little over two and a half years ago was at the time the world was saying, to reach human level capabilities, to reach AGI and beyond, all we need to do is scale up models, give them more data, do more next token prediction, and we are going to get there. And we didn't hold that point of view from a research perspective. We held the point of view that the scaling of compute was very likely to be directly correlated to the intelligence of the models. The scaling of compute during training. But that the missing key to be able to unlock human level capabilities on beyond was going to become reinforcement learning. And this was a very different point of view on research than the world had, you know, over two years ago. Today, I think most of the companies in the foundation model space have gotten around to this in the last six months. Uh, and so we had our own unique point of view on how to get to human level capabilities in models. Then we decided, we said, look, we're going to put on blinders. We're not going to go and care about all the general purpose capabilities that a strong foundation model needs to have. Right, the ability to write a great poem or to give the right answer on a on a medical test. Uh, but we're going to focus entirely on the on the software development capabilities. And the reason this was twofold. On one hand, we felt that software development are a great proxy for AGI and intelligence. Doesn't encompass everything, but it requires good understanding and knowledge of the world. It requires the ability to reason and plan. It requires quite a lot. Uh, and secondly, is that our view was that software development was going to become the first major economically valuable activity that was going to get impacted by AI. And from two sides. One from the rapid increase that we believe models were going to have on reasoning and their software development capabilities, but also from a very simple point of view, is that developers have been early adopters of technology for as long as they've existed. Uh, and they were going to be the first ones to push this forward and kind of be a canary in the coal mine for all of us and for the rest of the knowledge work. And uh, and so that's why we've been focused on on foundation models. Um, and they are actually very solid general purpose models. You can't build just for code or just for software development. You can't just do code in a model and expect intelligence to emerge. Uh, but we definitely bias our models towards their software development capabilities far earlier in their training, with far more compute spend on pushing those capabilities. [06:56:063] Tobi: Wasn't the fact that Jason owned that key to that Arctic vault with like basically all the code that uh was was open sourced already. Um, like a factor that also counted in, like basically training data availability? [07:09:663] Eiso: Not so much because this is the dirty little secret in foundation model building, right? The 98% of the data that we use and gather from external sources is publicly available on the web. Right? The the data of of Git repositories wasn't behind a lock and and key. It was available for anyone to to get cloned. And so a lot of people assume that proprietary data becomes a big part of the training of models. Uh, and effectively the proprietary data that becomes a big part of training foundation models is synthetic data that you generated a company. Uh, it is definitely some high valued data sets that you that you do partnerships around. But for software development, the vast majority of of of we we call it just pre-training data, you know, in our space, is just available to to crawl on the web. [08:01:423] Tobi: Understood. Before we dive deeper, um, maybe maybe we move a little backwards, um, and zoom out. Um, like from your your memories, what was the earliest memory when you were falling in love with tech? What was your basically your nerd path? Like why why did you get there? Like why why were you doing that? [08:22:215] Eiso: I don't think I've ever spoken about this publicly before. Uh, but it's funny because I was recently reminiscing on this because, uh, my parents mentioned it. And I hadn't thought about this. Apparently, when I was about 10 years old, we had one of those, you know, like it was a Pentium 1 machine and we had one of those printers where you still have the hole punch on the side that we just like the continuous page rolled out. I think there already there were newer printers at the time, but it was just the printer we had. [08:48:785] Tobi: Like the needle printers. [08:49:555] Eiso: And, uh, and apparently I was printing out a 100 page manual on hacking and phreaking. Uh, and, uh, I I for the life of me can't remember which one it was. I think I'd heard about it from some Kevin Mitnick interview or something back in the day. Uh, and I was always already kind of messing around with the system, right? Trying to see what you could do and and and DOS and and and uh, and just essentially kind of hacking things together. And and that's kind of where it started. To me it was, I was a kid who got bullied a lot growing up. The computer at that time was my best friend. It's where I spent all my time on. Uh, and it started there. Uh, and then quickly kind of evolved into, uh, someone else in my high school. Maybe, uh, a year later, a guy named Dennis. And I still make a point out of sending him a thank you message every couple of years. Because Dennis, uh, who was an engineer in the Netherlands, uh, who was a couple of years older than me, took it took me under his wing and said, you know, you're trying to learn this programming thing. I'm trying to write an operating system for these old laptops. So this guy was writing in assembly an operating system from scratch, completely crazy. No one should do this. No one should be doing this. Even I'm not that old, by the way. I'm I'm I'm 34. So if anyone's hearing this that sounds like I'm 50 or 60. But but but Dennis was doing this. And Dennis got smart because Dennis realized that writing the drivers for the keyboard and the screen is going to be a lot of work. So why doesn't he help teach ISO this this kid uh a little bit how to write some simple see to help out. I don't know how much help I ended up being, but Dennis taught me that and then subsequently Dennis taught me PHP. So you can't get two more worlds further apart, right? Very high level. [10:30:445] Tobi: Right. ASM and PHP. That's really weird. [10:33:685] Eiso: Exactly. But it set me up uh for just becoming a huge fan of programming and and and and building software. And that evolved into several years later, uh, also wanting to build things for others. So this was on forums back in the day, you would build CMSs for people or like then you get paid like 100 bucks. And then I ended up doing something that, uh, quite a few of your listeners will have probably had some touch point with. If you were Googling, this is, oof, 20 plus years ago more. Uh, about 20 years ago probably. If you were Googling and you landed on this page that had this very long sidebar of links, like blue coffee cups and red coffee cups in uh, you know, plastic and like and then those pages had a big block of AdSense and then they have this text that was Markov chain generated, like it was barely readable. Uh, I was responsible for a big part of the software that created that. I created something called RSSG, a really simple site generator, which was effectively a way, to be very honest, of of spamming Google with pages that, you know, could rank to the top and had all of these complex, like SEO networks behind them. But it's looking back from a dots connecting kind of point, that's where I kind of first got in in love with text generation. It was with Markov chains back in the day. Uh, and uh, and those things just snowballed. Then at the end of high school, I ended up working on a digital currency called Tlers. This was 2008, 2009. So before the Bitcoin paper came out. I became very passionate with like alternative currencies. Uh, and and yeah, so it kind of a a classic geek path of just going from project to project that I found exciting. Uh, some of them monetized, some of them, uh, were were all they were all very hacky. And then over the years, you know, you you learn how to become better at software development. I ended up building, you know, projects and businesses, uh, that were always software and tech first. Uh, and then all the way to 2015, 2016 when AI caught me because Karpathy wrote a post. Uh, that's actually how I got into this field. [12:42:615] Tobi: Okay. And you already explained like why you're building your own foundation model. But, um, how how does it feel these days? I mean, I I I mean, you also raised like shitload of money, right? You raised also like 700 million or something or almost? [12:56:595] Eiso: 626 million, yeah. [12:58:935] Tobi: 626 million and you, I don't know, how many GPUs do you own? [13:02:695] Eiso: So today we operate 10,000 H200s. [13:05:745] Tobi: 10,000. Wow. Um, so you must be somewhere on the higher higher end of the list uh of of Nvidia buyers, I guess. Uh, but but how does it feel these days? I mean, basically, um, you entered a space where I would say like OpenAI, Google and traffic, like everyone basically also throws in billions now or or like unlimited capacity. Uh, how does that feel these days? Like do you feel a lot of pressure or do you think like our approach is so different that um like we we have space to to breathe? Do you sometimes wake up at night and think, oh shit, what what what's the news? Like what what's in the news? Like who like what whoever published the next codex? Like how does that feel like? [13:53:725] Eiso: So, I think first of all, if you are working on a foundation model and you're and your mission is to reach human level capabilities and beyond, you are in a constant race. And and and you should and and time to breathe and and feeling that you have lots of time, I think is is the wrong way to wake up every day in this uh in this space. Doesn't matter if you are at OpenAI, if you're a pool side, if you're anyway. I think for all of us, it's the constant race to the frontier. And and I think the world in the last couple of years has also gotten around to the fact that we really are on a trajectory to reaching human level capabilities in front of us and across the vast majority of knowledge work and then even beyond. And so, uh, I think it is important though to understand, what are the ingredients in that race? And for us, we always looked at the ingredients as talent and compute to build advantages in research, engineering and data. And the advantages are compounding over time. Right? When you're working on on models, your last generation model is the foundation for your next generation model. Uh, your infrastructure that you're building, your for there's a model factory to be able to go quickly from idea to results and bells is is this constant factory that you're building up so that you can build more capable models quickly and faster. Uh, and within that, uh, you need to acknowledge one thing and we acknowledge this from day zero, is that the fuel in your factory is the size of your GPU cluster. Or the engine in your factory or whatever analogy we can benefit of thinking of here together. And the 10,000 H200s fully dedicated towards training, either model versus synthetic data generation for training or RL for training, like everything that essentially builds a model, uh, is right now one tick away from where the frontier is going to be, uh, and some already is and where it's going to be in the next six to 12 months. Uh, and a tick, I mean a magnitude. Uh, and so it is our job to make sure that we get there. And this is a really important part. If we don't scale up our computer an order of magnitude for training, uh, over the course of the next, you know, period, we won't be in that race anymore. Now, then the question becomes, what do you have to be able to do to show that you can get there? And here I think is where I I I do sleep, you know, I sleep through the night. I do wake up occasionally at 4:00 in the morning with, oh, we have to like think about this and then you write it down. I think everyone in tech does. I think that's when you're in a race, that's kind of common. [16:32:645] Tobi: Unfortunately, yes. [16:33:435] Eiso: Exactly. Uh, and uh, that's about, I don't know, four out of seven nights or so. But the the importance of of of getting to that next level, yeah, is about like what can you do with more compute? Do you have your scaling laws ready to go? Because a lot of the things that we prove out in our industry, you can prove out at a thousand GPUs, a couple thousand GPUs or 10,000 GPUs. So the next scaling access to unlock uh, as you essentially pour more compute into the things that you know can improve and scale. And for us right now, that really gets divided between the bread and butter of foundation model building, improving the data, scaling up the model size, improving the architecture, like everything that is effectively improving compute efficiency or data efficiency. And then really where we think the most important scaling access right now in the industry is, which is reinforcement learning. [17:27:145] Tobi: And how important is it to also build the interface to that? Like, uh, I mean, how how how important is the UI? Uh, is that coming from someone else? Is that like ultimately coming from cursor and Windsurf or others or do you also have to do that? [17:42:612] Eiso: So, I think we got to break UI down. Uh, if we think about the model, the next layer on top is the agent. Right? Agent to me is model loop and environment. It's effectively the model running in a loop, we provide it with tools in an environment in which it can execute, especially in software development. This is very clear, but this holds true as well for, you know, agentic products like a deep search product. And frankly all work that models are going to be doing. We have a saying at poolside that everything collapses into the models and we're seeing this, right? Agents today are increasingly thinner wrappers in terms of code around the model. But what is maybe not yet obvious in the world is that the agent experience that you are building for software development, the actual binary, the actual code that gets wrapped around the model and the tools that connect it to the environment, is the exact same binary that goes into the reinforcement learning. So the way that we are improving models in their ability to build software, reason over it, act, you know, run commands, change files, execute tests, all of the things we that essentially encompass software development is that the product, the agent that you're building is also the thing that the model is being trained with. And this brings me increasingly to the belief that for areas where foundation model companies and really improve model capabilities with training, is that the the best agents, the most capable agents are going to come from the foundation model companies themselves. Because that exact same code with those same prompts is being trained in the in the model. Does that mean there's not tons of amazing [19:18:612] Tobi: Which is basically specs, right? Like you have specs and then you generate code and you execute the code and you see if the results are are proper, right? As to simplify it. [19:27:662] Eiso: Yes, so the way RL through code execution feedback and particularly RL through code execution feedback with agents works is that you have tasks. Tasks will come in all different shapes and forms, from fix this bug, you know, with three words uh to uh build a whole feature, you know, with longer like descriptions. Think of this is how we interact, you know, with models ourselves today. Uh, and and the sheer scale and diversity of those tasks uh determines like how large of an RL run that you can do. So in our case, we have I believe it's the world's largest code execution environment. It's over 800,000 repositories that are fully containerized with their test suite. And in this environment, our model with its with the agent goes and does, you know, millions and soon hundreds of millions and billions of tasks effectively to, uh, to it's modify these repos, to fix bugs, add features, uh, across the different versions of their history. And this is teaching the model how to be a successful software development agent. Uh, which then when you bring it out in in product form, it's the model with the agent wrapper. But this is why I wanted to break down UI. Because on top of that, the form factor can be many different things. Uh, and we have focused on kind of a two prong strategy. On one hand, we want to bring our models and agents to a place that anyone can build on top of them. So a very API first centric world. Uh, we're getting there. Uh, we're not fully there yet, but we're making very fast progress to making that into a place where we want to bring this to the world that everybody can build on top of what we're doing. And then we have our own unique and opinionated views on what a user experience and form factor for software development with AI agents looks like today. And that is an extension to editors, uh, that's a CLI that we uh, we'll be bringing out, uh, and that is also uh uh a view into it from a web assistant. Uh, and that's kind of the way to think about. You have your own product experience, but you want to allow other people to build massively around you because the surface area is massive and the form factors are going to be changing. [21:39:272] Tobi: So the simplest form is basically like an editor where you basically code a prompt and then it generates you pull requests, like similar to to Codex or or or others new solutions, right? [21:50:822] Eiso: I I think the simplest form is a is an agent that's a binary that you can interact with at, you know, programmatic or CLI level. Uh, and you provide it a a prompt, which is usually a form of a task, you provide it a target. So that's a a code base. And then the third thing is you can provide an environment, right? Think of this is a dev container, uh, or a a more scalable environment where where the agent is doing this task. And not unlike by the way, what we do with developers, right? Task, codebase, environment, uh, we I think we can anthropomorphize quite a bit what's happening in models and agents. [22:26:882] Tobi: What's the biggest secret sauce that you have? [22:33:557] Eiso: The biggest secret sauce that we have is is the one thing we don't talk about. But it's it's really work that we do in reinforcement learning for non-verified rewards. What we do talk a lot about is our work in reinforcement learning from code execution. And while it's not a secret sauce, the scale at which we can do it and at which it operates is very large. Uh, I think the second part to, uh, the technology that we've built that really is allowing us to keep pushing models, uh, to to reach the frontier is is the way that we build models, this model factory. Right? We really go end-to-end from raw data sources and crawling and ingestion to fully automatically eval models. All of that is built as a factory. So when you're at pool side and you are working on improving a data set or you're working on changing, improving the attention mechanism of the model or you're working on improving the eval set that we're doing. When you make changes there, it impacts this very large directed acrylic graph that is what a model run looks like. This is actually very cool because what it means is when you're joining here, you can see the immutable end-to-end system. So the way that that the way that that manifests itself is, I see people do PRs, you know, before the weekend where it's, hey, I'm going to do a big sweep over these different data configurations over these, uh, different attention mechanisms, these architecture changes. And I'm just going to kick off the model factory and let it run a thousand small models so that by month by Monday, I can decide which ones of those are most promising and then I scale them up and I try them out. And this is the thing that's maybe not as obvious to everyone is that model building is really turning into a factory that you build in which all of us operate, we improve all the components in it, uh, instead of just like every individual new model that you do is starting, you know, in in a very artisanal manner. Actually, it's it's far more about this is an empirical science and we want to provide the tools to people that they can very quickly like test their hypotheses. [24:40:177] Tobi: How does does versioning of that really work? Like if you say like, I mean, at a certain point, you you you built your new model and then you decide what is good, what is bad? Like how do you version it properly and how do you like differentiate like or separate the good and the the bad? [24:56:078] Eiso: So, it starts with the word immutable. Uh, underneath our entire factory sits, uh, the Apache Iceberg data layer. Uh, where every single experiment, every trajectory, every model trained is its own is its own entity. Uh, and has its own version of the the the code of the factory. So, I think of this as a commit hash in a repo. Um, to give you an idea for a a production model run, about 450 different components are chained together to build a production model. Uh, so every single and a single component can be an asset that generates a certain amount of synthetic data. It can be an eval that is happening over the model. But effectively anytime any experiment gets run or built at pool side, code gets changed, data gets changed, you have that full end-to-end traceability. The second thing part of your question in terms of how do you evaluate what's good and what's not, is that you have a big evaluation framework. Uh, and for any model run that you do, either small or large or a big sweep with lots of small models, is that you determine how often you want that eval evaluation to run. So this can be every, you know, X steps is what we refer to. So every six hours while the model is training or only at the end of the model training. Uh, and and then you get effectively visualization tools where you go in and say, hey, from the 200 variants of model that I just trained on these different data sets, which one performs the best against our evaluation set? Our evaluation set goes from very general types of evaluation, language and reasoning and understanding to very specific software development, uh, you know, capabilities to very kind of user facing capabilities. Like is the model stubborn or is it able to, you know, uh, respond in the correct style and way that we expect it to. So a good eval set in a in a model company is benchmarks we've created. It's, uh, what we refer to as automated vibe checks. So these are often real user prompts that have happened in the past and is is the answer correct? And then models as a judge. You use other models to judge the responses of the new versions and models. [27:17:108] Tobi: And what what happens if a training run fails? Like, uh, like how does that work? Like, is it like really because it's everything is containerized and isolated, you really isolate the failing use cases and then you you you have like a human in the loop that basically double checks it or like how does it work? [27:34:618] Eiso: So often we think about foundation model company there's like the big model run that happens. But by the time you do the make the but the big model run is actually the smallest part of your time. Uh, what's actually before it is thousands and thousands of experiments that get run in the factory before you determine the final recipe and then you kick that off. And within those, there's tons of failures that happen. There's code that doesn't work. There is, you know, uh, effectively anything possible can can fail there because you're you're working on a system. But by the time you kick off a a big model run, so something that's running for two or three months, uh, it's it's very automated. It's actually very anticlimactic to be very honest, Toby. What you do is you have now determined your final recipe, your final config, your final version of the code that you're going to use, right, across your cluster. You allocate your your number of GPUs that you're going to use for that. Uh, in experiments, all of this is dynamically allocated. So experiments can be reprioritized, dynamically allocated. But a big model run, you you kind of take your cluster and say, hey, these 8,000 GPUs are only going to be for this run. You put that in your config, you hit enter, and then you sit back. Now what's going to happen is that about every 48 to 72 hours, some hardware is going to fail. A GPU falls, what we call falls off the bus. That's the actual GPU error. Uh, or, you know, some something goes wrong in your hardware because the hardware is still quite brittle. At that point, uh, you have several machines that are what we would refer to as hot swap machines that go into an automatic restart process. So in during training, there's lots of observability side cars that observe what the machines are doing, how many errors they're throwing, are they successful. So if something fails, 9 out of 10 times, it's an automatic restart and it will take about 15 to 20 minutes out of your training uh, to do so. Most of that is 10 minutes for observability, like did it really fail? And then 5 to 7 minutes to restart everything. Uh, because a training process is all the GPUs need to be interconnected with each other at all times. If you know, one machine fills, you need to swap that machine out. You do this automatically because you have hot swap machines ready to go. And usually during a training run, there will be something that breaks that requires manual intervention. Uh, there's something unexpected that could happen on the hardware. But you just have a regular on call schedule. It doesn't happen very often. It might happen every couple of weeks or or maybe two or three times during a big training run. But at this point, compared to several years ago when model building was very artisanal, a big training run is kind of an anti-climactic event. What you do do is you sit along the way and you see the evals every day, right? You're running evaluations, in our case, we run them every six to nine hours. Uh, and so you get updated evaluations along the training to see how the model is improving in its capabilities. But by the time you're running a big model run, you're you're not really going to stop it. You have experimented for every version of your data, architecture. You know what you what you're training. It's extremely unlikely that a month into training, you say, oh stop, I'm going to go and do something different. It is highly predictable at that point from your scaling laws what to expect from the model. [30:42:918] Tobi: Understood. And, um, how do you, how do you operate? I mean, I guess Jason is somewhere in the US, you're right now in Spain, uh, like you're distributed, like fully or? [36:04:778] Eiso: We're we're fully distributed team. Uh, we're about 120 people. Uh, our team sits between the US and Europe primarily. Uh, from a research and engineering perspective, our team is on the East Coast of US and Canada and in Western Europe. Uh, almost entirely with a few people just uh a little bit beyond. Uh, that for us was a really important decision we made early on. We said instead of hiring in the backyard of our competition in the Bay area, we're going to find the world's most incredible talent that sits outside of it. Uh, and we found incredible people all over Europe, incredible people on the East Coast. We have moved people from Asia to Europe as well to join us. Uh, and uh, and I think that's been uh, that was probably one of the best decisions we could have made. Uh, it allowed us to to have this incredibly strong team, uh, without having the the same level of uh, noise that you have in the Bay Area and and just the challenge of of of of competition there. Uh, today I think it's a global market. So two years down the line, right, our our team is getting uh, recruited by others as well and vice versa and people are moving. So that that's it's it's less of a matter now, but it mattered a lot two years ago when we started building. Um, and but yeah, we operate fully distributed, but we do something a little bit unique. Our entire research engineering and product or come together in person every month for four days. [37:29:438] Tobi: Okay. Yeah, that's very unique. I just wanted to ask like how much synchronous time do you have together? So you also realized that like fully remote has its challenges and uh especially in such a research heavy environment, you you need to meet up every once in a while or how did that happen? [37:48:038] Eiso: Remote is a function of the fact that if you want to hire the world's best people in our field, you want to do it outside of the Bay area, you're not going to find them in a single location. You're not going to find them all in London or all in New York or all in, you know, Zurich. Uh, and so by nature, if you put your talent bar where it needs to be, and I think it has to be, you are going to have to be remote. You're either going to have to be in the Bay Area or remote. There's just not really a a choice in the Western world for for either. Um, so that was I think that the the baseline. Now remote comes with advantages. It allows people to do a lot of deep work while integrating it with their lives. Right? We're in a race. People work at incredible amount of hours at poolside. Uh, and and so being able to do a lot of that remote and integrated with your life, I think has allowed people to do so in a way that doesn't burn them out. Uh, but still really manages to make a a massive commitment to the to the mission. Then, uh, but still at the end of the day, if you could have everyone, you know, in the same building would be amazing. Right? It it just allows for a certain level of communication that we don't get to do with remote. So while remote has certain advantages over being in an office, right, that deep work, being together with each other, having good communication, uh, building those relationships, strengthening the bonds so that you can go off and then have a five minute chat on Zoom with someone you already know, uh, in person doesn't, you know, can't be beaten there. And so we take a hybrid approach. Um, and the nice side effect of it is, because people are really away from their daily lives for four days, right, they travel, they're there, we're fully 100% together and just, you know, working through things. Uh, it also means that it's kind of like a it's a shift that you make. It's kind of like as if you go on, if you're on a plane or you go on holiday, somewhere all of a sudden like you you get out of your environment, you have a bit of time to step away to think differently. Uh, and so we've seen that combo work really well. Uh, and uh, I've been increasingly more excited about it. I think it can work well for lots of companies. It's an investment. It's a financial investment. I don't want to understate that. But I think it gets the best of both worlds. Uh, but if you ask me very honestly, if you could have everyone in one place in an office, do I think that it could be even better? Uh, I I think there's probably a delta there. [40:13:908] Tobi: Um, you and Jason, you're both like very senior tech leaders. How how do you divide responsibilities without stepping stepping on each other's toes? [40:26:418] Eiso: Very simple, MIT, most important thing. Uh, we have a so me, Jason, uh, and and Marguerite our CEO, we run a daily meeting. Uh, we every day that meeting starts, what's the most important thing? And then we understand what maps to each other's skill sets. Uh, now there are parts of the org that we've already mapped to like each other's skill sets and and we focus on that. So for me that's research engineering and and and product. Uh, for Jason, that's oriented towards the the go-to-market side where a CRO, Paul St. John. But effectively in this race, you both end up spending lots of time and lots of places. If that's with partners, if that's with customers, uh, if that's with investors. And here it just really becomes around like constantly shifting, uh, the, you know, where can each person be most effective, uh, while making sure that your core parts of your business that you're responsible for are, uh, are on track and having incredible leadership there that can can drive a lot of things forward. But also, I mean, it only just comes down like if you have, you know, we care about five qualities in the company for everyone we hire. And it also hold true for for me and Jason and and looking at each other. It's low ego, kind hearted, extremely hardworking, lots of intellectual horsepower, and deeply care about the work you deliver. And that's that first one, low ego, I think makes it pretty easy to to work with each other and divide responsibilities based on what's needed in the moment. Uh, we're all willing to sweep the floors at pull side. [41:54:908] Tobi: Um, when you're stuck, who do you call? Like, is there any any mentor in your life, uh, or any any other weird sources of advice? [42:04:888] Eiso: So, oh, it's, well, first of all, I think having been around for a little while, you definitely have a a big bench of people that you call up for different things. Right? And and a lot of that is often domain knowledge or domain expertise. But the one that is, uh, relatively new to me since, uh, what's it, August, September of last year, is that I finally found a coach that uh, that really works, uh, that really has a a good impact, I would say. Uh, or at least I feel very like glad to have. I tried this before, you know, you kind of, I think every couple of years you try a coach and it really never really stuck. And it was a lot about not having found the right coach. Uh, his name is Bill Beswick. Uh, you can you can Google him online. Uh, Bill is, I hope I'm I'm saying this correct because otherwise he's going to be upset. But I think Bill's well into his 80s by now. Uh, and uh, he's a a sports psychologist and former basketball coach. Uh, so he had roles with like the English football club, Manchester United, trains a whole bunch of primarily athletes, funny enough. Uh, a few people in business. And Bill's been really great to kind of take once a month, like a pause and talk through everything that's going on in the craziness of of AI, of the business, of of of of life and and have him kind of, you know, share mental models, anecdotes, things that help you kind of, uh, you know, think through whatever is the most important thing at that time. Uh, and and my favorite one of his, and Bill usually comes with the big stories, right? When I coach this, you know, coach here or I got this team and there's a beautiful story and that kind of, you know, the story narratives are a great way to remind you of the mental models to use. But, uh, a while back, I had a conversation with Bill around this race that we're in in AGI. And effectively I said to Bill, I said, look, I want to win. I be very honest. He knows this, like we we want to pool side is is never just been about software development. Pool side is about, you know, building intelligence and and reaching it to a level where we can scale intelligence on computing the world. It's the most magical thing you get to work on creating. Uh, and I said, Bill, I don't know what I would do if I lose. Right? Uh, I've just never really thought about it. Like we're just here with putting the work, uh, and and, uh, and I would I couldn't I can't think of anything else that my skill set, I can go after that's bigger. Like this is kind of the ultimate thing to get to work on. And so I was expecting this great anecdote from Bill, you know, if you lose, you go off to the countryside, you do this and you're like, etc. And they always have a nice story. And this time Bill just like sits there and he kind of looks at me and he's like thinking and he goes, yeah, ISO, just don't lose. He's like, I have no idea. He's like, you're right, it's going to suck if you do. Just don't lose. Uh, and and and and that's probably the best advice that I've gotten in a long time. Uh, and that hopefully tells you a little bit as well at the culture of pool side, we're we're here because we think AGI is in reach. We want to double down towards that. Uh, we think there's an incredible form factor to be built around the models as well. We didn't spend much time on this by the way, for businesses and developers at enterprises. But really at the core, uh, that's really what this is, uh, you know, what this business is about and he's probably been one of the most impactful mentors recently. But then there's tons of people you call up investors, uh, executives that you've worked with before and known in the past when it comes to very specific things. [45:41:408] Tobi: Any any crazy learning that you made uh that you want to pass on to to other CTOs here? In the last months? Any anything that really opened opened your eyes? [45:54:336] Eiso: I I'll give two. One is kind of, uh, maybe an obvious one and the other is, uh, is maybe more of a something that I'd like to stress. So the obvious one to me is when we're when we're building pool side, we are genuinely in a race. Not every business operates by being in a race. Sometimes you're just competing against yourself or against your runway. Uh, but you're not really in a competitive race against others. Right? In my race, it's very clear. It's it's Google, it's Open, it's Anthropic, it's XCI. That's effectively like the the frontier. Uh, that uh that we're, you know, looking to reach and and and we compete with. And in that race, it is genuinely an existential race. If you lose, you lose. Right? I think and and this that's how we frame it. And when you are very lucky to be able to be in a position where you get to recruit the world's best people to join you on that journey. And a lot of that comes from the fact that we have a lot of capital and we've been very privileged that people have been have been willing to give us a lot of capital. And so that also opens up the spectrum of who you get to recruit. Just be very transparent about this. And so when you get to look for bringing along the world's best people, uh, on like a very, very competitive mission and race that you're on, you can throw there's lots of things from how you do things in a business, you can throw out the window. Uh, you uh a lot of the processes that we've built in software development, uh, are built so that we can operate across the mean, across the average. Uh, and when you get to hire these incredible people who you empower, uh, you can start finding, well, what does this process look like when I have such very strong, like such strong talent everywhere? That it doesn't mean that as you're scaling up, there's not work you have to do on roadmap planning and project management. Like not everything goes out the window. But you can actually start taking kind of a view and say, well, let's break some of the rules of what how I used to do things. Maybe it doesn't need a daily stand up. Maybe it doesn't need X, Y or Z. No, no Scrum. Yeah. Exactly. Right? And and and we've done a lot of stuff in tech over the last few years to really build like the best way of doing things. And I do think a lot of them are really good and they're and they're very generally applicable. But, uh, it's really worth asking yourself the question of, in a certain team, in a certain setup, uh, do you need to do it exactly that way? Uh, and so there's places where I think we we've definitely made some changes there. Now, as we're growing as a larger organization, you also see that some of those things have to come back a bit. Right? A lot of the, and I think this is something you learn over and over again in in tech, being small is an advantage. And when you have that advantage, don't cargo call all of the big company processes. Right? Take advantage of being small where you where you sit. Take take advantage of when you're still when when when you can still do that. So that's probably I would say, uh, not a recent lesson learned, uh, but it's one that I think comes up over and over again at poolside. Um, and then the second is, and I've been saying this for for several years now. Uh, please, please, please operate from a place where you assume that in X years, for me the X is less than three, maybe for someone else it's five or 10. Like I I I personally think it's less than three. AI is going to reach human level capabilities across the vast majority of knowledge work that we do behind the laptop, including software development. And in that world, what what does your organization look like? What does software development look like? Now, it doesn't mean that there's tons of things you have to change overnight today. But make sure you stay very close to the technology so you can have a sense yourself of how far along we are to that journey. Uh, if you're still on the bandwagon of AI is never going to get there. AI is just a stochastic parrot that repeats just some statistical things back to me. I really, really urge you to spend time, you know, with the the latest reasoning models in the world. Try them out on on your tasks and and build your own evaluation set so that you can see how the progress goes. Because this is going to fundamentally change the nature of software development. It is going to fundamentally change how we operate, you know, small and large organizations. Uh, we are going to be living in a world where we are managing large workforces of agents. Uh, I think it's it's hard to imagine at this point that we're not. [50:42:018] Tobi: That's true. But like, what what will happen in let's say five years? Do you think we will still write code and review code or will it be different already? [50:54:158] Eiso: I think there will always be some people who write and review code. But I think the vast majority of software that gets built, uh, is not because we ourselves are writing the code. It's us managing and working together with agents, uh, who are working either with us synchronously, right, real time back and forth, asynchronously, we're sending them off to work and it's coming back to us that we're either reviewing, or even autonomously places where we place agents that are just working through autonomous tasks like trying to fix the latest UI bug that got reported or observing the logs and trying to auto heal and fix the system if it's if it's having an issue. Uh, and so I think the the qualities in in software development for us to be able to think through what to build, how to bring things, you know, what what is the agency that we're going to take to make certain changes is key, but a lot of the writing of the code is going to be increasingly done by AI. [51:53:938] Tobi: And, um, how will we solve the the context window issue? I mean, when when and how will we solve it? Like, um, because it seems kind of feels kind of unsolvable, but um, I guess it's not and you have the answer already. [52:08:292] Eiso: So, I will I will challenge if it's going to be an issue first and foremost. And what I mean by that is if you think context window as working memory, how much state that you can keep in your own mind, right? And state in a model at a request. Uh, you realize that our working memory is also relatively small. It's limited. It's it's probably not a million tokens. Maybe it is for some people, but it's it's it's not for me. And the reason I mentioned this is that when models reach human level reasoning levels in software development and effectively are are getting to a place where they can do so, then and first and they're on a trajectory models to be, you know, a hundred or a thousand times faster than us. You know, we had a I think when we had our our call before the podcast a couple of weeks ago to me I might have showed you our internal version of a pool side model running at 2,500 tokens per second. That's still very much in the research lab, but it's it's showing us a world where, uh, models are going to be so much faster than us. And the reason I mentioned this in relation to the context window is if you are able to very, you know, do high quality, reliable reasoning over a multi-step process, you can use the context window you have and offload and onload things onto that stack and off that stack. Right? [53:34:102] Tobi: So you basically, the the model basically decides, hey, this is the context I need to have. So let's switch to this and this and that file and this and this and that um, directory. [53:44:632] Eiso: And let me remember these things from the last trajectory I took and you know, let me make these things, let's store these in cold storage where I'm going to only be able to retrieve them through search. Now, I don't think that is the final end state of models. Absolutely. We want models to have continual learning. We want them to be able to have their weights change over a longer term memory. We want different agent is running for agents are running for years inside an enterprise to be able to have the model weights are evolving towards that. And we're doing quite a bit of work towards that at side to get ready for that secret where where the model weights are individual to the organization or individual to the, you know, one day maybe even individual to the developer because getting cheap enough. Uh, and but I do think that uh, right now we are on a trajectory that increasing the model general intelligence and reasoning and coding capabilities can work around a lot of the context window. And then there's other just very technical details that are interesting, right? We're we're we've seen very interesting work uh on our end by combining the advantages of a recurrent neural nets with the advantage of transformers to allow for bigger kind of fixed state spaces. Uh, and and so I do think that uh models with much larger context windows than we have today will be something that we'll see more of in the world. Uh, but I also think that for many tasks, we're actually going to prefer the model to iterate back and forth over many inference requests faster than trying to stuff everything in a massive context window. That's that's not the strategy we can visit as a winning one. [55:16:912] Tobi: Yeah, it also sounds somehow like not not logic to do that, right? I mean, because also like a human, um, are I think also not really able to do that, right? Like you you can't throw me like a large context and expect me to solve an issue, right? Um. It's to spot the issue. [55:35:882] Eiso: We have a bias. At poolside, we definitely take uh an anthropomorphized human-like view of the type of intelligence we're trying to build. Doesn't mean that models are the same as humans. There's definitely clear differences and there's parts where they're already stronger than us and parts where they'll maybe forever be weaker. But, uh, I do think that I don't have a million light context window and given enough, if I could do things 100 times faster, uh, being able to use some external tools and external, you know, retrieval, uh, together with complex reasoning can probably solve for most of what we're trying to solve for. [56:12:942] Tobi: Cool. What was the most incredible tech you discovered or learned about in the last six months? [56:20:008] Eiso: Oh, uh, I'm I'm instantly jumping to the research side. Uh, and here I actually can't comment on it. Uh, but uh, let me try to find something else from a product perspective or such. Uh, it's actually probably not a specific product. I'm I'm just in coming back to the phrase that everything is collapsing into the models. I'm finding myself increasingly more and more working directly with the models and the agents that wrap them than actually with other products. Uh, I'll give you a very silly example. Uh, I have a lot of travel for work and so my weight has been steadily going up over the last two years of poolside. And and I recently decided to do something about it. And instead of, you know, using my fitness pal or the weight loss tracking app, etc, I decided to just have a daily conversation, uh, with the model about what I should be doing. Uh, and so, you know, I told poolside, I'm like, hey, this is roughly what I want to follow in terms of nutrition, protein, other things. And and and throughout the day I'm having that conversation with it. Should I eat this? This is the menu of the, you know, restaurant. Should I like, what should I pick? Uh, because I'm on the road a lot. And and it's been very successful, uh, in in helping me lose weight over the last seven weeks that I've been doing this. But I think all of us are finding that the model is becoming a companion. Not just in our software development work, but it's becoming in lots of other parts of our lives. And I don't think that means there's not incredible software to be built on top of models. Text and and voice are not the perfect interfaces. Uh, but that's probably what I continue to be excited about, like discovering the next use case at which you're using intelligence on compute, because that's effectively what this is. Uh, and that's probably uh, yeah, the a bit of a non-answer, but uh, hopefully it's [58:32:028] Tobi: No, it's a good answer. It's a good answer. Um, I still had have a little, like, uh, slowly coming to the end. I still have a little surprise for you because, um, I also had a a prediscussion with Jason. Oh, nice. And, um, he basically told me about like a very early Easter egg, um, in your facepalm model, which is, um, a security hole. Um, like you can inject commands. Um, and there's one, one command that he recommended me to inject because like that's one thing that only your model can do and that's also the reason why like, um, it's it's it's it's very popular, um, or was very popular internally, it's the time machine feature, which basically or it's basically possible to inject like a a command to travel back in time. And imagine we now, um, are doing that. Like we inject the command and we're jumping back to 2016 when you were building sourced, like basically you just started, right? Um, and, um, you're trying to convince the world that language models can write code. And and you now have the chance to whisper something into young ISO's ears. What what would you whisper? [59:46:138] Eiso: Oh, uh, I would I would essentially tell it, uh, you're right, language modeling plus RL can go all the way to human level capabilities. Forget everything else that you're that you're, you know, like we were building product around, like all of these things, like just keep doubling down, never stop. Right? And and and and probably and and on top of that I would throw, uh, scale compute, scale compute, scale compute. Uh, if it's 2016, I would also tell it about Transformers. So if it was a little bit later, I would, you know, we were already on there, but I would tell it like, hey, LSTMs are not going to go all the way. Uh, but, uh, so I would I would either give it the recipe of transformers plus but a lot of this is a little bit the the four-minute mile. Right? This is I I I I think we hold a depth of gratitude to to OpenAI because it it proved the four-minute mile. Right? And and and if it wasn't for that, you know, I wouldn't have been able to get back into the space. Uh, and so, uh, I think this is an important thing. I think I would have just tell ISO that. It's a four-minute mile. Like, it's it can be done. It's going to be done. It's going to be done on November 2022 by someone else. Uh, scale up next, scale up next token prediction and trust your gut on on reinforcement learning plus language modeling. Right? That's what we we started working on reinforcement learning by code execution feedback back then in combination with the early language models, LSTMs and then Transformer. But a lot of it was just, you know, you were doubting yourself. Like you had this strong conviction, but like was it a statistical parrot? I got into this space because I read an article by Andrej Karpathy called The Unreasonable Effectiveness of Recurrent Neural Nets. That launched me into this space. And then a year later, AlphaGo came out. And it's a combination of those two things that's just had imprinted on me ever since like what is possible, right? And that's now uh, it's now a decade ago. Uh, and still today, that's actually it. It's the unreasonable effectiveness of language modeling with the the scaling of reinforcement learning. Uh, and that seems to be the recipe right now. Uh, and that's why we built this company and what we're going to keep pushing, uh, in the coming years.
008] Tobi: I wish you all the luck in the world, uh, that you succeed and and win ultimately. Um, so yeah, really like crazy, right? Uh, thanks Eiso for for sharing your story and and your thoughts. Um, really great. Um, any any shout-out you want to make? Uh, any anything you want to want to tell like other CTOs out there?
048] Eiso: I think the the biggest shout-out is, uh, we're moving towards a world where we're going to be collaborating with agents. And everything you can do today to spend time with models and agents to get a good sense of where that world is heading is is time really well spent. 148] Tobi: Thank you. Have a great day. Have a good flight. 968] Eiso: Thank you, Tobi. Appreciate it. 908] Tobi: See you soon. Bye-bye.
212] Tobi: Thank you for listening to the Alphalist podcast. If you like this episode, share it with friends. I'm sure they'll love it too. Make sure to subscribe so you can hear deep insights into technical leadership and technology trends as they become available. Also, please tell us if there is a topic you would like to hear more about or a technical leader whose brain you would like us to pick. Alphalist is all about helping CTOs getting access to the insights they need to make the best decisions for their company. Please send us your suggestions to [email protected]. Send me a message on LinkedIn or Twitter. After all, the more knowledge we bring to CTOs, the more growth we see in tech. Or as we say in Alphalist, accumulated knowledge to accelerate growth. See you in the next episode.