OpenRouter: The Only AI Tool You'll Ever Need | Founder Alex Atallah

Ejaaz: What if I told you there was a single website you could go to where you can Ejaaz: chat to any major AI model from one single interface?

Ejaaz: It's kind of like chat GPT, but instead every prompt gets routed to the exact Ejaaz: AI model that will do the best job for whatever your prompt might be. Ejaaz: Well, on today's episode, we're joined by Alex Atala, the founder and CEO of Open Router AI. Ejaaz: It's the fastest growing AI model marketplace with access to over 400 LLMs, Ejaaz: making it the only place that really knows how people use AI models, Ejaaz: and more importantly, how they might use them in the future.

Ejaaz: It's at the intersection of every single prompt that anyone writes and every Ejaaz: model that they might ever be. Ejaaz: Alex Atala, welcome to the show. How are you, man? Alex: Thanks, guys. Great. Thanks so much for having me on. Ejaaz: So it is a Monday. How does the founder of OpenRouter spend his weekend?

Ejaaz: Presumably you know out and about chilling relaxing not at all focused on the company oh Alex: I usually i love weekends with no Alex: meetings planned and i just go to a coffee shop and just have tons of hours Alex: stacked in a row uh to do things that require a lot of momentum build up so Alex: i did that at coffee shops on saturday and sunday and then i watched blade runner again.

Ejaaz: Again okay um well so Ejaaz: when we were preparing for this episode alex Ejaaz: um i couldn't help but think that you've had a pretty insane decade of startup Ejaaz: foundership right um so open router is kind of like your second major thing Ejaaz: that you've done but prior to doing that you were the founder and cto of OpenSea, Ejaaz: the biggest NFT marketplace out there. Ejaaz: And now you're focused on one of the biggest AI companies out there.

Ejaaz: So it sounds like you're at kind of like the pivot point of two of the most Ejaaz: important technology sectors over the last decade. Ejaaz: Can you just give us a bit of background as to how you ended up here? Ejaaz: And more importantly, where you started.

Ejaaz: Walk us through the journey of OpenSea and how you ended up at OpenRouter AI. Alex: Yeah, so I co-founded OpenSea with Devin Finzer the very beginning of 2018, very end of 2017. Alex: It was the first NFT marketplace. And... Alex: It was not dissimilar to OpenRouter in that there was a really fragmented ecosystem Alex: of NFT metadata and media that gets attached to these tokens.

Alex: And it was the first example of something in crypto that could be non-fungible, Alex: meaning it's a single thing that can be traded from person to person.

Alex: Most things in the world are non-fungible. A chair is non-fungible. a Alex: currency is fungible so it was Alex: back back in 2018 no Alex: one was really thinking about crypto in terms of non-fungible goods Alex: and uh and the problem with the non with Alex: non-fungible goods is that there weren't any real standards set up Alex: um there was a lot of heterogeneous like Alex: implementations for how to get uh like

Alex: a non-fungible item represented and tradable in a decentralized way So OpenSea Alex: organized this like very heterogeneous inventory and put it together in one Alex: place. We came up with like a metadata standard. Alex: We did a lot of like a lot of work to really make the experience super good for each collection. Alex: And you see a lot of those a lot of similarities with how AI works today, Alex: too, where there's also just a very heterogeneous ecosystem.

Alex: On a lot of different APIs and different features supported by language model providers. Alex: And Open Router similarly does a lot of work to organize it all. Alex: Um i was at open sea uh until 2022 um when i was kind of feeling the itch to do something new. Alex: And um i'm at the very end of i left in august and then chat gpt came out a few months later.

Alex: And uh and my biggest question around that Alex: time was whether it was going to be a winner take all market Alex: because opening i was very far ahead of Alex: everybody else and um you know Alex: we had cohere command we had a couple open source models um Alex: but opening i was the only really usable one i Alex: was doing little projects to experiment Alex: with the gpt3 api and uh Alex: and then llama came out in january um really Alex: exciting about a tenth the size one on a

Alex: couple benchmarks but it wasn't really chattable yet and Alex: uh and it wasn't until uh a few Alex: months later that somebody a team at Alex: stanford distilled it into a new Alex: model called alpaca um distillation means Alex: you you take the model and you customize it or fine tune it Alex: on a set of synthetic data that they Alex: made using chat gpt as a research project Alex: and uh and that was it was Alex: the first successful major distillation that i'm aware

Alex: of um and it was an actually usable model i Alex: was like on the airplane talking to him i was like wow this Alex: is if it only took six hundred dollars to make something like this then you Alex: don't need ten million dollars to make a model there might be like tens of thousands Alex: hundreds of thousands of models in the future and suddenly this started to look Alex: like a new like economic primitive a new building block that people that kind

Alex: of deserve their own place on the internet.

Alex: And there wasn't one. There wasn't a place where you could discover new language Alex: models and see who uses them and why. Alex: And that's how OpenRouter got started. Josh: That's amazing. So one of the things that we're obsessed with on this channel Josh: in particular is exploring frontiers and how to properly see these frontiers Josh: and analyze them and understand when they're going to happen.

Josh: And when I was going through your history, you have this talent consistently over time. Josh: And even as far back as early on, I read you were hacking Wi-Fi routers in a hackathon. Josh: You're very early to that. You were early to the NFTs. You were early to understanding

Josh: AI and the impact that it would have. And what I'd love for you to explain is Josh: the thought process and the indicators you look for when exploring these new Josh: frontiers, because clearly there's some sort of pattern matching going on. Josh: Clearly you have some sort of awareness of what will be important and why it Josh: will be important, and then inserting yourself into that narrative.

Josh: So are there patterns? Are there certain things that you look for when searching Josh: for these new opportunities and that led you to make these decisions that you have? Alex: I think there's there's a lot to be said for finding enthusiast communities Alex: and and seeing if you're going to join it. Alex: Like, can you be an enthusiast with them?

Alex: Like whenever something new comes out that has like some kind of ecosystem potential, Alex: there's there are going to be enthusiast communities that pop up. Alex: And the Internet has made it self-certain. You could just join the communities.

Alex: Um discord i think is a incredible Alex: and super underrated platform because Alex: the communities feel kind of private you're Alex: like getting you don't feel like you're you know Alex: seeing somebody trying to get s you Alex: know like advertise something for seo juice there's Alex: no seo juice in discord um it's it's Alex: just people talking about what they're passionate about and and it Alex: goes it gets really niche um and when

Alex: you find a like an interest group in discord that Alex: like has to do with some some new Alex: piece of technology that's just being developed right now and doesn't really Alex: work very well at all um you get people who are just trying to figure out what Alex: to do with it and how to make it better and i think that's like that's the first Alex: core piece of magic that jumps to mind, Alex: there's got to be like a willingness to be weird because if you jump into any

Alex: of these communities at face value it's stupid. Alex: Like oh this is like just a game or it's like a really weird game I mean I'm Alex: not really a collectible game so I'm going to leave right now and yeah. Alex: Not only do you have to be aware, but you have to be creative. Alex: Like, okay, these are just cats on the blockchain, and people are just trading cats back and forth. Alex: You can't look at the community as simply that. Alex: Think about what you could do with it.

Alex: Like, what is this unlock that wasn't achievable before?

Alex: Um and uh Alex: and and i think there are there are people who Alex: just are good who will do this and they'll join the communities Alex: and and brainstorm live and you can see everybody Alex: brainstorming uh in real time but like Alex: another incredible example of this was the mid-journey discord Alex: you know it became the Alex: biggest biggest server in discord by Alex: far uh and you know Alex: why did that happened well you could it started with Alex: something weird silly maybe

Alex: not super useful but you could see all the Alex: enthusiasts like remixing and Alex: brainstorming live how to turn it into something beautiful Alex: and how to how to make it useful and um Alex: and then you know just explode it like i it's the most it's the it's the most Alex: incredible like niche community uh i think that discord has ever seen because Alex: of like how useless it started and how insanely exciting it became.

Alex: So um like i mean i i Alex: think i saw big sleep i was like playing around with this model Alex: called big sleep in 2021 that uh Alex: let you generate images that Alex: look kind of like deviant art okay and Alex: uh you could see you could like they're all Alex: animated images and they none of them really made sense but you could get some Alex: really cool stuff not like potentially something you'd want to make your desktop

Alex: wallpaper and if you're really like deep in some deviant art communities you Alex: know you kind appreciate it and so and that that that was like oh there's like Alex: a kernel of something here, Alex: and uh it took like a like another year or two before mid-journey started to Alex: like pick up but that was like.

Ejaaz: Where were you seeing all of this alex like where were you scouring just random Ejaaz: forums or just wherever your nose told you to go Alex: But basically there's this twitter account I'm trying to remember what it's Alex: called that posts AI research papers and and like kind of tries to show what you can do with them. Alex: And I discovered this Twitter account in like 2021.

Alex: And I. Alex: I think it was not it was it wasn't at all like related to crypto but it was Alex: a way you know big sleep was like the first thing i saw that used ai to generate Alex: things that could potentially be nfts, Alex: so i started experimenting around like how how much you could direct it to make Alex: an nft collection that would make any sense it was very very difficult um but Alex: that was how uh that was like the first generative and.

Ejaaz: This was before you were even thinking about starting open router right Alex: Um yeah yeah this was back this was when i was Alex: full-time at openc um oh is Alex: yeah i got the it's a colic Alex: this twitter account all right Alex: i really recommend it they basically post papers and like explainate and explore Alex: how this paper gets useful um they post animations uh like they make they make

Alex: ai research like kind of fun to engage with and that was that was my first experience. Ejaaz: Okay, so I mean that's a massive win for X or formerly as it was known back Ejaaz: then, Twitter as a platform, right? Ejaaz: It gave birth to kind of like two of the biggest technologies crypto, Ejaaz: also known as crypto Twitter, and now apparently all the AI research stuff which Ejaaz: kind of put you on to the path that led you to OpenRatter.

Ejaaz: So if I've got this right, Alex, you were full-time at OpenSea with a Ejaaz: multi-billion dollar company loads of important stuff to do there, Ejaaz: but you still found the time to kind of scour this fringe technology because Ejaaz: that's what AI was at the time. Ejaaz: Prior to kind of GPT-2 or GPT-3, no one really knew about this.

Ejaaz: And you were playing around with these gen AI models, these generative AI models Ejaaz: that would create this magical little substance and maybe it came in the form Ejaaz: of a pitcher or a weird little cat. Ejaaz: And you kind of like jumped into these niche forums of enthusiasts, Ejaaz: as you say, and kind of explored that further. Ejaaz: And it sounds like you kind of like honed that even beyond your journey from OpenSea when you left.

Ejaaz: I remember actually meeting you in this kind of like this abbess between you Ejaaz: leaving OpenSea and starting OpenRouter where you were kind of brainstorming Ejaaz: a bunch of these ideas. And I remember a snippet from our conversation Ejaaz: In like one of the WeWorks here, where you just kind of like had whiteboarded a bunch of AI stuff. Ejaaz: And one of those things was kind of like the whole topic of inference.

Ejaaz: And if I'm being honest with you, I had no idea what that word even meant back then. Ejaaz: I was extremely focused on all the NFT stuff and all the crypto stuff, Ejaaz: my background's in all of that. Ejaaz: But I just found that fascinating that you always had your nose in some of the Ejaaz: early communities. And I think that's a really important lesson there.

Ejaaz: I want to pick up on something that you actually brought up when you said you Ejaaz: discovered kind of like your path to open router, Alex. Ejaaz: And that is, you said you were playing around with these early AI models. Ejaaz: So not the GPTs before Claude was even created. Ejaaz: You're playing around with these random models that you would find either on Ejaaz: forums, on Twitter, or on Reddit, right? and you would experiment with them.

Ejaaz: And I find it fascinating that back then, even when GPT became a thing, Ejaaz: you were convinced that there would be hundreds of thousands, Ejaaz: or did you say hundreds of thousands of AI models? Ejaaz: Back then, that wasn't a normal view. Ejaaz: Back then, everyone was like, you need hundreds of millions of dollars. Ejaaz: Maybe it was tens of millions of dollars back then. And it was going to be a rich man's game.

Alex: Yeah, it was basically the Alpaca Project that kind of put me over the sack.

Alex: On there being many, many, many models instead of just a very small number. Ejaaz: And can you explain what the Alpaca project is for the audience? Yeah. Alex: So the Alpaca project, after Lama came out, you really could not chat with it Alex: very well. It was a text completion model. Alex: There were a couple benchmarks where it beat GPT-3. Alex: And... It was about a tenth the size of what most people thought GPT-3 was sized at. Alex: So it was a pretty incredible achievement.

Alex: But it wasn't really like, the user experience wasn't there. Alex: And the Alpaca project took ChatGPT and generated a bunch of synthetic outputs. Alex: And then they fine-tuned Llama on those synthetic outputs. Alex: And this did two things to Llama. It taught it style, and it taught it knowledge. Alex: It taught it, like, the style is like how to chat, which was the big user experience gap. Alex: And it made it smarter. Alex: Like, you can, fine-tuning transfers both style and knowledge.

Alex: And the model would, like, respond to things that it had, you know, Alex: like, the content of the synthetic data, like, was reflected in the model's Alex: performance on benchmarks after that point.

Alex: So um so if you can do Alex: that without revealing all Alex: the data that goes in um now now Alex: there's like a way you could sell data via api without Alex: like like just dumping all the data out to the world and then never being able Alex: to to like monetize it again so there's like a brand new business model around Alex: data that emerges um yet like the ability to create just like work towards open intelligence, Alex: and uh and build like new

Alex: architectures test them more quickly and and and Alex: uh uh fine-tune them quickly basically you Alex: can build on top of the work of giants i mean Alex: you don't have to start from zero every time a lot Alex: of like the biggest developer experience innovations just involve like giving Alex: developers a higher stair to start walking up so they don't have to start at Alex: the bottom of the staircase every single time um and you know that was like the the the big.

Alex: Like generous give that llama had for the community um and it wasn't you know Alex: that wasn't the only company doing open source models, Mastral, Alex: came out with 7B Instruct a few months later. It was an incredible model. Alex: Then they came out with the first open-weight mixture of experts a few months later. Alex: It felt like actual intelligence, but completely open.

Alex: And all of these provide higher and higher stairs for other developers to kind Alex: of like, basically to crowdsource new ideas from the whole planet.

Alex: Uh and and let these new ideas build on Alex: top of really good foundations so and Alex: you know when that when that like whole picture started Alex: to form into place um it felt like okay this is going to be like a huge inventory Alex: situation you kind of like nft collections were a huge inventory situation obviously Alex: completely different really different market dynamics really different type Alex: of of goal that buyers have.

Alex: And so a lot of like my early experimentation, like I made like a Chrome extension called Window AI. Alex: I did like a few other things were just about learning how the ecosystem works Alex: and like what makes it different and how the like, like what people really want, Alex: what developers really want.

Josh: So that leads us to OpenRouter itself, right? So I kind of want you to help Josh: explain to the listeners who aren't familiar with OpenRouter what it does. Josh: Because I think a lot of people, the way they interact with an AI is they send Josh: a prompt to their model of choice. Josh: They use ChatGPT or they use the Grok app or they're on Gemini and they kind Josh: of live in these siloed worlds.

Josh: And then the next step up from the people are those kind of who use it professionally, Josh: who are developers. They're interacting with APIs. Josh: Maybe they're not interfacing with the actual UI, but they're calling a single model. Josh: And OpenRouter kind of exists on top of this, right? Can you walk us through Josh: how it works and why so many people love using OpenRouter? Alex: Open Router is an aggregator and marketplace for large language models.

Alex: You can kind of think of it as like a Stripe meets Cloudflare for both of them. Alex: It's like a single pane of glass. You can orchestrate, discover, Alex: and optimize all of your intelligence needs in one place. Alex: One billing provider gets you all the models.

Alex: Uh there's like 470 plus now uh Alex: like all the models like they sort of implement features Alex: but they do it differently and they also there's Alex: a lot of like intelligence brownouts as andre carpoffi calls them yeah where Alex: models just go down all the time even the you know even the top models like Alex: anthropic and gemini and and open Alex: ai um so what we do is you know we like developers need a lot of choice. Alex: CTOs need a lot of reliability.

Alex: CFOs need predictable costs. CISOs need complex policy controls. Alex: All of these are inputs to what we do, which is build a single pane of glass Alex: that makes models more reliable, lower costs, gives you more choice, and, Alex: and then and helps you choose between all the options for where to source your intelligence.

Josh: How does it work uh because i would imagine like what Josh: each as and i on the show we frequently talk about benchmarks right where Josh: a certain model is the best at coding and that infers that maybe you should Josh: go to that model to do all of your coding needs because it's the best at it Josh: but it would appear as if it's not true if you're routing through a lot of different Josh: providers so how do you consider which provider gets routed to when and how

Josh: to get the best result for what you're asking Alex: So we've taken a different approach so Alex: far which is instead of like focusing on Alex: a production router that picks Alex: the model for you um we try Alex: to help you choose the model so we Alex: we build lots we create lots of analytics both on Alex: your account and uh and on our Alex: rankings page to help you browse and discover the models that Alex: like the power users are really using successfully on

Alex: a certain type of workload um because we Alex: think like developers today primarily want to Alex: choose the model themselves um switching between all Alex: families can result in like a lot like very Alex: unpredictable behavior but once you've Alex: chosen your model um we try to Alex: help developers not need to think about the provider there are Alex: like sometimes dozens of Alex: providers for a given model uh all kinds

Alex: of companies including the hyperscalers like aws google vertex and azure um Alex: and uh like scaling startups like together fireworks deep infra um and a long Alex: tail of providers that provide, Alex: like very unique features, Alex: very like exceptional performance. Alex: There's all kinds of differentiators for them. Alex: So what we do is we collect them all in one place. And if you want a feature, Alex: you just get the providers that support it.

Alex: If you want performance, you get prioritized to the providers that have high performance. Alex: If you really are cost sensitive, you get prioritized to the providers that Alex: are really low cost today. and we basically create all these lanes. There's.

Alex: Innumerable ways you could get routed but Alex: you're in full control of the of the overall user Alex: experience that you're aiming for and that's Alex: what that's what we found that was missing from the Alex: whole ecosystem was just a way of doing that and uh Alex: and you know we get like between on average five to ten percent uptime boosts Alex: over going to um providers directly just by load balancing and sending you to

Alex: the top provider that's up and able to handle your request. Alex: We really focus hard on efficiency and performance.

Alex: We only add about 20 to 25 milliseconds of latency on top of your request. Alex: It all gets deployed very close to your servers up the edge. Alex: We overall get just We stack providers. Alex: We figure out what you can benefit from that everybody else is doing and just Alex: give you the power of big data as a developer just accessing your model choice. Josh: So it kind of allows you to harness the collective knowledge of everybody, right?

Josh: You get all of the data, you have all of the queries, you know which yields Josh: the best result, and you're able to deliver the best product for them. Josh: Now, in terms of actual LLMs, EJ has actually pulled this up just before, which is a leaderboard. Josh: And I'm interested in how you guys think about LLMs, which are the best, Josh: how to benchmark them, and how you route people through them.

Josh: Is there a specific... Do you believe that benchmarks are accurate, Josh: and do you reflect those in the way that you route traffic through these models? Alex: In general, we have taken the stance that we want to be the capitalist benchmark for models. Alex: What is actually happening? Alex: And part of this is that I really think both the law of large numbers and the Alex: enthusiasm of power users are really, really valuable for everybody else.

Alex: Like when you're routing to Alex: um like clod in Alex: let's say you're routing to clod 4 and you're Alex: based in europe um there you Alex: know all of a sudden there might be like a huge variance in in throughput from Alex: one of the providers and you're only able to detect that if like some other Alex: users have discovered it before you and so we route around the provider that's Alex: like running kind of slow in Europe and send you, Alex: if your data policies allow it,

Alex: to a much faster provider somewhere else. Alex: And that allows you to get faster performance. So, like, um... Alex: That's, like, on the provider level, how, like, numbers help. Alex: On the, like, model selection level, like, what you see on this rankings page Alex: here, power users will, like, when we put up a model, like, we put up a new Alex: model today from a new model lab called ZAI, Alex: like, the power users instantly discover it.

Alex: We have this LLM enthusiast community that dives in and really figures out what Alex: a model is good for along a bunch of core use cases. Alex: The power users figure out which workloads are interesting, and then you can Alex: just see in the data what they're doing. And everybody can benefit from it.

Alex: That's why we open up our data and share it for free on the rankings page here. Ejaaz: I'm seeing this one consistent unit across all these rankings, Ejaaz: Alex, which is tokens, right? Ejaaz: And Josh and I have spoken about this on the show before, but I'm wondering Ejaaz: how, like you've chosen this specific unit to measure how good or effective Ejaaz: these models are or how consumed or used they are.

Ejaaz: Can you tell us a bit more as to why you picked this particular unit and what Ejaaz: that tells you as like the open router platform as to how a user is using a particular model? Alex: Yeah, I think dollars is a good metric too. Alex: The reason we chose tokens is primarily because we were seeing prices come down really quickly. Alex: Open Router has been around since the beginning of 2023.

Alex: And I didn't want a model to be penalized in the rankings just because the prices Alex: are going down really dramatically now like there's a, Alex: There's a paradox called Jevons paradox, which is that when prices decrease like 10x, Alex: users' use of some component of infrastructure increases by more than 10x. Alex: And so maybe they didn't get 10x at all. Alex: But I thought there were some other advantages to using tokens,

Alex: too. Tokens don't have this penalty and don't rely on Jevon's Paradox, Alex: which can have a lot of lag. Alex: They also are a little bit of a proxy for time. Alex: A model that is generating a lot of tokens and doing so for a while across a lot of users. Alex: It means that a lot of people are reading those tokens and actually doing something with them.

Alex: And same goes for input. But if I really want to send an enormous number of Alex: documents and the model has a really, really, really tiny prompt pricing, Alex: I think that's still valuable and something that we want to see. Alex: We want to see that this model is processing an enormous number of documents. Alex: That's a use case that should show up in the rankings.

Alex: And so we decided to go with tokens. We might like add dollars in the future, Alex: but I think tokens are, you know, they don't have this like Jevons Paradox lag. Alex: And there wasn't anything else. Like nobody was doing any kind of like overall analytics. Alex: We didn't see any other company even do it until Google did a few months ago Alex: where they started publishing the total amount of tokens processed by Gemini. Alex: So we'll see which use cases really need dollars.

Alex: But tokens have been holding up pretty well. Ejaaz: Yeah, I mean, this dashboard is awesome. And I recommend anyone that's listening Ejaaz: to this that can't see our screen to get on OpenRouter's website and check it out. Ejaaz: I've been following it for the last two weeks kind of pretty rigorously, Alex. Ejaaz: And what I love is you can literally see...

Ejaaz: So two weeks ago Grok 4 got released right Ejaaz: and Josh and I were making a ton of videos on this we were Ejaaz: using it with pretty much everything that we could do and Ejaaz: then this other model came out of China pretty much a few days after called Ejaaz: Kimi K2 and I was like oh yeah whatever this is just some random Chinese model Ejaaz: I'm not going to focus on it and then I kept seeing it in my feed and I thought

Ejaaz: okay maybe I'll give this a go and I kind of like went straight to open rather than just Ejaaz: almost gauge the interest from a wider set of AI users. And I saw that it was skyrocketing, right? Ejaaz: And then I saw that Quen dropped their models last week. Ejaaz: And again, I came to Open Router and it preceded the trend, right?

Ejaaz: People had already started using it. So I love how you describe Open Router Ejaaz: as this kind of like prophetic orb, Ejaaz: basically, where the enthusiasts and the community itself can kind of like front Ejaaz: run very popular trends. And I think that's a very powerful moat. Ejaaz: And kind of on this path, Alex, I noticed that a lot of these major model providers Ejaaz: see the value in this, right?

Ejaaz: So if I'm not mistaken, OpenAI kind of like used your platform to kind of secretly Ejaaz: launch their Frontier model before they officially launched it, right? Ejaaz: Can you walk us through, you know, how that comes about and more importantly, Ejaaz: why they want to do that and why they chose OpenRoddy to do that?

Alex: Uh open ai will sometimes Alex: give uh early access Alex: to their to models to some of their customers for Alex: testing and we asked them if they Alex: wanted to try a stealth model with us which we had never done before um it involved Alex: like launching it as under another name and seeing how users respond to it without Alex: having any bias or sort of inclination for against the model at the onset. Alex: And it would be like a new way of testing it and a new way of...

Alex: It was like an experiment for both us and them. Alex: And they generously decided to take the leap of faith and try it. And we...

Alex: Launched gpt 4.1 with Alex: them at and we called it quasar alpha and Alex: it was a million uh Alex: token context length model opening us first very Alex: very long context model and it was also optimized Alex: for coding and the incredible Alex: there were a couple incredible things that happened first Alex: we have this community uh of benchmarkers Alex: that run open source benchmarks and we give Alex: a lot of them grants to help fund the benchmarks

Alex: grants of open router tokens they'll just run the Alex: suite of tests against all the models and some of them are very creative like Alex: there's one that tests uh like the ability to generate fiction there's one that Alex: tests um like how like whether it can make a 3d object project in Minecraft called MCBench. Alex: There are a few that test different types of coding proficiency.

Alex: There's one that just focuses on how good it is at Ruby, because Ruby is, Alex: turns out a lot of the models are not great at Ruby. Alex: There are a lot of like languages that all the models are pretty bad at. Alex: And so we have this like long tail of very niche benchmarks, Alex: And all the benchmarkers ran, you know, for free their benchmarks on Quasar Alex: Alpha and found pretty incredible results for most of them.

Alex: And so the model got like, you know, OpenAI got this feedback in real time. Alex: We kind of like helped them find it.

Alex: And they made another snapshot, which we launched as Optimus Alpha. Alex: And they could compare the feedback that they got from the two snapshots. Alex: Um, and, and then they, and then like two weeks later, they launched GPT 4.1 live for everybody. Alex: So it was like, uh, uh, was it an experiment for us? Alex: And, and we've done it, um, again since, uh, with, uh, another model provider Alex: that, uh, that's still working on it.

Alex: Um, and it, and it's kind of like a cool way of learning of like crowdsourcing, Alex: uh, benchmarks that you wouldn't have expected. and also getting unbiased community sentiment. Josh: That's great. So now when we see a new model pop up and we want to test GPT-5, Josh: we know where to come to to try it early. Josh: We'll see because rumor is it's coming soon. So we'll be, we're on your watch list.

Josh: But having, I do want to ask you about open source versus closed source because Josh: this has been an important thing for us. We talk about this a lot. Josh: You have a ton of data on this. Josh: I'm looking at the leaderboards there. There are open source models that are Josh: doing very well, closed source. Josh: What are your takes in general? How do you feel about open source versus closed Josh: source models, particularly around how you serve them to users?

Alex: Both models, both types of models have supply problems, but the supply problems are very different. Alex: Typically, what we see with closed source models is that there's there's very Alex: few suppliers, usually just one or two. Alex: Like with Grok, for example, there's Grok Direct and there's Azure.

Alex: Um with anthropic there's anthropic direct there's google vertex there's aws Alex: bedrock um and then we also like deploy it in different regions like we have Alex: an eu deployment um for customers who'd like only want their data like to stay in the eu, Alex: and uh and we do custom deployments for Alex: the for the closed source models too to just kind of guarantee good Alex: throughput high and high rate limits for people um Alex: but uh the

Alex: like a tricky part is that like the the demand usually the like the closed source Alex: malls are doing most of the tokens on open router um it's it's dominant you Alex: know it's probably 80-ish 70 to 80 percent closed source tokens today. Alex: But the open source models have a much more fragmented supply, like cell supply.

Alex: Side order book um and and like Alex: the rate limits for each provider is Alex: like a like less stable on average um it Alex: usually takes a while for the hyperscalers to serve a Alex: new closed source a new open source model um so we so the load balancing work Alex: that we do on um open source models tends to be a lot more valuable the load Alex: balancing work that we do for closed source models tends to be very focused Alex: on caching and feature awareness,

Alex: making sure you're getting clean cache hits and only transitioning over to new Alex: providers when your cache is expired. Alex: For open source models, there's way less caching. Very, very few open source Alex: models implement caching.

Alex: And so switching between providers becomes more common. and Alex: uh like we we also track a Alex: lot of quality differences between the the open Alex: source providers some of them will deploy at lower Alex: quantization levels which means like it's kind of like a way of compressing Alex: the model um generally doesn't have an impact on the quality of the output uh Alex: but and yet we still see some odd things from some of the open source providers.

Alex: And so we run tests internally to detect those outputs. And we're building up Alex: a lot more muscle here soon. Alex: So that like, they get pulled out of the routing lane and don't affect anyone. Josh: So closed source accounts for 80% or something like that, a very large amount. Josh: Do you see that changing? Josh: Because that post we just had, it's at nine out of the 10 fastest growing LLMs Josh: last week, they were open source.

Josh: And every time it seems like China comes out with another model, Josh: it was Kimmy K2 a week or two ago, it kind of really pushes the frontier of open source forward. Josh: And the rate of acceleration of open source seems to be as fast, Josh: if not faster than closed source, where it's just, it's making these improvements very quickly. Josh: It has the benefit of being able to compound in speed because it's open source Josh: and everyone can contribute.

Josh: Do you think that starts to change where the percentage of tokens you're issuing Josh: are from open source models versus closed source? Josh: Or do you continue to see a trend where it's going to be Google, Josh: it's going to be OpenAI that are serving a majority of these tokens to users?

Alex: In the short term, we're likely to see open source models continue to dominate Alex: the fastest growing model category on OpenRouter.

Alex: And the reason for that is that a lot of users who come for a closed source Alex: model, but then decide they want to optimize later, Alex: either they want to save on costs or try out a new model that's supposed to Alex: be a little bit better in some direction that their app cares about or their use case cares about, Alex: then they leave the closed source model and go to an open source model.

Alex: So open source tends to be like a last mile optimization thing, Alex: making a big generalization because the reverse can happen too. Alex: And so because it's a last mile optimization thing, Alex: the jump from this model is not being used at all to this model is really being Alex: used by a couple of people who have Alex: left Claude 4 and want to try some new coding use case will be bigger.

Alex: Than the closed-source models, which start at a really high base and don't have Alex: growth quite as dramatic. Alex: So the other part of your question, though, was whether there's going to be like a flippening of. Josh: Close or some sort of like chipping it away at that monopoly of close source tokens.

Alex: It's hard to predict these things because, you know, Alex: I think like the the biggest problem today with open source models is that the Alex: incentives are not as strong like the model lab and the model provider.

Alex: Um they've you know they're they're Alex: sort of established incentives for how to Alex: grow as a company and attract good high quality um ai talent and um and giving Alex: the model weights away impairs those incentives now like we might see yeah this Alex: is where we might see like decentralized providers, Alex: helping in the future.

Alex: A way for like, Alex: uh you know like a really good incentive scheme that Alex: like allows high quality talent Alex: to work on an open source model um Alex: that remains open weights at least uh like could fix this i like i you know Alex: i stay pretty i try to stay close to the decentralized providers um and like Alex: learn a lot from them there's some like cool on the provider side on like on

Alex: running inference i I think there's some really cool incentive schemes being worked on. Alex: But on actually developing the models themselves, I haven't seen too much, unfortunately. Alex: So I think if we see one, flipping in the radar. And until we do, I personally doubt it. Josh: TBD, do you have personal takes on how you feel about open source versus closed source?

Josh: Because this has been a huge topic we've been debating too. It's just the ethical Josh: concerns around alignment and closed source models versus open source. Josh: When you look at the competitors, China, generally speaking, Josh: is associated with open source, whereas the United States is generally associated with closed source.

Josh: And we saw Llama and Meta release the open source models, but now they're raising Josh: a ton of money to pay a lot of employees a lot of money to probably develop a closed source model. Josh: So it seems like the trends are kind of split between US and China.

Josh: And I'm curious if you have any personal takes, even outside of OpenRouter, Josh: of which you think serves better for the long term outlook on, Josh: I mean, the position of the United States or just the general safety and alignment Josh: conversation around AI?

Alex: I mean, like a very simple fundamental difference between the two is that an Alex: innovation in open source models can be copied more quickly than an innovation Alex: in closed source models. Alex: So in terms of velocity and like how far ahead one is over the other, Alex: that is like a massive structural difference. Alex: That means that closed source models should be theoretically always ahead until Alex: a really interesting incentive scheme develops, like I mentioned before.

Alex: Uh, I think, and I think that's, you know, I don't see like evidence that that's Alex: going to change in terms of China versus the U S. Alex: Um, it's, I think it's very interesting that China has not had like a major closed source model.

Alex: Um and i don't really Alex: see a great reason why i'm Alex: not aware of any reasons that's not that's not going Alex: to be going to be the case in the future um my prediction Alex: is that there's going to be a closed source model from china um Alex: and uh you know if uh uh you know if like it's possible that DeepSeas and Moonshot Alex: and Gwen have built up really sticky talent pools.

Alex: But generally with talent pools, after enough years have passed, Alex: people quit and go and create new companies and build new talent pools. Alex: And so we should see some of that. It's not the case that the AI space has NDAs Alex: or non-competes that the hedge fund space has. Alex: That might happen in the future too. But assuming that the current non-compete Alex: culture continues, there should be more companies that pop up in China over time.

Alex: And I'm betting that some of them will be closed source. Alex: And my guess is that the two nations will start to look more similar. Ejaaz: Yeah, I guess that's why you have Zuck dishing out 300 mil to a billion dollar Ejaaz: salary offers to a bunch of these guys, right? Ejaaz: One more question on China versus the US. I kind of agree with you. Ejaaz: I didn't really expect China to be the one to lead open source anything, Ejaaz: let alone the most important technology of our time.

Ejaaz: Do you think is their secret source to building these models, Alex?

Ejaaz: And I know this might be out of the forte of Ejaaz: open router specifically but as someone who has studied this technology for Ejaaz: a while now i'm struggling to figure out you know what advantage they had you Ejaaz: know they're discovering all these new techniques and maybe the simple answer Ejaaz: is like constraints right they don't have access to all of Ejaaz: nvidia's chips they don't have access to infinite compute so then maybe they're

Ejaaz: forced to kind of like figure out other ways around the same kinds of problems Ejaaz: that western companies are focused on But it's pretty clear that America, with all its funding, Ejaaz: hasn't been able to make these frontier breakthroughs.

Ejaaz: So I'm curious whether you are aware of or know some kind of technical moat Ejaaz: that Chinese AI researchers or these AI teams that are featuring on Open Rata Ejaaz: day in and day out have over the U.S.? Alex: Well, I don't know. Alex: There are certainly some that they've come up with that like DeepSeek had a Alex: lot of very cool inference innovations that they published in their paper.

Alex: But a lot of what they published in the original R1 paper were things that OpenAI Alex: had done independently themselves many months before.

Alex: So uh i like Alex: on the inference side and on Alex: uh some of the model side i think like deep seek we we Alex: had talked to their team for years before r1 came Alex: out they had many models before that and Alex: they were always like a pretty sharp optimum like Alex: team for doing inference um like they Alex: came up with like the best user experience for caching prompts Alex: long before deep cpr1 came out and they had very good pricing um they uh they

Alex: were just they were like you know by far the the strongest chinese team um that Alex: we were aware of uh well before that happened and so i'm guessing there was like some talent.

Alex: Uh accumulation that they were working on in china Alex: for people who wanted to stay in china and yeah that's Alex: that's a huge advantage like american companies are obviously not Alex: doing that there's a duck is very on Alex: point that a lot of this is just based on talent Alex: um there are a lot of Alex: ai is open and out there and just like and Alex: very composable like a big tree of knowledge Alex: there's a paper that comes out and it cites like

Alex: 20 other papers and you can go and read all Alex: of the cited papers and then you like have kind of Alex: a basis for understanding the paper but you really have to Alex: go one level deeper and read all the cited papers two levels Alex: down to really understand what's going on and it's.

Alex: Just that no very few people can do that um and Alex: it takes like a lot of years of experience to like actually Alex: apply that knowledge and learn all these Alex: things that have not been written in any paper at all and uh Alex: and there's just there's just such such it Alex: like a small number of people um who can Alex: really lead research on all the different dimensions that Alex: go on to making a model and uh um and Alex: and like the the border between china and the u.s is

Alex: is pretty defined you have to leave china move to the u.s Alex: and really establish yourself here um so Alex: i do think there's like country arbitrage there's like Alex: there's you know the head the hedge fund background arbitrage there's uh there's Alex: there's hardware arbitrage like there's like a ton of hardware that's only available Alex: in china but not here vice versa that creates an opportunity um and this this Alex: will just continue to happen.

Ejaaz: Yeah, I think this arbitrage is fascinating. Ejaaz: I read somewhere that there's probably less than 200 or 250 researchers in the Ejaaz: world that are worthy of working at some of these frontier AI model labs. Ejaaz: And I looked into some of the backgrounds of the team behind Kimi K2, Ejaaz: which is this recent open source model out of China, which broke all these crazy rankings. Ejaaz: I think it was like a trillion parameter model or something crazy like that.

Ejaaz: And a lot of them worked at some of the top American tech companies. Ejaaz: And they all graduated from this one university in China. Ejaaz: I think it's Tsinghua, which apparently is like, you know, the Harvard of AI Ejaaz: in China, right? So pretty crazy. Ejaaz: But Alex, I wanted to shift the focus of the conversation to a point that you Ejaaz: brought up earlier in this episode, which is around data.

Ejaaz: Okay, so here's the context that like Josh and I have spoken about this at length, right? Ejaaz: We are obsessed with this feature on OpenAI, which is memory, right? Ejaaz: And I know a lot of the other memory, sorry, a lot of the other AI models have memory as well. Ejaaz: But the reason why we love it so much is I feel like the model knows me, Alex. Ejaaz: I feel like it knows everything about me. It can personally curate any of my prompt.

Ejaaz: It just gets me. It knows what I want and it just serves up to me in a platter Ejaaz: and off I go, you know, doing my thing. Ejaaz: Now, Open Router sits on top of like kind of like the query layer, right? Ejaaz: So you have all these people writing all these weird and wonderful prompts and Ejaaz: kind of routing it through on towards like different AI models. Ejaaz: You hold all of that data or maybe you have access to all of that data.

Ejaaz: And I know you have something called private chat as well, where you don't have access to it. Ejaaz: Talk to me about like what OpenRouter and what you guys are thinking about doing Ejaaz: with this data, because presumably, Ejaaz: or in my opinion, you guys have actually the best mode, arguably better than Ejaaz: ChatGPT, because you have all these different types of prompts coming from all Ejaaz: these different types of users for all these different types of models.

Ejaaz: So theoretically, you could spin up some of the most personal AI models for Ejaaz: each individual user if you wanted to. Ejaaz: Do I have that correct? Or am I, you know, speaking crazy? Alex: No, that's true. No, it's something we're thinking about. Alex: By default, your prompts are not logged at all. Alex: We don't have prompts or completions for new users by default. Alex: You have to toggle it on in settings.

Alex: But the result, a lot of people do toggle it on. And as a result, Alex: I think we have by far the largest multi-model prompt data set. Alex: Uh, but what we've done today, we've barely done anything with it. Alex: We classify a tiny, tiny, tiny subset of it. And that's what you see in the rankings page. Alex: Um, but, uh, what it could be done on like a per account level is really, Alex: um, like three main things.

Alex: One memory right out of the box. You can, you can get this today by like combining Alex: open router with like a memory as a service. We've got a couple of companies Alex: that do this, like Memzero and SuperMemory. Alex: And we can partner with one of those companies or do something similar and just Alex: provide a lot of distribution.

Alex: And that basically gets you a chat GPT as a service where it feels like the Alex: model really knows you and the right context gets added to your prompt. Alex: The other things that we can do are help you select the right model more intelligently. Alex: There's a lot of models where there's like a super clear, like migration decision that needs to be made. Alex: And, and we can just see this very clearly in the data.

Alex: But we right now we just like, you know, we have like a channel or like some Alex: kind of communication channel open with the customer, we can just tell them Alex: like, hey, and we know you're using this model a ton.

Alex: It's been deprecated. This model is significantly better. you Alex: should move this kind of workload over to it or like Alex: this workload you'll get way better pricing if you do this um Alex: and and that's basically like that's the Alex: only sort of guidance and kind of like Alex: opinionated routing we've done so far and it could Alex: be a lot more intelligent a lot more out of the box a lot more Alex: built into the product um and then Alex: the the last thing

Alex: we can do i mean there's there's probably tons of Alex: things we're not even thinking about um but Alex: like getting really Alex: really smart about how Alex: models and providers are responding to prompts and Alex: uh showing you just the really coolest Alex: data just like telling you Alex: what kinds of of prompts um are Alex: going to which models and how those models are replying and Alex: just like characterizing the reply in all kinds of interesting ways

Alex: like did the model refuse to answer what's the refusal rate Alex: did the model um did the.

Alex: Model like successfully make a tool call or did it decide to Alex: ignore all the tools that you passed in that's a huge one Alex: um did the model like pay Alex: attention to its context did uh you know did what did did some kind of truncation Alex: happening happen before you sent it to the model So there's all kinds of like Alex: edge cases that cause developers apps to just get dumber and they're all detectable.

Ejaaz: I'm so happy you said that because I have this kind of like hot take, Ejaaz: but maybe not so hot take, which is I actually think all the Frontier models Ejaaz: right now are good enough to do the craziest stuff ever for each user. Ejaaz: But we just haven't been able to unlock it because it just doesn't have the context.

Ejaaz: Sure, you can attach it to a bunch of different tools and stuff, Ejaaz: but if it doesn't know when to use the tool or how to process a certain prompt Ejaaz: or if the users themselves don't know how to read Ejaaz: the output of the AI model themselves, like you just said, we need some kind Ejaaz: of analytics into all of this, Ejaaz: then we're just kind of walking around like headless chickens almost.

Ejaaz: So I'm really happy that you said that. One other thing that I wanted to get Ejaaz: your take on on the data side of things is, I just think this whole concept Ejaaz: or notion of AI agents is becoming such a big trend, Alex. Ejaaz: And I noticed a lot of Frontier Model Labs release new models that kind of spin Ejaaz: up several instances of their AI model. Ejaaz: And they're tasked with a specific role, right?

Ejaaz: Okay, you're going to do the research. You're going to do the orchestrating. Ejaaz: You're going to look online via a browser, blah, blah, blah, Ejaaz: blah, blah. And then they coalesce together at the end of that little search Ejaaz: and refine their answer and then present it to someone, right? Ejaaz: You know, Grok4 does this, Claude does this, and a few other models.

Ejaaz: I feel like with this data that you're describing, OpenRouter could be or could Ejaaz: offer that as a feature, right? Ejaaz: Which is essentially, you can now have super intuitive, context-rich agents Ejaaz: that can do a lot more than just talk to you or answer your prompts. Ejaaz: But they could probably do a bunch of other actions for you. Ejaaz: Is that a fair take, or is that something that maybe might be out of the realm of open router?

Alex: Our strategy is to be the best inference layer for agents. Alex: And what I think developers want... Alex: Is control over how their agents work. Alex: And our developers at least want to use us as a single pane of glass for doing Alex: inference, but they want to see and control the way an agent looks. Alex: An agent is basically just something Alex: that is doing inference in a loop and controlling the direction it goes.

Alex: So um what what Alex: we want to do is just like build incredible docs Alex: really good primitives that make that easy Alex: to do so that you know like i think like Alex: a lot of our developers are just people building agents and so Alex: what they want is they want the primitives to Alex: be solved so that they can just keep creating new Alex: versions and new ideas um without worrying Alex: about like you know re-implementing tool calling over Alex: and over again and um and and

Alex: and so like at least for this is like a it's it's Alex: a tough problem given how many models there's like a new model or provider every Alex: day and uh and people actually want them and use them so uh to standardize this Alex: like make make these tools like really dependable um that's kind of like where Alex: we want to focus and uh so that like agent developers don't have to worry about it.

Josh: As we level up towards closer and closer to getting to AGI beyond, Josh: I'm curious what Open Router's kind of endgame is. Josh: If you have one, what is the master plan where you hope to end up? Josh: Because the assumption is as these systems get more intelligent, Josh: as they're able to kind of make their own decisions and choose their own tool Josh: sets, what role does Open Router play in continuing to route that data through?

Josh: Do you have a kind of master plan, a grand vision of where you see this all heading to? Alex: You're saying like as agents get better at choosing the tools that they use Alex: what what becomes our role when like the agents are really good at that yes.

Josh: Yes and like where do you see open router fitting into the picture and what Josh: would be the best case scenario for this this future of open router Alex: Right now open routers bring your own tool, Alex: platform um we don't have like a Alex: marketplace of mcps yet uh and Alex: and i i do think like a lot of the i think most of the most used tools will Alex: be ones that developers configure themselves agents just work like they're given

Alex: access to it like i think like a holy grail for for open router is that.

Alex: The the ecosystem is going to like basically my Alex: prediction for how the ecosystem is going to evolve is that um Alex: all the models are going to be adding state and Alex: other kinds of stickiness that just make you want to stick Alex: with them so they're going to add server-side tool calls Alex: they're going to add like um you know web search that that is stateful they're Alex: going to add memory They're going to add all kinds of things that try to prevent

Alex: developers from leaving and increase lock-in. Alex: And OpenRouter is doing the opposite. Alex: We want developers to not feel vendor lock-in. Alex: We want them to feel like they have choice and they can use the best intelligence, Alex: even if they didn't before. Alex: It's never too late to switch to a more intelligent model. That would be like, Alex: you know, a good always on outcome for us.

Alex: And so what I think we'll end up doing is, is like partnering with other companies Alex: or building the tools ourselves if we have to, so that developers don't feel stuck.

Alex: That's how I, you know, there's a lot of ways the ecosystem could evolve, Alex: but that's how I would put it in a nutshell. Josh: Okay, now there's another personal question that I was really curious about, Josh: because I was also right there with you in the crypto cycle when NFTs got absolutely Josh: huge, was a big user of OpenSea. Josh: And it was kind of this trend that went up and then went down.

Josh: And NFTs kind of fizzled out, it wasn't as hot anymore, and AI kind of took the wind from the sails. Josh: And it's a completely separate audience, but a similar thing where now it's Josh: the hottest thing in the world.

Josh: And i'm curious how you see the trend continuing is this a cyclical thing that Josh: has ups and downs or is this a one-way trajectory of more tokens every day more Josh: ai every day is do you see it being a cyclical thing or is this a a one-way Josh: trend towards up into the right nfts Alex: Kind of follow uh crypto in a, Alex: indirect way um when crypto Alex: has ups and downs nfts generally lag a bit Alex: but they they have similar ups and downs and um

Alex: and crypto is an extremely long-term play on like building a new financial system Alex: and there are so many reasons that it's not going to happen overnight um and And they're like, Alex: it's very, very entrenched reasons. Alex: Whereas AI, there are some overnight business transformations going on. Alex: And the reason AI, I think, moves a lot, one of the reasons that AI moves a Alex: lot faster is it's just about making computers behave more like humans.

Alex: So if a company already works with a bunch of humans, then there's, Alex: you know, there's some engineering that needs to be done.

Alex: There's some like thinking about how Alex: to like scale this but Alex: but in general i think that it's not like Alex: after seeing what can be possible um inference Alex: will be the fastest growing operating expense for all companies Alex: it'll it'll be like oh we can just hire Alex: high-performing employees at a click of a Alex: and they they work 24 7 they Alex: scale elastically it's like you know Alex: it it's not that hard it's not like huge mental

Alex: model shift it's just like a huge upgrade to the way companies work today um Alex: in most cases so it's just completely different from crypto there's there's Alex: like other than both being you know than nfts i mean other than both being new Alex: they're fundamentally very different changes.

Ejaaz: You're probably one of very few people in the world right now that has crazy Ejaaz: insights to every single AI model. Ejaaz: Definitely more than the average user, right? Like I have like three or four Ejaaz: subscriptions right now and I think I'm a hotshot. Ejaaz: You get access to like 400 and what is it? 57 models right now on OpenRata.

Ejaaz: So an obvious question that I have for you is Ejaaz: I'm not going to say in the next couple of years, because everything moves way Ejaaz: too quickly in this sector. Ejaaz: But over the next six months, is there anything really obvious to you that should Ejaaz: be focused on within the AI sector? Ejaaz: Maybe it's like the way that certain models should be designed, Ejaaz: or perhaps it's at the application layer that no one's talking about right now.

Ejaaz: Because going on from our earlier part of the conversation, you just pick these Ejaaz: trends out really early. and I'm wondering if you see anything. Ejaaz: It doesn't have to be open-racket related. It could just be AI related. Alex: I've seen the models trending towards caring more about how resourceful they Alex: are than what knowledge they have in the bank.

Alex: Not all of, I feel like a lot of the applications, I think the model labs maybe, Alex: a lot of them, I don't know how many of them really deeply believe that, Alex: but a couple of them uh talk about it and i don't think it's really hit the Alex: application space yet um because people will will ask chat gpt things and if Alex: the knowledge is wrong they think the model is stupid, Alex: and that's just kind of a bad way of evaluating a model um

Alex: like whatever knowledge a person has whatever Alex: a person like where calls happen at a certain time like Alex: does not it's not a proxy for how smart they are um Alex: like the the intelligence and usefulness of a model Alex: is going to trend towards how good it is at using tools and Alex: uh and and how good it is at like paying Alex: attention to its context of a long long long long context and so it's like it's

Alex: it's total memory capacity and accuracy um so i think those two things need Alex: to be like emphasized more um the.

Alex: Like it might be that that models pull all Alex: of their knowledge from like online databases Alex: from like real-time uh scraped Alex: index indices of the web along with a Alex: ton of real-time updating data sources um and Alex: they're never they're always kind of like relying on some some sort of database Alex: for knowledge but relying on their reasoning process for for tool calling you Alex: know like we we put it We spend probably the plurality of our time every week

Alex: on tool calling and figuring out how to make it work really well. Alex: Humans, the big difference between us and animals is that we're tool users and tool builders. Alex: And that's where human acceleration and innovation has happened. Alex: So how do we get models creating tools and using tools very, Alex: very effectively? there's very little, Alex: There are very few benchmarks. There's very little priority. Alex: There's the Tau Bench for measuring how good a model is at tool calling.

Alex: But there's, and there's like maybe a few others. Alex: There's Swee Bench for measuring how good a model is at multi-turn programming tasks. Alex: It's very, very hard to run, though. It costs like, you know, Alex: for Sonnet, it could cost like $1,000 to run it. Alex: And it's like the user experience for kind of like evaluating the real intelligence Alex: of these models is not good.

Alex: And so like I love, as much as we don't have benchmarks listed on OpenRouter Alex: today, I love benchmarks. Alex: And I think like the app ecosystem and like developer ecosystem should spend Alex: a lot more time making very cool and interesting ones. Alex: Also, we will give credit grants for all the best ones. So I highly encourage it. Ejaaz: Well, Alex, thank you for your time today. I think we're coming up on a close Ejaaz: now. That was a fascinating conversation, man.

Ejaaz: And I think your entire journey from just non-AI stuff, so OpenSea all the way Ejaaz: to OpenRouter has just been a great indicator of where these technologies are Ejaaz: progressing and more importantly, where we're going to end up. Ejaaz: I'm incredibly excited to see where OpenRatter goes beyond just prompt routing. Ejaaz: I think some of the stuff you spoke about on the data side of things is going

Ejaaz: to be fascinating and arguably one of your bigger features. So I'm excited for future releases. Ejaaz: And as Josh said earlier, if GPT-5 is releasing through your platform first, Ejaaz: please give us some credits. We would love to use it. Ejaaz: But for the listeners of this show, as you know, we're trying to bring on the Ejaaz: most interesting people to chat about AI and Frontier Tech. We hope you enjoyed this episode.

Ejaaz: And as always, please like, subscribe, and share it with any of your friends Ejaaz: who would find this interesting. And we'll see you on the next one. Thanks, folks.