Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI compute

And given uh you know yearly prices of renting that compute, that would be like close to 50 gigawatts. Now obviously we're not putting on 50 gigawatts this year. So presumably that's paying for compute that is gonna be coming online over the coming years. So I have a question about what how to think about the timeline aro around when that cap maximum is online. Similar question for the labs where, you know.

OpenAI just announced that they raised a hundred and ten billion dollars. Anthropic just announced they raised thirty billion dollars. And if you look at the compute that they have coming online this year, um you you should tell me how much it is, but like is it not isn't another four gigawatts total that they'll have this year? It feels like the cost to rent the compute that OpenAI and Anthropic will have this year to like sustain their compute spend.

at, you know, ten, thirteen billion dollars a gigawatt. Th those individual raises alone are like m enough to cover their compute spend for the year. And then this is not even including the revenue that they're gonna earn this year. So help me understand first. When is the timescale at which the big tech CapEx is actually coming online? And two, what are the labs raising all this money for if like the the yearly price of a a one gigawatt data center is like thirteen billion dollars?

power purchasing agreements and down payments and all these other things that they're doing uh for further out into the future so that they can set up this super fast scaling, right? And and and this applies to all the hyperscalers. and other people in the supply chain. And so, you know, twenty gigawatts roughly deployed this year, um, a big chunk of that being hyperscalers, a chunk of not being and all of these companies, their biggest customers are Anthropic and Open AI. Um

Anthropic and OpenAI are in the, you know, two gigawatt and you know, two and a half gigawatt and one and a half gigawatts roughly right now. They're trying to scale to much larger, right? If you look at what Anthropic has done over the last few months, you know, four billion, six billion revenue added. And if we just draw a straight line, hey, yeah, they'll add another six billion dollars of revenue a month.

Uh people would argue that's bearish and that they should go faster. What that implies is that they're gonna add$60 billion of revenue across the next 10 months, right? And sixty billion dollars of revenue at the current gross margins that Anthropic had, at least last uh reported by media, um, would imply that they have, you know, roughly forty billion dollars of compute spend for that inference for that sixty bill of revenue.

That 40 billion of compute at roughly$10 billion a gigawatt um rental cost means that they need to add four gigawatts of inference capacity just to grow revenue. And that's saying that their research and development training fleet stays flat, right? So You know, in a sense, Anthropic needs to get to well above five gigawatts by the end of this year. And it's gonna be really tough for them to get there, but it's possible.

And OpenAI has k kind of got way more access to compute than Anthropic by the end of the year. And so what does Anthropic have to do to get the compute? Well, they have to go to lower quality providers that they would not have gone to before, right? You know, optimally you know, Anthropic, at least historically, has had the best quality providers been like Google and Amazon. Um whereas, you know, at least historically minded, you know, the biggest companies in the world

um now Microsoft and now they're expanding across the supply chain and going to other players that are newer. Um OpenAI has been, you know, a bit more aggressive on going to many players. Yes, they have tons of capacity from Microsoft. Uh they have Google and Amazon as well, but they also have like tons with Core Weave and Oracle and they've gone to like

random companies or, you know, one would think random companies like SoftBank Energy, who has never built a data center in their life, but you know, they're building data centers now for OpenAI. So they've gone to and and and many others like NScale and others that they're going and getting capacity from. And so there's this like

conundrum for anthropic because they were so conservative on compute, um because they didn't want to go crazy, right? And in in some sense, a lot of the financial freakouts in the second half of last year were like, OpenAI signed all these deals, but they don't have the money to pay for them. Um, okay, Oracle stock's gonna tank. Oh okay, Core Reef stock's gonna tank. Oh, okay, like, you know, all these companies' stocks tanked um and credit markets went crazy because people were like,

the end buyer can't pay for this. Now it's like, oh wait, they raised a ton of money. Okay, fine, they can pay for it. But in the sense anthropic was a lot more conservative. They were like, we'll sign contracts, but we'll be principled um and we'll purposely undershoot what we think we can possibly do and be conservative because we don't want to potentially go bankrupt.

It's available for anthropic and open AI to get last minute. And like basically what is the concrete advantage that OpenAI has gotten? If they end up at similar compute numbers by twenty twenty seven, um, is it just like they're gonna end this year with different gigawatts? If so, how many gigawatts is anthropic and opening AI gonna have by the end of this year? Yeah.

But they could, you know, there were there were many other customers that had one year, two year, three year deals, six month deals on demand. And as these contracts roll off, who is the participant in the market most willing to pay price? Um And in this sense, right, we've seen H one hundred prices inflect a lot and go up.

And people willing to sign long term deals for, you know, as above$2, even, right? Like I've seen deals where certain AI labs, I'm gonna be a little bit uh vague here for a reason, uh have signed at as high as two dollars and forty cents for two to three years for H100s, which if you think about the margin, a dollar forty for hopper when you release it.

uh or hopper to build it um across five years. And now two years in you're signing deals that are two to three years that are at two dollars and forty cents. Those margins are way higher. Right. And so now you can crowd out

all of these other suppliers, whether it's Amazon had these or Core Weav had these or Together AI or Nebus or whoever it is, right? You know, the the or these these neo clouds are the firms that had a higher percentage of hopper in general because they were more aggressive on it, A, and B, they tended to sign shorter term deals, you know, not Corey, but the others tended to sign shorter term deals. And so hey, if I want hopper, there is some capacity out there. And then also

While most of the capacity at like an Oracle or a core weave is signed for a long term deal in terms of Blackwell, uh anything that's going online this quarter is already sold. Um, and and in some cases.

They're not even hitting all the numbers that they promised they would sell because there are some data center delays, not just those two, but like Nebias and all the other folks, Mike, Microsoft, Amazon, Google. But there is a lot of Neo Clouds as well as some of the hyperscalers who have capacity they're building that they did not sell yet.

or capacity that they were gonna allocate to some internal use uh that is not necessarily super A A GI focused that they may now turn around and sell or they may, you know, in the case of anthropic They don't have to have all the compute directly, right? Amazon can have the compute, they can serve Bedrock, or Google can have the compute and serve Vertex, or Microsoft can have the compute and serve Foundry and then do a revenue share with Enthropic or vice versa. Okay.

There are not that many incremental buyers of compute yet because Anthropic hit the capabilities here first where their revenue is mooning.

Oh, what is the depreciation cycle of a GPU? And the bears, the Michael Burris or whatever, have said, look, people are saying that four or five years for these GPUs And in fact, if you uh maybe it's because the technology is improving so fast or whatever, in fact it makes sense to have two year depreciation cycles for these GPUs.

Which increases the sort of like reported amortized capex in a given year, uh, and so makes it maybe financially less l lucrative to building all these clouds. But in fact you're pointing at like

m maybe the depreciation cycle is even longer than five years'cause if we're using hoppers and then especially if AI really takes off and in twenty thirty we're like, fuck, we gotta like get the seven nanometer fabs up and we gotta like you we gotta go back to the A one hundreds. Like we turn on the A one hundreds again. Uh then it's like actually the depreciation cycle cycle is incredibly long. And um uh so I I feel like that's an interesting financial implication of what you're saying.

Through bedrock or through Vertex or through Foundry, uh, I think they'll be able to get to five or six gigawatts, uh, which is way above their like initial plans, right? Right. Um You know, and and anyways, that's that's sort of like an and an open AI will be r a little roughly the same, maybe a little higher. Um, actually a little bit higher based on our numbers, but

Anyways, the depreciation cycle of a GPU, right? Michael Burry was saying it's, you know, three years or or less, right? Is like sort of his argument. And there's sort of two ways and lenses to look at this. Like mechanically, um In in in this, you know, there's a TCO model, right, uh total cost of ownership of a GPU, where we sort of project

Pricing out for GPUs and uh build up the total cost of a cluster. But there's a number of costs, right? There's your data center cost, right? Um, there's your networking cost, there's your hand smart hands and people in the data center sh swapping stuff out, there's your spare parts.

Right. There's your actual chip cost. There's your server cost. All these, all these various costs get slumped together. And there's some depreciation cycles on it. You know, there's certain credit costs on it. Um, and you get to, okay, that's how you build up. Hey, an H one hundred costs a dollar forty an hour to deploy at volume across five years if your depreciation is five years. And then if you sign a do a deal at two dollars an hour

For those five years, your gross margin is roughly thirty-five percent. It's a little bit above that. But you know, if you sign it for a dollar ninety, it's thirty-five percent roughly. Um

And then at you assume at that fifth year the GPU falls off a bus, right? It's dead. Um and in some cases, you know, sort of the argument people are making is, well, If you didn't sign a long-term deal, because every two years Nvidia's tripling, quadrupling the performance while only 2xing the price or 50% increasing the price, Then the price of nature 100, sure, maybe the value in the market was$2 at 35% gross margins in 2024, but in 2026.

When Blackwell is in super high volume and deploying millions a year, you're actually now worth a dollar an hour. And when Ruben in twenty seven is in super high volume, right, even though it starts shipping this year, is in super high volume next year, um, doing millions of chips a year uh deployed into clouds. Uh you've got another three X in performance, another fifty percent or two X in price. Actually, the the hopper's only worth seventy cents an hour. Um

And so the price of a GPU would continue to fall. That's like one lens. The other lens is what is the utility you get out of the chip, right? Because if you could build infinite Rubin or infinite um of the newest chip, then yes, that's exactly what would happen. The price of a hopper would

Fall at a spot or a short term contract rate as the new chips come out and the per price per performance goes up. But because You are so limited on semiconductors and deployment timelines and all these things, you end up with actually what prices these chips is not.

Hey, what's the comparative thing I can buy today? It's actually what is the value I can derive out of this chip today? Right. And in that sense, Um let's take GPT 5.4. GPT 5.4 is both way cheaper to run than GPT-4, uh, has fewer active parameters, um, it's it's much smaller, right, in that sense of active parameters.

Plus, because it's a you know a sparser MOE versus GPT four being a coarser MOE. Um there's also been so many other advancements in training, RL, uh model architecture, et cetera, et cetera, data qualities. Um, all these things that have made GPD 5.4 way better than GPD 4 and it's cheaper to serve. And so when you look at an H100, it can.

serve more tokens per GPU of 5.4 than if you had ran for GPT four on it. Right. So so at some sense, it's producing more tokens of a model that is of higher quality. Interesting. And so In some sense, you know, obviously GPD four, what is the maximum TAM for its tokens? You know, maybe maybe it was a few billion dollars, maybe it was tens of billions of dollars. Adoption takes time.

For GPT 5.4, that number is probably north of 100 billion, but there's an adoption lag and there's competition, so other people are getting it, and there's the constant improvements that everyone else is having. So if If con if improvement stopped, you know, for here, the value of an H100 is now predicated on the value that GPD 5.4 can get out of it instead of.

the value that GPT four can get out of it and the margins and all that stuff that these labs are doing and they're in a competitive environment. So their margins can't go to infinity. Um so you sort of have this like dynamic that is quite interesting in that An H one hundred is worth more today than it was three years ago.

Um obviously in terms of memory, the human brain has way more. H100 is like eighty gigabytes and brain might have petabytes.

Imagine if we had a a human knowledge worker can produce six figures a year of value. And so if an age one hundred can produce Something close to that, if we had actual humans on the server, uh, the value of an H100 is like it can repay itself in the course of like a couple of months. So as I've been going through everything to prepare for taxes, I realized that I worked with over fifty different contractors last year, from cinematographers to audio technicians to editors.

and I owed all of them ten ninety ninths. In the past I've just used a spreadsheet and a big folder of invoices to figure out who I need to collect tax forms from. But with so many contractors, this takes a bunch of time and I've almost missed some people.

This year though, Mercury made my process way more straightforward. Whenever I paid somebody in 2025, I just hit a toggle to have Mercury request a W9 from them. Because of that, everything that I needed to issue 1099s got sent directly to Mercury. I literally just clicked a button and Mercury generated and sent them all out.

This is just one of the many things that I never would have assumed that a banking platform could just handle for me. Mercury has a bunch of features like this, which are gonna collectively save me multiple days this tax season. You can learn more at Mercury.com. Mercury is a fintech company, not an FDIC Insured Bank. Banking services provided through Choice Financial Group and Column NA members FDIC.

So when I interviewed Dario, um the point I was trying to make is not that I think the singularity is two years away and therefore Dario desperately needs to buy more compute. Although it's the revenue is certainly there that he needs to buy more compute. But the point I was trying to make is that given what Dario seems to be saying, given his statements that we're two years away from a data center of geniuses,

um certainly not more than five years away. And data server geniuses should be earning trillions upon trillions of dollars of revenue. It just does not make sense why he keeps making these statements about being more conservative on compute or to your point, buying being less aggressive than open AI on compute.

Um, and I g I guess that point got lost because then people were like roasting me about like, Oh, this podcaster was like trying to convince this like multi hundred billion dollar company CEO, like, why don't you YOLO it, bro? But no, I was trying to say that his internally his c uh statements are inconsistent. Um uh anyway, so it's it's i it it's good to iron it out.

Start to see that value skyrocket per GPU. And so in that sense, you should commit now to compute. Interestingly, in like in anthropic fashion, right? Um, you know, there there there's a bit of a meme that they are uh they don't they have problems with commitment issues and they're like sort of polyramorous. So not not like this is a bit of a meme.

that will make people choose the higher quality good on the margin. So to give a specific example, suppose the you know, better tasting apple costs two dollars and then like the shittier apple apple costs one dollar. Okay, now suppose you put an import tariff on them. And so now Uh now it's three dollars is it's two dollars for like good great apple, medium apple, right?

In the price of compute. Yes. As a result, that will push people to be willing to pay higher margins to for slightly better models. Because the calculus is I'm gonna be paying all this money for the compute anyways. I might as well just pay slightly more to making sure it's like the very best model rather than a model that's slightly worse.

I think that makes a ton of sense. Also, we I think we just see all of the volumes are on the best models today, all the revenues on the best models today. And in a compute limited world, um There's sort of two things that happen, right? A, companies that have locked up, you know, and and don't have commitment issues, you know, have these five-year contracts for compute.

Modern pricing, when you're priced to the value of models, the price is going to be up a lot more. Um and so in a sense, like the person who committed early has better margins in general. Um, and the percentage of the market that is in long term contracts is much larger than the percentage of the market in short term contracts that can be this sort of flex capacity that you add at the last second.

And and then at the same time, right? Um, so where does the margin go, right? Uh, because models get more valuable. Um, how much can the cloud players flex their pricing? Well, if in fact, like if you look at Core Weav, they're average term duration is like over three years right now. Um uh for like 90% plus of their compute, it's over three years. Um and so they end up with this like conundrum of like, well, they can't actually flex price. But

Every year they're adding incrementally way more capacity than they had previously, right? Um this year alone, right, Meta's adding as much capacity as they had in the entire fleet of compute and data centers for all purposes for serving WhatsApp and Instagram and Facebook in 2022.

And doing AI, right? Um, they're adding that alone this year. So, in the same sense, you know, you talk about Meta's doing that, Core Weave and and Google and my Amazon, all these companies are adding insane amounts of compute year on year on year. That new compute gets transacted at the new price.

Um, so in a sense, yes, you've locked in as long as we're in a sort of a takeoff, right? Oh, OpenAI went from six hundred megawatts to two gigawatts last year and from two gigawatts to, you know, six plus this year and you know, six to twelve next year, right? The incremental added compute is where all the cost is, not the prior long-term contract.

So then who holds the card is the infra providers for charging margin, right? So now the cloud players, the neo clouds or the hyperscalers can charge the margin. Oh, they can't because um or they can at to some extent. But then as you go upstream to, oh, well, who has access to all the memory and logic capacity? Well, it's it's NVIDIA for the most part. They've signed a lot of long-term contracts.

Um, you know, they've got like ninety billion dollars of s long-term contracts today, and they're negotiating three-year deals with the memory vendors today. Um, you know, you've got you've got you know, obviously Amazon and Google through Broadcom and their, you know, Amazon directly and all these companies, sort of AMD, these companies hold all the cards because they've secured the capacity. Um and and T SMC is not raising prices, but memory vendors are just like sort of

to some extent raising a lot at price, right? So they're going to double or triple price again. But then they're also signing these long-term deals. So who is able to accrue all the margin dollars is actually, you know, potentially the cloud, potentially the chip vendors and the memory vendors. Um until TSMC or ASML like break out and they like, no, actually we're gonna charge a lot more. Um but at the same time, do the model vendors get to charge crazy margins?

Um I think at least this year we're gonna see margins for the model vendors go up a lot, right? Because they're so capacity constrained, they have to demand destroy demand, right? There is there's no way they can continue anthropic can continue at the current pace without destroying demand.

How specifically has NVIDIA has been able to lock up so much of both? So if you I think according to your numbers, by 27, NVIDIA is gonna have like 70 plus percent of N3 wafer capacity or something like that. Um uh were around that area. And then I I I forget what the numbers were for a memory at SK High Next and Samsung and so forth. But um if you look at

So uh think about how the neo cloud business works and how NVIDIA works with that, or how the uh RL environment business works and how anthropic works with that. In both those cases, NVIDIA is purposely trying to fracture. the complementary industry to make sure that they have as much leverage possible. So they're giving

you know, allocation to random neo clouds to make sure that there's not one person that has all the compute. Similarly, Enthropic or OpenAI, when they're working with the data providers, they say, no, we're gonna just of seed a huge industry of th these things so that um we're not locked into anyone supplier for uh for data environments. And I wonder why

On the three nanometer process, there that that's gonna be Tranium three, that's gonna be TPU V seven, uh other accelerators potentially. And why is TSMC just giving it all up to NVIDIA rather than, you know, just trying to fracture the market?

Apple likely reduces demand slash moves to two nanometer faster, where two nanometer is only capable of uh sort of mobile chips today. Um and in the future AI chips will move there. So sort of Apple has that. And then Apple's also talking to uh third party vendors because they're getting squeezed out of T SMC a little bit.

Um because TSMC's margins on high performance computing, um HP C AI chips, et cetera, is higher than it is for mobile. Um because they have a bigger advantage in mobile, um in sorry, in HPC than they do in mobile. But anyways. When you look at what's what's TSMC running calculus here, actually they're providing really good

um allocations to companies that are doing CPUs, right? So when you think about, hey, Amazon has trained and Amazon has uh Graviton, both of those are on three nanometer, Graviton being their CPU, training being their their AI chip.

They're actually uh uh TSMC is much more excited to give allocation to Graviton than they are to train because they view CPU business as more stable long-term growth, right? And As a company that is conservative and doesn't want to ride cycles of growth too hard, you actually want to allocate to the uh The market that is more stable and lower growth rate first before you allocate all the incremental capacity to the fast growth rate market. Now,

That is that is the case generally. And so when you look at like, hey, same for AMD, right? The c allocations they get on um you know their CPUs is is like TSMC is much more excited about those than they are for GPUs. Um likewise for Amazon. And NVIDIA um is is a bit unique because all yes, they have CPUs, yes, they make switches, yes, they make networking. Um they make NV Link, they make all these different infiniband Ethernet, all these different products.

Um by and large, most of these things will be on three nanometer by the end of this year with the Ruben launch and all the chips that are in that family. Um, the GPU being the most important one. And yet NVIDIA is getting the majority of supply, right? Part of this is because you look at the market and you like sort of like, you know, TSMC and others, like they there are many ways that they forecast market demand. Um

But also it's demand market signal, right? The market signaled, hey, we need this much capacity next year. We need this much. We need this much. We'll sign non-cancelable, non-returnable. We may even pay deposits, right? Things like this. NVIDIA just did it way earlier than Google. Or Amazon.

And in some cases, Google and Amazon had stumbling blocks. You know, there was one one of the chips got delayed slightly by by a couple quarters, uh, training and and all these sorts of things happened. And then so in that case, there was

A huge sort of like, okay, well, these guys are delaying, but NVIDIA's wanting more, more, more, more, more. And we are checking with the rest of the supply chain. Is there enough capacity? Right. So they're going to all the PCB vendors and they're saying, hey, is there enough? Uh Victory Giant. Is there enough PCB? This is like one of the largest suppliers of PCBs to NVIDIA and they're a Chinese company. All the all the PCBs come from China, sort of from them. Um

Or many of them. And in any ways, they're like, Do you have enough PCB capacity? Great. Oh hey, uh memory vendors, who has all the memory capacity? Oh, okay, Nvidia does great. Um so when you look at Sort of in the same way, you know, who who is AGI pilled enough to buy compute in long timelines at levels that seem ridiculous.

to people who aren't AGI pilled, but nonetheless they're willing to pay a pretty good margin um and sign it now because they view in the future that that ratio is screwed up. The same thing happens with the supply chain for semiconductors, right? Nvidia was Well th I don't think NVIDIA's quite AGI pilled, right? You know, Jensen doesn't believe software's gonna be automated fully and all these things, right?

Google and Amazon, you know, Google especially, even though their, you know, their TPU is just better for them to deploy, they have to deploy a crap load of GPUs because they don't have enough TPUs to fill up their data centers. They can't get them fab.

And Google has DeepMind. This is the other third prominent AI lab. And if this is the big bottleneck, why would they sell it rather than just giving it to Deep Mind?

Um you you you know, you and I know the compute team. There's one guy from you know, both of them actually came from Google uh at at uh you know the main people on on uh the compute team at Thropic. They saw this dislocation, they negotiated a deal, and they were able to get access to these to this compute. Before Google realized. And so actually, the chain of events, at least from our data that we found, was in early Q3, we saw over the course of two over the course of like six weeks.

We we saw capacity on um anthropic or sorry, on TPUs go up by a significant amount. Over the course of those six weeks and it went up like multiple times in those six weeks, right? There were multiple requests. Google even had to go to TSMC and explain to them why they needed this uh increase in capacity because it was so sudden.

But that a lot of that capacity increase was for selling to Anthropic. Yeah. Because Anthropic saw it before Google. And then Google had Nano Banano and Gemini 3, which caused their user metrics to skyrocket. And leadership at Google was like, Oh and then they started making the statement of we have to double compute every is it six months or I don't remember the exact number that they said. Um, but they they really woke up a lot more and then they're like, Oh

Hey, T SMC, we want more. We want more. And it's like, well, sorry guys, like we're sold out for next year. Um, we can work on next year, we can maybe get like five, ten percent more for twenty six, but really we're gonna work on twenty seven, right? This is sort of like you know, there's this like information asymmetry of the labs in my mind, right? I don't know if this is exactly the narrative I've spun myself from seeing all the data in the supply chain on like wafer orders and like

what's going on with the data centers that, you know, Anthropic signed and FluidStack signed and all this. Like sort of it's it's it's it's pretty clear to me that Google screwed up. And you can see this from Google's Gemini ARRs, right? Um they had next to nothing in Q1 to Q3, uh Q three a little bit, right? Once they started inflecting, but Q4 they were at like five billion ARR.

Right. Um, exiting or something like this. So it's like or five billion revenue for Q4 uh on an ARR basis. Um and so it's clearly like Google didn't see revenue skyrocket. Um and in a sense, right, anthropic was Not willing, you know, has kind of had like a little bit of commitment issues before their ARR exploded, even though they have far more information asymmetry and see what's coming down the pipe. Google is going to be more conservative than Anthropic is.

A and B, Google had had even less ARR. Um, so they they sort of were like, I think just not willing to like sort of do it and then they realized they should do it. And so now since then, Google um Has gotten absurdly AGI pilled, right? Uh in terms of like what they're doing. They bought an energy company, they're buying, putting deposits down for turbines.

Uh, they're buying a ridiculous percentage of the powered land. Uh they're going to utilities and negotiating long term agreements. They're doing this on the data center and um power side. Um Very, very aggressively, right? So, you know, I think uh Google woke up towards the end of last year, but it took them some time.

Couple years, three, two to three years versus a data center takes uh less than a year, right? Uh we've seen Amazon build data centers in as fast as eight months. Right. So there's a big difference in lead times because of the complexity of the building, the fab that actually makes the chips. And then the tools, right? Those also have really long lead times.

And so the bottlenecks as we've scaled have shifted from, hey, what is the supply chain currently not what is it currently not able to do? Um, which was co-op. power and data centers, but those were all shorter lead time items, right? Coas is a much more simple process of packaging chips together. Um, power and data centers are ultimately way more simple than the actual manufacturing of the chips. And so there's been some sliding of of of capacity across

you know, mobile or PC to data center chips, but that's been somewhat fungible, whereas on in and whereas CoAs and Power and Data Centers has have sort of had to start anew as supply chains, but now there's sort of no more capacity for

the mobile and PC industries, which used to be the majority of the semiconductor industry, to shift over to AI, right? NVIDIA is now the largest customer at TSMC and NVIDIA is the largest customer at SK Heinex, the largest memory manufacturer, right? So it's sort of Impossible for this scaling or the sliding of resources away from the common person, right? PCs and smartphones to shift.

anymore towards the AI chips. And so now how do we scale the AI chip production? And that's the biggest bottleneck As we go to twenty thirty is those.

Um, and the selling price for those is three hundred, four hundred million dollars. And currently they can make about seventy. Next year they'll get to eighty. Uh, even under very aggressive supply chain expansion, they only get to a little bit over a hundred by the end of the decade. And so what does that mean? Okay, they can make a hundred of these tools by the end of the decade and um you know, seventy right now.

Wha how does that actually translate to AI compute, right? We we see all these numbers from Sam Altman and and many others across the supply chain, gigawatts, gigawatts, gigawatts, right? How many gigawatts are we adding? Um and we see, you know, Elon saying, hey, the hundred gigawatts in space.

And to make a gigawatt worth of data center capacity of NVIDIA's latest chip that they're releasing at the end of this year or towards the end of this year, you need, you know, a few different wafer technologies, right? Um you need About 55,000 wafers of three nanometer, you need about six thousand wafers of five nanometer, and then you need about a hundred and seventy thousand wafers of DRAM, right? Memory. And so across these three different buckets, um

Each of these requires different amounts of EUV, right? So when you manufacture a wafer, uh there's thousands and thousands of process steps where you're depositing material, removing them. But the sort of key critical step, which at least in advanced logic is like 30% of the cost of the chip.

something that doesn't actually put anything on the wafer, right? You take the wafer, you deposit photoresist, which is like a chemical that basically chemically changes when you expose it to light, and then you stick it into the EV tool, which shines light at it in a s certain way. It patterns it, right? Because there is what's called a mass

s which is a stencil effectively for the design. And so when you look at a wafer, um, you know, leading edge three nanometer wafer has 70 or so masks, right? Seventy or so layers of lithography, but twenty of them are the most advanced EUV, right? And that specifically You know, if you think about okay, well if I need fifty five thousand wafers

five nanometer and all the memory. You're at roughly um two million UV passes for a single gigawatt. You know, these these tools are very complicated. So um when you think about what it's doing across a wafer, it's taking the wafer and it's scanning

And it's stepping across, right? It's Danning, stepping across, and it does this hundreds of times across the entire or dozens of times across the whole wafer. And and so when you're talking about, hey, how many UV passes, that's the entire wafer is being exposed.

um at a certain rate. A wafer a a UV tool can do roughly 75 wafers per hour. Um and the tool is up roughly 90% of the time. Right. So in the end You end up with actually, I need about three and a half EUV tools to do the 2 million EUV wafer passes for the gigawatt.

So three and a half EUV tools uh satisfies a gigawatt. So it's funny to think about the numbers, right? Because we're talking about, oh, what's a gigawatt cost? It costs like fifty billion dollars, roughly, right? Whereas what does three and a half EUV tools cost? That's like one point two, right? Right. Um it's actually like quite a lower number. Which is which is interesting to think about, like oh fifty gigawatts of economic

you know, sort of CapEx in in the data center and what gets built on top of that in terms of tokens is even larger, right? It might be a hundred billion dollars worth of AI value into the supply chain is held up by this $1.2 billion worth of tooling that simply just cannot expand its supply chain quickly.

Your earnings last quarter was like forty billion. And so forty billion times four. So a hundred and sixty billion dollars. So NVIDIA alone is turning. Some small fraction of a hundred billion in CapEx that's gonna be depreciated over many years, not just this one year, into a hundred and sixty billion dollars in a single year.

And then that gets even more intense when you go down the supply chain to ASML, which is taking a billion dollars worth of machines to produce a gigawatt. And then of course those machines last for more than a year, right? So it's it's doing more than that. Okay, so now I wanna understand, okay, well, how many such machines will there be by twenty thirty if you include not just the ones that are sold that year, but are have been compiling over the previous years.

Um, and what does that imply about the Sam Altman says he wants to do a gigawatt a week in twenty thirty. Are are th uh when you add up those numbers, is that compatible with that?

Um assuming it's all allocated to AI, which it's not, but three and a half tools per gigawatt gets you to 200 gigawatts worth of AI chips for the data centers to deploy, right? So two hundred gigawatts, Sam wants fifty gigawatts, right? Fifty two gigawatts a year. He's only taking 25% share then, right? Obviously, there's some share given to um, you know, mobile and PC, uh, assuming that

you know, s for some reason we're allowed to even have consumer goods still. Um, you know, and we don't get priced out of them. But, you know, roughly like he he's saying twenty five percent, fifty per s l you know, twenty five percent market share of the total chips fab. That's that's kind of like very reasonable given You know, this year alone, I think he's gonna have access to twenty five percent of the black wall GPUs that are deployed, right? So it's it's not that crazy.

cleaning the wafer, you know, dozens of those steps before you do another EUV layer. Um there's a spec called overlay, right? Which is, okay, you did all this work, you know, you drew these lines on the wafer. Um now I want to draw these dots, right? Let's just say I want to draw these dots to connect this

l these lines of metal to and then dot you know, holes and then the next layer up is another set of lines that goes perpendicular. So now you're connecting wires going perpendicular to each other. Um

There you have to you have to be able to land them on top of each other. So it's called overlay. And overlay is a spec that's been improved rapidly by ASML. Wafer throughput has been improved rapidly by ASML. And also the price of the tool has gone up, but not as much as the capabilities of the tool.

Right. Initially the UV tools were like one fifty million and over time they're now like four hundred million. Uh, you know, as I as I look out to twenty twenty eight, but the capabilities of the tools have more than doubled as well, right? Especially um on throughput and overlay accuracy, which is the ability to stack you know, accurately align the the sub sub subsequent passes on top of each other. Um even though you do tons of steps between. And so this is this is um

You know, ASML is improving super rapidly. I think it's also something noteworthy to say. ASML is n you know, maybe one of the most generous companies in the world, right? They have this linchpin thing. No one has anything competitive. Maybe China will have some EU V by the end of the decade, but no one else, you know, has anything even close to EU V. Um And yet they haven't taken price and margins up like crazy, right? You know, you go ask. You know, that we talk to all the time.

Like, you know, for example, Leopold and they're like, Well, you know, let's let's you know, the let's let's have the price go up, right? Uh because they can. The margin is there. You can you can take the margin. Like NVIDIA takes the margin, memory players are taking the margin, but ASML has never risen the price more than they've increased the capability of the tool.

You know, m some players as of very recently have like woken up, but in general, no one really sees demand for two hundred gigawatts a year of AI chips or, you know, trillions of dollars of spend a year in the semiconductor supply chain. They're just like, They're not they're not AI pilled, right? They're not AGI.

has sort of their tool has four major components, right? It has um the source, right, which is made by Simer in San Diego. Um it has the uh reticle stage, which is made in Wilmington, uh Connecticut, right? It has the wafer stage. um and the um the optics, right? The lenses and such. And those two are made in Europe, right? And so when you when you look at each four each of these four, they're tremendously complex supply chains.

that A, they have not tried to expand massively and B, when they try to expand them, the Time lag is quite long, right? Um and so again, this is the most complicated machine that humans make, period, right? At at a volume, um at any sort of volume, but like let's talk about the source specifically, right? What is the sport?

It hits it three subsequent times with a laser perfectly. So the first one uh hits this tin droplet, expands out, it hits it again, so it expands out to this perfect shape, and then it blasts it at super high power. And um the tin droplets get excited enough that they release.

uh EU V light, thirteen point five nanometer, and then it's in this thing that is like basically collecting all the light and directing it into the lens stack, right? Then you have the lens stack, which is Carl Zeiss, right, as you mentioned, and and and some other folks, but Zeiss being the most important part of it. Um

They also have not tried to expand production capacity because they don't see any you know, they they they're like, Oh yeah, yeah, like we're growing a lot because of AI. We're going from sixty to a hundred.

Right. It's like, no, no, no, no, we need to go to like a couple hundred, but it's it's fine, whatever. Um, each of these tools has, you know, I think eighteen um of these lenses effectively. Um, mirrors. Um, they are they're multi layer mirrors which are perfect layers of molybdenum And r uh ruthenium, if I recall correctly.

um stacked on top of each other in many layers and then the light bounces off of it perfectly. But it's not just like, you know, like when we think about a lens, you know, it's it's like in a shape and it focuses the light. This is a m this is like a mirror that's also a lens and so it's pretty complicated.

Any defect in this perfect layer of sta in this in these like uh super thinly deposited stacks will mess it up. Uh any curvature issues, like there is a lot of challenges with scaling the production. Um it's quite artisanal, right, in this sense, right? Because you're not making

tens of thousands of these a year, you're making hundreds, you're making thousands, right? Uh, you know, talk about 60 tools a year, um, 18 of these per tool, you end up with, you know, you're still in the um, you know, hundreds of tools. uh or thousand you're at the thousand number roughly for these these lenses um and projection optics.

So then you and then you step forward to the reticle stage, uh, which is also something really uh crazy. This thing moves at I wanna say nine Gs, like it it will shift. 9Gs because as you step across a wafer, the tool will go. Um, and the wafer stage is complementary, it's the wafer part. So you you line these two things up, you're taking all the light through the lenses that's focused.

and and here's the reticle, here's the wafer, and you're passing uh the reticle's moving one direction, the wafer's moving the direction the other direction as it scans a uh 26 by 33 millimeter section of the wafer and then it stops. It shifts over to another part of the wafer and does it again. And it does that in just seconds, right? And it and each of them are moving at nine G's in opposite directions.

So each of these things is like a wonder and marvel of like chemistry, uh fabrication, you know, um, you know sort of like mechan mech m mech mechanical engineering, um optical engineering because you have to align all these things and make sure they're perfect. Uh all these things have crazy amounts of metrology because you have to perfectly test everything because if anything is messed up, the yield goes to zero.

'Cause this is such a finely tuned system. And by the way, you it's so large that you're building it in all these you're building in the factory in Heindhoven, uh, Netherlands, and then deconstructing it and shipping it on p many planes. to the customer site and then you're reassembling it there and testing it again. And that process takes many, many months. So like it's it's just there's so many steps in the supply chain, right? Whether it's Zeiss making their l uh lenses and projection optics.

or Simer, which is an ASML owned company, making the EUV source. And each of these has th its own complex supply chain, right? ASML's commented their supply chain has ten over ten thousand people in it, right?

you know, single digit nanometers or even smaller because the entire system, the overlay, right? Uh layer to layer uh variation has to be on the order of three nanometers, right? Um and so if the overlay is three nanometers, that means each individual part the accuracy of its physical movement has to be even less than that, right? It has to be sub one nanometer in most cases because the the error of these things stacks up, right? And and and so there's no way to like

you know, just like snap your fingers and increase production, right? You know, it's things simple as power, right? The US going from zero percent power growth to two percent power growth, even though China's already at thirty, was like so hard for America to do, right? Um And and and that's a really simple supply chain with very few people in the supply chain, right? Uh who make difficult things. And there's

you know, probably what, a hundred thousand electricians slash people who work in the supply chain uh of electricity um or more in the US. And, you know, when you look at oh ASML employs like so few people, Carl Zeiss Probably employs like less than a thousand people working on this. And all of those people are like super, super specialized. So it's, you know, you can't just train random people up for this.

like in the snap of a finger. You can't just get your entire supply chain to gat get galvanized, right? Nvidia's had to do a lot. Um To get the entire supply chain to even deliver the capacity they're gonna make this year, even though when you look go talk to Enthropic, they're like, Well, we're short of TPUs, we're short of training, we're short of GPUs. When you go talk to OpenAI, they're like, We're short of these things, right? Um so OpenAI and Anthropic, they know they need X.

NVIDIA is not quite as AGI pilled and they they're building uh you know, X minus one. Um and you go down the supply chain, everyone's doing minus one. And in some cases they're doing like divided by two, right? Because they just don't they're not AGI pilled, right? I think and and and so you end up with uh the time lag for this whip to react, right? You know, the the cur the sort of

AI pilledness is and and desire to increase production is so long. And then once they finally understand, hey, we need to increase production rapidly, right? And they they think they understand, oh, AI means we have to go from sixty to a hundred.

In in addition to the tools all just getting better and faster, you know, the source getting higher power from five hundred watts to a thousand and, you know, all these other aspects of the supply chain, you know, advancing technically plus increase of production. They think they're they're like actually increasing production a lot.

But if you float through the numbers of, hey, what does Elon want? He wants a hundred gigawatts a year in space by twenty twenty eight, is it? Um, or twenty twenty nine, and you know Sam Altman wants fifty gigawatt, fifty two gigawatts a year um by the end of the decade. And you look at, you know, Prayanthropic needs the same, and then, you know, Google needs that. You know, you you go across the supply chain.

Wait, no, the the supply chain can't possibly build enough capacity for everyone to get what they want on the side of compute.

And it's not always clear why. Labelbox realized that this was a huge bottleneck for their customers. So they built an evaluation pipeline called Echo Chain to help you diagnose and fix your voice model's specific failure mode. Echo Chain starts by feeding conversations into your voice model and then injects interruptions at specific intervals and classifies any failures into one of three different modes. One, did it acknowledge the correction but keep the old plan?

Did it adapt briefly but then slide back to old assumptions? Or three, did it abandon the old task entirely? This is extremely useful information because Labelbox can get your model the exact data it needs to fix whatever issue is preventing it from being a viable and competent voice model. So if you want to ensure that your voice model stays performant in real conversations, you should reach out to LabelBox. Go to labelbox.com/slash dwarcash.

So I feel like in the um in the data center supply chain for the last few years, people have been making arguments of this specific thing we are bottlenecked by. Therefore AI compute can't scale more than X.

But then as you've written about, oh no, if you know, say the grid is a bottleneck, then we just do um we just do behind the meter on the site, we do gas turbines, etc. If that doesn't work, there's like all these other alternatives that people fall back on. And I I wanna ask you a question about wh whether we can imagine a similar thing happening.

in the semiconductor supply chain. So if EU V becomes a bottleneck, well we're we you know, what if we just went back to seven nanometer and do what China is doing currently and producing seven nanometer chips with uh multi patterning with DUV machines. Um and you know, if if you look at a seven nanometer ship like the A one hundred, um there's been a lot of progress obviously since uh from the A one hundred to the B one hundred or B two hundred. But um

How much of that progress is just numerics? And then like if if you just hold constant, say uh FP sixteen from A one hundred to B one hundred. The B one hundred is like a little over one petaflop, and then uh A one hundred is like three hundred teraflops.

And so you have you have like basically three X uh holding numerics constant. You have like a three X improvement from A one hundred to B one hundred. And then some of that is the process improvement, some of that is just the accelerator design improving, which, you know, we could replicate again in the future. And so th then it just seems like that actually it's like very small effect from the process improving from seven nanometer to four nanometer. So I don't know, this is say we have uh

I don't know the numbers offhand, but let's say there's like a hundred and fifty K wafers per month of three nanometer and then eventually similar amounts for two nanometer. But then there's a similar amount for seven nanometer, right? So if you have all those old wafers. And then there's maybe a fifty percent haircut because the process you know, the bits per wafer area are like

What is it, fifty percent less or something? Um, then it's like it doesn't seem like that bad to just bring on seven nanometer wafers and then oh, that gives you a another fifty or a hundred uh another hundred gigawatts. Um yeah, t tell me why that's naive.

um ish of FP sixteen, um, or maybe it's two thousand and then Ruben is like five thousand or so F P sixteen. It's it's not a fair comparison because these chips have vastly different um you know, design targets, right? At at A one hundred, that's what that that is what NVIDIA optimized for was F sixteen, B float sixteen numerics. When you look at uh Hopper

They didn't care as much about that. They cared about FP8. When when you look at Ruben, they don't care about FP16 and BF16 as much. They care mostly about fp four and six. Right. Um, and so numerics like are what they've designed the search uh designed their chip for. Um

And so there's a couple like, you know, okay, let's just say let's redesign, let's make a new chip design on seven nanometer. Sure, we can do that. Like, and then it's optimized for uh the numerics of the modern day. The performance difference is still gonna be much larger than the flops different you mentioned, right? Um often it's easy to boil things down to flop uh per watt or flops per dollar.

But that's actually not a fair comparison, right? Um and so this is where sort of you can bring in, hey, let's look at Kimi K1 or Deep Seek. When you look at Kimmy or or Ki Kimi K2.5, sorry, and Deep Seek, when you look at these two models. And you look at their performance on Hopper versus Blackwell on you know very optimized software, you get vastly different performance, right? And

Most of this is not attributed to flops. A lot of this is a tr and or numerics, right? Because those models are actually eight bit. So it's not like Blackwell's uh and Hopper, they're both optimized for eight bit and Blackwell's not really taking advantage of its four bit there. Um

You know, the the performance gulf is is actually much larger. And you know, the way you can sort of compare them and think about them is sure it's one thing to you know shrink process technology and make the transistor smaller and each chip has X number of flops.

But you forget the big gating factors, which is these models don't run on a single chip. They run on hundreds of chips at a time, right? If you look at Deep Seek's production deployment, which is well over a year old now, they were running on 160 GPUs. Right. Um and that's what they serve production traffic on. And so they split the model across 160 GPUs.

Every time you cross the barrier of a chip to another chip, there is an efficiency loss because you now have to uh transmit over, you know, high-speed electrical Certies. And there is a latency cost, there's a power cost, there's a, there's all these um dynamics that hurt. As you shrink and shrink and shrink the process node, you've

increase the amount of compute in a single chip. Now in chip, right, uh movement of data is, you know, at at hundreds or of of at least tens of terabytes a second, if not hundreds of terabytes a second.

Um whereas between chips, you're on the order of a terabytes of second, right? Um and and and so this this movement of data between chips that are super close to each other physically, and then you can only put so many chips close to each other physically, so you have to put chips in different racks. the order of uh data uh between that is on the order of hundreds of gigabits a second, right? 400 gig or 800 gigasecond.

Um so a a hundred gigabytes a second, roughly. And so you've got this like huge ladder of like, oh, on chip I can communicate communicate at super fast speeds. Within the rack I can communicate at you know order magnitude speeds. Outside the rack I can communicate at an even order of magnitude lower than that. And as you break the bounds of chips.

you end up with this performance loss. So anyways, the reason I explained this is because when I look at when you look at Hopper versus Blackwell, even if both of them are using, you know, a rack worth of chips, the hopper is significantly slower because the amount of performance that you have

leverage to the task within that, you know, within each domain of, hey, tens of terabytes of seconds of communication between these transistors or or these processing elements, and terabytes a second between these processing elements is much, much higher and therefore the performance is much higher. So when you look at inference at let's say 100 tokens a second for Deep Seek and Kimi K 2.5, Hopper versus Blackwell, the performance difference is on the order of 20x.

Up, there is some performance loss. It's not just perfect, but it is way better than different entire packages. Now, how large can advanced packaging scale? Um, The way NVIDIA's doing it is co ops, the way g uh, you know, Google and uh with Broadcom and MediaTech and, you know, Amazon, Trainum, all these chips are doing is called coas. But actually you can go and look back at what um

What Tesla did with Dojo, right? Dojo, uh, which they canceled and restarted. I don't know. Anyways, Dojo was a chip that was the size of an entire wafer. They had 25 chips on it. Um, and there were some trade-offs, right? They couldn't put HPM on it. Um, but the positive side was it that they had 25 chips uh on it. And so to date, it is still probably the best chip for running convolutional neural networks.

Um, it's just not great at transformers because the, you know, the sort of the shape of the chip, the memory, the arithmetic, all all these various specifications of it are just not well suited for transformers. They're well suited for CNNs. Um And anyway, so so you know, Dojo chips were optimized around that they made a bigger package, but at the same time

You know, as you make packages bigger and bigger and bigger, you have other constraints, right? Networking speed, uh, memory bandwidth, cooling capabilities, all of these things start to rear their heads. It's not simple, but yes. you will see a trend line of more chips on the package. And yes, you're gonna be able to do that on seven nanometer. In fact, that's what Huawei did with their um Ascend 910 C or D. Uh they put they put they were initially just one and then they did two.

Um, and they're focusing on scaling the packaging up because that is an area where they can advance faster than sort of process technology where they can't shrink. But at the end of the day, that's still, you know, that's something that you can do on the leading edge chips too, right? Anything you do on seven nanometer, you can also probably do on three nanometer in terms of packaging.

process technology, but it has not ramped it up as much. Whereas China, I don't know if you think by twenty thirty, they would have UV and I don't know, two nanometer or whatever. But they are semiconductor pills. So they're producing in mass quantity. Um, basically I'm wondering what the year is where there's a crossover where our advantage in process technology has faded enough and their advantage in scale has increased enough.

And also their advantage in like having one country that has the entire supply chain envisionized rather than having random suppliers in Germany and Netherlands and whatever would mean that China would be ahead in its ability to pr like produce mass flops.

They're they're trying to do all these things. The question is how fast can they advance um and and scale up production as well as quality? And to date, we haven't seen that. Now I'm quite bullish that they're gonna be able to do these things over the next five to ten years, right? Really scale up production, uh really uh kick it into high gear. They have more engineers working on it. They're um they have more uh desire to throw capital less across

Uh scaled for high production for high volume manufacturing, reliable enough. And then they had to ramp production and that all took time. Production hell takes time, right? Which is why it took another five to seven years to get EUV into mass production at a fab rather than just it working in the lab.

stages from Japanese vendors, they need to build it internally. So it's really tough to say where they'll be able to get to. Like I honestly think it's like a shot in the dark. But it's it's probably not unlikely that they'll be able to do, you know, on the order of a hundred DUV tools a year, uh, whereas ASML is doing hundreds of e DUV tools a year currently. No one's made

a process node, no company has a process node where they make a million wafers a month, right? Um Elon says he wants to do it and China's obviously going to do it, right? Uh And I don't think the you you know, T SMC is trying to do that. Um the memory makers may get there as well, right? To the million wafers a month, but not in a single fab. It it's it's it's sort of mind boggling to think of that scale um and challenging to Yeah.

So I'm not sure, you know, I don't want to doubt, you know, China's capability to scale. Right.

all the d w w and put in the input of your model when they'll have uh do via machines at scale, when they'll have UV machines at scale. Because I think there's this like question around if you have long timelines on AI, by long meaning twenty thirty five, which is not that long in the grand scheme of things.

Um, should you expect a world where China's just like dominating in semiconductors? Which I think I don't know, it doesn't get asked enough visit in San Francisco, where just like thinking on time scale of like, you know, d weeks. And then if you're outside of San Francisco, you're not thinking about AGI at all.

And so this question of like, okay, what if we have AGI? What if you have this transformational thing that is commanding tens of trillions of dollars or hundreds of trillions of dollars of economic growth and weight, you know, uh token output and so forth? Uh, but then it happens in twenty thirty five. And like, what does that imply for the West versus China? I think it's just like I don't know. The semi analysis has got to write the definitive uh model on this.

Um, we can only make like reasonably accurate estimates for data center capacity based on, you know, land purchasing and, you know, permits and turbine purchasing and all these things. And we know where all these things are going and we like that's what the data we sell is, but like

you know, as you go out to like twenty thirty five, you know, things are just so radically different and, you know, your error bars get so large it's kind of hard to make an estimate. Uh, but at the end of the day, like you know, there is if takeoff or timelines are slow enough, right? Um, then certainly China I I don't see why they wouldn't be able to catch up drastically, right? Um, you know, in in some sense we've got like this valley, right? Of where

you know, call it three to six months ago, Chinese models were or maybe even now, Chinese models are as competitive as they've ever been. Uh I think I think Opus four six and GPT five point four have really pulled away and made the gap a little bit bigger, but I'm sure, you know, some new Chinese models will come out. But As we move from, you know, hey, these companies are selling tokens where they provide the entire uh reasoning chain and all that to uh selling automated, you know.

white collar work, right? Automate a software engineer, send them the request, they give you the result back, and there's a bunch of thinking on the back end that they don't show you. The ability to distill out of American models into Chinese models will be harder. A. B

As the scale of the compute that the labs have, right? Uh OpenAI exited the year with roughly two gigawatts last year. Um Anthropic will get to, you know, two plus gigawatts this year. And and by the end of next year they'll both be at like ten gigawatts of capacity. Um China has is not scaling their AI lab compute nearly as fast. And so at some point, you know, when you can't distill the learnings from these labs.

into the Chinese models plus this compute uh race that open ianthropic Google etc meta are all racing on, at some point they end up getting to a point where, you know, the model performance should start to diverge more. Um and then all of this CapEx that's being spent on, you know, data centers and all that, right? Amazon, you know, two hundred billion, Google one eighty, you know, so on and so forth, all these companies are spending hundreds of billions of dollars of CapEx.

Um, you know, there's there's, you know, nearly a trillion dollars of CapEx being invested in data centers in America this year, roughly, right? Um You you end up with, okay, well, what's the return on invested capital here? Uh, you and I would think that the return on invested capital for data center CapEx is very high. Um and at least if we look at anthropics revenues in, you know, January they added like four billion, in February, which is a shorter month, they added like six.

Um we'll see what they can do in March and April, um, given compute constraints are what's bottlenecking their growth, right? The reliability of quad code is actually quite low because they're so compute constrained. Uh but if this continues, then the ROIC on these data centers. is super high. Yeah. Um and at some point the US economy starts growing faster and faster over the next

you know, n this year and next year, because of all this capex and all this revenue that these models are generating, um, and downstream supply chain versus China doesn't have that yet, right? Um they have not built the scale of infrastructure to then invest in model uh to invest in models to get to the capabilities, to then deploy these models at such scale, right? Uh'cause when you look at like anthropics, hey, they're at call it twenty billion ARR.

Of that, you know, the margins are sub fifty percent, at least the last reported by the information. So then, you know, you're at okay, that's like thirteen, fourteen billion dollars of compute that it's running on rental cost wise, which is actually like fifty uh billion dollars worth of capex that someone laid out for anthropic to generate their current revenue. Um and China has just not done this if

If and when anthropic 10x is revenue again, uh, and I think our our answer would be when, not if, um, then China doesn't have the compute to deploy at that scale. And so there is some sense of like, oh, we're in fast takeoff ish. Right. And it's not like we're talking about, you know, Dyson Sphere by X day. It's more like the revenue is compounding at such a rate that it does affect the ec economic growth. Um and the resources these labs are gathering are s going so fast that

You know, and and China hasn't done that yet. So in that case, the US uh and the West is actually diverging. The flip side is actually the these these infrastructure investments have middling returns. Maybe they're not as good as as as hope.

You know, maybe Google is wrong for wanting to take free cash flow to zero and spend three hundred billion dollars on CapEx next year. Maybe they're just wrong. Um and, you know, people on Wall Street who are bearish and people who don't understand AI are correct. Right. Um, and in which case, then the US is building all this capacity, it doesn't get really great returns, and China's able to build the fully vertical indigenized supply chain, not you know, US.

Japan, Korea, uh Taiwan, Southeast Asia, you know, Europe, all these all these countries together building this like less vertical supply chain. Um And in a sense, at some point China is able to scale past us if AI takes longer to get to certain capability levels than you know, I would say the vast majority of your guests on this podcast believe.

So we've got this memory crunch, as you're talking about. And earlier I was asking about, oh, could we solve for the EUV tool shortage by going back to seven nanometers? So let me ask a similar question about memory. Um HBM is made of DRAM but has three to four X less. Uh bits per wafer area than the DRAM it's made out of.

Is it possible that accelerators in the future could just use commodity DRAM and not HBM? And so just we can make much more uh capacity out of the the DRM we get. And the reason I think this might be possible is look. If we're gonna have agents that are just going off and doing work and it doesn't really you don't it's not a synchronous chatbot application. then you don't necessarily need extremely high uh fast latency kinds of things anymore. And so maybe you can have the low, low bandwidth.

uh because you the the reason you stack DRAM into stacks and make HBM is for higher bandwidth. And so is it possible to go to HBM uh accelerators and um and basically have the opposite of clot code fast, like have clot code slow and

Sure, Anthropic could actually release a slow mode, right? They could release Claude Slow Mode and have an increase in tokens per dollar by a a significant amount. Um, they could probably like reduce the price of Opus four six by you know, four X, five X, and reduce the speed by another by maybe just like two X. Like the curve on inference throughput versus speed is there already just on HBM. Um, and yet they don't uh

n because no one actually wants to use a slow model. And furthermore, on these agentic tasks,

You know, it's it's great that the model can run at this time horizon of hours. That's kind of like, okay, well if if the model was just running slower, that hours would become a day, right? Um or vice versa, right? If the model's running faster, that hours becomes hour. Um And yet no one really wants to move to that day-long wait period because the highest value tasks also have some time sensitivity to them, right?

And and so I'm I struggle to see, you know, yes, you could use DDR, um, but then there's a couple like things that are challenging with this, right? You could use regular DRAM. Um One is you're you're still limited, you know, one of the like core constraints of chips, even though they're, you know, sort of like, you know, the there's an a chip is like a certain size.

All of the I.O. escapes on the edges of the chip, right? So oftentimes, you know, what you see is the left and the right of the chip are HPM. The I/O from the chip to the HBM is on the sides. And then the top and bottom are I/O to other chips, right? Um And so if you were to change from HBM to DDR,

then all of a sudden this I/O on this edge would have significantly less bandwidth, but it had significantly more capacity per chip. Yeah. Because and and and so yes, you're making less um You know, the the the metric that you actually care about is bandwidth per wafer, not bits per wafer.

for tool calling or whatever, how much do I um how many chips do I paralyze this on? Obviously these are like the the search space of this is like very broad, which is why we have like inference X, like this is like an open source model like searches all the optimal points on inference for a variety of eight different chips. um and models. Um anyways like the point is you don't necessarily you're not always necessarily constrained by memory capacity.

Uh you can be constrained by comp flops, you can be constrained by network bandwidth, you can be constrained by memory bandwidth, uh, or you can be constrained by memory capacity. There's sort of like four, if you're really to simplify it down, there's like four constraints. And each of these can break out into more, but

In this case, if you switch to DDR, yes, you produce four X the bits per DRAM wafer, but all of a sudden the constraints shift a lot and your system design shifts a lot. You go slower, yes. Is the market smaller? Okay, maybe possibly. But also now all of a sudden all these flops are wasted because they're just sitting there waiting for memory. It's like great, I don't need all that capacity because I can't really

increase batch size because then the KV cache is going to take even longer to read. And so you never you can yeah.

And in that shoreline, um you have 2048 bits transferring at 10 gigatransfers per second. Uh you multiply those together and you divide by eight bits to bytes, you're at roughly two and a half terabytes a second per HPM stack. When you look at DDR um in that same area, it's maybe 64 or 128 bits wide. And that DDR five is transferring at any, you know, anywhere from 6.4 gigatransfers a second to maybe eight, 8,000 gigatransfers a second.

So your your bandwidth is like significantly lower lower, right? It's sixty-four times eight thousand divided by eight. Um you're at sixty sixty-four gigabytes a second. Um and even if you take a generous interpretation of one twenty eight. Times eight gigatransfers, you're at 128 gigabytes a second for the same shoreline versus two and a half terabytes a second. There's a there's an order of magnitude difference in bandwidth per

edge area. And if your chip is a square or it's twenty-six by thirty-three, right, is the maximum size for a chip, individual die, um, you only have so much edge area. And then on the inside of that chip you put all your compute. Um there's things you can do to try and change, right? More SRAM, more caching, blah, blah, blah. But at the end of the day, you're very constrained by bandwidth.

Sorry for the run on question. I think on on your newsletter you said thirty percent of the capex in twenty twenty six of big tech is going towards memory. That's insane, right? Yeah. Like of the six hundred billion or whatever, you're saying thirty percent is going just to uh just to

a hundred dollar increase in cost on Apple. Also Apple has some margin. They're not just gonna eat the margin. So now that's a hundred dollar cost increase. That's also just on the DRAM. The NAND also has this same sort of like market. So in fact, you know, it's probably a hundred and fifty dollar increase on the on the iPhone.

Apple has to either pass it on to the consumer, A, or B, they have to eat it. I don't see Apple reducing their margin too much. Maybe they eat a little bit. But at the end of the day, that means the end consumer is paying$250 more for an iPhone. Um, and now that's on like, hey, what is last year's memory pricing versus today's? Now there is some lag for Apple to have to feel the heat because they have tended to have, you know, three, six or a year long contracts for a lot of memory.

But at the end of the day, Apple gets hit pretty hard by this. But they won't they won't really adjust until the next iPhone release. But that's the high end of the market. Actually, that's only a few hundred million phones a year, right? Apple sells what, two, three hundred million phones a year. The bulk of the market

Is this mid range, low end, right? Uh used to be one point four million smartphones were sold a year. Now we're at like one point one. Uh but our projections are we maybe get down to like eight hundred million this year and next year are like six hundred or five hundred million because

And and and we look at like, you know, there's some data points out of China from some of our analysts in Asia and Singapore and Hong Kong and Taiwan. They've they've been trekking this and they see Xiaomi and Appo are cutting low-end and mid-range smartphone volumes by half.

Because yes, it's only a hundred and fifty dollar price increase on a thousand dollar smartphone or a hundred and fifty dollar bomb increase on a hundred on a thousand dollar iPhone where Apple has some larger margin. But if we look at the smaller phones, the percentage of the bomb that goes to

memory and storage is much larger and the margins are lower. So there's less capacity to even eat the margins. Um and they have like generally tended not to do as long-term agreements on memory. Um And why this is like a big deal is if smartphone volumes, let's say half The halving will frankly happen in the low and mid range, not in the high end. So it's not like the bits released are halving, right? Um, you know, currently consumers more than half of memory.

Demand, even if you half the smartphone volumes because of the shape of the halving, right? It's like low end gets cut by more than half, high end gets cut by less than half. Because you and I will buy, you know, the high-end phones that cost north of a thousand dollars, we'll buy them, even if they get a little bit more expensive. And Apple's volumes will not go down as much as like a low-end smartphone provider. And and the same applies to PCs. And what this does to the market is.

quite drastic, right? Um DRAM gets released, goes to AI chips who are willing to do longer term contracts, willing to pay higher margins, et cetera, et cetera, because at the end of the day, the margin that they extract is much larger from the end user or whatever. Um And so this this this probably leads to like

People hating AI even more, right? Because they're gonna start being like, today you already see all the memes like uh on on like PC subreddits and PC like Twitter, gaming PC Twitter is like, you know, cat dancing videos and it's like this is why memory prices is doubled and you can't get a new gaming GPU. Right, or you can't get a new uh desktop and and and it's gonna be even worse when memory prices double again, especially D RAM.

Another dynamic that's quite interesting is it's not just DRAM, it's also NAND. Um NAND is also going up in price. Both of these markets have expanded capacity very slowly over the last few years. Um NAND almost zero, but smartphones. Uh the percentage of NAND that goes to phones and PCs is larger than the percentage of DRAM that goes to phone and PC. So as you destroy demand, you unlock

More, you know, mostly for the DRAM purposes, you unlock more NAND that gets allocated and can can sort of go to other markets. And so the price increases of DRAM will be larger than those of NAND. because you've released more from the consumer. Uh in fact you've produced more memory for AI.

uh over the next few years are it's it's it's quite similar to what's preventing us from producing more memory wafers. In fact, like literally the same exact machine. This EUV tool is needed for memory. So I I guess, yeah, maybe uh there's a question that somebody could be asking right now, like, well, why can't we just make more memory?

Three to four years, these vendors have just not built new fabs. Um, that's because memory prices were really low. Um, their margins were low. And in fact, they were losing money in twenty twenty three on memory. Um so they were like, oh, we're not building new fabs. And then like the market slowly recovered over time, but never really got amazing until last year.

Um, you know, in 2024 we were like banging on the drums that like, hey, reasoning means long context, which means large KV cache, which means you need a lot of memory demand. Um, and we've been talking about that for uh like a year and a half, two years. Um and people who understand AI like

went really long memory then, right? Um, you know, and and so you've seen that sort of like dynamic, but now it finally played out in pricing. It took so long for what was obvious, right? Hey, long context, KV cash gets bigger, you need more memory. And accelerators, half their cost is memory. So of course they're just gonna start uh you know, they can they're gonna start like going crazy on it. Um

It took a year for that to actually reflect in memory prices. Once memory prices reflected, then it took another six months, three months for the memory vendors to start building fast. And those fabs take two years to build. And so we don't have really meaningful fabs that you can even put these tools in until twenty late twenty-seven or twenty-eight. Right. Um

And so instead what you've seen is like some really crazy stuff to get capacity, right? Uh Micron bought a fab from a company in Taiwan that makes lagging edge chips, right? Um Heinnix and Samsung are doing, you know, some pretty crazy things to try and expand capacity at their existing fabs, uh, that also have like very not large knock on effects in the economy. And so, hey, why can't we build more

Capacity is like there's nowhere to put the tools, right? And it's not just EUV, there's other tools involved in DRAM and logic, right? Like L logic, you know, N3, 30% or so of the cost, you know, 28% of the cost is UV of the wafer, of the final wafer. When you look at like DRAM,

It's it's it's in the teens. Um and it's going up, but it's in the teens. So it's as much smaller percentage of the cost as DRAM or or is UV. These other tools are also bottlenecks, although their supply chains are not as complex as ASMLs. Um And so you see applied materials in LAM Research and all these other companies also expanding capacity a lot. And anyways, you don't have anywhere to put the tool because the most complex building that people make is FAPS. And Fabs take two years to build.

Even beyond the noise, the nature of the signal changes constantly in reaction to things like pandemics and elections and new regulations and even changes in sentiment. There's this unremitting game of trying to figure out whether your old models still reflect the real world, and if not, what to do about it.

If you're interested in working on this sort of thing, Jane Street is hiring ML researchers and engineers. They're also accepting applications for their summer ML internship program with spots in London, New York, and Hong Kong. And if you happen to find yourself at GTC, which is happening the week after this episode drops, Jane Street's GPU performance team is giving a talk. Go to JaneStreet.com slash SporeCash to learn more.

Uh I interviewed Elon recently and his whole plan is that I guess they're gonna build this Giga Fab, terrafab, uh some power of ten, and they're gonna they're gonna build a clean room. I won't even ask you about the dirty rooms thing, but like let's say they build the clean rooms. Um and fur okay, I have a couple of questions. One

Do you think this is the kind of thing that Elon Co. could build much faster than people are conventionally building it? But this is not about building the end tools. This is just about building the facility itself. How complicated is it to just Build the clean room and do it extremely fast. Is this something that like Elon with his uh move fast thing could do much faster if that's what we're modeling on this year or next year?

And two, um, does that even matter if in two years your view is that we're not bottlenecked on clean room space, but we're bottlenecked on the tooling?

And in the space of fabs, right, clean rooms are the biggest bottleneck this year and next year. Um, and as we get over time, twenty-nine, you know, twenty eight, twenty nine, thirty, there will be still constraints there. The thing about Elon is I think he's had an a tremendous capability to garner physical resources and really smart people to build things.

And the way he's able to recruit really amazing people is just try and build the craziest stuff. Right. In the case of AI, that's not really worked because everyone's trying to build AGI. Everyone's very ambitious. But in the case of like we're gonna make you know, we're gonna go to Mars, we're gonna make li rockets that land themselves, or we're gonna make fully autonomous cars that are electric, right? Um, or we're gonna make human aid robots, right? Like these are methods of recruiting

The people who think that's the most important problem in the world to work on that problem because he's the only one trying really hard. In the case of semiconductors, I want to make a fab that's a million wafers per month. No one has a fab that big. That's what he stated, right? He wants to make a million wafers a month. You know, it's possible that he's able to recruit a lot of really awesome people and get them on this heroically, you know, this crazy task.

of trying to build a fab that does a million wafers per month. Step one is to build the clean room. And I think that he probably can do. Right. I think, you know, there's some mindset, you know, his his mindset around like delete things. It can be dirty. It's fine. Probably not right. Or actually I I think 100% it's not right. You like need the fab to be very clean. I think the entire air the entire all of the air in the fab gets replaced like every three seconds.

It's like that fast and it there's so few particles per but I think he can build the f uh clean room. It'll take a year or two, maybe. Initially it won't be super fast, but then over time we'll get faster and faster at it. But then the really complex part is actually developing the process technology and building wafers. And I don't think he can develop that uh quickly. I think that has a lot of built-up knowledge. It's again like the most complicated like integration of

very expensive tools and supply chain that's done is a T SMC or an Intel or a Samsung. And those some of the these two other two companies aren't even that great and they're like tremendously complex.

I'm sure as an industry insider that sounds like a totally naive question, but do you see what I'm asking? Like is it pa like how What probability should we put on, oh, something totally out of the left field comes out and none of this is relevant?

like seven nanometer or whatever uh wavelengths of light to then use in lithography tools. But those things are like massive partic particle accelerators that are then generating this light. So a very complicated thing to build. So there's a couple companies and I think that that could be a big disruption to the industry.

uh beyond what EU V is. I don't necessarily think that like we're gonna just magically build something new that is like direct right and super simple um and can be manufactured at huge volumes, although there are some attempts to do things like this.

um and you want the pitches to be tighter and all these things. So so generally people are still trying to do a EUV, but what 3D would do is it would take the c you know, hey, a single EU V pass, how many bits can it make, right? If you do this sort of like calculation. And that number would go up drastically if you go to 3D DRAM. That is the hope, but right now everyone's roadmap is sort of like you go from current

Uh it's called a six F squ uh cell to a four F squ uh cell and then finally three D D RAM like by the end of the decade or early next decade. So there's still like a lot of R and D and manufacturing and integration to be done. Um I wouldn't call that out of the cards. I think it's very much likely going to happen. It also is going to require a huge retooling of fabs, right?

The the breakdown of tools in a fab are very different, right? Actually, the lithography tool is the only thing that isn't like that different, but the number of them relative to different types of chemical vapor deposition or atomic layer deposition or uh dry etch or different kinds of etch chambers with different chemistries, all of these things.

You have all these different kinds of tools for different process nodes. You can't just like convert a logic fab to a DRAM fab or vice versa back and forth or a NANFAB to a DRAM fab in in a short amount of time. And in the same way, existing DRAM fabs. require a lot of retooling just to go from one B or one alpha to one b beta to one gamma process nodes because now they have to add DEUV and change the uh chemistry stacks for when you're using EUV in in terms of deposition and X.

And the EV tool has to be there. And furthermore, like when you change to 3D DRAM, there's gonna be an even larger shift. And so there's a lot of retooling of these fabs that needs to happen. um in terms of the tools. And so that would be a big disruption. Um that would make EU V demand generally lower, but as we've seen across time, UV demand as a percentage of wafer cost has trended up initially, or lithography, right? Lithography initially

I wanna say in like twenty fourteen ish era was like sixteen percent of the wafer cost, seventeen percent, and it's gone to thirty over the last, you know, fifteen years. Um and for DRAM it was in the mid teens as well or low teens, and now it's trended towards the high teens. Um And before we get to 3D DRAM, it'll likely cross into the twenties percentage range. But then if we get to three D DRAM, it tanks again in terms of the total end wafer cost as a percentage of UV.

y it it doesn't really matter in some sense what TSMC does. And in fact, even if you have Intel and Samsung building more foundries, i in in the long run you're still gonna be bottlenecked by ASML and other tool makers and other other material makers. So

First, is that correct interpretation? And second, then why should should basically should Silicon Valley people be going to the Netherlands to try to pitch ASM? Like right now, should they be trying to pitch ASML to make more miss tools so that like in twenty thirty they can have more AI computers?

In the sense that like, you know, these turbines are much cheaper than UV tools and there's many more of them produced, right? Especially once you like get to like industrial gas turbines or like, you know, not not just combine cycle, but like the cheaper, smaller, et cetera, less efficient ones. People put down deposits for these. So in a sense, someone could do this, right? Someone should go to the Netherlands and be like, I'll pay you

A billion dollars, you give me the right to purchase 10 EUV tools two years from now. Right. And I I have I'm first in line two years from now. Um and then over those two years, you then go around and wait for everyone to realize, oh crap, I don't have enough EV tools. And then you try and sell your option.

at some premium. But all you're effectively doing is you're saying, ASML, you're dumb. You weren't making enough margin on these. I'm going to make a margin. And the question is like, will then will ASML even agree to this? Right. And I'm like, I don't think so, right?

and their capability to build. So then you arbitrage this by locking up the capacity and then sort of doing like a forward contract and and and then trying to sell it at a later date once other people realize actually shit, everything is fucked and we don't have enough capacity. And then you'll have like this insane margin that ASML and TSMC should have been charging. But the thing is, I don't know if ASML and TSMC will ever agree to this.

And only a fraction of that goes to AI, I assume. So w yeah, if if in twenty thirty we have enough logic and memory to do two hundred gigawatts a year. Is it do do you just think that these things are on a path to ramp up to more than two hundred gigawatts a year, or w what what do you see?

But there's losses along the chain, right? There is loss on the transmission, there's l losses on the conversion, uh, there's losses on cooling, etcetera. And so you should f gross this factor up, you know, from twenty gigawatts for this year or two hundred gigawatts by the end of the decade um to

Some number 20, 30% higher. And then you have capacity factors, right? Turbines don't run at 100%. In fact, like if you look at PGM, uh, which is the largest grid, I think, in America, um, sort of the Midwest sort of northeast kind of area-ish. Not not the full northeast. But anyways, PGM, they rate in in their models for like

Hey, turbines, how much capacity? We want to have excess, you know, roughly twenty percent capacity. Uh, in addition, in that twenty percent excess capacity, we're running all the turbines at ninety percent because they are derated some for reliability. Oh, things go down, maintenance, et cetera, et cetera, et cetera. So then

In reality, the nameplate capacity for uh energy is always way higher than the actual end critical IT capacity because of all of these factors. Um, but it's not just turbines, right? If you were just making power from turbines, like that's simple, boring, easy, right? Um we're, you know, humans and capitalism is far more effective. And so the whole point of that blog was, yes, there's only three people making combined cycle gas turbines.

But there's so much more we can do, right? We can do error derivatives, right? We can take airplane engines and turn them into uh turbines as well. And there's even new entrants in the market, like Boom Supersonic's trying to do that, right? And they're working with Crusoe and also there's all the other ones like that already exist in the market. There's

automobiles manufacturing's going down, these companies all have capacity and could scale and convert that to for data center power, right? Stick all these reciprocating engines. Yes, it's not as clean as combined cycle. Maybe you can you can convert them from diesel to get to gas if you want.

Um, but at the end of the day, these spinning engines, oh, what about ship engines, right? All of these engines for these massive cargo ships, those are great. Nebbius is doing that for a data center in Micro for Microsoft.

You know, and then and then there's solar plus battery, which as these cost curves continue to come down, those can come online. There's wind. And you know, of course the derating of those, you know, hey, when you put on a wind turbine, you might say, Oh, I'm only gonna expect fifteen percent of the maximum power because things like

just oscillate, but yeah, batteries. There's all these things. And then the other thing is that like the grid is scaled for um, you know, hey, we are not gonna cut off power at peak usage, which is like the hottest day in the summer. Um But in reality, that's a load spike that is 10, 15, 20% higher than the average.

Well, if you just put enough utility scale batteries or you put peaker plants that only run a small portion of the year, then all of a sudden, you know, and those could be gas, they could be industrial gas turbines, they could be combine cycle, they could be any of the other

sources of power I mentioned, um, they could be batteries, then all of a sudden you've unlocked 20% of the US grid for data centers because most of the times that capacity is sitting idle and it's really only there for that peak, right? Which is a day or two.

Right. And it's a few hours of like maybe a few a few days of the full year is that peak. And so you just r have enough capacity to absorb that that peak load and all of a sudden you've transferred all. And today data centers only three, four percent of the power of the US grid. And by twenty eighth, they'll be ten percent.

But if you can just unlock twenty percent of the US grid like this, like it's like not that crazy. Um, you know, and and the US grid is terawatt level, not hundreds of gigawatts level. Right. So we we can add a lot more energy. It's not easy. I'm not saying it's easy. These things are gonna be hard. There's a lot of hard engineering. There's a lot of risks that people have to take. There's a lot of new technologies people have to use. But

Elon was the first to do this behind the meter gas. Um, and since then we've seen an explosion of different things that people are doing to get power. And they're not easy, but people are going to be able to do them. And the supply chains are just way more simple than chips.

Okay, the hopper that was a dollar forty is now a dollar fifty in cost. Oh, I don't care because The models are improving so fast that the marginal utility of them is worth way more than that ten cent increase in energy.

Um but we're tracking 16 different vendors and we have all of their orders and things like that. And it turns out there is just hundreds of gigawatts of orders to various data centers. As we get to the end of the decade, we think like something like half of the capacity that's being added will be behind the meter. And when we look at like a lot of this is

Actually behind the meter is almost always more expensive than grid connected, but there's just a lot of problems with getting grid connected and uh you know permits and interconnection cues and all this sort of stuff. So it ended up being even though it's more expensive, people are doing behind the meter and then what they're doing behind the meter with

ranges widely, right? It could be reciprocating engines. It could be ship engines. It could be aeroderives. It could be combined cycle although combined cycle's not that great for behind the meter. Um it could be uh bloom energy fuel cells, it could be solar plus battery, right? Like it could be any of these

And if you think about the US labor force of how many electricians there are, how many construction workers there are. Yeah, I guess there's like eight eight hundred K electricians. Uh I don't know if they're all substitutable in this way. There's m millions of construction workers. But if if we're in a world where we're adding two hundred gigawatts a year, are we gonna be crunched on labor eventually, or do you think that is actually not a real constraint?

Electricity, you know, power power moving across the data center, right? Something like this, right? Um humanoid robots maybe start to or robotics at least start to, but the main factor is going to be for reducing the number of people is modulizing things and making them in factories in Asia. Um Unfortunately, but you know, like at least for America, but

you know, Korea, um, Southeast Asia, in in many ways China as well. But you know, the these these areas are going to do are are going to ship more and more built out sub uh um built out sections of the data center and those will be shipped in, right? Maybe today you, you know, you currently ship servers in or a rack in and then you plug that into, you know, different pieces that you're shipping from different places. But now you'll ship it to a factory and integrate

the entire, you know, hey, maybe this is a two megawatt block. And this block goes from, you know, high voltage power to uh the, you know, the voltage power that you the voltage and uh and maybe DC that you deliver to the raft. um instead of being A C and high voltage, right? Or something like this, right? Or cooling, you take

you you ship a fully integrated thing that has a lot of the cooling subsystems already put together. Or because plumbers are also a bit constraint here. Or furthermore, you take instead of just a single rack, And now you have people wiring up all these racks of power and electricity and blah blah blah blah blah. You take a skid and you put an entire row of ex of of servers and that is shipped from the factories. Um

And and today a single rack may be 120, 140 kilowatts. But as we get to, you know, next generation, you know, NVIDIA Kyber and things like that, it's almost a megawatt. And then in addition, if you do an entire row, it'll have the the rack, it'll have the networking and it'll have the cooling and the power racks all integrated together. So now when you come in, actually you you have much less stuff to cable, whether it be um networking with a fiber, whether it be

Um the power, right? There's viewer power that power things to connect. and then there's fewer plumbing things to connect, right? And so this drastically can reduce the amount of people working in data centers and therefore the capability to build these will be much larger. And along the way there will be, you know, new things mean

You know, some people move faster to new things, some people move slower, right? Crusoe and Google have been talking a lot about this modularization, as has people like Meta and

You know, many others, right, have been talking a lot about this modularization and others are gonna be slower to doing it, but at the end of the day, you know, and and and people who move faster to new things may have more delays or people who are slower have labor problems. So there will always be dislocations in the market because this is a very complex

Supply chain. At the end of the day, it's still simple enough that we will be able to solve it through capitalism and human ingenuity on the time skills that are required.

If you're s right that power is not a constraint on Earth. I guess the r other reason they would make sense is that even though you can phys i there is enough there'll be enough gas turbines or whatever to build it on Earth. I think Elon's next argument then is like you can't get the permitting. to build hundreds of gigawatts on Earth. Do you buy that argument?

Uh and so, you know, Elon, Elon had to deal with a lot of like this complex stuff in Memphis and then building a power plant across the border and all these things for Colossus one and two. But at the end of the day, there's a lot more you can get away with in the middle of Texas, right?

It was a whole host of different constraints. It was probably an area where electricians and things like that were easier to find. But at the end of the day, I'm not exactly sure why they chose that site. I bet Elon would have chosen somewhere in Texas if he could have like gone back. But

Yeah, and because of the regulatory faces he's challenged he's uh challenges he's faced. It's it's it's ultimately like permitting is a challenge, but America is a big place and there are fifty states and things will get done, and there are a lot of small jurisdictions where you can just transport in all of the workers that you need for a temporary period of six months to a year.

um depending on the type of contractor. It can be even three months for depending on the type of the contractor uh that's coming in and put them in temporary housing, pay out the butt because labor is very cheap relative to the GPUs and the power, or not the power, but the GPUs and the like the networking and so on and so forth and the end value of the tokens it's going to produce. So all of these things have plenty of room to like be paid for.

And so I think people are figuring out how to build these things and permitting like I I just like ultimately like permitting in red tape in middle of nowhere Texas or middle of nowhere Wyoming or middle of nowhere like New Mexico is probably a hell of a lot easier than sending

And what we've seen that disperses, right? We have Cluster Max, which rates all the neo clouds and we test them. We test over 40 cloud companies, including the hyperscalers and neo clouds. What differentiates some of these clouds the most outside of software is their ability to deploy and manage failure, right?

GPUs are horrendously unreliable. Uh, even today, 15% of black wells or so that get deployed have to be RMA'd. You have to take them out. You have to, you know, maybe just plug them and plug them back in. But sometimes you have to take them out and ship them to NVIDIA, or rather, there are partners who do these RMAs and such.

Three additional months, probably six. Let's say six additional months, then that is ten percent of your cluster's useful life. And and because we're so capacity constrained, that compute is most valuable.

theoretically in the first six months you have it because we're more constrained now than in the future, because that compute now can contribute to a better model in the future or can contribute to revenue now, which you can use to raise more money to get better, you know, all these all these sorts of things. Now is always the most important moment. And so

You've delayed your compute deployment by six months potentially. And and the thing that separates these clouds is we see clouds that take six months to deploy GPUs today on Earth, right? We see clouds that take a lot less than six months, right? And so the question is, where does space get in there? I don't see how you would test them all on Earth, deconstruct them and ship, ship them and shoot'em into space, and it not take longer than just putting them in the spot that you're testing them.

Um even as we continue to advance into the future. And so then you end up with this problem of Well now you need to tra you know, you need to connect all these satellites together com comms wise as well. Yeah.

are more expensive than the you know, a cluster like twenty percent of the cost or fifteen percent of the cost is networking. All of a sudden now you're making it like space lasers instead of like pretty simple like lasers that are manufactured in millions of volumes with, you know, pluggable transceivers.

And those things are very unreliable as well. More unreliable than the GPUs, by the way. Across the life of a cluster, you have to unplug, clean it all the time, right? Um unplug, replug it just for random reasons. These things are just not as reliable. So you've got that also that problem as well. Like you've got a more expensive, complicated space laser to communicate instead of this pluggable optical transceiver that's been in super high volume.

uh period where we're adding a terawatt a year globally of of various types of power. At some point we do cross the chasm where space data centers make sense, but it's not this decade, right? It is it is much further out once you have Energy constraints actually being a big bottleneck, once you have space, land permitting be a much bigger bottleneck as it subsumes more and more of the economy. Um and and chips are no longer the bottleneck.

because chips are the biggest bottleneck and so you want them deployed working on AI the moment they're done ma being manufactured. And so there's a lot of things people are doing to increase that speed faster and faster, whether it be modulizing data centers or even modulizing racks where you actually put the chip in.

at the fa at the data center, but only the chip and everything else is already wired up and ready to go at the data center. So there's things like this that people are doing to decrease that time that you cannot do in space. And at the end of the day, All that matters in a chip constrained world is get these chips working on to uh producing tokens ASAP in a world

you know, maybe twenty thirty-five, once the semiconductor industry and ASML and Zeiss and all these other suppliers, land research applied materials, fab manufacturers like pendulum swings and they're able to make enough chips.

And really we're optimizing every dial and like it makes sense to optimize the ten percent of energy cost or fifteen percent of energy cost or as we move to A6 potentially. Uh and NVIDIA's margins aren't seventy plus percent, maybe that energy cost is thirty percent of the cluster. um and fab construction, all this like these are the things our data center construction, these are the things to optimize. But that's not a you know, Elon doesn't win by doing

you know, twenty percent gains. Elon never wins that way. Elon wins when he swings for the fences and does 10x gains. Right. That's what SpaceX is about. That's what Tesla was about. That's what all of its success has been about. Right. It's not a been about these chasing the twenty percent. So I think I think space data centers will eventually be a 10X gain, uh, potentially as as Earth. resources get more and more contentious. But that's not this decade.

2x the performance, you may only get tw twenty percent more performance. And that requires much more exotic cooling, right? It requires uh more complicated cold plates and very complicated liquid cooling and or maybe it requires like things like immersion cooling. But in space, higher watts per millimeter is very difficult, whereas on Earth, these are solved problems. And one of these things enables you to get a lot more tokens, maybe it's twenty percent more tokens.

Wait for that's manufactured. And that's a humongous way.

Scale up domain is this like tight domain where the chips are communicating on the order of terabytes a second. And so for NVIDIA, previously this meant an H100 server had eight GPUs and those eight GPUs could talk to each other at terabytes a second. With Blackwell, NVL72, they i implemented RackScale.

scale up. And that meant all 72 GPUs in the rack would connect to could connect to each other at terabytes a second speed. And and the speed double gen on gen, but also the most important innovation they did was going from eight to seventy two in the domain.

When we look at Google, their scale-up domain is completely different, right? It has always been on the order of thousands, right? With TPU V4, they had pods the size of 4,000 chips. With V8, they have pods even at, you know, or V7, they have pods in the 7,000 or sorry, 8,000, 9000 range.

And there is always some blocking of resources when you do that. So because that one TPU is only connected to six other TPUs. And so there's a difference in topology and bandwidth. And there are trade-offs and advantages of both, right? Google gets to have a massive scale-up domain. But then they have the trade-off of you have to bounce across chips to get to from one chip to another. Um, you can only talk to six direct neighbors. And so there is like this trade-off in Amazon.

It has has mutated their scale-up domain. Uh, they're somewhere in between NVIDIA and Google effectively, where they're trying to make larger scale-up domains. Um, they try and do all to all to some extent, which is what with switches, which is what NVIDIA does, but also to some extent they use Taurus topologies like Google does. Um

And as we as we advance forward to next generations, b all all three of them are moving more and more towards o uh a dragonfly topology, which means there's sort of like there is some fully connected elements and there's some elements that are not fully connected. So you can get the scale up to be hundreds or thousands of chips, but also have it not contend for resources when you're bouncing through chips.

Let's say you have a five T model running at FP eight. So that's five uh five trillion gigabytes. Yeah. And then you have the KV cache, let's say it's like Okay, let's say it's the same size for one batch. So you need ten gigabytes, sorry, ten terabytes to be able to run.

Um and in development specifically you can split you split it between pre-training and uh RL, right?

And so when you think about, hey, what exactly is happening? Well, the model, the compute efficiency gains you get from research are so large, you actually want most of your compute to go to research, not to development. Because You know, all these researchers are generating new ideas, trying them out, testing them, and continuing to march along this and push the proto-optimal curve of scaling laws further and further and further.

And at least what we've seen empirically is like model cost gets ten X cheaper every year or even f more than that. Um, which at the same scale gets ten x cheaper or I get, you know, at you know, to get to reach new frontiers it costs the same amount or more, right?

So you don't want to train you don't want to allocate too many resources to pre uh to pre-training and post an RL. You actually wanna allocate most of your resources to research. Um and then in the middle is this sort of this like development period. If you pre-train a five trillion parameter model. Now you have to spend all this time. How many rollouts do you have to do in these RLs?

And these rollouts for a trillion parameter model versus a five trillion parameter model are five times larger, which then means it takes if you wanted to do as many rollouts, maybe the larger model is more sample efficient. Let's say it's two X more sample efficient. Okay, great. Now you need two and a half X much time. of RL to get the model smarter.

Um, or you could RL the smaller model for 2x the time and you'd be it'd be you know you'd still have a 50 percent or you'd still have a 25% difference in the big model, which is two X more sample efficient, uh, and doing X number of rollouts versus the small model, which is a trillion parameters.

doing, although it's less sample efficient, is doing twice as many rollouts, it's still done faster. And so you get the model faster, sooner, and you've done more RL. And then you can take that model to help you build the next models, help your engineers train and do all these research ideas. And so this feedback loop is actually weighed towards smaller models.

in in every case, no matter what your hardware is. And then as you look to Google, Google does deploy the largest production model of any of the th the major labs, right? Uh with Gemini Pro. It is a larger model than GPT uh 5.4. It's a larger model than Opus. And and so you end up with, yes, Google does this because they have a unipolar set of compute, right? All t almost all TPU. Um

Whereas Enthropic is d dealing with H one hundreds, H two hundreds, Blackwell, Traniums, TPUs of various generations, right? And and uh OpenAI is dealing with mostly NVIDIA right now, but going towards uh having a AMD in train as well. The fleets of compute like Google can can just optimize around a larger model and they can leverage a thousand chips in a scale-up domain to get you know the RL time speed much faster.

So that you can actually uh have this feedback loop be fast. But at the end of the day, in isolation, you almost always wanna go with a smaller model that gets RL'd faster and gets deployed into research and development so you can build the next thing and get more compute efficiency wins.

And then this compounding effect of, oh, I made a smaller model that I RL'd more that I then deployed into research and development earlier, and I spent less compute on the training itself because I was able and I was able to allocate more compute to the research. This like compounding effect of being able to do the research faster and faster and faster is potentially a faster takeoff. And that's all these companies want is fastest takeoff possible.

Um, you know, you're explaining you make the uh a semi-analysis sells these spreadsheets and you're always like, uh, six months ago or a year ago you told people the memory crunch, or now you're telling people the clean room crunch and then the in the future the tool crunch. Um Why is Leopold the only person that is using your spreadsheets to make outrageous money? Well w what is everybody else doing?

uh almost ad nauseum. Um, you know, whether it's a hyperscaler saying, Hey, that other hyperscaler, their numbers are too high, you know, and we're like, nah, that's it. And they're like, No, no, no, no, it's impossible, blah, blah, blah. And then you're like, Finally have to convince them through all these facts and data when we're working with hyperscalers or AI labs that in fact no, that number isn't too high. Um, that's correct.

But eventually and eventually like sometimes it's like six months later it takes them to realize or a year later. Um I think I think other clients like on the trading side also use our data, right? We we sell data to a lot of Um yeah, I think roughly sixty percent of my business is industry. So AI labs, data s uh data center companies, hyperscalers, semiconductor companies, uh you know, the the whole supply chain across AI infrastructure.

Um but then like forty percent of our revenue is like hedge funds, right? And l and you know, I'm not gonna comment on who our customers are, but I think a lot of people use the data, it's just How do you interpret it? And then what do you like view as beyond it? And I will say Leopold is pretty much the only person who tells me my numbers are too low always.

Um and sometimes he's too high, sometimes I'm too low, right? Uh but in general, I think other people are, you know, doing that and you can check certain you can you can look across the space at hedge funds and look at their thirteen Fs and see actually they own

Maybe not exactly what Leopold does, uh, because it's always like a question of like, what is the most constrained thing? What's the thing that's gonna be that's most outside of expectations? And that's what you're really trying to exploit as inefficiencies in the market. And in a sense, what our data shows is like is is like making the market more efficient by making the base data uh of what's happening more accurate versus like and but but in a sense, I think many, many funds do trade.

on information um that is out there. And it's not I I don't think I don't think Leopold's the only person. I think he has the most conviction on the entire uh in the entire like about the AGI takeoff though, right?

obvious. KV cache explodes as context lenses get longer. So you need more memory. And then you do the math and you then and you also have to have a lot of supply chain understanding of like what fabs are being built and what data centers are being built and how many chips and all these things. And so we we we we track all these different data data sets like very

Leopold was the only person buying like memory companies. I think there were a lot of people buying memory companies. He of course sized and positioned and did things in a better ways than some. Um, may maybe most, right? I I don't wanna comment on whose returns are what. Um, but you d certainly did well. Um, but other people also did really well, right? Um

You're trying to be like this is wow, you've made me diplomatic for the first time ever. No, no, you're fine. You're fine. I think it's hilarious, right? I'm being a diplomat, you know, whereas usually I'm like spicy. Yeah.

N three is mostly gonna be AI accelerators, but then there's N two, which is mostly Apple now. And then in the future, I guess AI would also want to go on N two. Can Can they kick out Apple if NVIDIA and Amazon and Google say, Hey, we really we're willing to pay a lot of money for N two capacity?

Right. What they're gonna do is when Apple orders X, they may say, Hey, we project you only need Y or X minus one. And so that's what we're gonna give you is X minus one and then that flex capacity Apple's kind of screwed on. Um whereas traditionally Apple's always

overordered by like ten percent and cut back by ten percent over the course of the year. And some years they they hit the entire ten percent, just, you know, volumes vary, right? Based on the season, a macro, blah, blah, blah, blah, blah. Um

But then it it falls dra drastically, right? Just like for N three, they were half um we'll we'll we'll see in and when I say N two, that includes A sixteen, which is a variant of N2. Um Over time, those those nodes will be the majority. And what's also interesting is traditionally Apple's been the first to a process node.

Two nanometer is actually the first time they're not. Well, bef besides Huawei, right? Huawei back in twenty twenty and before was the first with Apple, but they were both making smartphones. Now with two nanometer, uh You've got AMD trying to make a CPU and a GPU chiplet that they used advanced packaging to package together in the same timeframe as Apple.

And this is a big risk for AMD that causes potential delays potentially because it's a brand new process technology. It's hard. But at the end of the day, this is this is a bet that they want to do to, you know, scale faster than NVIDIA and try and beat them. As we move forward, actually, when we move to the A16 node, the first customer there is not even Apple. It's AI.

And as we move forward, that will become more and more prevalent. Not only will Apple not be the first to a node, there will also not be the majority of the volume to the new node.

And then they'll just be like any old customer. And because the scale of TSMC's CapEx keeps ballooning, but Apple's business is kind of not growing at the same pace, they become a less and less relevant customer. And They also will just cut their orders because things in the supply chain are kicking them out, whether it be uh packaging or materials or DRAM or NAND, these things are

increasing in cost. They can't pass on all the cost to customers likely because the consumer is not that strong. And and you end up with like this conundrum where they are just not Apple T S M C's best bud like they have been historically.

like two months before the TPU and like four months before um Nvidia's, I wanna say, was it V one hundred or A one hundred? A one hundred, I think. And so, you know, I mean that's just moving to a process no. That doesn't imply uh software, it doesn't imply hardware design, all these other things. But Huawei is arguably the only

company in the world that has all the legs, right? Huawei has cracked software engineers. Huawei has cracked networking technologies. That's in fact their biggest business historically, right? And they have cracked AI uh talent. Um, but furthermore, beyond NVIDIA, they actually have better AI researchers. And furthermore, beyond NVIDIA, they have their own fabs.

And furthermore, beyond NVIDIA they have, you know, their own, you know, end market of like selling tokens and things like that. And Huawei tends to be like they're able to get the top, top, top talent Nvidia is as well, but not as not as in c and much concentration. And Huawei has a bigger pool in China. It's very arguable that Huawei, if they had TSMC, would be better than NVIDIA. And there are areas where China has advantages outside of um

in areas that NVIDIA can't access as easily, right? Around not just scale, but also like some things around um tech you know, certain optical technologies China's actually really good at. Um so there's there's certain I think it's very reasonable that if Uh in twenty nineteen that ish that that was not that Huawei was not banned from using T SMC.

Huawei would Huawei would had already eclipsed Apple as the biggest TSMC customer, and Huawei has huge share in networking and compute and CPUs and all these things, they would have kept gaining share and they'd likely be TSMC's bigger biggest customer.

Any thoughts on what that implies? What would be required for that?

are determined by a l much more capable model in the cloud that runs at very high batch sizes. And then it pushes those directions to the robots who then interpolate between each subsequent action or is given like Hey, pick up that cup and then the model on the robot can pick up the cup. And it's like as it's picking up, it's like, oh, you know, in fact, this you know, you know, things like weight and all these things might have to be and like sh force may have to be like

Determined by the model on the robot, but not everything needs to be like, you know, hey, pick up the robo uh, you know, this, right? Or like, hey, that's a headphone. Actually, I'm the supermodel in the cloud. I know that this headphones are, you know, Sony XM sixes, which is not a Dorcash ad spot, but you know.

And it may get told by the model in the cloud every second, every 10 times a second, maybe, you know, it depends on the hertz of the action, but a lot of that can be offloaded to the cloud because otherwise if you do all of the processing on the device.

I believe it would be more expensive because you can't batch. Two, you couldn't have as much intelligence as you do in the cloud because the models will just be bigger in the cloud. And three, we're in a semiconductor shortage world and any robot you deploy needs a leading edge chip.

because the power is really bad for robots, right? You need it to be low power and efficient. And all of a sudden you're taking power or and chips that would have been for AI data centers and you're putting them in robots. Yeah. So now that 200 gigawatts gets lower if you're at if you're deploying millions of humanoids.

Your compute like there's eight uh eight billion humans and their compute is on on their heads, on their person. And in a future, even with robots that are out of uh physically in the world, I mean, obviously knowledge work will be done in a centralized way from data centers with huge Like in hundreds of thousands of instances or maybe millions of instances.

But even for robotics, the future you're suggesting is um one where there's like more centralized c uh thinking and centralized computation that's driving, you know, millions of robots out in the world. Um And so I think that's just like, yeah, th th that's uh there's an interesting fact about the future that I think people might not appreciate.

And because of that and the centralization of resources in Taiwan, him having his robot chips in Texas and also being a separate supply chain that is not as constrained by no one's making AI chips really on Samsung besides Uh NVIDIA's new gr uh LPU that they're launched. Uh they're launching it next week, but we're recording it the week before.

And he's not as competing as much with with the like willingness to pay of infinity of the data center of geniuses.

You know, these tools actually use a lot of semiconductors, which are manufactured in Taiwan. So it's like a it's like a you know, a a snake eating its own tail sort of like meme, because you can't make the tools without the chips from Taiwan, which you can't use without the tools in Taiwan. You know, there's there's obviously some diversification there, but

And and they don't use super advanced t uh chips in lithography tools, but at the end of the day there is some tail eating the dragon. Um just shipping out all the engineers and blowing up the fabs means China has a stronger semiconductor supply chain than the rest of the world, right? In terms of verticalization, now that you've removed Taiwan. And Now you've got all the know-how, but you've got to replicate it in, let's say, Arizona um or wherever for TSMC.

And it's going to take a long time to build all the capacity that TSMC has had built over the years. And so you've drum drastically slowed US. and global GDP, not just growth, you've shrunk the GDP massively. And you've got a lot bigger problems uh and your incremental ability to add compute goes to almost zero, right? Instead of hundreds of gigawatts a year by the end of the decade. Let's say by the end of the decade something happens to Taiwan. Now you're at maybe like

ten gigawatts across Intel and Samsung or twenty gigawatts. It's like nothing. Right. Um, and now all of a sudden you've like really cost some crazy dynamics in AI. Uh of course you have all the existing capacity, but that existing capacity pales in comparison to the capacity that's being expanded. Yeah.