[Ben Bajarin]: Hello everyone, welcome to another episode of The Circuit. I am Ben Beharon. [Jay Goldberg]: Hello world, I'm Jay Goldberg. [Ben Bajarin]: We, I had a joke a couple of days ago, Jay and I were talking about where [Ben Bajarin]: I would introduce myself as Jay Goldberg, just to throw everybody off and see [Ben Bajarin]: what the response would be. But then we concluded that might not be a good [Ben Bajarin]: idea.
[Jay Goldberg]: I think you've just accomplished confusing everybody. [Ben Bajarin]: Yes. All right. Ben Beharon is the one talking now. There you go. All right. [Ben Bajarin]: So it's been an interesting week on the back of a couple of semiconductor earnings [Ben Bajarin]: calls, ASML and TSMC. This created a lot of different talking points, questions. [Ben Bajarin]: Something that we're going to dive into today is something we've talked a
[Ben Bajarin]: little bit about before. I wish we had the wherewithal to go even deeper on [Ben Bajarin]: this thesis, but it's something that Jay and I are working on as well. Relative [Ben Bajarin]: to the impact that AI has on a number of different computing segments, not [Ben Bajarin]: just data center, but edge, we've done some specific, like I said, episodes on [Ben Bajarin]: this, but Jay recently wrote a thesis. on this at Digits to Dollars, which
[Ben Bajarin]: we can link to in the show notes. So I'm just going to let Jay outline his thesis [Ben Bajarin]: because I followed some of the people that were questioning him on Twitter [Ben Bajarin]: and seeing the discussion. So lob the thesis out on us and then let's unpack [Ben Bajarin]: it. [Jay Goldberg]: So I approach this topic of AI semis. Obviously, everyone's talking about [Jay Goldberg]: it a lot. We've done a few episodes on it. And I realized that we sort of addressed
[Jay Goldberg]: different parts of the elephant. And so I wanted to take a step back and sort [Jay Goldberg]: of look at the whole thing. And I think from a high level strategic perspective, [Jay Goldberg]: there are really three questions around AI semis today. And the, the first one [Jay Goldberg]: is will AI be additive to the addressable market, the TAM for semiconductors, [Jay Goldberg]: or will it just cannibalize CPUs and other stuff and market stays the same?
[Jay Goldberg]: Second question is how will the market for edge inference, or excuse me, [Jay Goldberg]: for inference in general, shape up? What's that competitive dynamic gonna look [Jay Goldberg]: like? And the last question is can anybody displace NVIDIA, who is clearly [Jay Goldberg]: the market leader and the dominant player in this space right now? And I think [Jay Goldberg]: those are the three sort of fundamental questions that are gonna determine
[Jay Goldberg]: some important trends for the next decade of semiconductor. market and who [Jay Goldberg]: the winners and losers are going to be. [Ben Bajarin]: Okay, I'm gonna start actually with your third point as a starting point. [Ben Bajarin]: So this is something I've been thinking about and I haven't asked you this
[Ben Bajarin]: question but I've been talking to other people. We talked a couple episodes [Ben Bajarin]: ago when we talked about ARM and the data center about everybody sort of, [Ben Bajarin]: everybody who's in the know, unanimously agreeing that much of the workload [Ben Bajarin]: in AI is gonna move from training to inference. And in some cases I've seen folks [Ben Bajarin]: and investors on the sell side, estimate this to be, you know, 77, 78% of the
[Ben Bajarin]: market today in terms of dollars is training. Um, but that could move to a 40, [Ben Bajarin]: 60% split 50 over the next five years. So people are assuming that training [Ben Bajarin]: will still be important. Let's again, let's say it's 50%, 60% great. but a
[Ben Bajarin]: huge portion of that moving dollars moving to inference. So the question is, if [Ben Bajarin]: we believe that, and then perhaps over the 10 year timeframe, even more moves [Ben Bajarin]: to inference, maybe that's a 20% training, 70% inference, does it even matter
[Ben Bajarin]: if people compete with Nvidia in training? Because there's so much other [Ben Bajarin]: opportunity for inference, why would we not be focusing more on that versus [Ben Bajarin]: trying to compete with Nvidia? [Jay Goldberg]: So I think there's a few parts to that. A lot of that is around inference, [Jay Goldberg]: which is kind of my second question. But in terms of Nvidia's position in the market,
[Jay Goldberg]: I think we should probably break down a little bit what dominance means. Today, [Jay Goldberg]: they are clearly dominant in training, close to 100% of the market. The [Jay Goldberg]: only real [Ben Bajarin]: Yep. [Jay Goldberg]: competition they have there are a few internal things at like Google, who [Jay Goldberg]: also is using Nvidia for training as well. So that's part of it. then the question [Jay Goldberg]: is, can they extend their position and training into inference?
[Ben Bajarin]: Mm-hmm. [Jay Goldberg]: And I think we're at a place now where most people assume that AI equals [Jay Goldberg]: GPU [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: and GPU equals Nvidia. [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: We can debate who has a better product and is AMD's MI 300 competitive versus [Ben Bajarin]: Right.
[Jay Goldberg]: whatever. But I think fundamentally, it comes down to the fact that the big customers [Jay Goldberg]: in particular are not going to ever settle for a world in which they are [Jay Goldberg]: completely dependent on Nvidia, especially when Nvidia controls both the silicon [Jay Goldberg]: and the software layer around CUDA. That's just an unacceptable, untenable [Jay Goldberg]: position for the customers. And so no matter what, this current status
[Jay Goldberg]: quo has to change. Now, I'm not bearish on Nvidia by any means. They're [Jay Goldberg]: gonna do fine, but absolutely, they are not going to be able to extend that [Jay Goldberg]: 100% share into inference. There's no way that's gonna happen for a lot of reasons. [Jay Goldberg]: Can they get displaced out of training? I think... I think everybody's exploring [Jay Goldberg]: that right now. And [Ben Bajarin]: All right.
[Jay Goldberg]: I think it's gonna be tough. I think it's actually gonna be pretty tough. [Jay Goldberg]: But if you start to look at some of these alternative software frameworks
[Jay Goldberg]: that are coming out, PyTorch and Triton are on everybody's lips. I think [Jay Goldberg]: there are a lot of people in marketing who just assume, okay, PyTorch is [Jay Goldberg]: coming and it's going to displace CUDA and suddenly we're going to have a little [Jay Goldberg]: bit of evening out in market share and you can run PyTorch on an AMD or [Jay Goldberg]: whatever. And that's going to disrupt Nvidia's hold on training. I actually,
[Jay Goldberg]: I'm not convinced. I'm not convinced. I think [Ben Bajarin]: I [Jay Goldberg]: there [Ben Bajarin]: agree. [Jay Goldberg]: is, there's a world in which you just run PyTorch on CUDA. Right. Uh, [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: they don't necessarily, in some ways they compliment each other. Uh. I think. [Jay Goldberg]: You have the software on training is still very complex. I know a lot of companies [Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: have tried to get into it. It's just, it's going to, it's going to be, it's [Jay Goldberg]: going to be a battle. Um, and I think it's just for a lot of companies, it's [Jay Goldberg]: just not going to be worth it as, especially because like you said, inference [Jay Goldberg]: is going to become. The majority, I would say, I would argue the vast majority [Jay Goldberg]: of spend over [Ben Bajarin]: Yep. [Jay Goldberg]: time. And [Ben Bajarin]: Yeah.
[Jay Goldberg]: so it there's how much effort do you want to spend trying to disrupt training [Jay Goldberg]: when that's, you know, 10% of your spend five years from now, [Ben Bajarin]: Exactly. [Jay Goldberg]: right? You're gonna be much more focused on the inference side and training [Jay Goldberg]: stuff is pretty complicated. [Ben Bajarin]: agreed. [Jay Goldberg]: Unquestionably big companies, Google comes to mind in particular, but others
[Jay Goldberg]: as well, they're gonna find alternatives. They're not gonna use training. [Jay Goldberg]: They're not gonna do all their training on Nvidia long-term probably. But [Jay Goldberg]: I think for the enterprise, like why, you know, if you're an enterprise, [Jay Goldberg]: you're a bank and you wanna run training on your data, Like that's hard
[Jay Goldberg]: enough. You have to find AI people, developers to just do that. And then [Jay Goldberg]: to go the extra mile and like fight the trend and create new software frameworks [Jay Goldberg]: and integrations. Some people will do that, but I think a lot of people just [Jay Goldberg]: find going with the default Nvidia for training is the easiest solution.
[Ben Bajarin]: Yeah. So that was kind of like roundabout where I was getting to, right? Which [Ben Bajarin]: is, I see the allure in competing with them today and trying to compete with [Ben Bajarin]: them. There's at least half a dozen companies that we know of. There's probably [Ben Bajarin]: more attempting this. It's not a trivial problem. PyTorch, you're right, is [Ben Bajarin]: gaining... significant sort of share in terms of academia and research and
[Ben Bajarin]: people who are using it. But again, that's not the only thing here that's [Ben Bajarin]: being used. So it really does feel like there's a still strong reason to [Ben Bajarin]: be bullish on Nvidia for GPU. But what you said is exactly right. One of [Ben Bajarin]: the big questions is, what do they do with Grace Hopper and how do they continue
[Ben Bajarin]: to move their companion CPU part? Um, because if you, if you unpack Jensen's [Ben Bajarin]: vision, and this is where I, this is the thing I think is going to be the [Ben Bajarin]: hardest for most companies to really kind of grasp in their, their strategy [Ben Bajarin]: to compete with Nvidia is Jensen labels this under accelerated computing. And [Ben Bajarin]: in my mind, he does not limit accelerated computing to even the GPU or the
[Ben Bajarin]: CPU there's networking parts. And I think there's other elements of AI Asics [Ben Bajarin]: Nvidia can make. So if they are still. a dominant platform player and a dominant [Ben Bajarin]: seller than of GPUs who compete, who bring companion parts to diversify these [Ben Bajarin]: workloads. It's actually again, a really strong story because his focus is [Ben Bajarin]: just, let's just accelerate computing and any bit of Silicon that can do that. Let's
[Ben Bajarin]: just go make it Nvidia. And I think it's hard because who else is going to [Ben Bajarin]: have that stack? Very few people, maybe AMD, right? Maybe Intel could have all [Ben Bajarin]: those pieces, but Jensen's much more focused. And that's the part I think is [Ben Bajarin]: just gonna be really, really tricky at the end to bite off all those pieces. [Ben Bajarin]: Whether he's successful in those other areas, I don't know, that's debatable.
[Ben Bajarin]: But I think his vision's very clear and his knowledge of the problem is also [Ben Bajarin]: very clear. [Jay Goldberg]: Yeah, so let me break that down. There's a couple of things there. One, today [Jay Goldberg]: when people ask me, like, why is Nvidia so dominant in AI? The sort of default [Jay Goldberg]: knee-jerk answer is to say CUDA. CUDA is a software layer that in between the [Jay Goldberg]: operating system and the chip, and you can get a whole bunch of optimizations
[Jay Goldberg]: to run your systems much better because you have CUDA. And I think the advantage [Jay Goldberg]: that CUDA has conveyed to Nvidia I think we could reasonably argue that is slowly [Jay Goldberg]: being diluted by all the things I talked about before. CUDA was really important [Jay Goldberg]: in the early days of AI for enabling all of this. Its sustainability as [Jay Goldberg]: a durable competitive advantage is probably peaking. That's how I'll put it. [Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: It's very strong, but it's probably at its peak and it's going to wane. So then [Jay Goldberg]: the next question will be, oh, does that mean I can shorten video now? And [Jay Goldberg]: I think the answer is no. setting aside cyclical factors because they're going [Jay Goldberg]: to blow up in a few quarters because they always do but like in terms of [Jay Goldberg]: Secular trend I think Nvidia is still in a really good position and You're
[Jay Goldberg]: right. It's all the reasons you stated they have this whole stack They [Jay Goldberg]: have they have all the pieces it will become very easy to just buy an AI [Jay Goldberg]: system from Nvidia if you can afford it You just get everything from them [Jay Goldberg]: and for a lot of companies that will be very appealing maybe not the hyperscalers [Jay Goldberg]: but maybe them too, but certainly for the enterprise like you just You buy everything,
[Jay Goldberg]: you buy a rack or two of Nvidia solution. Plus on top of that you have all [Jay Goldberg]: their software offerings. Their models which are trained, their software frameworks [Jay Goldberg]: which are trained for specific industries and they have a dozen now. I think [Jay Goldberg]: that's a really compelling vision and I think that speaks to Jensen, like you [Jay Goldberg]: said, really understands this and knows where it's going. He's multiple steps
[Jay Goldberg]: ahead. The one pushback I would give on that though is not everybody's going [Jay Goldberg]: to want that. Because again, it conveys a pretty high degree of lock-in, [Jay Goldberg]: right? If AI is as important as everybody seems to think it is. there's [Jay Goldberg]: a risk in being so dependent on somebody for an entire solution. And historically,
[Jay Goldberg]: those kinds of dependencies don't last. That being said, it will appeal [Jay Goldberg]: to enough people and Nvidia is so far ahead on so many software fronts [Jay Goldberg]: that I think Nvidia will do just fine, even if CUDA goes away. I think [Jay Goldberg]: they're in [Ben Bajarin]: Oh, [Jay Goldberg]: a really [Ben Bajarin]: for sure. [Jay Goldberg]: good position. [Ben Bajarin]: Totally.
[Jay Goldberg]: There's gonna be a lot of friction about people complaining about locking and [Jay Goldberg]: all that, but still there's a lot of strong appeal there. [Ben Bajarin]: I totally agree. I mean, on the merits of just the product quality themselves, [Ben Bajarin]: right, that they're building, it's hard to do what they're doing when it
[Ben Bajarin]: comes to these specific workloads. So by there alone, right, I think my broader [Ben Bajarin]: sort of just view is, one, we're probably a couple years away, or sorry, not [Ben Bajarin]: a couple years away, maybe closer than that, but NVIDIA is much now more quickly [Ben Bajarin]: going to be... a $100 billion company in revenue sooner than we thought, [Ben Bajarin]: thanks to this trend. That increases their capex. That increases their ability to
[Ben Bajarin]: become a priority share at TSMC. I think they're number four priority now or [Ben Bajarin]: so, but increasing in that. It just gives them so much leverage with, again, [Ben Bajarin]: well-execution, good vision, and a large software stack that it's hard to [Ben Bajarin]: be displaced where they are today. is part of my view. But that leads me to this [Ben Bajarin]: sort of broader question you and I have been circling around, which is,
[Ben Bajarin]: there are other areas of growth to the data center that's not GPU. And I think [Ben Bajarin]: we both believe GPU spend of that is going to grow probably faster than [Ben Bajarin]: CPU, but both are relevant. But we've asked this question before about how [Ben Bajarin]: much additional lift to the data center, Tam, is AI going to bring? If it's [Ben Bajarin]: a big number, like I've seen numbers anywhere from, you know, 30 billion
[Ben Bajarin]: to an extra a hundred billion over the next, you know, 10 years. So if it's [Ben Bajarin]: a big number, I kind of feel like if I was somebody trying to increase my [Ben Bajarin]: strategy in the data center, I would want to go after this green field of [Ben Bajarin]: growth versus go for these areas where people are more entrenched. [Jay Goldberg]: Yeah, I think, well, I think this gets into sort of the second question,
[Jay Goldberg]: which is inference, right? I think going after training right now, especially [Jay Goldberg]: for a startup is, um, I don't [Ben Bajarin]: Yes. [Jay Goldberg]: want to say suicidal, let's call it challenging, [Ben Bajarin]: challenging. [Jay Goldberg]: very challenging, but the, the bigger market will be inference. [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: It's going to be a very big market and it just can't be run on GPU entirely. [Jay Goldberg]: Right. The [Ben Bajarin]: Yes.
[Jay Goldberg]: economics will not work out even if, [Ben Bajarin]: Right. [Jay Goldberg]: you know, especially with supply conditions the way they are today. But there [Jay Goldberg]: are other factors too, right? A lot of these AI, I mean, AI is just software. [Jay Goldberg]: And so you're going to run your normal corporate software workload, and you
[Jay Goldberg]: can have some AI functionality in it. There are a lot of times when the software [Jay Goldberg]: architecture is going to dictate that means running the AI on CPU alongside [Jay Goldberg]: the other part of the workload. So AI is not all GPU. Some of it's going to [Jay Goldberg]: be on CPU. A lot of companies are going to build accelerators for this. The [Jay Goldberg]: hyperscalers I think are, you know, are pretty invested in, in accelerators.
[Jay Goldberg]: That's an important category for a lot of them. And they're going to, they're [Jay Goldberg]: going to, that's, they're going to run inference. In the cloud on accelerators, [Jay Goldberg]: because the numbers just work out much better that way. [Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: I haven't quantified this yet. We've been talking about this a lot. But I [Jay Goldberg]: think my intuition is that the market for inference for generative AI is going [Jay Goldberg]: to be the economics are going to be so challenging that the only way it's [Jay Goldberg]: going to work at is if you can push a lot of that inference onto device, onto
[Jay Goldberg]: the edge. Because there and the key thing there is the consumer, the customer [Jay Goldberg]: is paying for the CapEx, they're buying a phone, they're buying a PC that has [Jay Goldberg]: some AI functionality and running it on their device. [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: That's going to offload it. Doesn't have to be run in somebody's cloud. [Jay Goldberg]: And I think that's the only way this really works out given the way that
[Jay Goldberg]: generative AI is taking off. And so, yeah, that's, that's where the opportunity [Jay Goldberg]: is in those, in those areas around inference. [Ben Bajarin]: And I'd add another element that we highlighted last week that I still think [Ben Bajarin]: is just one of the most fascinating things to think about is the other argument [Ben Bajarin]: that why all of this can't continue to be done in the cloud is scarcity of resource [Ben Bajarin]: of energy. And [Jay Goldberg]: Right.
[Ben Bajarin]: so to your point, right, CapEx, it gets offloaded if I'm now using another [Ben Bajarin]: device, my edge device, my car, my PC, my automotive, my camera that's sitting
[Ben Bajarin]: right on a stoplight, um, it's, it's the one handling that power, right? So [Ben Bajarin]: I'm offloading power as a part of that as well, because I think you could [Ben Bajarin]: make the strong argument like we have that we just don't have the grid for [Ben Bajarin]: all of this to be run, you know, in, in the cloud, especially amongst the top [Ben Bajarin]: three hyperscalers, let alone people include in, in broad terms, Apple as
[Ben Bajarin]: a top four U S hyperscaler, you know, they, it's, you just can't run all this [Ben Bajarin]: in the cloud. So I think that's an important reason, which again, goes back [Ben Bajarin]: to your other point, right? A thesis of, of on device, which we did a whole [Ben Bajarin]: segment on. And I think now we've seen a handful of demos and a bit of extra
[Ben Bajarin]: conversations on device. It's definitely not there today. But at some point [Ben Bajarin]: in time, you'll be able to do a whole lot more of this on device, and it [Ben Bajarin]: will feel not that far off from the things you've experienced in cloud-centric [Ben Bajarin]: experiences. [Jay Goldberg]: Yeah. I think, I think that's, that's how this is going to work out. And I [Jay Goldberg]: think that it's just, it's just too cumbersome and the, the workloads are too
[Jay Goldberg]: big to really be run any other way. It has to be some significant portion [Jay Goldberg]: of offload from the cloud.
[Ben Bajarin]: So the question here that sort of plagues me is, it feels like it's gonna [Ben Bajarin]: be very hard to run this hybrid AI environment that a lot of people are talking [Ben Bajarin]: about, meaning that I provide a service, I'm a cloud provider, and I realize [Ben Bajarin]: that I've got tons of capabilities here to do what I wanna do in terms of this
[Ben Bajarin]: service, what people are paying for. But yes, I want to offload that to the [Ben Bajarin]: device, but I need to know how much that device can be offloaded to that device. [Ben Bajarin]: And that's not going to be an all things equal scenario, right? Devices that [Ben Bajarin]: are five years old are going to need a whole lot more cloud help than devices [Ben Bajarin]: that are one year old. But people talk about this hybrid environment. I want
[Ben Bajarin]: to have some of it in the cloud and some of it on device. I feel like that's [Ben Bajarin]: a really tricky architecture. to talk about because again, it feels like [Ben Bajarin]: the service needs to know, well, I can't offload that or it will be a terrible [Ben Bajarin]: experience or it's a capable device, I can't offload 80%. I just don't know [Ben Bajarin]: how this gets worked out but that's kind of how people talk about this hybrid
[Ben Bajarin]: cloud on device today. Could change in three years, but it feels very tricky, [Ben Bajarin]: very complex to me to do that. [Jay Goldberg]: I think you just described Android. [Ben Bajarin]: Well, that's a whole different issue in my brain, [Jay Goldberg]: Yeah, [Ben Bajarin]: but you're 100% right. [Jay Goldberg]: I won't start my Android rant, but I think when you talk about the edge, we're
[Jay Goldberg]: really talking about three things. We're talking about PCs, iPhone, and Android. [Jay Goldberg]: Yes, there's cameras, those will come, and then there's automotive someday [Jay Goldberg]: further out. But for the moment, what we're really concerned about is laptop [Jay Goldberg]: and phone. And at their event, what? Two months ago, AMD actually... started [Jay Goldberg]: talking about that, including some, some neural processing blocks in their
[Jay Goldberg]: seat, in their laptop CPUs. Apple's been doing it for a while. Obviously [Jay Goldberg]: they have some in the phone as well. [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: Right. That's, that's really where this edge inference is going to take [Jay Goldberg]: place is in those kinds of devices. And so what's going to, what's it going to [Jay Goldberg]: take to accomplish that is, uh, Microsoft has to get windows to the point [Jay Goldberg]: where it can do it. [Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: Apple has to, it has to do likewise for both Mac OS and iOS. And I would say, uh, [Jay Goldberg]: Microsoft is clearly fully invested in it and they're, they love generative AI. [Jay Goldberg]: And I think [Ben Bajarin]: Yeah. [Jay Goldberg]: I have to imagine that there are teams inside Microsoft working pretty heavily [Jay Goldberg]: to bring that. transform or to support into Windows sooner rather than
[Jay Goldberg]: later. Apple already done it for the Mac. And so once those frameworks get [Jay Goldberg]: set up, I think it actually can happen pretty quickly. We will have this [Jay Goldberg]: problem with Android where there are going to be a lot of devices that can't [Jay Goldberg]: run it for years. [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: And I think that's just one more thing that has to add that to the list
[Jay Goldberg]: of problems that Google has with Android. Because when Apple thinks it's, [Jay Goldberg]: you know, consumers really want generative AI support on the iPhone, they'll [Jay Goldberg]: launch it, you know, if it's not in this iPhone, it'll be, you know, whenever [Jay Goldberg]: six months after they think consumers are ready for it, right? [Ben Bajarin]: Yeah. Well, and [Jay Goldberg]: So [Ben Bajarin]: it's.
[Jay Goldberg]: I would be surprised if they don't talk about something along these lines [Jay Goldberg]: in September with the new iPhone. [Ben Bajarin]: Sure. So there was a report, I'm sure everybody who listens to this saw [Ben Bajarin]: that Apple, Mark Gurman wrote it at Bloomberg and just saying that they are [Ben Bajarin]: working on their own GPT model. It sounds like it's a pretty large model in
[Ben Bajarin]: terms of overall size, like definitely large enough that whatever they're. building [Ben Bajarin]: data set wise is not going to run on device at that size. I mean, roughly [Ben Bajarin]: the stuff we've tried today seems to be a successful if it's in the 10 to 15 [Ben Bajarin]: billion parameter range. Anything north of that is crashing devices and running [Ben Bajarin]: exceptionally slow. But that's today, right? That's not where we'll be in
[Ben Bajarin]: two to three years. But it shows kind of what's possible on device versus the [Ben Bajarin]: size of cloud. But Apple, 100%. right, is going to want to do this. And arguably,
[Ben Bajarin]: like I said, they are a top four hyperscaler. They can create that cloud to [Ben Bajarin]: device infrastructure as good as anybody, if they want, um, to handle their, [Ben Bajarin]: their device fragmentation, which as you point out is not as, not nearly as, [Ben Bajarin]: as tricky as Google's.
[Jay Goldberg]: Yeah, Apple technically is capable of doing it today. There's no question. If [Jay Goldberg]: they wanted to get generative AI working on the iPhone, it could happen today. [Jay Goldberg]: I think the question for Apple is, or the question that Apple is asking is [Jay Goldberg]: more, what are we going to use this for? Apple doesn't like to add features [Jay Goldberg]: that consumers don't care about. [Ben Bajarin]: Right.
[Jay Goldberg]: How much do consumers really care about generative AI? How important is [Jay Goldberg]: it to the user experience, the human experience, excuse me, [Ben Bajarin]: Right. [Jay Goldberg]: on an iPhone? And there are a lot of people who think that Apple is behind [Jay Goldberg]: in AI as well, because they don't have generative AI today. I think they're, [Jay Goldberg]: they're trying to figure out as am I, and a lot of people, like what is
[Jay Goldberg]: actually really useful for. And I actually had a Twitter debate with somebody [Jay Goldberg]: who was saying, you know, AI is going to be really important, everyone's [Jay Goldberg]: going to want it. And I said, well, how much more would you pay for a phone [Jay Goldberg]: that does [Ben Bajarin]: Sure. [Jay Goldberg]: chat GPT or stable diffusion on the phone? And he says, well, I already
[Jay Goldberg]: paid chat GPT $20 a month. And I'm like, yeah, I understand that. But that's [Jay Goldberg]: not the question. You like generative AI, you're willing to pay for it. How much [Jay Goldberg]: though are you willing to pay for it to work on your phone in airplane mode? [Jay Goldberg]: And I don't think anybody knows the answer to that question yet. [Ben Bajarin]: So there's another though, part of this that feeds into this that I think is
[Ben Bajarin]: fascinating. And this again speaks back to companies who are prioritizing on device, [Ben Bajarin]: those who are not be playing both sides of cloud to device in terms of their [Ben Bajarin]: roadmap, is you exactly rightly point out, I don't think anybody is gonna pay [Ben Bajarin]: more for these things. I think it's gonna have to just be an evolution of
[Ben Bajarin]: the silicon's... roadmap to increase those features. But if I'm a silicon designer [Ben Bajarin]: today, so let's just say this is MediaTek, Qualcomm, and Apple, to some degree [Ben Bajarin]: Intel and AMD on PCs, you do have to sort of make some decision about how [Ben Bajarin]: much transistor budget you're gonna throw to something like an NPU, because [Ben Bajarin]: it's a relevant decision, right? If I believe that I need that to compete, I
[Ben Bajarin]: gotta take that from something else, right? I gotta take it from my GPU blocks. [Ben Bajarin]: my CPU blocks, right, something else, right? So they need to make calls on [Ben Bajarin]: how important that is by how much diary are they're gonna commit to these
[Ben Bajarin]: things going forward. And that's a fascinating trade-off, right, that I think [Ben Bajarin]: people are stuck with these next two years, when again, you and I can argue [Ben Bajarin]: all day, people will 100% be willing to pay for those silicon bits in the
[Ben Bajarin]: data center more. They're not gonna pay for them on device. And so that's [Ben Bajarin]: just a fascinating dynamic of maybe how fast the capabilities develop at [Ben Bajarin]: the edge, when again, you've got to make these trade-offs with your transistor [Ben Bajarin]: budget.
[Jay Goldberg]: Yeah, I think that's a really good way of framing it. And I think that sort [Jay Goldberg]: of throws light on what I was saying about Apple is, Apple at this point [Jay Goldberg]: doesn't seem to see the need to throw those blocks, those resources at it. [Jay Goldberg]: They just don't see it as necessary. [Jay Goldberg]: And I don't think anyone knows when that will change. [Ben Bajarin]: Well, we'll see. I mean, I think we'll see with two fundamental things in my
[Ben Bajarin]: opinion, that'll happen this fall, right? Qualcomm is going to unveil their [Ben Bajarin]: new PC chip, PC chip architecture. My hunch is that's going to be loaded with [Ben Bajarin]: tops. Um, Apple has historically only bumped their tops up. If I recall two [Ben Bajarin]: to three, you know, uh, I think they went from 10 to 15. Right. So maybe five [Ben Bajarin]: or so. So, so what they throw at tops in next generation products will be
[Ben Bajarin]: telling. Right. If it's goes from 15 to 30, that's a pretty big jump. Right. [Ben Bajarin]: But if it goes from 15 to eight to 18, you know, you'll, you'll again, you'll [Ben Bajarin]: just sort of see how they're prioritizing. So, um, that's an interesting [Ben Bajarin]: bit, but, but you're right. I think that's, that's really the call they've [Ben Bajarin]: got to make is, is how much C block they throw to this stuff and it, and [Ben Bajarin]: could it be premature, you know?
[Jay Goldberg]: Yeah, Apple, I think, typically tends to be a little bit conservative in jumping [Jay Goldberg]: on the latest feature, right? The original iPhone was a 2G phone deep into [Jay Goldberg]: the 3G era. That's the classic example. And they've had AI on their phone [Jay Goldberg]: longer than anybody, right? They don't call it that. It's a neural engine.
[Jay Goldberg]: And it's there for a very specific purpose to help with image processing. I don't, [Jay Goldberg]: yeah, it would be interesting to see it to what degree they change it, right? [Jay Goldberg]: Or do they just sort of keep going at their steady incremental pace? [Ben Bajarin]: Yeah. So I want to end on just circling back to the question at hand about [Ben Bajarin]: is there even a way to sort of put a model around the additional lift of dollars
[Ben Bajarin]: that's coming to AI? I think we agree we can't do it on device. But to that [Ben Bajarin]: point, in a number of earnings calls, as well as some commentary of some of [Ben Bajarin]: the people who are doing this, it appears that it The answer is others haven't
[Ben Bajarin]: done this work either. That these models actually haven't been built or at least [Ben Bajarin]: in a way that they're confident they will portray it to customers and or investors [Ben Bajarin]: that they think these custom AI parts or their accelerator bits or whatever [Ben Bajarin]: could lead to this much more revenue. Like in terms of, I guess their model [Ben Bajarin]: guide, it seems like everyone kind of believes like we do. There's something
[Ben Bajarin]: there. There will be some additional tam in the data center, but no one's really [Ben Bajarin]: done that work yet is basically what I'm saying. So as an answer to the question, [Ben Bajarin]: I'm floating the no yes, it could be, but no, that hasn't worked been done. It's [Ben Bajarin]: very vague speaking in vagueness. [Jay Goldberg]: So I will start by saying, the answering the first part of your question,
[Jay Goldberg]: which is, yes, somebody can build this model. I don't think anybody has [Jay Goldberg]: yet. You and I have toyed around with this. I think our model is as advanced [Jay Goldberg]: as pretty much anybody's. And it's an important, important area. I tend to [Jay Goldberg]: think, and I've been debating this a lot with you and with other people and [Jay Goldberg]: myself lately. I'm coming down on the side that AI is additive to the semiconductor [Jay Goldberg]: TAM.
[Ben Bajarin]: Mm-hmm, agree. [Jay Goldberg]: And we're recording this on Friday, July 21st, TSMC reported last night. [Jay Goldberg]: And they made some comments about seeing AI servers cannibalizing CPU servers [Jay Goldberg]: in their, in the data they track. But, but what they're really saying is what, [Jay Goldberg]: what they, what they really saying is because hyperscaler capex budgets [Jay Goldberg]: for new data centers are fixed. We haven't gotten into the new budgeting cycle.
[Jay Goldberg]: that's going to accommodate this increase in [Ben Bajarin]: Mm-hmm. [Jay Goldberg]: AI needs. So that's a very short time window because they say long-term AI is [Jay Goldberg]: going to be huge in the data center. I will say anecdotally, from what I [Jay Goldberg]: can tell, I've heard a lot of the big hyperscalers are accelerating, pulling [Jay Goldberg]: forward their data center physical plant expansion because they need AI, they
[Jay Goldberg]: need it soon, right? So I was talking to somebody recently who owns a plot [Jay Goldberg]: of land, it's sort of a... tier two or tier three data center location. [Ben Bajarin]: Hmm. [Jay Goldberg]: Critically, it has power, it has electricity and they're in a region where [Jay Goldberg]: the main tier one location is out of power. So they're on a grid, they have
[Jay Goldberg]: electricity and they just won the lottery. They've been sitting on this piece [Jay Goldberg]: of land for years and now suddenly they have customers, they have all the hyperscalers, [Jay Goldberg]: all those usual suspects plus many more knocking on the door saying, hey, [Jay Goldberg]: can I get in? Let's get going, I need this, right? And... it's, you know,
[Jay Goldberg]: they have to build a plant, they have to build the building. So it's not [Jay Goldberg]: going to come in, you know, next three months, but [Ben Bajarin]: Right. [Jay Goldberg]: it is clearly to me, additive to whatever the hyperscalers are doing, they're
[Jay Goldberg]: adding data centers that they know they're adding to the plan. And so it's [Jay Goldberg]: not going to happen this quarter, but over the, I would say over the next year, [Jay Goldberg]: we're going to see this spike.
[Ben Bajarin]: Yeah. Nope. I agree. I'm, I'm aligned with that. I think what questions I've [Ben Bajarin]: heard, which again, I'm it's fine that nobody knows this, but just in terms [Ben Bajarin]: of, of people being aware of the questions is really just how, how much [Ben Bajarin]: additional capex could be thrown to this. Like there's a reasonable amount, [Ben Bajarin]: again, knowing that we're up against limitations of physical space, we can't
[Ben Bajarin]: get enough wafers to meet that demand. So yes, it will grow, but it's not [Ben Bajarin]: going to go. It's not going to be a hundred percent. right, growth year over [Ben Bajarin]: year. So I think the understanding the amount at which it can grow and then where [Ben Bajarin]: are those pockets that might get spent quickest I think are at least helpful [Ben Bajarin]: if you're trying to come up with where might these dollars go over the next
[Ben Bajarin]: year or two, knowing where the constraints are. And I think that's a better [Ben Bajarin]: way to kind of. look at this question about how much, because again, I go [Ben Bajarin]: back to this, people are throwing these astronomical numbers out over the next [Ben Bajarin]: five years. And I keep asking, well, do we even have the land and the grid?
[Ben Bajarin]: Can they even build out fast enough to meet that revenue number? But there's [Ben Bajarin]: some reasonable amount of growth that's going to come from AI to these data [Ben Bajarin]: centers. And so I think if the CapEx goes up, I don't know, I'm just making [Ben Bajarin]: this up, but let's just say we landed on it's 5% to 8% a year in terms of
[Ben Bajarin]: your flexible budgets. it's a helpful way to look at where that growth can [Ben Bajarin]: come from, being, again, additive to a number where they were already spending. [Jay Goldberg]: Yeah, I agree. I agree. We need a little bit more work to put a precise number [Jay Goldberg]: on it, but we're getting closer. And [Ben Bajarin]: Yeah. [Jay Goldberg]: it's going to be, I think, a meaningful amount for a lot of companies.
[Ben Bajarin]: Yes, no, I agree. Um, I will be at an event next week with one of the top [Ben Bajarin]: three hyperscalers and, uh, we'll have a chance to talk to many of their customers. [Ben Bajarin]: So we can, we can talk about that maybe on the next episode or episode after [Ben Bajarin]: that. When, uh, when I can share more of what I've learned, but this is top [Ben Bajarin]: of mine. This will be amongst my top questions in, uh, in, in CapEx spend
[Ben Bajarin]: for AI specific stuff, so more on that later then. Um, all right. Well, thanks, [Ben Bajarin]: uh, everybody for listening. Review our podcast, give us likes, tell your [Ben Bajarin]: friends, share us on socials, et cetera. Uh, we appreciate everybody listening. [Jay Goldberg]: Thank you for listening everybody.