Support for Decoder comes from Stripe. Stripe is a payments and billing platform supporting millions of businesses around the world, including companies like Uber, BMW, and DoorDash. Stripe has helped countless startups and established companies alike reach their growth targets, make progress on their missions, and reach more customers globally. The platform offers a suite of specialized features and tools to fast track growth, like Stripe Billing.
which makes it easy to handle subscription-based changes, invoicing, and all recurring revenue management needs. You can learn how Stripe helps companies of all sizes make progress at Stripe.com. That's Stripe.com to learn more. Stripe. Make progress. Support for Decoder comes from Huntress.
Huntress is one of today's fastest growing cybersecurity companies. Its platform was designed from the ground up to work for small to mid-sized businesses, and it promises enterprise-grade security driven by technology, services, and expertise. seated to defend against today's cyber threats, all at a price that makes sense.
Today, even the least sophisticated hackers can still do a ton of damage to your small business. And that's why Huntress built a fully managed, highly sophisticated security platform for its customers to guard against potentially devastating threats. Plus, you can rest assured that the elite human threat hunters running their 24-7 security operations center will offer real protection all day, every day.
So, if you want cutting-edge cybersecurity backed by experts who monitor, investigate, and respond to threats with unmatched precision, you can visit Huntress.com to learn more and start your free trial. Support for the show comes from Alex Partners. If you're in the tech industry, wondering how artificial intelligence is going to affect your business might seem like the new normal by now.
Alex Partners is a global consulting firm dedicated to helping you navigate the changing headwinds of AI without getting lost in the noise. Learn how your business can navigate AI while making sure your strategic initiatives are aligned by reading Alex. www.alexpartners.com In the face of disruption, businesses trust Alex Partners to get straight to the point and deliver results when it really matters.
Hello and welcome to Decoder. I'm Nilay Patel, Editor-in-Chief of The Verge, and Decoder is my show about big ideas and other problems. Today I'm talking with Mustafa Suleiman, the CEO of Microsoft AI. He was one of the co-founders of DeepMind, which got acquired by Google in 2014 and is now known as Google DeepMind, the center of that company's AI efforts. He was a Google VP for several years before leaving to found another AI startup called Inflection in 2014.
Then earlier this year, Inflection cut a deal with Microsoft to license its core technology in a weird and kind of controversial not-quite-acquisition that sent Mustafa, his co-founder, and a majority of their employees into Microsoft. made Mustafa CEO of Microsoft AI.
As CEO of Microsoft AI, Mustafa now oversees all of Microsoft's consumer AI products, including the Copilot app, Bing, and even the Edge browser and MSN, two core components of a lot of people's web experience that feel like they're radically changing in the world. with AI. That's a lot, and it's a lot of decoder bait. I am continually fascinated by Microsoft's org chart and all of the little CEOs that report up to Satya Nadella.
And of course, I'm also obsessed with what AI might do to the web at large. I also took the opportunity to ask Mustafa to compare and contrast working at Microsoft and Google, since he has direct experience at both. And I would say his answer is pretty revealing.
It was also important to me to ask Gustavo about AI training and the data it requires. He's caught some heat for describing content on the web as, quote, freeware before, and Microsoft and its partner OpenAI are in the middle of major copyright lawsuits about training data. I'm generally curious about how AI companies are thinking about the risky and seemingly uncertain legal foundations of their work, and I wanted to know how Mustafa was thinking about it now.
But before we got into all of that, I needed to ask about AGI, or Artificial General Intelligence. That's the idea that these AI systems will be able to handle tasks as well as a human, or even better in some cases. Sam Altman in OpenAI, which, again, is a huge partner with Microsoft, has said he thinks AGI is achievable on our current computing hardware.
and his most recent comments at the Dealbook conference seem to lower the bar for how he defines AGI entirely, which makes it easier to argue that it will arrive sooner than we think. On top of that, there's a lot of reporting that says OpenAI can get out of its Microsoft deal when it says it's achieved AGI. So Altman has a lot of incentives to say it's happening.
So I asked Mustafa straight up if he agrees with Sam Altman and if AGI is achievable on current hardware. Because if the answer is yes, maybe a bunch of word chart questions are a little secondary to what should humanity do to deal with AGI? You'll hear Mustafa be optimistic about achieving AGI in a much more diffuse time frame than Sam Altman, but you will also hear him pull away from the idea of AGI being a superintelligence, which feels like that same kind of redefinition.
There's a lot going on in this episode, including a discussion of what I have started calling the DoorDash problem. We'll see what I mean. Okay, Microsoft AI CEO Mustafa Suleiman. Here we go. Mustafa Silliman, you are the CEO of Microsoft AI. Welcome to Decoder. Nilay, great to be with you.
Yeah, I'm very excited to talk to you. I have a lot of questions for you about how Microsoft AI is structured within Microsoft, about what it means to be the CEO of Microsoft AI at a company that appears to be all about AI lately.
about how you make decisions, all the decoder stuff. But I want to start hot out the gate. I hope you're ready for this because I realized that the answer to this question from you is if you answer one way, this whole interview goes a different way. So I'm just going to ask. Right at the gate. Very recently, Sam Altman said in a Reddit AMA that he thinks we can achieve AGI on current hardware. Do you think that's possible? What does current hardware mean?
Within one or two generations of what we have now, I would say. I don't think it can be done on GV200s. I do think it is going to be plausible at some point in the next two to five generations. And I don't want to say, I think it's high probability that it's two years, but I think...
Within the next five to seven years, which is, you know, each generation takes 18 to 24 months now. So five generations could well be up to even 10 years, depending on how things go. And we really are facing increasingly tough challenges with these chips. I don't think it's going to be as linear in terms of its progress cost per dollar and so on, as we've seen in the past. But things are accelerating very fast. So I agree with that sentiment.
So between two and 10 years, you think? I think that's the right. The uncertainty around this is so high that any categorical declarations just feel sort of ungrounded to me and over the top. Yeah. You and I have spoken several times in the past about a lot of things, and I want to follow up on a lot of those ideas. But it just occurs to me that if we think between two and ten years, very much in the spans of our lifetimes, there will be AGI.
Maybe we shouldn't be working on anything else. Like that seems like it will complete. That's the paradigm shift, right? We're through the singularity. Now there's AGI. Everything will be different on the other end of it. How do you approach that? And then also think about, well, I need to launch the Copilot app on the iPhone.
Well, so that depends on your definition of AGI, right? AGI isn't the singularity. The singularity is an exponentially recursive self-improving system that very rapidly accelerates far beyond. anything that might look like human intelligence. To me, AGI is a general purpose learning system that can perform well across all human level training environments. So knowledge work.
By the way, that includes physical labor, the definition. So a lot of my skepticism has to do with the progress and the complexity of getting things done in robotics. But yes, I can well imagine that we have a system that... can learn without a great deal of handcrafted prior prompting to perform well in a very wide range of environments and i think that is
not necessarily going to be AGI, nor does that lead to the singularity. But it means that most human knowledge work in the next five to 10 years could likely be performed by... one of the AI systems that we develop. And I think the reason why I shy away from the language around singularity, artificial superintelligence, I think they're very different things. And the challenge with AGI is that it's become so dramatized that...
we sort of end up not focusing on the specific capabilities of what the system can do. And that's what I care about with respect to building AI companions. getting them to be useful to you as a human, work for you as a human, be on your side, in your corner, on your team. Like that's my motivation. And that's what I have control and influence over is to try and create systems that are accountable and useful to humans.
rather than pursuing the purely kind of theoretical superintelligence quest. Yeah, one of the reasons I'm particularly curious about this is the notion that... All human knowledge work can be performed either with the assistance of a very capable general AI or by the AI itself. It sort of implies that we will build a new kind of AI system, one that will be able to be as creative as a human knowledge worker at the 99th percentile. And I don't see that in our systems now. The way an LLM works.
They don't necessarily come up with a bunch of individually creative thoughts. You can prompt them to do surprising things. But that turn, I have not experienced it. Do you think that the way that the current LLMs are built and trained and deployed?
Is it a linear path to the kind of AGI you're describing, or is there another kind of thing we need to build? It's funny because two or three years ago, people would often say, well, these systems are destined to regurgitate the training data that they were trained on. and that there is some one-to-one mapping between query training data and output. And it's pretty clear today that they're actually not doing that. The interpolation...
of the space between multiple n-dimensional elements of their training data is in itself the creative process, right? It's picking some point in this massively complex space to produce or generate a novel form. of the response to the question that it has never seen before. We've never seen that specific answer produced in that specific way. To me, that is the beginnings of creativity, is the kind of glimmer of a truly novel invention.
which is obviously what we're trying to produce here, right? Just to level set here, intelligence is the very thing that has driven all of our progress in the world in history. There's power to synthesize vast amounts of information. aggregate it into conceptual representations that help us reason more efficiently in complex space, make predictions about how the world is likely to unfold, and then take actions on the basis of those predictions.
making a table or you're playing baseball with your friend Every single one of those environments that you experience has those characteristics. And so if we can distill those sort of moments, if you like, into an algorithmic construct, then of course... there is huge value there. And what I think we see in this little mini moment in the last three or four years...
is the glimmers that they really can be creative, that they can exert real judgment, that they can produce novel ideas, these LLMs. And your point about can they do that proactively? I think is a good one. Like, can they do that unprompted? Can they do it independently? Can they do it with very subtle or nuanced or lightweight guidance? I think that's kind of an open question. I feel very optimistic about that myself.
Much of the infrastructure to sort of ensure that they can do that is kind of an engineering issue now. So stateful memory and meta reasoning about the current... context of a model are things that we know how to do in software today. We know how to introduce a second or a third system to observe the working state of an LLM in its activity.
and use that to then steer or re-steer a prompt that it is operating to. And if you can do that kind of asynchronous meta reasoning, which is what the initial... quote unquote, chain of thought methods seem to show in the last sort of six to 12 months or so, then you can imagine that it could string together actions in these continuous environments.
and then orchestrate and coordinate with other parts of its working memory, other parts of its system, some of which are designed to do more short-term things, some to draw from long-term memory, some to be a bit more creative, some to be more adherent to the...
behavior policy or the safety policy that you're designing to. So it's obviously not done and dusted, but there are very, very clear signs that we're on the right path, I think. Those orchestration systems are fascinating to me because the models themselves... are never going to produce the same output twice. A lot of the things we want computers to do are insanely deterministic. We definitely want them to do the same thing twice over and over and over again in a variety of situations where.
an AI might be really helpful, right? If you want to just do tax preparation, like you want the AI to be very helpful and understand all the inputs, but you also want it to follow the rules 100% of the time. And it seems like. Connecting our more traditional logical computer systems to control the non-deterministic AI systems is a big pathway here, more so than...
making the AI more capable. And that feels like a new way of talking about it that I've only just recently seen. Does that feel like the kinds of products you need to build? Are you still focused on the capability of the model itself? It's a good framing, but let's tease apart what you mean by determinism, right? So determinism operates at layers of abstraction. So at the very lowest layer, each token is being generated non-deterministically.
But as those outputs become more recognizable with respect to a behavior policy or a heuristic or a known objective, like in your case, filling out the tax form. then that knowledge can be stored in representations which are more stable and deterministic, right? And this is exactly how humans operate today. No matter...
how well you might memorize something. If I ask you to do it a hundred times over, you're most likely going to have some variation in the output. We don't really store things deterministically. We have... co-occurring conceptual representations, which are quite fluid and abstract, which we then reproduce and fit into a schema of words and language in order for us to be able to communicate with one another.
These models are actually very similar to that architecture. They can store stable information that can then be retrieved in quite deterministic ways and, like you said, integrate with existing computer systems and knowledge bases. But it's not true to say that one approach is going to sort of trump another. The models are going to get way more capable
And the methods for retrieval and information access and the use of existing databases or making function calls to third-party APIs to integrate that information, those are going to advance simultaneously. And by the way, we're going to open up a third front.
like a whole new front of capability, which is that now these LLMs can speak natural language, they're going to be able to go and query in real time other humans and other AIs. So that's like a third... paradigm for quote-unquote retrieving information or verifying that information or accessing new knowledge or checking state on something so That in itself is going to drive like huge gains in addition to straight up model capabilities and integration with existing systems.
I want to talk about the agent component of that at length, because that seems to be where so many companies are focused, including to some extent Microsoft. And it raises a million questions about how our computer systems and our networks should work. But now that I've just level set on... We think we're headed towards AGI between two and 10 years. We think we can do it with an increase in model capability, but also some novel approaches to how we use those models.
I want to talk about how you're actually doing it at Microsoft. But it occurred to me from the jump, if we didn't agree on what the goals were, the structure conversation would be kind of ungrounded from reality. So those are the goals. Those are huge goals. At Microsoft AI, how are you structured to accomplish those goals? First and foremost, my organization is focused on the consumer AI part. So it is about Bing, Edge, MSN.
and co-pilot so for consumer facing products that have many many hundreds of millions of dow lots of user data and lots of direct commercial surfaces where we can deploy into production get feedback, drive large scale experimentation. So for me, that's mission critical because five years ago, we were in a state with LLMs and AI more generally. where we were still relying on the benchmarks to drive progress. Everything was happening. Evaluation was taking place in basically academic environments.
albeit in commercial engineering labs, the models weren't good enough to actually put them into production and collect feedback from the real world. That has completely shifted now where... All of the innovation is happening by optimization and hill climbing in production. So I think that's the first thing to say. Second thing to say is obviously our Azure business and the immense number of customers that we have that are using.
M365 Copilot every day provides another huge experimentation framework, which is very different to the consumer experimentation framework. And it's actually a great opportunity for me because I'm learning a lot from how many, many businesses are integrating. true AI agents in their workflow today. Because they have more visibility and control of their internal data, and in many cases they have tens or even hundreds of thousands of employees, they're able to introduce novel...
co-pilots into their workflows, be it for training sales agents, upskilling, underperforming sales agents, providing marketing feedback. I've seen HR co-pilots. There's all kinds of customers. service copilots happening. So that gives me a sort of window into all the different flavors of testing and pushing the limits of these AI models in.
third-party production environments in the enterprise context. The third arena, of course, is our collaboration with OpenAI, our great partners. I think this is going to turn out to be one of the most successful partnerships in computer history.
five years old that partnership now and many many years to run and we get models from them we get ip and they get compute and obviously funding and it's obviously a huge source of support for us And then the fourth area is that we've just spawned, since I arrived eight or nine months ago now, our own core effort to develop these models at scale. inside of Microsoft AI. And so we have some of the best AI researchers and scientists who are pushing the frontier of post-training and pre-training.
for our weight class which is you know so we're choosing a flops match target that really suits the kind of use cases that we care about and making sure that we have absolutely world-class frontier models that can do that Let me just unpack some of the vocabulary there. You said weight class. Does that just mean giant corporation or do you mean something more specific by weight class?
oh sorry weight class is the way that we refer to comparing frontier models with one another so obviously your flops need to be matched to your sort of your competitor model that you're evaluating yourself against so size is really the by far the overriding predictor of capability performance in these models. So you can't compare yourself to something that's 10x larger by flops. You sort of have to treat them as weight classes or flops classes, if you like.
Yeah, that makes sense to me. And then you said you want to target it towards the applications you're using, right? So you're making many models that are geared towards specific Microsoft products? That's right. So if you think about it, Copilot under the hood is a whole collection of different models, different sizes that adapt to different contexts. If you're in a speech setting, it's a different type of model. If you're on desktop.
if you're actually in the native apps on Mac or on Windows, they're all slightly different models. And then there's different models for search, reasoning, safety. And I think that that is going to get even more heterogeneous as we go. And then I just want to be very clear about this. It sounds like you're developing a frontier model that can compete with Gemini, with GPT-4 or 5, whatever it is. Are you working on that as well?
For the current weight class, yes. So at the GPT-4, GPT-4-0 scale. But it depends on how things turn out over the next few years, because each order of magnitude increase... is really a phenomenal piece of physical infrastructure. You're talking about hundreds of megawatts, soon gigawatts of capacity. There will really only be three or four labs in the world that have the resources to be able to train at that scale.
by the time that we get 10 to the 27 flops for a single training run. So we won't duplicate that between us and OpenAI. OpenAI is our pre-training frontier model partner for those things. And hopefully that continues for a long time to come. So you're not going to compete with the next generation of model size, right? Right. You're going to let OpenAI do that. The reason I ask that is because Microsoft runs the data centers. That is a partnership. It is ongoing.
But Amazon runs its own data centers and Google runs its own data centers. And it seems like there is just a core tension here, regardless of how good the partnership is, between we are going to build these data centers and restart nuclear power plants in the United States. supply power to some of these data centers. And maybe it's better to sell that to someone else versus build the models ourselves. Do you feel that tension?
You know, every partnership has tension. It's healthy and natural. I mean, they're a completely different business to us. They operate independently. and partnerships evolve over time, right? So, you know, back in 2019, when Satya put a billion dollars into OpenAI, I mean, it seemed pretty crazy. I didn't think it was crazy, but I think a lot of people thought it was crazy.
And now that has paid off and both companies have massively benefited from the partnership. And so partnerships evolve and they have to adapt to what works at the time. So we'll see how that changes over the next few years. Do you have a backup plan if OpenAI declares AGI and walks away from the Microsoft deal? There's some credible reporting that says if they declare AGI, they could walk away from the deal. No. So, look, I mean...
It's very unclear. That's why I said to you, it's very unclear exactly what the definition of AGI is. I mean, we have inside of Microsoft AI, one of the strongest AI research teams in the world. If you look at the pedigree of our crew. My own co-founder, Karen Simonian, for example, led the deep learning scaling team at DeepMind for...
eight years, was behind many of the major breakthroughs. Nando De Freitas has just joined us. He previously ran audio video generation at DeepMind for 10 years. So, you know, we have really an exceptional team.
And we'll make sure that whatever happens, we'll be in a position to train the best models in the world. It does seem like you have some uncertainty there. You've said whatever happens several times now in the context of the opening ideal. Does that feel like something that you can rely on?
It definitely does. Look, they're an exceptional company. They're on a tear. There are many companies in the world that have grown as fast as they have. And so during that kind of meteoric rise, things are going to be brittle.
some of the bits and pieces are going to fall off occasionally. And that's what we've seen in the last 12 months. So that doesn't really change their trajectory. They're going to be incredibly successful and we're going to do everything we can to help them be successful because they've helped make us successful. That's genuinely...
what's going on here and that's naturally in any partnership there's little tensions here and there but fundamentally we will win together yeah i want to come back to the cooperation competition dynamic there when we actually talk about products but i want to stay focused on Microsoft AI inside of Microsoft for one more turn. You obviously started Inflection. Microsoft sort of reverse-equihired all of Inflection. They brought over all the people. They issued you all shares.
Why do the deal that way? Why join Microsoft and why structure that deal in that way? I've known Satya for a very long time. He's been trying to get me to come and be part of the Microsoft crew for a long time. For a good long while, as far back as 2017, when we first started hanging out. And, you know, I've always been inspired by his leadership, particularly. And I think the company is actually in an incredibly strong position. The investments that we're making in compute.
Like I said, the distribution that we have with so many enterprise partners now deploying M365 Copilot and what you can learn from that is a real game changer. You know, a lot of people are talking about these actions, right? Clearly, you want your consumer co-pilot experience to have these seamless interactions with brands, businesses, opportunities for getting stuff done, buying things, booking, planning, and so on.
And so having that kind of protocol built in-house and available to the consumer side is super important. The thing I realized about where we were at with Pi and Inflection, I mean, we had really unbelievable engagement with Pi, very high intensity. DAO, the average session of the voice interaction lasted 33 minutes a day. It was pretty remarkable. So it was well up there. But I think the challenge is the competition is going to invest.
for years and years and years and basically keep it free, if not reduce it to like nothing and basically make it widely available to hundreds of millions of people. And so from a consumer perspective is a very, very, very competitive landscape. And look, when Satya made me the offer to come and run all the consumer stuff here, it was just an offer that we couldn't refuse. It sort of enabled us to pursue our life.
long-term vision of actually creating a true ai companion that has a lasting relationship with hundreds of millions of consumers that is really useful to you and to me that's going to shape the future that is really the thing that is going to shape our long-term trajectory so i couldn't turn that down We have to take a short break. When we come back, we'll get into all the juicy questions about structure. Support for Decoder comes from Banta.
Do you know the status of your compliance controls right now? Like right now? It's known that real-time visibility is crucial for security. But when it comes to GRC programs, we often rely on point-in-time checks. But it doesn't have to be that way. More than 8,000 companies like Atlassian and Quora have continuous visibility into their controls with Vanta.
Here's the gist. Vanta brings automation to evidence collection across 30 frameworks like SOC 2 and ISO 27001. They can also centralize key workflows like policies, access reviews, and reporting. and might be able to help you get security questionnaires done five times faster with AI. Now that's a new way to GRC. Learn more about Vanta at vanta.com slash decoder. That's vanta.com slash decoder. Support for Decoder comes from Shopify.
It's the holiday shopping season and every business is competing for customers' attention and cash. They're offering wild discounts, spending millions of dollars on advertising, and praying that the social media intern can make something go viral on TikTok. time for Christmas. It's ferocious out there and your business needs a partner who can help you reach and convert more customers this time of year. And that's where Shopify can help. Thank you.
and sell more over time. And their ShopPay feature can boost conversions by 50%. There's a reason companies like Allbirds turn to Shopify to sell more products to more customers. Businesses that sell more sell with Shopify. Want to upgrade your business and get the same checkout Allbirds uses? You can sign up for a $1 per month trial period at shopify.com slash decoder, all lowercase. That's shopify.com slash decoder to upgrade your selling today.
Shopify.com slash decoder. Support for Decoder comes from Grammarly. Try to guess the number of apps that most companies use. If you guessed 231, well, great guess, because that's right. But all those different apps can lead to a lot of context switching that distracts employees and can cost your company money. Luckily, Grammarly can help. Grammarly's AI works in over 500,000 apps and websites, making it easy for your team to crank out clear on-brand emails, documents, messages, and more.
In just one click, they can go from blank page to finish brainstorm or even first draft, all without leaving the page they're on. It's like having a built-in writing assistant that can keep your workflow smooth and stress-free. Grammarly has 15 years of experience building effective, secure AI that can be implemented in days with minimal IT overhead.
It goes beyond basic grammar to help tailor writing to specific audiences, whether that means adding an executive summary, fine tuning tone, or cutting out jargon. You can join 70,000 teams who save an average of $5,000 per employee per year using Grammarly. You can go to grammarly.com slash enterprise to learn more. Grammarly, enterprise-ready AI. Welcome back. I'm talking with Mustafa Suleiman, CEO of Microsoft AI, about Microsoft's somewhat unusual leadership structure.
You are the CEO of Microsoft AI. Microsoft is an interesting company in that it has a CEO. several other CEOs. Phil Sensor is the CEO of Microsoft Gaming. Ryan Roslansky is the CEO of LinkedIn. We just had Thomas Domke from GitHub on. He's the CEO of GitHub. What does it mean to you to be the CEO of Microsoft AI? Look, Microsoft's an enormous organization, a quarter of a trillion dollars in revenue, about 280,000 employees. So the logic of making...
single individuals accountable for our own P&L, I think is very rational. There's about 10,000 or so people in my org. We have full integration from Training the models, building the infrastructure, running the ads platform, managing all the sales leaders, making sure that our content is high quality, getting that integrated across four platforms.
That's the logic here. And that's very much how Satya runs it, extreme accountability. One thing that strikes me here is GitHub is a product. LinkedIn is a product. It has a beginning and an end. It's very tangible. People can understand it. Microsoft AI is the company. There's just a lot of AI at Microsoft that is infusing into all of these products. I think Sacha has agreed that AI feels like a platform change. There's enormous opportunity inside of a platform change.
You've obviously got your core products in Bing and Edge and MSN and all that. But when you think about the relationship to the rest of the AI efforts at Microsoft, where does the line begin and end for you? That's a good question. Right now, the company is so focused on winning on Azure OpenAI, for example, getting our models into production and getting them into the hands of hundreds of thousands, millions of businesses.
involved in a lot of the reviews on the enterprise side. But really, I play a role as an advisor and support and our MAI internal models so far haven't really been focused on those enterprise use cases. My logic is that... We have to create something that works extremely well for the consumer and really optimize for our use case. So we have vast amounts of very predictive and very useful data on the ad side, on consumer telemetry and so on. And so my focus is on building.
That's a product focused structure, it sounds like. Have you reorganized Microsoft AI to be a more product-driven team? I think the business was focused on the product before. What we've done is bring the kind of AI sensibility into the heart of each one of our products. So, you know, we have a lot of ranking. We have increasingly conversational and interactive surfaces. We're trying to bring the voice of Copilot.
to bing to msn you know we want to make it a core part of the search experience so that your first thought is Let me just ask my AI. What does my AI think about that? My AI can remember that for me. It can save it, it can organize it. And so making sure that it shows up in deeply integrated ways that really support the surface rather than sort of just are an adjacent kind of add-on. an afterthought that's the craft that we're kind of working towards
You are a unique person to have on the show because you also founded DeepMind and you worked at Google. We've had Dennis, the CEO of DeepMind, on the show before. Google is a challenging place to work at. He is the CEO of Google DeepMind. Google doesn't have CEOs like the way that Microsoft has CEOs in that particular way. Can you compare and contrast these two companies? You worked at one huge company. You were at a startup for a minute. Now you work at the other huge company.
They are very different culturally and structurally. Do you think Microsoft has advantages over Google's approach? I do. I think that at Microsoft, there is a lot of discipline around revenue and P&L. And I think that is a very healthy attitude because it really focuses the mind on what is a consumer going to find truly valuable and be prepared to pay for.
Second, I think there's a real long-term thinking in terms of like, where does this platform shift take us? And what does the five to 10 year horizon look like? So there's a kind of planning attitude, which... at least during my time at Google, felt more instinctive. I mean, their instincts are really good. It's a very incredibly creative company, and many times they've made long-term bets.
but it was kind of reactive, instinctively reactive, whereas I think there's a lot more thought in the kind of scenario planning and really kind of thorough deliberation. I would say the third thing, I guess I would say, is like, you know, Friday's... senior leadership team meeting with Satya is just like a phenomenal experience. I mean, it runs basically 8.30 till 2.30 in the office in Redmond and everyone's there, all the leaders.
And, you know, we sort of review all the big businesses or all the big strategic initiatives in detail. And the SLT cross-functionally is in the weeds. And that is a pretty remarkable thing. because they're sort of reviewing these things week after week, like security, huge priority, genuinely, like...
a number one focus for the company, AI and infrastructure, then reviewing all of the businesses. It's very cool to see the other leaders ask the questions or probe, and I kind of see the world through their eyes. which is sort of slightly different. So although there's sort of lots of CEOs, everyone's looking at everyone else's businesses and giving advice and feedback. So it's quite an intellectually diverse group. And then the other thing I would say is that...
Because there's obviously an enterprise DNA to the company, there's a real focus on what does the customer want. Google is like, what would be a cool technology for us to build? Whereas Microsoft's like, how would this actually help the customer and what are they asking for? I think...
Both of those strategies have their own benefits. But if you swing one way or the other to an extreme way, there's sort of real problems. And so I've certainly enjoyed learning from the fact that Microsoft is very much like, what does the consumer want? What does the consumer want? What does the customer want?
You mentioned security at Microsoft. The renewed focus on security is because there were a bunch of lapses earlier this year. This has been an issue. Can you just explain, you have an outsider perspective. You're building a lot of products that might go out into the world and do things for people. You're building a lot of products that require a lot of customer data to be maximally useful. As you go into these meetings and you talk about Microsoft's renewed effort.
on security because there were some problems in the past, how that's affected your approach to building these products. Yeah, I mean, I definitely think that the company culture is security first. I just want to be very clear to the audience. Sacha has started saying that now, but it's because there were these enormous security lapses just in the past year. That's true. That is very true. I'm just saying since I've got there...
I sit in a weekly security meeting where literally all the heads of the companies, various different divisions, are singularly focused on what we can do. And it is the number one priority. There's nothing that can override that. No customer demand. no amount of revenue. It is the first thing that everybody asks. So culturally...
as far as I've known it, it is the central priority, which has been good for me too. I mean, for my businesses are also like mission critical that we preserve consumer trust, right? And trust means that... People just expect us to be able to store and manage and use their data in ways that singularly benefit them and are in their interests. That is a central part of the culture. And you're right, maybe that's a refocusing.
of late. But it certainly is the case now. You also mentioned you have P&Ls as CEOs. I sort of understand how LinkedIn has a P&L. They have a product, they have some engineers, they make some money, people pay for premium. Microsoft AI, it feels like kind of a lot of losses and not so many profits. How are you thinking about bouncing that out? Oh, we're very profitable.
We're very popular. Well, I'm just saying there's a lot of forward investment in AI, right? Like that stuff hasn't paid off yet. That's true. That's true. The AI stuff hasn't paid off yet. I think it's fair to say. But remember, I spend over half my time focused on the Bing business. Bing Business is doing incredibly well. I mean, we grew 18% last quarter, and we actually took gains from Google, which means we're growing faster than Google.
which makes everybody feel happy. And that's kind of the main goal. So the product is deeply integrated AI. There's generative search results in the context of your search experience. There's increasing conversational experiences there. And just the general quality that we've been able to level up with LLMs has been very impressive. And I think that's translating into... to revenue improvements as well so in that sense
AI itself is actually in production across the company. It's not like we're just waiting for these sort of chatbots to suddenly miraculously generate a new business model. LLMs are being used at all sizes across the existing business for all kinds of things. Like even in Edge, for example, for transcription and summarization built into the browser. There's so many different ways that AI is showing up. I think you've got to think of it more as like a new...
high bar in terms of the table stakes of the features that we offer. The part where the LLMs are integrated into a bunch of products like Bing or Edge. Are they driving more revenue from those products or are they just taking share away from Google? The way I think about it is like it's improving the quality of ads that we show, improving the relevance of those ads. And so it's making the experience more useful for the consumer. And that is obviously the overall pie.
is growing which is that's the nature of the growth because obviously google's growing too so the entire market is continuing to grow the point is that we're growing faster than google for this quarter and i think that's a huge achievement the team's done an amazing job and it's a it's not it's not about me but by the way, that's a product of them, many years of them investing in quality and relevance and just generally doing a great job. Famously, when being with...
Co-pilot was introduced. I sat down with Sacha. He said, I want to make Google Dance. And then I went and asked Sundar about that. He said, he just gave you that quote so people would run that quote. And that was kind of his response. Sundar was very calm in that way.
You came into it after that whole situation, and now you run the products that are directly competitive with Google. Do you think that you're growing faster than Google in some places? Do you think that you are actually posing a competitive threat to Google in either? Bing with Search or Edge with Chrome? I think one of the things that I've realized as I've got a bit more experience to mature over the years is that you have to be very humble about how the landscape changes. On the one hand,
This really is an opportunity to re-litigate some of the battles of the past. The chips are going to fall in a completely different configuration in the next two or three years. At the same time, that's a very challenging thing to do. habits die hard and so on. But our goal with this completely new interface is to make it 10 times easier for people to access information, advice, support in a truly conversational way and to do things that our competitors won't do.
that are truly useful to everyday consumers. And I think that's actually going to be one of the differentiators. It's like, what is the personality and the tone and the emotional intelligence? of an AI companion. Because remember, most people do love information and they like getting accurate and reliable information, but that's going to be commoditized, right? All of these models are going to have that.
And despite what we like to think in Silicon Valley, surrounded as we are by, you know, sort of nerds and information obsessives who, you know, read all the content that you can get access to, most people... really connect to brands, really connect to ideas in a social way, right? They connect to it because It is friendly, kind, supportive, emotionally reassuring. And I think that's going to form a big part of the way these models actually turn out to be successful in a few years' time.
I need to ask you the core decoder question, but then I want to come back to the idea that the information will be commoditized. You've described a lot of change, right? You were at one company, you were at a startup, you're at Microsoft, you're learning how Microsoft works. You have big decisions to make about how to deploy these products. The way that I like to operate is in a six-week rhythm. So I have a six-week cycle, and then we have a one-week meetup.
for reflection, retrospectives, planning, brainstorming, being in person. The reality post-COVID is that people work from all kinds of places and they like that flexibility. So my rhythm is to keep people in person two to three days a week. and then really come together for that seventh week of retrospectives.
So my general framework is to try to be as in the weeds as possible. Like I really spend a lot of time in our tools, tracking telemetry, hearing feedback from people, and then creating this very tight operator. rhythm where in the context of a cycle six seven week process we have a very falsifiable mission every single team can express in a sentence exactly what it is they're going to deliver
And it'll be very falsifiable at the end of that. So we'll know. And then when we observe, you know, whether or not that happened, that's a moment for retrospective and reflection. And I like to really write. I'm a writer. I think by writing. And I like to broadcast my writing. So every week I write a newsletter to the team that is just like a reflection on what I've seen, what I've learned, what's changing, what's important.
And then I document that over time and use that to kind of track and steer where we're going. That's kind of the basics of how I practically implement my process for reflection and stuff like that. But in terms of the framework... One thing is to really tune in to the fact that no matter what product you invent, no matter how clever your business model is, we're all surfing these exponential waves. And the goal is to predict which capabilities fall out of the next large training model.
Overthink that and assume that actually there's some genius new ecosystem incentive or new business model or new UI style. All of that is super important. But if you think that it's really only going to be that. or that that's going to be the overwhelming driver, I think that's a mistake. And maybe this comes from sort of my 15 years experience of trying to build these models. You know, remember at DeepMind 2014 to 2020.
I was banging my head against the table trying to ship machine learning models, ship CNNs in the early days, find classifiers, do re-ranking, try and predict what to watch next on YouTube, trying to do activity classification on your wearables, trying to ship... crash detection algorithms inside of Waymo, like every single applied practical machine learning objective I had explored there. And now we actually have the tools.
to be able to do those things and do them really, really well. They're really working. So we're basically surfing those tides. The goal is to really nail those waves because we already have models which are... giving us more than we can extract and apply into products. That's quite a profound state that we're in. We actually haven't yet completely extracted all the gains.
from the current class of frontier language models. Every week, there's still some new capability or some new trick or people have crafted or sculpted them in post-training in a new way. And I think that that...
that is going to continue for the next few years to come, many years to come, in fact. So in terms of the decision-making framework, The goal is to be very focused on model development and scaling those models, getting them to be practical and useful, really aligning them and getting them to behave in the way that you need for your product.
development and we need to get more of the models we have now, there's a little bit of attention there. There's a notion that the scaling laws are going to run out, that the next class of models is not significantly outperforming the models we have now. And I think you can track that in just the way we're talking about the products. A couple years ago, it was AI is an existential risk. We have to stop it so we can make sure it's aligned before we kill everyone. And now we're kind of at like...
well, we've got to get more out of the models we have now and actually ship some products and make some money, hopefully, and figure out what it's all good for and how to best use it. Because it doesn't seem like the next generation of models is actually running away as fast as we think they might. Is that your view?
the sort of frontier models are not getting better as fast as we thought they might, and so we have to get more out of what we have? No, I don't think that's true. I think that they're going to continue to deliver the same... you know, seismic gains that we've seen in the previous generations. Remember that they are more costly and they're more fragile and they take longer to train this time around.
So we're not going to see them happen in the same sort of 12 to 18 month timeframe. It's going to shift to 18 to 24 months and then a bit longer. But I don't see any sort of sign that there's this like structural slowdown. I kind of see the opposite is that there's huge gains to extract from where we are today, but it's very clear to me that there's huge gains to extract from the next two orders of magnitude of training as well. We have to take another quick break. We'll be back in a minute.
Support for Decoder comes from Stripe. Payment management software isn't something your customers think about that often. They see your product, they want to buy it, and then they buy it. That's about as complex as it gets, but under the hood of that process, there are a lot of really complicated things happening that have to go right in order for that sale to go through.
Stripe handles the complexity of financial infrastructure, offering a seamless experience for business owners and their customers. For example, Stripe can make sure that your customers see their currency and preferred payment method when they shop. never feels like a chore. Stripe is a payments and billing platform supporting millions of businesses around the world, including companies like Uber. And reach more customers globally. The platform offers a suite of specialized features and tools.
to power businesses of all sizes, like Stripe Billing, which makes it easy to handle subscription-based charges, invoicing, and all reoccurring revenue management needs. Learn how Stripe helps companies of all sizes make progress at stripe.com. That's Stripe.com to learn more. Stripe. Make progress. Support for this show comes from Grammarly. 231. That's the staggering number of apps the average company juggles every day.
All that context switching is a productivity killer, putting employees off track and draining resources. The good news? Grammarly has the solution. Grammarly's AI works in over 500,000 apps and websites. making it easy for your team to crank out clear on-brand emails, documents, messages, and more. That means you can go from blank page to finish brainstorm, our first draft, all without leaving the page you're on.
Grammarly has 15 years of experience building effective, secure AI that can be implemented in days with minimal IT overhead. And it goes beyond basic grammar to help tailor writing to specific audiences. Whether that means adding an executive summary, fine tuning tone, or cutting out unnecessary jargon, you can join 70,000 teams who save an average of $5,000 per employee per year by using Grammarly.
Just go to grammarly.com slash enterprise to learn more. That's grammarly.com slash enterprise. Support for Decoder comes from Mint Mobile. Some companies have forced their customers to do some pretty wild things to get a great deal. Like those restaurants that will give you a free meal if you can finish their flaming death burger without, you know, dying.
Phone companies aren't that different. They add piles of fine print to contracts and make it almost impossible to get a deal you thought you signed up for. But Mint Mobile just wants to give you what they promised. When you purchase a new three-month phone plan with Mint Mobile, they say you'll pay just $15 a month. That's it. All Mint Mobile plans come with high-speed data and unlimited talk and text delivered on the nation's largest 5G network.
You can keep your phone, your contacts and your number. You can get this new customer offer and a three-month premium wireless plan for just $15 a month by going to mintmobile.com slash decoder. That's mintmobile.com slash decoder. You can cut your wireless bill to $15 a month at mintmobile.com.
$45 up front payment required equivalent to $15 per month. New customers on first three-month plan only. Speed slower about 40 gigabytes on unlimited plan. Additional taxes, fees, and restrictions apply. See Mint Mobile for details.
Welcome back. I'm talking with Microsoft AI CEO Mustafa Suleiman. Right before the break, we were talking about extracting gains from current AI models. Mustafa mentioned the idea of commodifying information, and I definitely wanted to dig into that a little bit more. I want to make sure we talk about the thing you mentioned, the commodification of information. And then I want to make sure we talk about agents very quickly to bring this all around to the products to come.
The commodification of information is, I think, the big story of the internet that we have today, the platform internet, for lack of a better word. You go to Google, you ask it a question. Now it might spit out an AI-generated answer. You go to MSN, you ask it.
for the news, and it might algorithmically or with AI sort a bunch of news and summarize that news for you. Everyone's headed in this way. We've been talking about this for a long time. To train the next generation of models, we need even more information.
You've gotten yourself into some trouble, I would say, saying that the information on the internet is quote-unquote freeware and the expectations that you can use it to train. There are a lot of lawsuits, including several pointed at Microsoft. Where do you think that next body of information comes from before we sort out the copyright implications of using all this stuff to train?
So one way of thinking about it is that the more computation you have, the more time these models can spend attending to the various relational components of all that training data. So think of flops as a way to spend understanding time. learning the relationship between all these various sort of training inputs. So first of all, you can still gain more from just having more computation to learn over all the existing data. Second thing is we learn a vast amount from interaction data.
You know, users tell us implicitly and explicitly how they feel about an output. Is it high quality? Is it used? Is it ignored? Third is we're generating vast amounts of synthetic data. That synthetic data is increasingly high quality when you ask an AI teacher or a rater to compare two or three different examples of the synthetically generated output.
and the human written output, it's extremely difficult to detect those precise nuances. So the synthetic data is increasingly high quality and used in a whole bunch of different settings. Then the fourth is I can imagine AIs talking to other AIs, asking for feedback, AIs that have been primed for different areas of expertise or different styles.
prompted in different ways, you can imagine those interactions producing valuable new knowledge, either because they're grounded in different sources or just because of their stylistic output, they're actually producing novel interactions. So I don't necessarily see data being the limitation anytime soon.
I think that there's still huge benefits to come from scale for a foreseeable future. So that's all new data, right? You're going to get a bunch of interaction data. Maybe the synthetic data will be high enough quality to train the next generation models. But the original data sets...
More the web, right? It was a bunch of web content. It was the entire internet. Maybe it was to video platforms to some extent from some of the model providers. The quote I have from you in June, I think you were speaking to Andrew Ross Sorkin.
Here's a quote. You said, I think that with respect to content that's already on the open web, the social contract of that content since the 90s is that it's fair use. Anyone can copy it, recreate with it, reproduce with it. That has been, quote, freeware, if you like. That's the understanding.
I'm curious, right? You said that. That was the understanding for search, right? And there was a lot of litigation around search and Google image search and Google Books that led there. Do you think that that is still stable enough? For you in the age of AI with all of the lawsuits outstanding.
I mean, what I was describing in that setting was the way that the world had perceived things up to that point. And I think that my take is that just as anyone can read the news and content on the web to increase their knowledge. under fair use, so can an AI. Because an AI is basically a tool that will help humans to learn from publicly available material and all the material that has been used for generating or training our models has been...
scraped from publicly available material. But publicly available and copyrighted are very different things on the internet, right? Like, publicly available does not mean free of copyright restriction. Oh, yeah. I mean, look, obviously, we respect the content providers, right? So let's not...
You know, that's an important distinction. But, you know, I guess what I'm trying to say is from our perspective, there's certain types of content that, for example, in our co-pilot daily or MSN daily that is paywall publisher content that we pay for directly. And that is what MSN has been doing since the beginning of time. It's what we've decided to do with Copilot Daily for high quality content because we want those publishers.
to create an information ecosystem that really works for everybody. And I just think this is one of those things where You know, things will play themselves out in the courts. Like at any time when there's a new piece of technology, it changes the social contract as it is at the moment. There's clearly a gray area in terms of what constitutes fair use and whether an AI can have the same face.
use as as a human and you know we'll just have to sort of play it out you know over the next few years and i think we'll we'll have some perspective over that in in the next few years as as things land One of the reasons that I asked that as directly as I'm asking it is the cost of training the next generation models is very, very high. But that cost is built on a foundation of, well, the training data is free.
And if a couple of court decisions go a couple of ways, the cost of the training data might skyrocket, right? If a court says it's not fair use to use the New York Times as content or it's not fair use to use these books from these authors. Suddenly, you might have to pay a lot of money for that data as well. Do you think that that is something that the business can sustain? Yeah, I mean, we already pay...
for books at huge scale. And if it's a copyrighted book, we're not hoovering that up from the internet. Well, Microsoft might not be, but there's a very big lawsuit from a bunch of publishers who say that... For example, open AIs, right? And that's the model that you are reliant on. So it just seems like there's a, maybe legally, we'll see what the answer is. But economically, there's also a lot of uncertainty here, right? Because of the costs of the underlying data.
Yeah, that's true. And I think our focus has been to make sure that we pay for the really high quality copyrighted material from publishers. both news publishers and book publishers and others. And I think that's going to continue. And that's definitely what we're committed to. Who decides what's high quality? That's actually an interesting question. Quality is...
Actually, something that we really can measure. We want to make sure that the content, especially from a nonfiction perspective, so we're particularly interested in academic journals, academic textbooks. We can verify the source and citations for that knowledge. And that is one of the big measures that we consider to be high quality. But, you know, the visual artists, the nonfiction artists, visual effects artists, the movie industry.
Right. They're saying, hey, we're going to get pushed out of work because we are not compensated for any of the work that's going into these models. Like, how do you think this plays out for that? Because. Again, I agree that the law here is deeply uncertain. These cases are going to play out. But I'm looking back at what you're describing as the social contract of the web, and what I see is, oh, Google litigated a million of these lawsuits.
We didn't just all wake up one day and decide this is how it's going to work. Google went to court 15 times. and they were a bunch of kids who had slides in the office, and they had just made Google, and they were positioned as a company in a moment, and they had a product that was so obviously useful in so many different ways.
that they kind of got away with it. And I don't know that the tech industry is in that position anymore. I don't know that the products are so obviously useful the way that putting Google on the internet for the first time ever was so obviously useful. And I certainly don't know that...
the feelings from particularly one set of creators are as mixed or to positive as they were for Google back in the nineties and early two thousands. And that to me feels like, I mean, you're on the board of the economist, right? Like that to me feels like the.
The people that make the work are having the most mixed emotions of all. Because, yes, I think a lot of us can see the value of the products, but we also see the value transfer to the big tech companies, not the upstarts, not the cute kids with the slides in the office. I think that this is going to be more useful and valuable than search. I think search is completely broken. And I think it's a total pain in the butt. And we've just kind of got used to using a terrible experience.
Typing a query, just sort of think about what a query is. We had to invent the word query effectively to describe this really weird, restricted way that you express a sentence or a question.
into a search engine because of the weakness of the search engine. And then you get 10 blue links and then those things are vaguely related to what you're looking for. You click on one and then you have to go and refine your query. I mean, it is a painful and slow experience. I think that... if we can get this right, if we can really reduce hallucinations to de minimis amount, I think we've already demonstrated that they don't have to be toxic and bias and offensive and all the rest of it.
pretty good it's not perfect but it's getting much much better and i think it's only going to get better with more stylistic control then these conversational interactions are going to become the future of the webs. Quite simple. I mean, this is the next browser. This is the next search engine. It is going to be a hundred times easier for me to just turn by voice to my co-pilot and say, hey, co-pilot.
What's the answer to this? I already do it five times a day. It is my go-to. It's made my bottom right hand app in my iPhone. my thumb instantly goes to it. I use the power button to open my favorite app like I did with Pi. I mean, it is clearly... the future, that conversational interaction. So to me, the utility is phenomenal. And I think that is going to weigh into the cases.
as they make themselves make their way through the core. So that leads us, I think, directly to agents, right? Where you're just going to ask some app on your phone or some part of the operating system on your computer to do something, and it will go off and do it. It will bring you the information back or it will accomplish some task. on your behalf and bring you the result. You and I have talked about this before in various ways.
It commodifies a lot of the service providers themselves, right? You say, I want a sandwich, and now I don't know if it's DoorDash or Uber Eats or Seamless or whoever is going to bring me the sandwich. My AI is going to go out and talk to them. That implies that they will allow that to happen. They will allow the agents to use their services. In the best case, they would provide APIs for you to do it. In the worst case, that...
They let people click around on their websites, which is a thing that we've seen other companies do. And sort of in the medium case, they develop some sort of AI-to-AI conversation. Not quite an API, not quite we're just literally clicking around on a website and pretending to be human. RAIs are going to have some conversation. What is the incentive for those companies to build all of those systems or allow that to happen to become disintermediated in that way? I mean, people often ask that.
When there's a new technological scientific revolution and we're going through it and it's causing a massive amount of disruption and people are curious, it's like, well, why would someone do that in 10 years? And then if you look back for centuries...
It's always the case that if it is useful, it gets cheaper and easier to use, it proliferates, it becomes the default. And then the next revolution comes along and completely turns everything on its head. My bet... is that every browser and every search engine and every app is going to get represented by some kind of conversational interface, some kind of generative interface. The UI that you experience is going to be automatically produced by an LLM.
in three years, in five years, you know, and that is going to be the default. And they will be representing both the brands, businesses, influencers, celebrities, academics, activists, organizations, just as each one of those. stakeholders in society you know ended up getting a podcast you know getting a website writing a blog you know maybe building an app like using the telephone back in the day you know so
The technological revolution produces a new interface which completely shuffles the way that things are distributed. And, you know...
Some organizations adapt really fast and they jump on board and it kind of transforms their businesses and their organizations and some don't. And so there will be a kind of... you know an adjustment we'll look back by 2030 and be like oh that really was the you know the kind of moment when there was this true inflection point because these conversational AIs really are the primary way that we have these interactions and so
You're absolutely right. Like, you know, a brand and a business is going to use that AI to talk to your personal companion AI, because I don't really like doing that kind of shopping. And some people do, and they'll do that kind of browsing experience, direct to consumer. And many people don't. And it's actually super frustrating and hard and slow. And so increasingly you'll come to...
work with your personal AI companion to go and be that interface, to go and negotiate, find great opportunities and adapt them to your specific context. And that'll just be a much more efficient protocol. because AIs can talk to AIs in super real time. And by the way, let's not kind of fool ourselves. We do already have this on the open web today, right? We have behind the scenes, this real time negotiation.
between buyers and sellers of ad space or between search ranking algorithms. So there's already that kind of marketplace of AIs. It's just not explicitly manifested in language. It's operating in vector space. That's the part I'm really curious about. The idea that natural language...
Is the platform shift? Is the paradigm shift? I think it's very powerful. I don't think it has been expressed very clearly, but the notion that actually the next form of computing is inherently based in natural language, that I'm just going to talk to the computer, it's going to go off and do some stuff because it understands.
me is very powerful. I buy it. How that actually plays out on the back end is the part that to me still feels up in the air, right? I'm going to ask for a sandwich that necessitates there to be companies. that are in the business of bringing me a sandwich. And how they talk to my AI and how they stay in business seems very challenging. Right now, those companies, they're in business because they can sell ad space on my phone.
The other companies actually make the sandwiches. They have upsells. There's a million different ways these companies make money. If they abstract themselves down to their AI talks to my AI and says, okay, here's a sandwich. and I take away all of their other revenue opportunities, I'm not sure that that ecosystem can stay relevant or even alive. I'm not sure about that. I mean, your sandwich making AI is still going to want to sell itself, be persuasive, be entertaining, produce content.
to the consumer, right? It's not that it kind of completely disintermediates and disconnects because brand and display advertising is still super relevant. And, you know, there will be ways that that... Sandwich making AI shows up in the context of your personal AI context in maybe a sponsored way too. So, you know, there'll still be that core framework of keyword bidding, paying for presents.
paying for awareness there's still going to be ranking that is still going to be relevant to some extent it's just that You are going to be represented by a personal AI companion that is going to be that interlocutor and negotiator. And those two AIs are going to have an exchange in natural language, which is what we would want. We would want to be able to go back and audit that negotiation.
and check where the error came from and see if it really was a good price in hindsight and all the rest of it. As you start to build these products in Copilot, have you had these negotiations with these other providers? Have they started to say what they would want? We've talked, I wouldn't describe them as negotiations. I mean, I think lots of brands and businesses are building their own AIs. Today, they're characterized as customer support AIs that pop up in your website.
But tomorrow, in two or three years' time, they're going to be fully animated, conversational, rich, clever, smart, digital co-pilots that live in social media. They're going to appear on TikTok. They're going to be part of the cultural space. So I think that there's not so much a negotiation to happen there. I think it's just this inevitable tide of the arrival of these co-pilot agents.
You run MSN. You obviously have peers at Microsoft who run other kinds of social networks, other kinds of information products. I see a flood of AI slop choking out some of these networks. I've searched Facebook for Spaghetti Jesus, and I have seen the other side of the singularity, my friend. How do you – right? We already had one conversation about determining high quality and the answer is sort of I know it when I see it. But if you run these networks and you're faced with a bunch of –
agent AIs who are talking or AI influencers on TikTok. Can you label that stuff effectively? Can you make it so that users can only see things from other people? You certainly can. It would require a shift in the identity management system of the platform, which has a lot of pros and cons.
You can certainly tell which accounts come from a human and which are AI generated. To some extent, I think there can be digital watermarking and signing for verified human content or verified AI content from a specific source. And then to some extent, there can be detection of synthetically generated content because that does have a specific signature. Long term, I don't think that's a defense. I think it is going to be perfectly photorealistic and very high quality.
And it's going to be a game of cat and mouse, just as it has been in security and privacy and information for decades and centuries, actually. So I expect that to continue. It is going to get harder and more nuanced. But this is the sort of natural trajectory of things. Do the people who run LinkedIn or do your folks at MSN say, this is a problem that we can't stop? We need to make sure we don't have too much AI content here.
Right now, it's not good enough, right? I can see it a mile away. I see those bullet points. I think someone made this with ChatGPT. I don't even want to read it. Is that a problem that you're facing right now? Is that a problem to come? I think it's a problem to come. But the thing I would say is, you know, we humans are behaviorists, right? We observe the output of other humans and we evaluate and we decipher trust.
based on the quality of information with respect to our own assessment. Is it accurate? Is it reliable? Is that person consistently doing what they said they would do? And so we can observe their actions. And rather than sort of introspecting, why did this happen? Why did this neural network generate this output? Why did this person come to this conclusion?
And that's actually an important distinction because I think a lot of purists are kind of fixated on the causal explanation for why an output has been produced rather than the... kind of more observational assessment of like was it useful does it do the same thing over and over again that's what drives trust
So I do think that poor quality content will be detectable in that sense, or sort of AI content that is deliberately misrepresentative or misinforming will be detectable because I think we will have better models and we're getting them all. all the time for ranking down and deprioritizing certain types of content. One of the things that I've been thinking about a lot throughout this conversation, you're in charge of Microsoft's consumer business.
right now in 2024 is built around not making the iPhone, right? Like that's the thing that Microsoft missed in consumer famously. It has nothing to do with you. But the iPhone happened, Microsoft pivoted to being an enterprise business, and it's now slowly coming back because I think the company rightfully sees a platform shift, a paradigm shift in computing.
Apple still exists. The iPhone still exists. You said, I've got this icon on my iPhone. It made it onto the home screen and it's in this preferred position, the position everybody wants in the bottom corner. Apple has a pretty big distribution advantage here. They have a deal with... with OpenAI to use ChatGBT. Can you make products so good that you overcome the iPhone's distribution advantage? They're bundling into the operating system?
It's a great question. I mean, distribution and defaults really matter. And so from our perspective, obviously, we're focused on distribution deals. But fundamentally, we are also focused on building something that's truly differentiated. And to me, that... AI companion really is going to be differentiated. The tone and the style of that interaction matters.
The fact that it will be able to remember you and what you've said over time. The fact that it will reach out at that opportune moment just before a difficult moment in your life where you're starting a new job or your kids just having their birthday party or something. You're in a moment where...
having your companion reach out, be supportive is a differentiator. And that's how a lot of people make their decisions and it's how a lot of people seek support. So I think that's a really big opportunity to spread, you know, a good vibe and spread kindness. And I think... Most apps and most kind of product thinking in Silicon Valley doesn't really engage with that kind of emotional plane in the way that the advertising industry in New York.
just thinks of that as second nature, for example. And I think that's a big shift that we're making as an industry. And it's certainly one of the areas that we're going to be focused on in Copilot. And we have to build something that is really beautiful and differentiated. It's going to be a real challenge. It's not easy.
an opportunity to build consumer hardware again. Not a phone, but whatever comes after the phone. I am open-minded about that. The voice-first experiences are going to be transformational and they do represent a new platform. And I think we're increasingly tired of our screens. I'm frankly sick of looking at a grid of multicolored icons on my iPhone. I think many people are. You sort of feel trapped down this sort of, you know, structured.
fixed kind of unit of tapping these things and i don't know i think people are looking for more opportunities to go hands-free and to be away from keyboards and screens leave your phone at home So I think there's a lot of opportunity there. I'm very, very interested in that space. Have you played with the products that are out now, the humanes, the rabbits?
I have. I played with all of them, yeah. And I've actually just come back from an extended trip to China where I visited all the big manufacturing companies, got to know those guys. Very impressive what they're doing out there, moving at light speed. It's very, very interesting to see. Should we expect hardware from you? No, not anytime soon. But I think we're a huge company. We're keeping open mind about lots of things.
and we'll see how things go. Very good. Well, Mustafa, we're going to have to have you back very soon. I have a million questions here I didn't get a chance to ask you. This was great. Thank you so much for being on the show. This has been a lot of fun. Thanks, Eli. I really appreciate it.
I'd like to thank Mustafa for taking the time to join me on Decoder. We spoke over the Thanksgiving break, so I appreciate it. I'd also like to thank you for listening. I hope you enjoyed it. If you'd like to let us know what you thought about this episode or really anything else, drop us a line. You can email us at decoder at the verge.com. We really do.
read all the emails and for our last episode of the year i'll be answering some listener questions so now's the best time to send some in if you want to ask anything You can also hit me up directly on Blue Sky or on Threads. I'm at Reckless1280 on Threads and at Reckless.bsky.social on Blue Sky. We also have a TikTok. Check it out. It's at DecoderPod. It's a lot of fun. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts.
of production of The Verge and part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Statt. This episode was edited by Travis Larchuk and Callie Wright. Our supervising producer is Liam James. The Dakota Music is by Breakmaster Cylinder. We'll see you next time.
Support for this show comes from Alex Partners. You don't need us to tell you that AI is reshaping how business is done. But during critical moments of disruption, it can be hard to figure out how to leverage cutting edge technology. With a focus on clarity, direction, and effective implementation, Alex Partners provides essential support when decisive leadership is crucial.
you can discover insights and learn how to convert digital disruption into revenue growth by reading the 2024 digital disruption report at In the face of disruption, businesses trust Alex Partners to get straight to the point and deliver results when it really matters.
Support for this show comes from Elf Cosmetics. Listen, makeup melting off your face like a very beautiful but very sad candle has no place in your workout. Elf's Power Grip Primer can keep your look together with its sticky gel formula and hydrating ingredients. It's designed to lock your makeup in place for comfortable, long-lasting wear through all the day's activities. And of course, it's e.l.f., so you get it at an incredible value too. Power Grip Primer is only $10.
You can find ELF's Power Grip Primer at elfcosmetics.com.