Mozilla Firefox CTO on Browser War Stories and the Path to Distinguished Engineer

AI is coming to consumer technology like water running downhill. This is Bobby Hawley. He grew from an intern to the CTO of Mozilla Firefox and lived through the browser wars with Google. Every time Google would do something that would hurt Firefox, it would always be framed as unintentional. He also had interesting stories from his projects including some difficult security bugs he tackled.

I was disassembling it. And then I had this realization that I knew who had written this exploit. How'd you know that that security researcher wrote that vulnerability? He shared tips helpful for any career in tech. One thing that I think that everybody... can work on at whatever level they are is writing. And I think it is one of the most powerful things. There's this new era of browser wars, bringing AI into existing browsers. I have pretty strong opinions about this. Here's the full episode.

Before actually working at Firefox, I saw that you interned at NVIDIA. I'm kind of curious, why did you pick Mozilla over NVIDIA? Yeah, so I mean, my first internship after freshman year was at NVIDIA.

It was a great internship. I think it was much more standard of an internship. You show up, there was orientation, they showed you what to do, you sat at your desk, gave you a very basic starter project, said, well, we'll check on you in a couple hours. It was really designed to be a sort of thoughtful, you know, growing experience for somebody who didn't really know what they were doing. And Mozilla was, I would say, quite different from that. I remember showing up...

Really, I think on my first day, my manager was the director of platform who ran all of Gecko. It was this guy, Damon Secor. And he really, he had a very particular vibe. He sort of embodied, I think. the dude from The Big Lebowski in like all of the best ways. Like it's just extremely laid back. And I remember asking him, you know, what should I work on? And he was like, I don't know, like go get on IRC and...

get on Bugzilla and like start figuring out some things to do. And that was just a very big contrast to the sort of more standard internship. And what that spoke to was the nature of how Mozilla operated. Because Mozilla had been an open source project that largely happened in some ways by accident, where lots of people just showed up and started working on things.

And there was not a lot of support and there wasn't, you know, this was like way before video conferencing. There was just the bare minimum of what you needed if you were really dedicated and really creative and really persistent. to figure out like how you would download the code you had to figure out how to download it you have to figure out how to build it you have to figure out where to go to you know file bugs and like get help and there was just like not a lot of guardrails

And what that selected for was a particular type of person who really just needed the bare minimum and was very motivated to just go in and figure things out. And so the management structure at the time... was very much oriented towards nurturing and empowering this culture, right? Very much like let's get the right people together and let's just get out of the way. And so this for me, I would say initially was a little bit disorienting.

But I ended up getting really inspired by all of the people who were working on it, some of whom were paid, some of whom were not paid. And a lot of these people, maybe they'd started being paid recently as Mozilla started to have money.

but had been, you know, getting involved in the project from their college dorm room, just trying to figure out how to make their browser crash less. There was this guy, sort of one of my longtime heroes, Boris Zabarsky. We didn't really, I don't think there were really titles at the time, but... He eventually was one of Mozilla's first distinguished engineers. And he was just very awe-inspiring to me because I only, you know, for I think several years, I only met him on IRC. But...

Mozilla was a wild west, you know, the Firefox code base was a wild west of code, just like tens, you know, millions and millions, probably about 10 million lines of code that were inherited from Netscape and largely unowned. And...

The major challenge was just dealing with all of this code and figuring it out. And Boris probably owned like half of it. He was the person of last resort that you would always ask if you didn't know what to do. And you would realize that you'd be having this conversation with him on IRC.

And for me, it would be like taking all of my focus of trying to understand and like, you know, keep up with him. And then I would realize that he was in like five other channels at the same time, like having these conversations with like other senior Mozilla engineers, other interns. contributors users who were just having bugs and he was just a machine so i'm kind of curious at this time how does this tie into the the timeline of

google chrome and and the browser wars was this before google chrome yeah so this was before chrome um i guess as a brief history lesson so the original round of the browser wars was in the 90s and this was largely defined by internet explorer versus netscape

And they competed with each other by trying to ship different proprietary APIs and make the site work better in IE or work better in Netscape. But eventually, Internet Explorer won, largely because Microsoft bundled it for free as part of Windows. and was very aggressive with respect to distribution and really pushing Netscape out of distribution. And Microsoft eventually was taken to court and lost and had an antitrust judgment about precisely this.

But the way the market works, it didn't matter. IE had already won. And Netscape was going under, and as a parting shot, they open-sourced the code. and created this Mozilla foundation to steward this gargantuan mess of browser code that was Netscape Navigator. And then what happened was that Microsoft...

decided that this web thing was not particularly aligned with the strategic interests of Microsoft, right? Because this web thing was a new platform where you could have the same experience across different operating systems.

And so they largely declared that they had won and they put the browser in maintenance mode. But there were a lot of people who really saw the potential of what the web could be and what it already was with like Internet Explorer 5 and Internet Explorer 6, but that it could be so much more.

And a lot of these people, like Boris, were just drawn to, I want to make this experience better, and I want my browser to stop crashing. And so people started downloading this source code for Netscape Navigator and working on it. And gradually improving it and adding... new features like the ability to block pop-ups and the ability to have browser tabs rather than a new window for every website. And this sort of gradually turned into a really more useful browser than Internet Explorer was.

But it was still very much a suite, right? It had an email client in it. It had an address book. It even had an IRC client for all of the developers. It was kind of built into the browsers so they could talk to each other. And eventually...

a few, I think, relatively junior people, you know, just like some, you know, contributors, interns, you know, came along and decided to build a stripped down version of it. And that was originally, that was just a browser, the bare minimum of what you need for a browser.

And that originally they called it Phoenix, but then that was a BIOS. Then they renamed it Firebird, but that turned out to be a database. And then finally they called it Firefox. And Firefox just took off like wildfire because the...

world really wanted a better browser and it was a better browser and so Firefox became you know just this thing that everybody you know the first thing you do when you got a computer what you do for all your friends what you do for your family members is install Firefox because it's just such a better experience. And Google was extremely well-aligned with this, right? Because Google was this burgeoning product and business that was largely built around

a search engine that was about discoverability of the web. And so that the more that the web was discoverable and usable and useful, the better it was for Google. And Google quickly realized that Internet Explorer was a big problem and that Firefox was really this coming solution to this. And so Google was very, very invested in Firefox's success. When I started as an intern, we all received

badges to go over to the Google campus and eat lunch whenever we wanted. There were snacks in the kitchen that I heard, I never quite confirmed, were directly delivered by the Google facilities people. And of course, the most important was that Google signed a deal with Mozilla to be the default search engine for Firefox. And this was perfectly fine with Mozilla because everybody liked to use Google anyway and this was a great user experience and it also...

suddenly allowed Mozilla to be financially sustainable and to hire a lot of these contributors who'd basically been working nights and weekends. And Google had engineers that were directly working on Firefox and contributing to it. And there was really this sense that Google and Mozilla together were working to unlock the future of the open web. And when I started in June of 2008, that was sort of the...

peak of that sentiment. But shortly thereafter, Google announced that they were building their own browser, which was Google Chrome. And I think that you can obviously see why in retrospect, right? The real, the business strategic... posture of Google as a company was really hinging on this ragtag group of very weird people who were working on this open source browser. And there was a desire to have a little bit more control over their own destiny.

And so when they initially announced Chrome, you know, we had a meeting, John Lilly told everybody who worked at Mozilla about this, but indicated that it was still very friendly. The desire was to take market share away from Internet Explorer. And that we would still continue to be close partners and on the same side. And initially, it really did feel that way. But as time went on, things started to change. And I think it was like 2009.

There was a New York Times article that I think really captured the zeitgeist. I think the title of it was something like, between Mozilla and Google, group hugs are getting tricky. And that referred to this very positive... hippie atmosphere of working together on the future of the open web. And the degree to which Google's more direct business interest in owning and controlling the user's journey on the web was starting to create tension.

You mentioned that at some point you had Google badges and you had free food through Google and you had a good working relationship with them. How did it transition from that to a full... competition and your personal experience so my sense is that as time went on things changed slowly and i really don't think that it's a situation where any individual person at google

was like, ha ha, now we are going to get them and they don't know. But it's just the way that organizations work. And so we continue to have strong collaboration with many engineers over at Google. But there were two things that really happened at the same time. The first is that Google really ramped up its attempt to acquire market share with Chrome. And this was, you know...

Probably very truthfully aimed at Internet Explorer, but Mozilla was very much collateral damage. And they were spending hundreds of millions of dollars a year on advertising and distribution. And if you were to include the effective cost of the advertising that they put on Google.com, which was their own homepage, the advertising spend, I think, would reach into the billions.

And there was very much a sense that whenever we talked to them, that it wasn't supposed to be directed at us. Jonathan Nightingale, who was a VP of Firefox at the time, you know, a couple of years ago, he did a tweet thread. explaining his point of view on this, and I think he referred to it as the year of 100 oopses, where every time Google would do something that would hurt Firefox, it would always be framed as unintentional.

right and the obvious way this would be is that like firefox was sending source traffic over to google and so when users were in firefox they typed something in the address bar they would end up on google And then Google would show these Firefox users an ad that would say, or a big banner directly on google.com, like not even on the search end result page, saying, download Chrome, it's better. And they always claimed that it was unintentional, but these things just kept happening over and over.

And I do think that on a sort of organizational strategic level, it was unintentional. But the individual incentives of the people involved and what they got by doing this and then, you know, maybe fixing it in a release a couple weeks later. you know, they still won as far as their metrics were concerned. So that was one piece. But there was definitely another piece where Firefox had legitimately fallen behind.

Part of this was that when Chrome came out, it had a lot of really interesting and new architectural innovations that created gaps with Firefox. Some of these included having a multiprocess architecture, and they had a plugin architecture for things like Adobe Flash that was much more robust against ability issues.

And they had a very fast JavaScript engine. Firefox also had a fast JavaScript engine. And we traded back and forth, you know, taking the performance crown. But they had some real structural advantages. And at the same time, Mozilla made... a big and ambitious bet in the early 2010s to try to build a mobile phone operating system because there was this sense that mobile is coming and both Android and iOS are potentially hostile platforms that are not going to allow

Firefox to do what it did on desktop on their platforms. And the belief was that the only possibility was to create a new mobile operating system that was based on web technology. And there was something to that idea. and there was quite a lot of interest from the industry, a very surprising amount of interest, and we had a lot of partners, but ultimately it didn't work. And the big bet that Mozilla made on Firefox OS created a lot of resource competition with

the work that needed to happen to keep Firefox competitive exactly at the time when it was needed the most. And so Firefox really had a lot of gaps with respect to Chrome. I think the multiprocess architecture was a big one. The stability issues. particularly coming from Adobe Flash, which was widely used particularly for streaming video, which was becoming a thing. That was another big gap. And Firefox also had some legacy baggage around its browser extension API.

where the way that you would make a browser extension, the whole concept of browser extension really arose from the Firefox architecture where you could just inject anything anywhere. And the Firefox application was written in...

JavaScript and markup and you could just modify it with an extension by just adding more JavaScript and more markup and more CSS. And this was really cool and that's where people first got the notion that you could configure and change your browser and extend it with new functionality.

But the problem was that there were no well-defined interfaces. And so all of these extensions were depending on all of these internal details of how the browser worked. And anytime we changed anything, we would break an extension. And so that was a very real and significant drag with respect to trying to architecturally modernize the browser. You mentioned prior to Google's funding, this was a group of...

A bunch of people that were working for free in many cases. I'm kind of curious because I haven't done a lot of open source stuff. What is the incentive structure? Why would people spend so much time working for free on a project like this? So I think it probably varies. But the world was quite different back then. And open source, there were not, first of all, a lot of opportunities for people to contribute to something major and meaningful.

Right. Like these days, much of the software stack that everybody uses, many projects are open source. But there were not so many of those back at the time. And so I think that it was an interesting opportunity to have impact. And I think for a lot of people, like I mentioned Boris.

He literally got involved in Mozilla because he had no way to browse the Internet in his dorm room. One option was he was like at MIT at the time, I think. And, you know, he could either VNC into some like Solaris thing with Internet Explorer 5.

Or he could use this build of Netscape that he downloaded that was just constantly crashing. And so he got involved just because he wanted to fix something for himself. And he had this vision of how he wanted it to be better for him. And then he realized...

that there were so many other people out there that were also looking for the same things and trying to fix this stuff. And once you dive into a code base like this, it is just an amazing intellectual challenge because the code base was just so vast and so complicated. And so he was like a... math major, right? But this was in some ways like even harder. And I think that that challenge was another major aspect.

But I think the biggest aspect was the motivation of the Mozilla mission and what we were trying to achieve. Because the... I think a lot of principles that people take for granted about, you know, how the web should work, that it should be open and should be interoperable, that it shouldn't be controlled by one company. And even that that about the Internet itself, right? You know, the Mozilla mission.

is about making sure that the internet is a global public resource that is open and accessible to all. And I think if you grew up in the 90s when Microsoft was eating the world, that was not a given. That was not the dominant way of thinking about how technology platforms should work. And Mozilla had a vision of how it could be different. And it had a means of executing that vision, right? It wasn't just an advocacy organization arguing that tech companies should do something different.

it was actually going out and doing something and succeeding and creating a browser that gave users choice to block pop-ups and control over their experience and the ability to block ads and the ability to have tab browsing and all these things that they weren't getting before and creating that reality and making the world better in that way i think was very inspiring to a lot of people yeah i'm kind of curious like you know once you got there i want to hear a little bit about the

projects that you're working on and the various stories of things that happened in the early days. Are there any projects that you felt were particularly interesting or contributed to your career? Yeah, so I got started... working on all sorts of things. Like I think as an intern, the first thing that I worked on was the front end for the new mobile browser. But this was before there was Android and iOS. So it was actually for like Nokia Memo. And...

That's what I started on. Then I started working on the graphics engine. I worked on the image rendering library. And there was very much a sense back then, just because there was so much code that, you know, was largely unowned. that as soon as you show up and you start working on something and you fix one thing, you start looking around and you're like, hey, who owns this? And they're like,

You do now. And so I ended my first internship owning quite a lot of code in the Firefox browser. And that felt to me like a really big and important responsibility to make sure that this stuff went well. I think... I was particularly motivated to work on the really like the hardest and gnarliest parts of the code, the parts that felt most critical. When...

After Chrome came out and then Firefox was coming back with its first counterattack of trying to build a browser that was more performance competitive, particularly around JavaScript. So Chrome launched with this JavaScript engine called V8 that was... had a just-in-time compiler and it was very fast. And we were intending to ship our own just-in-time compiler for our JavaScript engine. But there was a lot of challenge at the end of actually getting this thing shipped.

particularly around this very complicated area of how the JavaScript engine talks to the rest of the system. You know, how the bindings between the JavaScript and the DOM, which were referred to as XP Connect. And so literally there were months of delay of getting Firefox 4 out the door because of all of the tangled web of that architecture. And I, you know, also with the advice of Damon at the time.

decided to dive in and try to help make that stuff better and so i ended up working on that code and the thing about that code was that it was very uh tied up in a lot of security considerations in the browser. When the browsers and the web were first designed, JavaScript was famously designed in 10 days, people did not fully think through all of the security implications of this architecture.

both in terms of the web platform and the browser itself. So Firefox as a browser, one of the cool things about it was that all of the application on the front end was also written in markup and JavaScript. But all of that relied... from a security perspective, on making sure that a website would be properly isolated from touching other websites and from touching the browser application itself. And this was all a very complicated mess of security enforcement.

in this DOM binding code. And it was not perfect. And there was this cat and mouse game between people working on Firefox and this external community of security researchers who... we're constantly finding new ways to poke holes in this right and you do some very very clever thing with you know function.bind and you end up slithering up the prototype chain and suddenly you're like in the address bar and tab code and once you do that you know the browser

and perhaps the computer compromised. And so there were a number of researchers. There was one, I think their name was Mozbug RA4. We never learned what their actual name was. Rumor had it that they worked in, they lived in a country where security research was illegal. But there was very much like a weekly back and forth between people at Mozilla working on Firefox and Moz Bogari 4 of new...

potential paths of vulnerability. So Firefox 4 definitely, I think, introduced a much more robust architecture. And that was when I got involved with some of the stuff. But... There started to be other and additional security researchers who showed up who were finding new ways to poke holes in it. And what really started to worry me was that there were...

certain categories of things that we didn't really have a way to fix in a robust manner. You know, we could put a band-aid over that particular part, we could wallpaper it over... But there were usually ways that you could apply the same tack in another part of the codebase. And I started to get pretty worried about this. And I launched a very large cross-functional effort that I spent, you know...

at least over a year on, which I referred to as slaughterhouse. And that was because, you know, it was a joke because there were these things called Chrome Object Wrappers that we needed to get rid of. But there was generally a sense that this could be a really big problem if...

these types of exploits were to be weaponized in the wild before we did something about it. And there was one particular researcher who I worked with quite a bit who was really good at identifying and finding these issues. And so what would happen was these external researchers, they would send us something, they would file it in a confidential bug in Bugzilla in our bug tracker.

And then we would look at it and be like, yeah, this is a real issue. And then we had a bounty program where we would pay them a small amount of money as a thank you and an incentive to bring it to us and not sell it on the black market. the black market certainly paid more and sometimes there would be zero day exploits where you'd have to sort of jump on some new thing and fix it and ship a fix within you know 24 hours to

make sure that users were protected. And sometimes these attacks got ahead of us. And so there was one situation where somebody flagged me that there was a zero-day exploit in the wild. that had been discovered, I think it was on MoscowTimes.com, and I was disassembling it. And then I had this realization that I knew who had written this exploit because it was one of the security researchers that we worked with.

And without thinking too hard, I fired off an email to this guy being like, hey, you're double crossing us, right? You're not supposed to sell these things if you report them to us. And... He got back to me, and the thing about security researchers is that they're extremely paranoid. He's like, no, no, no, I didn't. This must mean that my devices must be compromised at a very deep level. And the only thing I can think to do here is to go off the grid. And...

We then discovered, basically like an hour later, that actually it was Mozilla that had been compromised, that Bugzilla, our bug tracker, had been compromised by a threat actor. And I tried to write back to the researcher, but at that point he'd already gone offline. and we couldn't reach him. And so it was several weeks later that we got a phone call from a bowling alley in upstate New York where this researcher had gone into hiding and was trying to throw off the people that were after him.

And we managed to explain that actually, sorry, this was our bad and had to help him get back on his feet. But it was a really big problem that Bugzilla had been compromised because that is where you have the records. of all of the interesting security attacks that people have discovered against the browser. But the amazing thing was that Slaughterhouse had just wrapped up and that out of, I think, 45 of the...

I don't know, something on the order of 45 of the bugs that the threat actor had exfiltrated. We had fixed all but two of them. And with this sort of new architecture that basically fixed it. And so that really impressed upon me.

the importance of getting ahead of things on security and making sure that you have the right architecture, because sometimes you find yourself in a situation where you don't have the luxury of the time to fix it. How'd you know that that security researcher wrote that vulnerability? People have very particular styles. And in this particular area of security or of exploits, which is different than other kinds, there was just a lot of people's fingerprints on it, right? You could tell.

because these were things that were handwritten logic attacks where you are using very particular parts of like i'm going to take this api and then i'm going to bind this function to it and then i'm going to send this call like around async through the event loop twice and then i'm going to do this right like there was a very particular way and these were all handwritten there is

a different type of security attack that was also a major issue at the time, which were memory safety vulnerabilities. And these were much less of an artisanal thing and more things that you would produce by tools. particularly by fuzzing tools. And so this was also a major issue that Mozilla was grappling with because you would have people who would run these fuzzers against the code base and the code base was written in C++ and...

There are lots of ways to have memory hazards in C++. And really any one of them is the sort of like web page takes over your computer situation. And so we were really struggling. to find ways to deal with that in a robust manner and that was really one of the things that motivated mozilla's investment in rust so i could see and i'm just trying to understand here the the security issue here is

There's some front-end code that abuses or does some special path in Firefox and then can take over Firefox in that case in the past. But how does it take over the computer? Wouldn't the operating system kind of shield?

ideally from Firefox doing crazy stuff? So certainly not at the time. Like originally, particularly on Windows, applications largely ran with system privileges. And if you took over the... know, the classic thing that security researchers would do to prove that they'd pwned your machine was to launch the calculator.

like the Windows calculator, right? And so you'd run this exploit. You really have to trust the person who sent it to you that it wasn't going to do anything bad. But, you know, calc.exe, that was the proof. As time went on and... you know, Chrome had a major hand in improving the table stakes of how browsers worked, there were additional layers of protection that were put on, right? And so some of these include in-process sandboxing.

as well as operating system level sandboxing. And so the architecture that browsers like Firefox and Chrome have today involves a very, you know, defense in depth, multiple level of... multiple levels of protection, including operating system level sandboxing that is applied to things like the rendering and content processes. That said, this was all bolted on over time on Windows.

And Windows, unlike a platform like iOS, iOS is designed from the beginning with an application sandboxing model where applications are just applications and they have their own little world. That was not how Windows worked. And so evolving Windows and evolving the security critical applications that sit on top of Windows, like browsers, to deal with these sorts of attacks is honestly an ongoing challenge. Did you ever figure out how Bugzilla was compromised?

I think it was a credential leak from an employee like that. I'm 85% sure that that's what it was.

And that was, I think, before we had things like two-factor authentication for logging into Bugzilla. Bugzilla was a very old tool that dated back into the 90s. And so similarly, it was something that had to evolve with the threats over time. One thing that I was curious about is... as an organization how do you prioritize security because for instance at the time there was also these big investments in performance and i'm sure there's a bunch of other things

For sure, yeah. Security is one of those long-term things that is very easy to take a short-term perspective on and ignore. And that is perhaps a decision that some products will make. even where they are, but it's not something that you can do as a long-term browser. And so honestly, from my perspective, it's a cultural thing in Mozilla that...

From the very beginning, back before there was any sort of management structure and performance measurement and company objectives, it was just understood that we had an obligation to protect our users and to keep them safe. And I think that persists. That means that there's a lot of, I would say, ongoing bottom-up energy towards making sure that we improve security.

There's a lot of pride that people at Mozilla take in the security of Firefox because Firefox does not have, you know, I think that the Chrome security org, you know, on its own is like on the order of, you know, 70 plus people.

And we don't have anything close to that, but it is considered to be a shared responsibility. And we have a lot of people who are very dedicated to that and trying to think ahead, right? Like Slaughterhouse, the project that I led, that was not any kind of top-down edict of, we need to go fix this.

It was that I was responsible for the code, and I felt that this was important, that we needed to get it right. And one of the things that I'm very proud of at Mozilla today is that there's an annual security... competition called Pwn2Own where researchers show up and test their exploits and receive prizes for compromising the browsers. And generally speaking, they always find a way to compromise every browser, right? Like no browser is perfectly secure.

And Firefox, for the last two years, has won the award of being fastest to patch on the order of 24 hours. And this is only possible because we have people literally in time zones around the world. who are just working nonstop to get it out. And it's not something that we ask people to do. There's no management edict that somebody needs to stay up late or work on a weekend to do it. People are just motivated to do it because they care.

You mentioned no performance reviews and this scrappy bottoms up culture. How does that kind of culture work? I think in the world of prior to being much management, it was...

just uneven, I think, as you would guess, right? And I think that it was generally things worked well because the caliber and dedication was very high. But it was very much a bottoms-up culture, right? And if there was somebody who was owning that particular part of the code that wasn't doing a good job, everybody just kind of looked with that. And also, I would say, there was very much a notion of...

people having ownership of areas and that being somewhat final, right? Mozilla had a governance structure that was very much insulated against any sort of executive power. It was this module system and people were owners of the modules and... Even the initial access control system, I think, was called despot. And it was very much a notion of distributed and decentralized decision making. I think that that culture...

was very powerful. And there are still elements of that culture that I think are important to preserve today. But there are also ways in which it fell short and needed to change. One of the things that I have spent a lot of thought on over recent years is finding myself in a leadership position is how do we preserve those aspects of the technical culture that make it great while also addressing some of the shortcomings.

You mentioned a little bit about C++ and memory safety issues. That's a very classic thing. And I know that Rust was developed at Mozilla to tackle those problems in large part. So I'm curious. Can you tell me a little bit about the adoption of Rust and how it solves the problem? Right. So in the early 2010s, it was pretty clear that Mozilla and Firefox, the code, had some...

real structural issues. And a lot of these were issues that we could address through sheer hard work, right? This included building a multi-process architecture and evolving our extension architecture such that stuff, you know, didn't have its tendrils into all the undocumented interfaces within the browser and building native video playback.

so that we didn't have to rely on Adobe Flash anymore. These were all large but incremental projects to address specific architectural issues. But there were two architectural issues that were very difficult to address in that way. And the first one I already mentioned is memory safety.

Because when you have a code base that's written in C++, a large portion of the defects that you can make are the sort of defects that compromise the browser and, in the absence of sandboxing, compromise the computer. And that was just a very difficult thing to deal with in a non-whacamole fashion. And there's another piece, which was that particularly at the time, there was an increase.

in the number of cores that were shipping on devices. But browsers were still largely single-threaded. And this meant that you were leaving a lot of the available performance on the table if you didn't have multi-threaded code. But if you did have multithreaded code, especially in C++, you just had all sorts of very difficult, non-deterministic bugs that were very difficult to track down. And some of these also became memory safety issues that became exploitable defects. And...

There was this notion that both of these were rooted in deficiencies of the C++ programming language. And all browsers were written in C++. But, you know, in 2010, Firefox had just taken on the world and opened up the web. And there was a sense that we could do the impossible. And so we began an effort, you know, sponsoring some research that some people internally had around building a new programming language.

And this programming language was Rust. And the idea of Rust eventually became that we would have a programming language that had static guarantees at compile time. That number one, you didn't have memory safety issues. And number two, that you didn't have race conditions. And... This idea was that first we would build a new programming language with these properties, which was difficult to do. And then we would use this programming language to build a new browser engine for Firefox.

So the Rust project co-evolved with this project called Servo, which was a ongoing evolving testbed for the Rust programming language with the intention of building a browser.

that was memory safe, but more importantly was parallel. Because there was a sense that we could solve these architectural issues and close the gap with Chrome, but that also we needed something to leapfrog Chrome. And there was a belief that concurrency and safe concurrency powered by a new programming language was the way to do that yeah i want to talk about the competition with chrome at the time because you said they launched with a

more performant browser and also they were hammering you guys with distribution advantages. I'm curious, what was the strategy at that time from the Mozilla team to compete with Chrome? So the strategy to compete with Chrome was multifaceted. One of the parts was a recognition. I think there was a little bit of denial in the very early days about...

the deep value that some of Chrome's architectural advantages would provide. And this was, you know, they came out with a JavaScript engine with what's referred to as a method JIT. We had a tracing JIT. We thought the tracing JIT was good. And... was going to be better, but in the end it wasn't. Chrome came out with multi-process architecture. I remember the first thing that I heard was, yeah, good luck doing that on Mac. And then they figured out how to do it on Mac.

And there was a belief that some of these things were not possible until they were. And then once they were, that they weren't the most important thing. But eventually it became clear that things like a multiprocess architecture and things like native video playback. were really essential and we had this issue particularly on video playback where because chrome first had a better plugin architecture for adobe flash and then was also early to introduce native video playback they were able to have

a much more stable experience on YouTube, which was blowing up at the time. Whereas Firefox on YouTube, which was what everybody wanted to do on the internet, was just crashing left and right. And it wasn't stuff that we could fix. It was Adobe's fault. And... we really couldn't do anything about. And so we had to work on a lot of these really hard but discrete challenges of closing the gap with Chrome.

But then at the same time, there was a desire of you can't just close the gap. You have to do something new and you have to do something that is better. And Rust and Servo were our strategy for doing better. So you mentioned the strategy with Servo and the rewrite in Rust. How did it go? Was it successful? We know that Google's distribution eventually dominated, but how did the efforts with Servo go?

So on the one hand, Servo and Rust had, against all odds, captured something really real. And Servo, as a browser engine prototype, had a lot of really amazing stuff in it, right? It was not complete, but it did have the skeleton of certain features that had never been done before, including a multi-threaded CSS engine. But the problem was that...

We had a team of, you know, a dozen people working on Rust and Servo. Google had hundreds and hundreds and hundreds of engineers who were working on Chromium. We had less than that, but still most of Mozilla's efforts were directed. towards Gecko and Firefox because that was the product that we had in market. So we had this situation where in order to build a competitive browser engine, it's not like you need to just do this one time. It's the accumulation.

of all of the work over all of these years of hundreds of engineers working on improving it and improving the architecture, improving the performance, improving the features. And so every year there was more and more stuff that server would have to catch up with. And I think if you just look at it, it was very difficult to believe that we would ever end up in a situation where Servo would be competitive with a regular production browser engine.

And so circa like 2014, 2015, that was a situation where there was really a sense that we needed to deliver a fully refreshed version of Firefox that really got us back in the game. That included... this parallel CSS engine that we uplifted from Servo. And this was a project that I personally led and was really the highlight of my career working on it.

because it was so exciting to be doing something really new in a new programming language that, and every step of the way, there was this notion that, like, that's definitely never going to work, right? Like, how are you even going to hook this thing up? And then we hooked it up and we had it traversing the DOM over C++. We have a C++ DOM and we're having this Rust CSS engine and how you get these two things to work together.

There was no tooling for it. We had to build it all ourselves. At the end of the day, it was really a dramatic performance improvement. I think it improved rendering time on Amazon.com on the order of 25%, which is just enormous in terms of... web performance. And because it was parallel and it can run on all of your cores, it was and still is the fastest CSS engine in the market. Firefox really got back in the game and was

really performance competitive with everything else out there. And that was a really exciting moment for everybody, realizing that if we just coordinate and we do all the technical effort and we make the right bets, we can win. Okay, so you mentioned that this was one of the...

most proud moments of your career and like a major launch very exciting project i'm kind of curious to talk about the career parts of it so where were you in your career at this point and maybe you can talk about how it grew as you worked on these projects Sure. So I think when I started, I don't know if Mozilla had levels. I certainly wasn't aware of them if they existed. It was a culture of choose your own title.

The title that I chose for myself was negative entropy. That was sort of what I wanted to do on this incredibly complex and gnarly code base. As a fun fact, my favorite intern made his title entropy, but we were really on the same side. I think by the time we were shipping Firefox Quantum, Mozilla had developed, I would say, a more industry standard career progression. And so, you know, you had you come in as.

you know, just title of engineer, and then you've got senior staff, senior staff. At the time when I was working on Quantum CSS, I believe I was my... I had reached the level of senior staff. And upon the successful launch of Firefox Quantum, I and the engineer who led the other half of Quantum, Quantum Flow, were promoted to the level of principal engineer. And at the time that was, there were not very many principal engineers, you know, like probably fewer than five. And so that was pretty.

significant jump. But my long-term dream was to be like Boris Zabarski, and Boris Zabarski was one of Mozilla's three distinguished engineers. And so that was really what I had my sights on in terms of the level of impact that I wanted to have. And the way that I really aimed at having this impact was by trying to contribute to as many parts of the code as possible and just be prolific.

And so after working on quantum CSS, I went and worked on web render and I worked on mobile and multiprocess and layout and all sorts of different parts of the code. And I remember my manager at the time, Joe Hildebrand, who ran engineering, told me that he wanted me to spend one day a week when I didn't open my terminal. And I was just shocked.

I was like, what are you asking me to do here, right? You're asking me to not work for an entire day. And he was encouraging me to start thinking about it in the impact that I could have in a much broader way, as opposed to direct contributions to the code. And so this was something that I started to think about, but it was really not the natural way that I thought to work. When you talk about the individual contributions, because you mentioned that was your way of having a lot of impact.

I was very much motivated to work on whatever felt like the biggest problem. And so that was early on. That felt like... the DOM bindings and XP Connect and all the security stuff, and then at some point it was the media playback to solve the YouTube problem, and then it was multiprocess, and then it was this Rust and Servo CSS stuff, and after that...

I thought that WebRender, which was the new graphics I back-end that we were uplifting from Servo. So I went and worked on that. Then Mobile. So I was always drawn, I think, to what felt like the most pressing problem. And my model of how to have impact on that was to jump in and learn all of the code in the most pressing problem and help work on it. And not just necessarily typing all of the code myself. but working with the other leaders in those areas on figuring out what

the right things to do would be and reviewing code and setting direction and setting priorities. So there were aspects of leadership, but it was all very much oriented around the code that needed the most help. So you were prolific, but... It was a guided search specifically on the biggest problems for the organization. That was how I thought about it, yes. One thing that I'm curious about, because in the industry, when it comes to career growth, I often hear two...

methodologies one is where people they focus on the promo first and foremost and as a byproduct to get the promo they you know grow and level our skills and various other things behaviors And in the other school of thought, it's ignore the promos, but focus on the behaviors, focus on the skills, and then the promos will come. I'm curious your thought.

between the two methodologies and which one you think is better for structuring an engineering career? I have pretty strong opinions about this. I am very much of the mind that it is important to focus on. impact and not the title. And I think there are a number of reasons for this, but that is certainly what I look for now that I'm in a situation when I'm helping people with their careers and I am often in the decision of...

figuring out who's the right person to advance. And what I am looking for is I am looking for the impact. And my general philosophy is that if you are having that level of impact... you don't need to sell yourself in that way. Or sell yourself is not quite the right word, but there is a particular philosophy where you want to promo hack.

You want to pin your management chain down on what are the things that I need to do to receive this promotion. And that, just speaking frankly, is very annoying. There are aspects to this that I think are just intuitive to how I, at least as a person, perceive things. You know, like, I'm a father of kids, and when my daughter asks me, like, when I ask her to do something, and she's like, uh...

you know, will you give me this if I do that, right? It just does not land well with me. But I think that there's also a substantive aspect to it, which is that if you're really focused on the impact, that is how you're really going to achieve things.

Right. Whereas if you're focused on checking some boxes that somebody has presented to you, then you're in this awkward situation where it's like, well, I did technically say that if you did this, you know, that would maybe make a case for this next level.

but the way in which you did it and was very sort of focused on checking the box as opposed to having the impact. And at the end of the day, impact is what matters and impact is what needs to be recognized and rewarded. One thing that I hear often, because I think this is a... failure mode for engineers that are a little more soft-spoken is they're great at writing code and landing it and having impact but they're a little

weaker on the you know selling yourself or kind of marketing your work and so that kind of bites people and so sometimes i hear career advice that is focused on you know hey make sure you create a brag document and you share the wins when you land the thing and and i think a lot of those people don't have an ulterior motive of let me you know

kind of connive my way into the promo um but i think there is this this real thing of promos are decided by people and there's some value and i guess a balance because obviously if you were completely conniving you say hey i'm gonna talk to my manager and sell my work to him that's kind of not ideal but you know making posts or sharing it and thinking about the levels and stuff seems to be

a good way for those a little quieter or less proactive engineers in structuring their careers. So, you know, how would you kind of balance those two frames of thinking? I think there are often multiple ways to frame. similar activities, but that the framing of how you think about it is going to have a big impact on how you carry it out and therefore what it actually achieves. And so I think a good example...

of this is this notion that you brought up selling yourself or brag documents or something like that. I think it is important that people have visibility and visibility around the work that they're doing. And this is just a very like a basic fact about impact, because if you want to have impact, you need to be connected to people who understand what are the biggest challenges and you need to be.

coordinated and you also need to make sure that people understand what you are capable of such that they can put you in a position of having more impact. But if you think of it as I am trying to brag more about me versus I am trying to communicate and coordinate and create visibility for work that I think that is important, you will have very different outcomes.

Yeah, it's interesting because it's subtle. I feel like you could have two people with different incentives. One goes into it saying, I'm going to create this update so I get visibility. And the other person comes in to say, I'm going to create this update so... The right people know the right things. And they get to the same path at the end of the day. But it's like this subtle... It's worded slightly differently. And people can tell. A lot of it is the degree to which...

how much you make it about you versus how much you make it about other people and the work and the team. And everybody knows that the person sending the update likely had a significant hand in the work. And a very important thing to do as part of those updates is to make sure that you give visibility to other people and to thank the team and not make it so much about you. Because also that is a way to work.

well with people, to have people feel like you are a humble leader and somebody that they want to work with. And that is an important part of leadership. making sure that you create recognition for other people's contributions and don't make it all about you. It is an ideal world that people have impact and they get recognized exactly as much as their impact.

one thing that i can imagine happening though is you know there's that idiom that the squeaky wheel gets the grease basically if you had a loud person that is really vying for their promo and you know making all aggressively trying to make the posts and you know constantly bothering their manager hey i want to get promoted that oftentimes assuming they have impact if they don't have impact then it's not gonna nothing's gonna happen

but assuming if they're also a high performer and they're doing that compared to the high performer that's you know reasonable quiet and solid not selfish that actually that loud person, unfortunately, I think in most organizations will get rewarded first or faster. What do you think about that? I think it does depend on the organization and it depends on the technical culture.

You know, one of the things that we try really hard to do in our culture is to not reward that. But I think there is a balance to be struck because my message is not don't be ambitious, but rather. Think about your ambition in terms of the impact that you want to have as opposed to the status you want to achieve. And if you have the impact and create visibility around the impact, the rest will follow.

And yes, there are edge cases and there are probably some situations where people lobbying for themselves will kind of grudgingly get themselves a little more consideration just because it's top of mind.

But the best kinds of conversations that I like to have with people are not conversations where people come and ask, how do I get promoted? It is how do I grow my career and have more impact, right? Like, am I working on the most important thing? Like, what could I do to grow? What could I do to have more impact?

Right. And I think that those are important conversations to have with your leadership to make sure that you're focused on the right things, because the worst and saddest thing is when somebody is just working so hard at something that is just not that important.

and not going to have the type of impact that they ultimately want to do. And they might do a great job at it, but it's just not the right thing. And so I do think that some coordination of like, this is the type of impact that I want to have, and not everybody wants to have that kind of impact.

and making it known that you want to have that impact and you're looking for swings at the bat and you're looking for opportunities to do more, I think that that is helpful. But trying to pin people down of like, what are the boxes that I can check? And... you know i did this like are you good to promote me now is just honestly extremely annoying from my perspective uh and i just think zooming out at a level of life you don't want

the people in leadership to find you to be annoying. So in your case, you were focusing on impact. You had these role models of these DEs that you wanted to be like. And I'm kind of curious what...

eventually did the growth look like and how did your impact grow? I was really focused on continuing to work on like what was the hardest and gnarliest challenge in the browser and moving on from team to team. And that was really the model that I had in my head. And I did have in my head this idea that I eventually wanted to have the level of impact that Boris had. And I assumed that that would eventually mean becoming a distinguished engineer. And then at some point...

The CTO at the time, Eric Rescorla, sat me down because at this point, you know, Boris and David and this guy, Robert O'Callaghan, had been distinguished engineers kind of since the beginning, you know, since maybe like 2010 or something. And we hadn't made any new ones since.

And the CTO sat me down and he said, look, I want to let you know that for the first time in almost a decade, we're going to make some new distinguished engineers. And I thought you should know that you're not going to be one of them. And this was... I think a little bit surprising to me because I do think that you know I thought and that I think anybody at the time would say that at that time like probably maybe not quite Boris level but that

In terms of knowledge of all of the Firefox code, that was something that I had more than almost anyone else. But what he explained to me was that that was a more traditional model. of distinguished engineer inside of Mozilla, but that he was looking for something different now, and that he was looking for something that was more oriented around industry impact. And that was very much...

how he saw the world. The definition of impact seems to have shifted while you were thinking about your DE journey. Did you ever debate the definition of impact or what was your thinking?

when you saw that the criteria was updated honestly it was just kind of inspiring for me i think there was you know some people were concerned that i would be upset but i think it was more like you know you work out and you think that you're really strong and you gotta go do some pilates move or something you're like oh my gosh like this muscle that i have i didn't even know that i have but it is extremely weak um

And the truth is, is that there are lots of different ways to define DE and it's what matters for an organization, right? The thing that mattered early on in Mozilla and what was the make or break for Firefox's success. was just the management of this incredibly complex and sprawling codebase by very few people. And the thing that was needed most was a prolific contributor and enabler of that.

And that was the archetype of distinguished engineer at Mozilla. There are other companies where distinguished engineer means something else, right? I know that some companies generally it means you built a billion dollar business, but that is not. what Mozilla is about. And what Mozilla is about is changing the world. And so the criteria for DE are really focused around that. And this was, I think, an inspiring thing for me. And I still think that I, you know...

didn't ultimately, you know, I still probably have not had the level of industry impact of some of our other distinguished engineers. And so probably in the end, when I got to DE, it was maybe a combination or a hybrid of the old and the new definitions. But understanding that model of thinking about the world and that model of impact, I think was very profound for me and really oriented how I directed my efforts.

I'd love to hear some examples of what a DE level of impact looks like. Can you talk about some of the people who have done really impressive engineering work? Sure. Yeah. So at the time...

There were, you know, when we had this conversation, there were three new distinguished engineers that were promoted. So the first one was Martin Thompson. And Martin still is at Mozilla today. And Martin is an international standards extraordinaire in terms of the development of the technical standards that power the internet. And so his name is... you know, in the author line of protocols like HTTP2 and QUIC and WebPush, and he was heavily involved in the creation of WebRTC.

and geolocation. He authored other protocols like oblivious HTTP. Like, I don't know. I'm honestly not super familiar, but it wouldn't surprise me if there's really nobody else who has had that level of prolific impact. on all of the core technical standards of the internet. And this isn't just about doing some editorial work on making sure that all of the drafts are formatted correctly.

This is about core design challenges of making these protocols better in a way that advances Mozilla's objectives. And so similar to the situation around TLS, there were generally problems around the... privacy and security and performance of the legacy HTTP protocols. And by designing new ones that made it faster, while at the same time making it more private and having less metadata leaked.

and having fewer security issues meant that there was a business incentive for the entire industry to move onto these protocols because they were faster. And by designing them that way, he managed to really increase the privacy and security of the internet. That was one archetype. The second was Luke Wagner. Luke was many people... Luke was probably, if there was anyone in the world, the most single-handedly responsible.

for the development of web assembly so historically people wanted to run code in browsers that was not written in javascript maybe you wanted to run autocad and They didn't want to rewrite the entire codebase in JS. And the original way of doing that was through plugins, where there was this basically native binary blob that got shunted into the browser window.

and it was awful had lots of terrible security properties and bad user experience and so we tried to move away from that and the first idea was actually from there was an engineer who worked on a side project alone sky who made this

tool at Mozilla called MScripten. And the idea of MScripten was we just take the C++ and then we compile it to some very naive JavaScript and then run the JavaScript, which technically worked, but it was extremely slow. And Luke was one of the lead engineers of the spider monkey javascript engine and firefox at the time and he wanted to make this something that was actually viable so the first thing they did was there was some back and forth of well how do we optimize the javascript engine for

what mscripten is emitting. And that was obviously not a sustainable approach. But then they had this idea of asm.js. And the idea was that mscripten would output a subset of JS, which would be detected by the JavaScript engine. And the JS engine would then verify that it was from this restricted subset, which was sort of a proto bytecode format.

and then run it in a more natively unoptimized way. But, and so this was what got the flywheel going of people actually being able to say, hey, we can deploy these amazing native. experiences that are compiled and they're running in the browser because the ultimate goal was that we wanted basically everything on the desktop to be running in the browser or most things on the desktop to be running in the browser and there was some stuff that just couldn't and

That flywheel then created enough industry momentum to say, why don't we actually create a new bytecode format, which was WebAssembly, where you could have code that was native, but that was also sandboxed and portable. and running inside the browser. And this was at a time when Google was pushing a very different technology that was more akin to plugins. They had this thing called Native Client or NaCl, and then they had this thing called Pinnacle.

And there were a lot of very complicated industry dynamics of getting all of the browsers to get on board with this WebAssembly train. And so Luke managed the technical pieces and the political pieces to eventually forge a browser consensus on WebAssembly.

which has been a very transformative technology in terms of what browsers can do and even what can be done outside of browsers because WebAssembly was built with such great technical properties. It is now being used all over the place on the back end, things like serverless. This is a bunch of the work that the Bytecode Alliance is involved in. So that is another archetype of impact of the creation of WebAssembly. The third distinguished engineer at the time was Tim Terabury.

who was our codec lead. And so one major problem that Mozilla and honestly the web had back in the early days was that all of the video playback happened through proprietary plugins like Adobe Flash. And you would think that the obvious answer, which is the obvious answer, is to build native video playback into the browser.

But the problem was that the formats that are used to compress video are heavily patent encumbered. And there are lots of reasons for this, but it's just the way the world works, where in this particular space, everybody is fighting over patents. And... This meant that an open source browser like Mozilla Firefox couldn't ship native support for the H.264 codec, which was...

really what was powering most of the video on the internet at the time. It was the most efficient codec, and that was what YouTube was using. And this was a pretty significant problem for us. And so Mozilla worked over a number of years to... develop truly open and free and non-patent encumbered and royalty free codecs. And the team led by Tim first developed an audio codec called Opus.

And then they worked to develop a video codec called Dalla. And Dalla was then incorporated into AV1, which became an industry standard video format. that is now developed by the Alliance for Open Media, which is a consortium across many companies like Mozilla and Google and Apple and Netflix. And so this is basically what powers video today.

a lot of that vision around what would it mean to actually develop truly patent encumbered video codecs came from tip. So those I think are three archetypes of the types of world-changing impact that you can have.

as an engineering i see at a place like muzulam yeah the the achievement of these engineers is absurd i mean you mentioned lab assembly i mean i think of a business like figma as possible because of a lot of that work um and that's a tens of billions of dollars industry or i mean a company just by its public market cap and so it

the the value that's being created by these distinction is absurd what's interesting to me is i guess because it's a more open source company the value capture is very different I'm curious, is that something that people at Mozilla are ever not feeling great about? Because they've created so much value, but the capture is happening by these capitalist other companies.

Yeah, I think from Mozilla's perspective, we don't have shareholders, right? We're not trying to make gobs and gobs of money. The Mozilla mission and what we are here to do at the end of the day is to... create a better internet and one that works for people and that's open accessible for all right and sure it is very convenient for big tech companies to have a royalty-free codec for video streaming so they don't have to pay anybody but they totally could pay

right? What Mozilla's angle here is to make these formats and make these changes in a way that really empowers everybody. And obviously that empower, that creates economic value for everybody. And some people capture that more than others.

I think from Mozilla's perspective, we want to be able to continue to exist. And that does mean that we have to think about revenue. And at the moment, you know, a lot of Mozilla's revenue comes from one of the dominant advertising players on the internet, which is Google.

And so we would like to change that, but it's hard. And it's particularly hard because it's not necessarily where our heart is, right? Our heart is around having impact on the internet and making that better. And so it is a tension that I think... does exist within Mozilla of how much to focus on one and how much to focus on the other. But I think that Mozilla at its best, we have some necessary business stuff that we have to manage, but...

The end goal of what Mozilla is about is having this impact that benefits everyone. So then after these examples of distinguished engineers, I'm curious what advice you would give to someone who wants to become a distinguished engineer.

In terms of one thing that I think that basically everybody can work on at whatever level they are is writing. This is a particular hobby horse for me because I think that this was an area where... Mozilla and myself included, did not have a lot of developed capability. And I think it is one of the most powerful things. We have this internal memo, which is actually, you know, public on the internet.

directed at Mozilla engineers called Writing Things Down. And one of the things that this memo does is it makes the case that writing is both about the artifact that you produce, which is useful for coordination and wide review and getting feedback. but also useful in the process itself. And that is because the process of writing is the process of thinking and refining your ideas and articulating them.

This is just an infinite game that you can always get better at. And I think it is one of the most powerful ways to create clarity of thinking for yourself, because clarity of thinking is, in my view, one of the most important things. And... You only can really get that by forcing yourself to write down your ideas. And so I suppose one, you know, pertinent or timely piece of advice is if you care about this, do not use an LLM.

to produce text for YouTube. Because first of all, if you're putting your name on this thing, at least today, the LLM's like, it's pretty easy to tell when somebody does that. But more importantly, you are losing the opportunity. to do the thinking yourself. And that thinking that happens when you write is one of the most important types of thinking for growth. Do you have tips on how to write well? Yes.

I would say one of the most important things is that it doesn't need to be any longer than what you have to say. I think I always prefer... shorter documents. And I think there's some notion where people run out of things to say. And at that point, you either need to refine your ideas more. Or you're just done. And it's never useful to anybody to add fluff. And dealing with documents that are fluffy and overly overwrought is one of my least favorite things. So...

I think that's one piece. Related to that, I think that it's important to focus on substance rather than rhetoric, which is sort of related to fluff. But I think that some people say, oh, well, I'm not... a native English speaker and therefore I'm never going to be able to write as beautifully as others. And I think that's the wrong way to think about it. Like there is some basic baseline that you need to work on in terms of articulation of the ideas.

But it is not about using the most flowery words. It is about clarity of thought. And clarity of thought, once you have it, is relatively easy to express in written form if you practice. Yeah, I think every single person I've ever asked this question to has always said, be concise. It's shocking to me how unanimous that advice is. Okay, so going back to the browser wars, I think...

We all know how things have played out. And I see that there's this new era of browser wars that things are heating up again. There's a lot of competition for...

an ai browser or bringing ai into existing browsers and i'm kind of curious for your take on what the future of the browser wars is going to look like as the cto of firefox so i think one thing that's really worth clearing up is that you hear that there's, every month you hear about a new browser from this AI company or that. And what I think is really important to understand is that these are not...

browsers as I think of browsers. These are different front ends, different window dressing and monetization keys on top of Chrome. Because... much of what powers a browser is the browser engine and browser engines are also extremely complicated and difficult to build and there have been fewer and fewer browser engines as time goes on and today firefox is the only

independent browser that has no dependency on a browser engine from somebody else. So you've got Safari which is pretty vertically integrated with WebKit. Nobody else really uses WebKit directly. Then you've got Firefox where we build Firefox with Gecko underneath it. And then you've got Chrome on top of Chromium and all these other browsers like Opera and Perplexity and even Edge, they all use Chromium under the hood. Because what they're seeking to do is not necessarily to evolve.

the web platform and the web experience, they are trying to control the touchpoint with the consumer in this world that is coming with AI. Because on desktop... The way consumers experience the internet is through their browser and everybody is jockeying over what this experience is going to look like and people want to control that experience. So I am very much of the opinion that...

AI is coming to consumer technology like water running downhill. And that this is not something that is going to be stopped or reversed. because there is so much disruptive potential of the technology. But where that water goes and how it ends up, it can end up over here in somebody's reservoir, or it can end up in the ocean, or it can end up as a river.

there are lots of opportunities now to steer how it plays out and there is also a lot of desire for people to capture it for themselves and chrome as an example is making a very aggressive play to control the AI stack by leveraging its dominance in Chrome. And they're doing this at multiple levels. They're obviously doing it as part of the Chrome browser, where there's 5000 different integrations with Gemini as part of Chrome now.

But they're also doing it as part of the web platform. They are advancing some proprietary APIs that we do not support to allow websites to have local... to access to an LLM inside of the browser. And this is attractive to web authors because it allows you to offload your compute and you don't have to pay for the inference and you can just ask the browser to do the inference. But the thing that is different about this from other web standards is that

there is no way to specify an LLM or to have it have deterministic behavior that is interoperable. And so the API calls that you have into this LLM are fundamentally unspecified. But as the owner of the world's dominant web browser, they have a lot of market power to get websites to tune their usage of the LLM. to what Gemini expects. And there is obviously a co-evolution of AI experiences and the models underneath them and creating a world where more and more AI experiences

are built on top of Gemini is part of Google's strategy to achieve and dominate control of the space. And we at Mozilla are not building our own foundation model. We don't have any particular horse in the race. But what we are trying to do is to make sure that the space remains open in the way that the web has been open and that users have choice of how they integrate with these things and which ones they use. And also that the web remains important and relevant.

Because there is a big risk that the web becomes a backend detail that's used by RAG from your chat GPT query or that's used by operator on the backend, where it is no longer the focal point of the user's experience. But if this were to happen... there would be much that would be lost because the web is a very unique system in the sense that it provides users agency and control that they would not necessarily achieve elsewhere, right? It is not a direct experience.

that is produced and delivered in a way that can't be modified. It is reinterpretable content that the browser can modify and render as the user chooses.

right and there are lots of ways that this matters in practice you know things like ad blockers are a great example of how the reinterpretable nature of the web allows users to control their experiences in important ways and that the web is a safe environment where it's safe to open a link even from an untrusted source and that all of this is not created and controlled by any one company but that it is built as an interoperable system where

Anybody who produces code that implements the right protocols can show up and be part of the system. And this leads to a multi-stakeholder environment where no single entity controls the evolution of the web platform. And that is not how most platforms happen, right? It's not how the platforms happened before the web, and it's not how, you know, the native mobile platforms happened after the web. It is, in some sense, a very historical accident that we got something that is this.

well aligned with the vision that Mozilla has for how the internet should work. And it's very important to us to make sure that the changes that happen with AI do not sacrifice these properties. You mentioned all the components of the browser and there's the web engine and you know a lot of those ai browsers and um like and also other ones like opera etc they're actually built on chromium at the end

i'm curious what what do you think would happen if firefox was built on chromium like what is the advantage of firefox building its own uh web engine so it's a lot of work and we like doing hard things. But the strategic advantage from the perspective of Mozilla's mission is that it gives Mozilla a seat at the table when web standards and the future of the web is decided. And...

It used to be the case that Standard's bodies were wide groups of many different vendors. You had Opera back in there with its own engine. You had Internet Explorer. You had Safari. You had Mozilla. And as time has gone on... engines have dropped off the map because if you look at it from a bottom line perspective, there's basically no justification for doing it. It is a giant cost center from a business perspective. But from a mission perspective, the...

benefits of being part of the conversations that shape the web are profound. And that is why Mozilla is the last independent player that is doing it, because we are not driven by that bottom line. We're driven by the impact. And so imagine... if there was a world where there was no Firefox at the standards table, you would have Apple running WebKit, and you would have Google in charge of everything else. And Apple and Google, very...

Capable and innovative companies do not get along particularly well and are very entrenched in their own interests. And in a bilateral environment, as you can even see in the mobile space. there is a sense of entrenchment that can prevent progress. And one unique thing that Mozilla brings to the table, even though we have relatively small market share compared to these other browsers,

is that we bring a fresh perspective and the ability to tilt the conversation back towards what is best for the user and what is best for the web. One example that I'm particularly proud of is not necessarily one of web standards discussions per se. These happen all the time in working groups all over. The editor of the HTML specification is a Mozilla employee who works with everybody.

continuing to refine HTML. But the example that I would go to is actually one about performance. So it's long been the case that competition is what drives the advancement of browsers getting faster. And it is just this amazing achievement that browsers continually get faster, right? It's like the semiconductor supply chain. It's something that people don't even think about. And just like every year, the stuff underneath that is powering all the experiences is getting faster and faster.

And the thing, if you look at the history, the thing that makes browsers get faster is competition from other browsers. And the way this has historically been measured is that different browsers would produce a benchmark that they would design in a way that was...

you know, particularly well-suited for their engine, and then they would optimize it. And then other browsers would say, well, yes, that's your private Cook benchmark. And there was an initiative a couple years ago around this benchmark called Speedometer 3. to try and produce a shared benchmark that we could all agree on that would be representative not of just what worked well in any particular browser, but would be representative of the web and the experiences that we wanted to get faster.

And this is a type of engagement where if you just have the bilateral relationship between Google and Apple, it would predictably go nowhere because there are too many opportunities. for somebody to tilt it in their own interests in a way that the other side says, aha, I see what you're doing there. But I think that Mozilla's involvement was really critical in keeping everybody focused on the stated mission of...

the operation which was not to make any particular browser look good but to provide us a shared incentive to make browser engines faster together in order to make the web faster and in firefox the Impact has been pretty incredible of what has been driven by this benchmark over the last couple years. I think that really across all of the browsers, at least Firefox and Chrome, both Firefox and Chrome have improved their score on the benchmark by a factor of two.

So I've gotten twice as fast. And we have actually measured in Firefox the impact that this is translated to on real-world performance on the same machines in terms of page load and... time to first paint and all these things. And it's on the order of 30%. So I think it is reasonable to say that having this benchmark has made the web 30% faster. And that is something that benefits everyone. And I don't think it's something that would have happened without Mozilla in the room.

Coming to the end of the interview, I just want to ask you a few questions kind of reflecting over your career because I feel like you have a lot of experience. When you look back on your career and you made a lot of contributions, you were an IC through and through of her.

almost all of your time at Mozilla. I'm curious, is there any technical decision that in hindsight you wish you'd change? One thing that comes to mind is a situation where I think the outcome... The technical outcome was good, but I think in some sense the process wasn't worth it. And this was around WebRender, which was the graphics backend in Servo. When I was working on...

Quantum CSS to take the Servo CSS engine and put it in Firefox, I was very excited about the momentum of uplifting other parts of the rendering pipeline of Servo into Firefox. And I advocated very heavily. that the graphics team should replace its rendering and rasterization pipeline with the one from Servo. And this was not based on technical expertise on my part around it.

it was very much a vibes thing of this is where it's going and that vibes thing was infectious and i was not necessarily strictly in a decision-making point at this time but there were I think that point of view really caught on with management and I remember the basic issue was that the team, many engineers on the team, had very strong doubts about this. They thought that it was not ready, that it was too much.

that it was a very risky project. And they tried to express some of these concerns. And there was an engineering leader at the time who heard them and responded with C++ is not the future. And that vibes-based enthusiasm for a risky technical direction over the concerns of the people who knew it the best. ended up being very toxic and a number of our most senior and most impactful graphics engineers ended up leaving. Now, the rest of the team ended up rallying and WebRender did turn out to be...

a major and difficult and expensive and many-year project. The actual technical result is awesome. It is a fantastic engine and it is really performant and safe and does all these great things and I'm really proud of the team for what they built. But I think that pushing it on the team when the team wasn't ready and wasn't aligned on it had collateral damage that I regret. Is there a top book that made an impact on your career?

I would consider myself to be an avid reader. But the truth is, I don't read a lot of books. I do read some, but honestly... You know, the books that I can think of that probably have had the most profound impact on my thinking are not books that I've actually read. And that's just a personal style of mine. I tend to be very wide in terms of what I consume.

reading a whole book on a topic is a depth-first approach. And I certainly wouldn't want anybody to take this as encouragement not to read books. And I think that there's a lot of value that can be gained from that, and it's something that... many people do and do very successfully. But it's not the only way. I think, you know, there was this book that I just read a book review of called Good Strategy, Bad Strategy. And I just read three pages.

And that has just stuck with me as a description of what is wrong with many strategy documents that I read. Just a very clear and crisp articulation of the model that gets created and what the right thing looks like. And that's something I never actually read the book. I just read a three-page summary. Other, I think, ideas that have had a profound impact on me from books.

His book, Turn the Ship Around, which I also haven't read, which is about empowering teams on the subject we just discussed. Innovator's Dilemma, Clayton Christensen. I think it's very powerful analysis of how the industry works. I definitely do like to consume... content from a lot of different areas. I really encourage people to look wide, not just within tech. Within tech...

You know, I do really appreciate Ben Thompson's writing, I think is really good for understanding the tech industry. But I like to also spend a lot of time consuming writing from other intellectual disciplines. One example is I've found that the whole geopolitics, national security space is an area where there's actually a lot of thoughtful writing.

that I had never been exposed to. And there are people who basically spend their lives really thinking about what happens on a battlefield and what goes well and what doesn't. And a lot of those ideas actually translate very similarly or you know it certainly rhymes with situations um you know when battles between big tech and how do you manage and motivate teams and

So I would definitely encourage people to take a wide view of the thought that is out there because there are many things that can be learned from things that are outside of just tech and business. Last question for you is if you could give yourself some advice when you just entered the industry, is there anything you'd say knowing everything you know now? At the end of the day, to be humble is the most important.

I think that I had a particular attitude not strictly about myself, but about engineering and the role of engineers and how important engineers were. And there was definitely this attitude of that was the center of the universe. And as I've gone on, I've started to realize that there are many other important things that go and that you don't necessarily understand when you're first starting out at an organization of...

why there are certain processes and why there are certain functions and what do we need a legal team for and all these kinds of things. And as I've progressed and found myself more in a leadership role, I have really managed to develop an appreciation and empathy for all of the different players within the organization and within the industry and realizing that

everybody has a point of view and a place that they're coming from and that working to understand where people are coming from is the way to make progress. All right. Well, thank you, Bobby, for your time. I really appreciate it. With that being said, is there any last words you want to say for the audience? Yeah, I think the most important message I want to send is that the web very much matters.

And the web is something that underpins so much of what we do. And I think for a lot of people, it is invisible in something that they take for granted, right? It is like water flowing through the taps or the semiconductor supply chain. of this amazing resource that continues to get better over time and continues to deliver us more value that we take for granted. But it doesn't have to be that way and it easily could not and we could easily backslide into a situation

of unilateral control by big tech over the experiences that people have. And I think that simply losing focus on something because we've had it so good for so long would be a tragedy. And I encourage people to think about what they gain from the existence of an open and free web and to participate in trying doing your best to keep it open. And, you know.

from a selfish perspective and from the perspective of the web, a great thing that you can do if you are building websites and experiences is making sure that they work in all browsers and that you don't do the lazy path and build something that only works in Google Chrome, because that is the first step. to ending us back in a situation where we were 25 years ago. you can support by engaging with the content on youtube or on spotify if you want to drop a review that'll be super helpful

And if there's any guests that you want to bring on to, please let me know. I feel like sourcing very senior ICs. There's no... well studied list out there on Google that I can just search this up. So if there's someone in your org or at your company who you really look up to and you want to hear their career story, let me know and I'll reach out to them.