The startup behind ChatGPT voice - Russ d'Sa from LiveKit

Right? Like, I I guess I always thought of grounding as, like, that the that the ground itself was was negative or something, and so I that there was this flow from positive to negative. But the bottom of the cloud ends up getting negatively charged, And what it does, that negative buildup of electrons at the bottom of the cloud, it pushes the because likes repel. Right? So it pushes the electrons and negative particles at the actual ground on, you know, Earth.

It pushes those farther into the Earth. And so the bottom of the ground ends up getting so the negative and the negative repel, so it pushes the the electrons at the Earth into the ground deeper, and thus, the surface of the earth becomes more positively charged. And then when that builds up enough, then there's this lightning bolt that is the flow of electrons from the negative, at the bottom of the cloud to the to the positive charge at the surface of the Earth. It's I think that's right. I don't know.

You're gonna have to check.

The negative charge at the bottom of the cloud seeks out a positive charge, usually on the ground, and that's why lightning often strikes tall objects. Make sense?

That agent is then relaying that audio into g p t four o. It's thinking about what you're saying. And then when the response is generated, that's going back through our agent framework, back through our cloud network to your phone, and then played out. You can kinda think of it sort of like a sort of like Cloudflare for for multimodal AI or or media streaming.

And then we have servers all around the world that kind of give a user the lowest latency connection to the fiber backbone of the Internet, carry their audio data to wherever OpenAI has compute, and then they take that audio data or that video data. So if you're sharing your screen or pointing your camera at something, so they take that audio visual data, and they can send it into their model, and then the model generates a response. So the model can generate transcriptions. It can generate an audio response in the case of a multimodal model like GPT four o. And so you kinda think of it as, like, GPT is this brain that's answering all your questions and can understand the nuances of your speech and things like that.

And then LiveKit is the nervous system that carries the information to that brain and from that brain.

It's not true point to point if you look at, like, network topology, but it's there's no intermediary storage happening. It's just going my bytes are going straight through to to your computer, and and your bytes are coming straight through to me. In the AI use case, it's the bytes are flowing straight through to an agent that that's connected to you, in a similar way that we're connected, and then the agent is taking those bytes and processing them like your brain is doing right now.

This seems expensive, but then I built an integration with them in about twenty minutes that I had spent two days banging my head off the wall trying to build it directly with Okta. And then with Workhorse, I then have, like, many, many SSO providers, like, support instead of just one. So, yeah, like, for me, Workhorse is one of the nicest developer experiences I've encountered in the last like five years probably. And it's not surprising because a bunch of developer team are ex GitHub

Netflix does a flavor of it, but it's not real time. It's you know, you when you watch Netflix, it just buffers. You sit there, and, like, sometimes if your Internet's weak, it'll buffer for thirty seconds. What what's happening there is there is a file intermediary in the Netflix case, and it's actually going and downloading a part of the file. And then it starts to play the video.

And as you're playing the video, it's downloading more of the file and sticking that onto the data onto the back. And, eventually, you play through, you know, the the entire movie or TV show that you're watching with Netflix, but real time is a bit different. There is no storage or file that you're downloading. All the data is generated dynamically from my camera and my it's captured from my camera and my microphone in real time and then sent over a network to you and then reconstituted and played out through your screen or through your speakers on the other side. And so I think not a common thing that people have had to do, but now with this multimodal AI and the computer getting really smart and us building, like, HAL 9,000 or whatever, it's I think streaming is gonna become a much more common way that you interact with computers.

So it's gonna be in more places.

One is actually answering questions with ChatGPT live, and then the other one will be the story, but I'll I'll censor it a little bit. So there was a really large company. I remember so I think it was, like, 08/04/2023 or 08/03/2023, '1 of those two days. I think it was the fourth. And I went to this lunch for five hours with this really large company, and they kind of ran through this gamut of, like, options for me.

They were like, can we buy you? Can we license your cloud product? Can we contract you to build the same thing at our company? Or can we maybe wrap your stuff and launch our own version of it to compete with you? Because we're an open source project. Right? All of our stuff is open source. And so I was kinda taken aback, and I and I asked. I said, well, you know, if you were gonna buy us, like, how much would you buy us for? I'm just curious.

And the economy was bad, and, like, they effectively, in the end, they didn't wanna tell me because they thought it would be an insulting offer, but I pushed. I insisted anyway. And they said, well, we'll buy you for we can buy you for $20,000,000. And I was like, well Wow. I mean, that's awesome.

And, you know, thank you for that, but we've also raised 15, so it's it's not great. You know, just mathematically, it's not gonna be great. And, they're like, well, look. If you can tie what you're doing at LiveKit to GenAI, two hundred million. And I was like, one, that's kinda ridiculous you said that, but two, I I no.

Like, we're like a video conferencing, live streaming infrastructure company. Like, I can't I can't tie it to GenAI. Like, I don't have I don't have I can't I can't position it in that way. It doesn't make any sense. So I left that I left that meeting, and I was, honestly, like, a bit terrified because they're a big company.

They're very respect like, respectable. Like, I admire a lot of the stuff that they do. And and so I was like, wow. I mean, this is, like, existential. Like, these guys are gonna destroy me. Like, it's time companies is gonna be dead. And so I'm driving home on the 880. It's terrible highway. Never take it if you can if you can manage that at rush hour. And then I get this email, and it's like, hello from OpenAI.

And I was like, what? And so I chat with them, and we start to talk. And they wanted to build a voice interface to ChatGPT and had been exploring options. And they'd signed up for for LiveKit Cloud a few weeks before, and I didn't know because they used a personal Gmail account. Wow.

But they ended up, you know, building kind of the first proof of concept for for VoiceMode on LiveKit Cloud and got excited and convinced that it was a solid platform from a small team, and it was it was good software. And I guess, you know, definitely being open source and then being able to see our code helped convince them of that. But we started to have commercial discussions, and, you know, ever since, we've been working with them on all of the multimodal real time stuff. And so that's an amazing story for sure. And what's interesting is after I talk to them, I'm like, wait a sec.

I do have a Gen AI story. Hey.

Like, I I think that in that moment when we started to work with OpenAI on voice mode and when I tried the first prototype of it, you know, I guess it was well yeah. I I tried the first prototype. It it's just it's just funny because they had, you know, Skye's voice in there. And I tried it, and I was like, man, this does sound like ScarJo. But it it was not.

It is not ScarJo. It wasn't. And, like, I know that for sure because I was, you know, there in the early days of it. But I also was like, I like this voice. It really does sound soothing.

In any case, the realization and I've said this, you know, in other conversations before, but, like, the realization was, okay. OpenAI is gonna build AGI. AGI is a synthetic human being. Right? If you take it all the way to its end, maybe there's some people who don't believe that that's gonna happen, but let's just say, for at least my opinion, for talking purposes, AGI is this synthetic human being.

It's a computer that can behave like, communicate with you like a human being, teach you things like we just learned about lightning, all those kind of stuff. If you have a computer that's smart, how are you gonna interact with it? You're probably not going to type you're probably not gonna, like, use a keyboard and a mouse to to interact with that computer. Like, that's what you do when the computer doesn't, isn't very smart, can't understand you naturally, so you have to adapt yourself to give it information so it can work for you. Mhmm.

But when the computer is really smart and like a human, indistinguishable from a human maybe at, you know, at the end of all of this, gonna interact with it the way that you interact with other human beings. And the way you interact with other human beings is by through your eyes and your ears and your mouth. Like, you talk to other humans. You see them move. And the equivalent sensors for a computer, a human like computer, to eyes, ears, and a mouth are cameras, microphones, and a speaker.

And it just turns out that kind of what we were working on for video conferencing and live streaming can be used as the eyes, ears, and mouth of of multimodal AI or, like, a human like computer. It's the you know, we're basically taking the camera and the microphone and hooking those things up to the Internet and allowing the data that streams out of those to be transported anywhere else in the world. And and so that's what our infrastructure was designed to do, albeit for a very different use case, and now it's gonna be used for this use case. And I think that this use case, particular, is gonna be everywhere. The future, of course, will contain humans connecting to other humans in applications, but I think the vast majority of real time is going to be humans connecting to machines.

Whether those machines are personal devices, whether those machines are models running in the cloud, or whether those machines are humanoid robots walking around, you know, the the earth and trying to help us in the physical world, In all those cases, you're going to interact with that computer using your natural human IO. And and, yeah, we're like kind of like the wires of the nervous system in that in that brain analogy. The model's a brain and and we're the nervous system.

And I remember just being as I got into, like, looking into the Alexa ecosystem, it just got, like, it's kind of just demotivating. It felt like they'd kind of abandoned it. And it was just there there just wasn't much going on there. It was like the kind of things were just about novelty and, you know, I still use it to like set alarms and ask for the time and stuff like that. But like Yep.

What what do you think the future can could be with how we interact with with how we do our work, how we live with, you know, with GenAI stuff.

Because when you're when you were interacting with Alexa, like, ten years ago, right, probably when you had one, That's when I I think I got mine. And when you interact with that system, it's just this, like, giant expert system. It's like a ton of if else statements. Right? And so Yeah.

If you hit a branch in that tree that it can't handle, it says, like, sorry. Can't do that or whatever the response is. The penalty from a UX perspective of not being able to answer a query is so high that it erodes your trust in that system where then you just stop asking it pretty much anything. You only ask it the things that you know explicitly it can do, and you probably only know or are aware of, like, five things. It can definitely do more things, but you only know probably about five of those things, and that's the only thing you'll you'll use it for because you had some experience where you asked it for the sixth thing and it couldn't do it.

And so you're just like, forget this thing. It sucks.

You get something, and it's the something is not I don't know. And so you don't you kind of almost avoid hallucination helps you almost avoid this negative UX penalty that erodes your trust, and you'll kinda keep trying. Or maybe phrase your question or your prompt a little bit differently, etcetera. So I think that that's a really important feature. Where I think this is all going, right, with voice interfaces, and, you know, you you talked about how pain in your arms, like, maybe RSI or something from, like, typing on a keyboard a lot, etcetera.

You know, we have this vibe coding now. Like, you know, maybe it's gonna become like voice coding, but vibe coding sounds a lot better, so I don't think we'll change the term.

I'm vibing right now. But, you know, I think that the first stage of this, especially for creatives and people that are building stuff, is it's like the copilot use case. So what's interesting, I I I was at a dinner yesterday, and we were talking about this a lot because there were the people at the dinner were all kind of working in the cogen space, or a lot of them were working in the cogen space. And one of the parts of the conversation was that you have something like Cursor today where it's your copilot, and it's helping you write code. Right?

And then there's the vibe coding version of this where you're kind of just hitting accept whatever changes it has and and kind of correcting it or trying to reprompt it if it's if it's getting a a bit lost. But you are kind of like the editor in that experience where it's doing a lot of the mundane work, and then you're the editor hitting accept, accept, accept. But it's still driven by you. I think that that's the, that's the the key part of the relationship with the with the AI model today and the way that you write software, still driven by you. There is a flip that is happening, though, and I think, like, Replit agents is one example of this.

Maybe to a degree, v zero is also like this, but Devon is certainly like this, and Cloud Code is like this where you're still pair programming, but you're not the one driving. It's actually the AI that is driving. And so you're kinda sitting there, sitting back, watching, supervising, making sure that it's doing the right thing, you know, changing the right files, making the right decisions, testing it properly, all of that stuff. And so that, I think, is like it's still a copilot experience. You're still supervising and, you know, synchronously working with the AI or looking over its shoulder.

That's the first step, I think, to where things are going. And I I think, like, all this reinforcement learning kind of verification of the results is is gonna help very quickly accelerate cogen and, like, the AI's ability to to be the driver in that in that pair programming example or use case. Then after that comes, like, the agentic the full agentic experience, right, where it goes from copilot to coworker, where you say, hey. You know what? I wanna build this app, and here's kind of how I'm thinking about it and, like, go, you know, work on it.

Give me, like, a first draft of it or come back with, like, a proof of concept. And then it goes and it starts to do its stuff, and you're not really you're going off and using your time for other things, and then it comes back, and it's like, okay. I've got our, like, an initial proof of concept here that I've put together and all that stuff. And what do you think? Like, what should we tweak?

What should we refine? Let's have a meeting and sit down and and talk about it. And and just like you would with a human, you would do the same thing with with your AI coworker, and and then it'll go off and continue doing its work again and the refinement steps. So I think that that's that's, like, the direction that these things are going. And I think a lot of interfaces, for creatives especially, it's gonna be less important to be an engineer, and it I think more product designers are going to or rather product design is going to become the skill, to hone for a human.

Kind of, you know, I guess, like, a very clear example of this is if you watch, like, the Iron Man movies, it's it's the way that he, you know, even just designs a suit in the first place, yeah, using Jarvis to do most of the mechanical work. Yeah. I thought you were waving your hand in front of the camera.

I think it still requires, like, human oversight. And, ultimately, like, the things that we are building, we're building for other human beings. We're building for each other. So there's gonna be a human in the loop somewhere.

And I was I definitely lost a lot of nights of sleep through that process. But I think what I was so proud of the team about was that, like, when we first launched Voice Mode, it was for all it was for all paid users. And it was like there was some other limiting factor to it as well. I can't remember what the limiter was, but, you know, it wasn't it wasn't to every ChatGPT user. And so that was comforting to begin with.

But I remember that I remember that the whole thing with Sam happened, and the

It had been running smoothly, and we've been doing well. So and, you know you know, it's not like OpenAI was just trying to, like, rug pull me or anything like that. It was just they were just like, hey. Like, you said it scales, Russ. You told us it scales, so, like, we're gonna we're gonna ramp it up.

We're ready to go. And so my cofounder was in the, like, Cayman Islands on vacation, and I already told the team, like, no shipping anything this week. And and so this message came in the middle of that, his his one one week vacation in the Cayman Islands. And so I, like, send this message. I I pause my interview.

I'm like, hey. Can I can I take a five second five minute pause right now? I gotta, like, ping the team. So I ping the team internally on Slack, and, like, they look at the stuff. Like, they look at the graphs, the Grafana charts, and they're like, yeah. We're good. We're ready to go. And so I'm like, are you sure? And they're like, yeah. Yeah. We're ready to go. So they ratchet it up, and it just it just scales. It's like no problems at all. Wow. And I was like, just team's amazing.

I have not I had nothing to do with it. But, yeah, team is incredible. Wow.

And Twitter was just just terrible. Like, the the code, the architecture, everything was just so bad. It was, like, melting down constantly, like the Fail Whale thing just constantly happening. And a lot of people ask, like, well, why did you write it this way? Why didn't you plan for this scale?

And I think, like, I think there's this tendency for engineers to try to imagine and architect this, like, perfect system, but hitting a scaling wall or having something that you designed break, is actually like a rite of passage. I think it takes way longer. It delays your go to market, and finding product market fit so much more to try to devise a system or an architecture that would scale, and you don't even know if you need to. Like, Twitter, literally, I'm not even talking about just from a software perspective, but I'm very grateful grateful to have worked there. Like, it's an amazing company, probably the best time in my entire career, including I mean, I am enjoying LiveKit as well, but it was such a fun time.

I was an IC engineer. I was young.

But what it taught me was that, I mean, I guess it taught me exactly that is get the product market fit right. That's the most important thing. You can go down. You can fail. You can design the wrong system.

You're gonna rewrite it. Like, that's all a rite of passage that proves that you have found some semblance of product market fit or you've built something that people want. Mhmm. And so now you have the honor or the privilege to be able to rewrite it. And so I think that my advice would be the learning I mean, yes, our system scaled and we prepped for it.

Or, you know, my teammates did it very fast, so it's full credit to them. But but that was a part of the journey too. Like, I was proud that we scaled to the, you know, the certain level, but then we did hit a wall, and we had to, you know, we had to do stuff differently to get to the next level. And that's just part of it's part of growth. It's part of, you know, building things.

But the whole purpose is to uncover the un to make the unknown unknowns known known I don't know. Known unknowns? I I'm that I don't know how to word that one, but it's to understand what makes Xerox explode so you can move as quickly as possible to put the patches in or to to build something that prevents all of these these negative consequences or adversarial effects from happening. And and that's I think that that's, like, a model for how I think about things too is go out there and, like, figure out what the requirements are as fast as you can, right, so that you can solve those problems. There's almost no way to envision all of the different failure modes, in what you are building, especially in, a complex system, which LiveKit is.

And so, yeah, I think we I think we did it right.

That makes sense for Twitter, but then does it make sense for, like, life care or, like, you know, because because you're, like, kind of selling, like, kind of almost, like, you used to write, like, the kernel. It's, like, almost, like, your kernel is, like, we don't wanna worry about this going down. Like

But infrastructure kind of has this expectation that it's behind the scenes, and there are people that are building on top of this infrastructure. And while the the developers building on top of your infrastructure might make mistakes and go down on their end users, they definitely don't want you to go down on them because, you know, that's something that they don't control. Yeah. And so it's not quite the same thing as Twitter, and we do have to be more careful. And, you know, the stakes are also very high too for us.

Like, I don't wanna be I don't wanna suggest that, like, we're like willy million. Like

So, it there's also that. We're we're pretty careful, but we're not trying to engineer around every single possible scenario or scaling, you know, kinda wall that we may hit. I think the important part is to be able to see it coming and then be able to react appropriately before you actually just smash into that wall. Yeah. You know?

And we've we've we've done that successfully where we've seen it coming and worked around it or scaled over the wall. And then, you know, I mentioned the character AI one, like, we we definitely smashed into the wall at least once.

So find me find me on x. I always wanna call it Twitter, but, yeah, find me at find find me on xx.com/dsa. Right to left on your keyboard, three letters, helps to have been an early employee.

But one of my coworkers, I guess, I used to pair program with him a lot. He went into the database, and the person who has Sierra Nevada, he, like, changed their username to something, and then he he gave Sierra Nevada, like, at Sierra Nevada. And then a week or something later, this random guy comes in through the elevator to the office, and he's just got, like, 10 in like, with, like, this Dolly, and he's got 10 boxes, like, 24 packs of Sierra Nevada beer. And he's like, is this person here? And, like, we're like, yeah.

He's right here. And then, like, this guy wheels in. He's like, this is a thank you from Sierra Nevada, and, like, drops it off in the office. Oh

I think I was, like, the seventy fifth or eightieth employee or something. I was the only person who just started to scan all of the single letter username, and at you was open, and so was at x. I should have taken at x. I mean, I could have sold that to to Elon, but maybe. But at you is open, and so I tried at you, and I just got so much spam from people that did not know how to at mention people on Twitter.

So I was like, hey. Whatever. And they would write their username without the at symbol and then say at you. And so my notifications were just destroyed with that name. So now I'm like, I need to change it to something else. Okay. I'll change it to my last name. It'll be at d s a. Yeah. And at d s a collides with two groups.

One is the Down Syndrome Association Down Syndrome Association. And so that one, maybe I would give up my username too. But the other one is the Democratic Socialists of America, and that DSA has been pinging me for, like, ten years, like, trying to get DSA from me. It's like it's like Bernie Sanders folks and stuff. Know, I don't talk politics, but and I have nothing against them, but they they definitely have been trying to get my username.