Lessons from the early days building Kafka and Confluent | Jay Kreps

You know, engineers say there's a problem when there's a problem and then you know at the director level people say, oh we have challenge and senior director level people say, oh we have an opportunity. And then at the sea level people say we have a big problem. So it goes full circle, right? And I don't think you have to be kneeling right now. I think you say there's a problem, but I think being aware of what is the impact I'm trying to have and getting good at that.

That's a skill where you develop over time, but only if you try. And then you know, that's the only way you can have impact on trying to get a large group of people to do something differently. Because by developing that if you don't do that, you know, you're just kind of shouting into the wind. Welcome to the Software Misadventures podcast. We are your hosts, Ronak and Guang. As engineers, we are interested in not just the technologies, but the people and the stories behind them.

So on this show, we try to scratch our own edge by sitting down with engineers, founders and investors to chat about their path, lessons they've learned, and of course the misadventures along the way. Jay, super excited to have you with us today. Welcome to the show. Really excited to be here. Thanks for doing this. So we thought we would start with asking you about a video I saw on Twitter and this person, Eric Wissriya. I hope I'm pronouncing his name right.

There's a video of you shaving this person's head. You and a couple other people and Eric is on the board of content to find out, right? Can you tell us what's going on? Yeah, happy to do that. Eric was our first investor. So just as we were starting the company, we went and talked to him. We actually didn't have a name for the company yet. So I think we pitched to him as Nuko and nonetheless the investor. And so yeah, I think I'm not sure what the milestone was.

It was some revenue target or ARR target or something I think that we beat. And so he had offered to shave his head. He actually hit our hit. I think it was if we hit our plan for the year or something. And so yeah, we made a video of it, which was it was originally meant to be just inside the company. And yeah, you didn't look bad, although you know, if you've got the shaved head, you've got to, you have to eventually tan the head otherwise. But you look kind of pale.

You look like you maybe just came out of the hospital versus you did intentionally. So in my case, you know, it's just from balding. It's not necessarily from breaking any plans or anything like that. Hashtag relatable. Yeah. So it's pretty cool actually that you went through with it and actually publish a video. And put it in this case, I'm hoping that X family was okay with the specific choice. I heard it was not that popular and that was why the shave Ted looked in stick around.

I think I see. What was the reverse bed, by the way? What if he won the bet? Was there something that you would have to do? I think it was one sided. You know, it was the people doing the shaving was me and the head of marketing at the time. Both of whom were, you know, pretty bald. So we had nothing to lose in that respect. Yeah. Nice. So you mentioned that initially when you pitched to a RQ, you named this company NUCO. So Q does like how did that evolve to conflict?

Yeah, well, we, you know, we were really picky about the name that we wanted to give. So we, and we don't want to put down something that wasn't good. And so, you know, you got to put something on the slide deck. We just came up with NUCO, although apparently there's a bit of a tradition of that, you know, companies that don't have a Navy at our often call themselves the NUCO. But we, I guess we independently invented that name for the interim.

I mean, my tip to people would be if you're pitching a company, you know, you should probably just come up with a name you can change it later versus like having not decided. But the specific sequence of events was kind of unusual because, you know, we, we talked to the people at LinkedIn as we were leaving. And they said, oh, you know, we would be interested in investing. But you would need to go get kind of VCE that would price the round.

And so we went out and pitched a bunch of VCs much sooner than we'd actually expected to. We'd expected to leave LinkedIn and really kind of spend some time and kind of get everything baked. So we were effectively doing everything a couple months ahead of schedule at the time that we were fundraising because we wanted to make sure that LinkedIn folks didn't lose interest, kind of strike while the iron saw it. Yeah. And so the result was a bit of a rushed initial fundraising process. Thanks.

I think it's pretty cool that you guys have a playful, you finally say that our relationship, right, with this VCE, I guess, in particular. How has that relationship like, was it always like this from the beginning, or how has that evolved over the past years? Yeah, I think we've been lucky to have, you know, really good early investors. And I think that was really helpful. Anybody who's involved or on the board has a stake in the company, you know, they're going to be part of the early team.

And, you know, that can be a, you know, that can be a great addition, or it can be a bit of a pain really depending on who the person is. You know, it's, it's, I think some investors add significant value. And then I think some probably subtract some value, right? So, so I think the, the trick is trying to figure it out when you're raising money. Because, you know, at least for us, we had, you know, very little experience with venture capital. Didn't really know that ecosystem of people.

Yes, we were coming into it a little bit blind and trying to make sense very quickly of who all these people were. Again, because of this rushed process to try and keep LinkedIn interested as we were leaving. So, in this case, like you went from being an engineer at LinkedIn to starting the company. And obviously, Kafka existed at that time. It was open source and you wanted to build the company around it. Fundraising is not a skill that engineers typically have.

And especially when you were doing it, like you said, a little bit rushed in the process. Did you have any guidance at the time in terms of how to go about this and have these conversations? No, absolutely not. There was some information on the internet, not nearly as much. You know, now there's so much information on starting companies that you can go listen to like 10,000 hours of podcasts and blog posts and books and so on. Yeah, there wasn't a ton out there, to be honest.

There was some at that time and so yeah, I'd read some stuff online. That was probably the extent of it. You know, so yeah, we definitely did not know what we were doing. I don't think that's necessarily a bad thing. I mean, I actually think the skill for early stage investors is kind of being able to see the diamonds a little bit in the rough. Because the reality is the level of polish may not be a perfect predictor of the quality of the team or the idea, you know, in that very early stage.

Later on, it should be. And so yeah, you know, I think actually the fundraising process was pretty successful. We had a number of people who were interested. Yeah, despite us having no idea what we were doing. And it also being just three very technical people who had not really even worked in an enterprise business, you know, let alone founded a company. Interesting. How did you guys think that happened?

Who wrote all the emails to reach out or do they just one person or two VCs know that you guys are doing this and then more just kind of, um, yeah, uh, yeah, we sourced the bunch. So, you know, we kind of went out to our connections and tried to figure out who knows these people basically will. We didn't know any of the VCs. So we just looked at companies we thought were good. And then tried to figure out who invested in those companies. And then just met with a bunch of people, right?

So they tried to get introductions to them. Neha, who was one of my co-founders, you know, I think it was her cousin was the CEO of a company who was extremely helpful in introducing us to people. There's a few other paths that we had, but we just collected a bunch of intros by whatever path we had to the people that we thought might be good. But of course, it's pretty hard because we didn't really know who to talk to.

We're just trying to figure out like, oh, who covers this kind of space and might be interested. I heard there's like a quote, not a quote. Someone told me that, oh, for VCs, right? Like, their product is money and money is like the most commoditized product there is. So how do you differentiate a big question? Was it clear to you during that process, like some VCs were just so much better than others? Or did it all feel kind of like middle of the road?

Yeah, for us, you know, I think board members can definitely be very helpful to a company. So I don't think it's commoditized. I think that can be very valuable, but it could also not be valuable. Kind of as I was saying, so the question is how do you tell? And the answer is, you know, mostly you talk to other people who work with the person, probably the same as any hiring decision, and you talk to the person and try to assess it.

So yeah, so I think probably for us, I think a lot of the decisions were like, hey, how do you wage the brand of the firm, the kind of seniority and experience of the individual, and like what they've done, plus your conversations with them, like how much they seem to understand the company, and what you're trying to do, plus the references that you do, you know, how do you kind of weigh all those different factors? I don't know if we came up with the perfect system, but it worked out well.

You know, but yeah, I think we definitely thought a lot more about it in the first round. I think we actually took like a week to pick, which is probably about six and a half days longer than we really did. That's nice. Usually it's the other way around, but once you've gotten enough traction, right, then it's like you're picking the VCs.

I know this is a long time ago, but do you remember any question that you asked everybody and then you were kind of judging, like their answers to that question? Yeah, yeah. I don't think it was so much that. I mean, the conversations are on both sides, right? They're going to ask you questions and you would ask them questions. I actually go, I put a lot of weight in the questions the person would ask.

So I actually like people who could understand the business we were describing, why it could be valuable and understood some of the problems that we would probably face. The more that somebody could pick it apart and have like kind of a very critical take on what could go right or wrong, the more I thought they really got what we were doing, it would be helpful. Even though in some sense, I could come across as negative, at least for me, that kind of shows, oh, okay, they really get it.

I've always thought that way. I actually feel that way. If we interview candidates to work at Confluent, I always, sometimes people don't really have any questions at the end, which is fine. But when people do have questions, they're actually quite insightful. And you're like, yeah, that is our biggest problem. I always feel like that's actually a really good sign that they kind of have gotten right to the heart of the matter.

And to be able to do that, you actually have to have a pretty good understanding of the space and the competitive dynamics and the type of business and what really matters. And it actually shows a lot. It's like you actually care. Yeah, yeah, yeah. Both care, but also can put all the parts together. I think so. I thought that was important. The other thing for us, we felt that there was a bigger opportunity around data streaming.

And so the kind of cookie cutter thing was like, hey, this is the next message queue. Like, it'll be like, TIPCAL. And we felt like, no, there's more here. Do real-time stream processing of the data. You could build really a whole platform around this in the way a database is about. But so, we wanted to try and find people who were signed up for the bigger thing, rather than the smaller thing. And in theory, VCs always want the bigger thing.

But then in practice, the cookie cutter thing is much easier to see. Right? And so, I think that was another factor for us as well. How much did it seem like people got the bigger opportunity and vision? Did that naturally mean that you had a preference for VCs that had a more technical background? Or did you also see that people that didn't have that, but were able to ask really good questions? Yeah, it's interesting.

Eric, who we went with for benchmark, you know, I'd say he's technical, but not hyper technical. You know, his background is not necessarily software engineering. I would describe him as a very smart generalist. And so, yeah, I don't know that it was so much, oh, somebody who knows a lot about the building of software. So much as somebody who was willing to dive in and understand this space. You know, at that time, there was not going to be anybody who would like study stream processing.

It was also a venture capitalist. That was like seven computer science papers and, you know, whatever, it was just not a thing that you would expect somebody to already know about. So, but yeah, that kind of willingness to really dive into an idea, I definitely think that that, I think that's important. I think generally, if you have somebody invested in the company, you want them to understand what it's about.

And the reason for that is that's what will drive conviction that it can be successful. If you don't really understand something, then you may be excited when it's hot, but you will not be excited when it's not because you're going off of other people's judgment. And, you know, so I tried to maintain that in later fundraising, just the people who are willing to dive in and kind of get into it more.

Not necessarily the most technical people, but sometimes, you know, I think that's actually really valuable in as kind of a test of how they approach these things. And I think it was useful for us. Yeah, nice, nice. We were talking to Josh Willes and one thing that he said that stood out to me was that it was like, all engineers don't make good engineering masters because they're very pessimistic. It's hard to see, right? Like so many years down the road that what's the

billion dollar thing that you're trying to build. Did you find that to be true in your experience? Yeah, I actually, I probably had the opposite problem as an engineer. I was always like, I don't know how to describe it. I'm not like a super optimistic person. I think I heard myself to describe this kind of grimly determined, but which is not quite the same. But the, you know, at least for me, I always feel like it's very clear the bigger thing that

you could do, right? And especially in technology, you can easily prove to yourself that it could be done. There's nothing fun. I mean, maybe there's some fundamental problem. I can't be done. But once you see that it can be done, like once you proved to yourself that the core problems are solvable, then I feel like there's, you know, the only thing standing between you and that outcome is kind of the execution. You know, like, are you going to go do it, right?

Right. And so yeah, I do think like engineering operations, some areas of finance, you do tend to have more, call it pessimistic realists. You know, to some extent these are people who their work is in direct contact with reality. So they see all the ways things go wrong. And but some people I think over index on that versus seeing the thing that you would describe like for an angel investment, which is kind of, is there any fundamental reason why this couldn't go right?

And I think for a lot of big technology projects, that's the thing that actually gets a team through it. As you see that, I instate, you're like, we can, we can definitely make this happen, right? We might fail, but there's nothing fundamental that will prevent it. It might be hard, but we can definitely do it. And I always felt that about these big systems engineering projects. And it kind of big change or transformation in a company you're trying to do internally with

technology, externally in terms of creating a startup or a new product. And I think it is a little bit of a test of personality, you know, whether people believe that you can kind of accomplish some larger change. But I think it's also a question of experience. What I've seen with people is

people who've had a lot of failures tend to be kind of beaten down around it. But somebody who in their career, even if they're able to accomplish some smaller change, which led to them being doing some larger change, maybe in an organization, which led to something bigger, you know, they tend to actually feel, no, you can do these big things. There's nothing preventing you. Worst case, it doesn't work, but like the, there's nothing, there's no fundamental problem. And so

if there's something very valuable that should happen, then, you know, you should go try. And so, you know, I always think for people in organizations, you want to find a way to try and build that experience for that, give them something that they can go do. That's kind of within the scope of accomplishment that they may be able to achieve. And as they've done a few of those, they tend to get more ambitious in terms of what they want to go fix. And I think that's a really powerful thing

to build. And to some extent, a company is very much like that, right? So, yeah, that was definitely our take on Confluent. From first principles, we were like, hey, this kind of streaming model for data, it makes a lot more sense than batch and batch processing. To get from A to B is really hard. It's, there's a whole ecosystem around data warehousing that's very batch oriented, right? You move the data in at the end of the day, you have a bunch of jobs that turn over, you know,

you have to recreate all of that. And so, when you sit down and you try and think of like, oh, how would you do it? It seems like, oh, that's really a lot of work. But then, all you have to know is that it would be much better if it worked that way. And there's no fundamental technical problem that limits it from happening. Once you know that, then you would say, well, okay, there's kind of an inevitability that somebody's going to do it. And so, if we try and do this, you know, what

would be the steps? Can we come up with a first step that we could do that would be valuable, right? That would actually have some use cases that people would use. And then if we had that, could we come up with a second step that would be a little bit further along, that would also be valuable. And you know, as you start down that thought process, then you're like, oh, yeah, this could absolutely be done. It's a long journey, right? It's not going to be like a one-year

project, but it's definitely doable. So, yeah, I think that ability to do that I think is a very powerful time, especially for senior people in an organization. You want them to have that that kind of belief and the possibility of important changes because most of the big valuable things in companies are that kind of change. Going along that, like, can you remember like the specific milestones in your career that led you like, like, the first few steps that finally led

to you, you know, quitting LinkedIn and then starting the company? Yeah, yeah, I had, when I came to LinkedIn, I actually came to try and work more with data than on data systems. So, my background had been in machine learning and I was really interested in trying to find a place where I could do that in my job. And so, and I thought these social networks would be a really good place for that. And so, I came to LinkedIn for that reason. So, the, the first project I had at LinkedIn was

this kind of news matching thing that we were trying to build in. It was, it was the most ridiculous bad, you know, ridiculously bad approach. Is that the biggest thing in issuing? Unranked. Yeah, it was worse than that. It was like an unranked keyword search based on your company name for news. So, like, everybody working on it was like, this is the dumbest idea. Like, it's going to produce terrible results. A lot of company names are just words, right? So,

you can't, you can't do this. But anyhow, so we, we did that, you know, I had some other efforts that were in the, you know, the use of data. But at that time, it was very hard to do that at LinkedIn for, you know, a couple of reasons. I think one, the infrastructure was very immature.

Like, LinkedIn was basically running on a very old centralized technology stack. So, you know, there was this social graph system that would be get called like a zillion times to try and figure out the routes between people, you know, it had the whole social graph on one computer. And so, and there was replicas of that computer that could also be queried. But the number of edges in a graph, you know, grows quadratically with the number of people.

And so, there was like this continual engineering effort to try and shrink that data set. So, it was like, okay, first let's write it in C++. And then it was like, well, let's try and compress it. And the urgency of this was very high. So, you know, it was like, if this project is not completed, LinkedIn will like go off the internet. And so, I was on this larger team. That wasn't my project, but, you know, it was a kind of peer team. And, you know, there was just a lot of that where we're,

we're wanted to do these very sophisticated things with data. But the underlying infrastructure was like, you know, not the right thing. Not the right thing. So, you know, you would spend like 99% of the time just trying to get or, you know, hook up to systems or come up with any way you could do anything. And then you would end up with these solutions in the end that were just this like dumb keyword search or something. Because that was the only thing that was possible. So, over time,

a lot of my efforts and switched to kind of infrastructure. And I liked it for two reasons. One, there was, you know, there was no product managers. And so, you had more degrees of freedom and how you would solve the problem. And I was so frustrated from that keyword search thing, where I was like, this is like, not, this is not the right way to build a new development system. But nobody would listen. I was so frustrated about that. I was like, oh, this is better. You

know, in the infrastructure world, you can just, the customers are internally. You can just, you know, come up with the right approach. You can reason that the second you're not in that front, in order to target managers. Yeah, yeah. And so, you know, and at that time, the technology stack at LinkedIn was extremely frustrating. So, you know, so I was riding the train back and forth to work from San Francisco. And, you know, to do something on the train, I had a programming project

which was to build a database. And I was interested in some of the distributed systems. They'd been a paper that came out of Amazon on DynamoDB. And it was like a key value store. And so the first, you know, the first project was, oh, could I implement that? And, you know, if I did that, I thought, oh, maybe that can be, you know, a better replacement because we're trying to do everything on top of these Oracle databases that were like really, really difficult to scale. And so I thought,

hey, you know, if you did that, you could have like really fast access to data. You could do a lot of better front end features. And, you know, at the time, everybody thought I was kind of nuts because, you know, it wasn't, it wasn't a thing that you would do to like go build the database. But this is a very simple database. I thought, oh, there's no reason you couldn't do this. So, sometimes I want to pause you for a second. Yeah. Exactly this part. Like today if someone goes

and says, hey, I want to go write my database at any company. Or even friends like, it's like, friends don't let friends write databases, for example. So, yeah, it's a terrible. Yeah, it's a terrible. And the other thing is, I think this, if you place this back in time, the source like somewhere between 2007, 2009, somewhere around that, I think. Yeah, yeah, that's right. The open source technologies weren't as mature as they are today. So, yes, you know, there was

nothing. There was nothing. Very few options to work with. When did you get that conviction from to say, I'm going to go write this database? Oh, well, you know, I'd read this dynamo DB paper and then, you know, the underlying storage engines, there was these things like Berkeley DB, you know, it was like an embeddable. Now everybody would use like ROXDB, but it was the equivalent at the time. And that was already there. And so, I was like, you know, look, these systems are not

that complicated. You know, a lot of the data access patterns for live systems are pretty simple. And so, if we could just scale that one thing, that would be pretty good. And, you know, so, so yeah, it's definitely true that you should not write your own database. And I later learned more about why? Because if you do that, then you also have to run your own database. You should be

exact to be the hard part. But, you know, it, directly, it was right, like LinkedIn needed more scalable data infrastructure, like, you know, buy, build, borrow, whatever, like there needed to be some solution. Like the running indefinitely off Oracle was not viable. And so, yeah, it was a bit of an internal battle to release the thing, but it was used internally. I linked it for a long time. It was open source to this project. And, you know, for a while, there was this, you

know, a whole storm of key value stores. And it was quite popular. But I think it was eventually displaced by the ones that had some company that was, you know, investing in it continuously. And, this is what I had kind of moved on to other. Yeah, that's right. Yeah, it's called Baltimore. So, we're talking. It also had a weird name. That was the other trend. I think a lot of classics at the time at LinkedIn, for some reason, had references to Harry Potter, like, I think

Baltimore. Yeah, yeah, that was my, that was my naming scheme. Oh, I see. So Kafka, I think diverges from that. Yeah, we ran out of good Harry Potter names. You know, the name has to, like, sound cool in a ridiculous way. And so, you know, some of the names were good, like, Baltimore. And I think that's true. Yeah, there was a, there was a, there was a Nimbus, which I think is like the broom or something. I don't know. There was a, there was a few of those. But then after a while,

we kind of ran out of cool Harry Potter names. And so, there was switched to writers. So, there was that. There was Kafka. There was Kafka. There was Camus. There was a few others. But, you know, these, these things turn over over time. So, I'm sure Kafka is probably the only one that exists now. Well, wait, on naming it, so, like, did you read these authors at the time and knew of, like, I didn't know of these authors. I learned about Franz Kafka through the project Kafka. So,

cute. It's like, how, how did yeah, yeah, I had, I had read Kafka. You know, I'd, I'd, I'd done like a literature minor almost. I think I did, it was like maybe one class short of it. So, I'd run, I'd read some, some Kafka, I thought it was interesting. And it's, it's also sort of synonymous with complexity. So, the, you know, the idea was to unite a lot of the different data flows inside LinkedIn. And it was indeed a very kind of Kafka-esque setup. Like, we had all these different

scripts and little hacks to get data out of one system and into another. So, this was meant to, you know, kind of unify those. And so, you know, Kafka-esque, that was the problem with Kafka. Oh, I think. So, when you're talking to Felix, who actually introduced us to the email, he mentioned one legend that he was basically talking about, Uwe Nis, which is the success of Waldemort. And he was basically saying that legend has that J. Krips wrote Waldemort on a

cattle train. So, that part is, yeah, that's true. That's true. There's a second one that we heard about. And maybe you can help us confirm if that's true or not, that Kafka, you started writing Kafka with a Christmas break. Is that true as well? Yeah, yeah. So, I, you know, both these projects were a little controversial internally. And, you know, obviously, it wasn't just me. Like, I wrote

some of the initial code. A bunch of people did the later Waldemort work. You know, for Kafka, it was tricky because, of course, the, each one of these data pipelines was owned by a different team. And so, you know, my plan was to like replace all the projects with one thing, not in an aggressive way, but just because, you know, for a company like that that was growing very quickly, they were all breaking all the time. And then I owned some of the downstream infrastructure at the time.

So, I've gotten the Hadoop setup going. And, you know, the, the things you can build in that kind of data lake area, they only work if the data is reliable. And both reliable in terms of like shows up and doesn't get lost, but also reliable in terms of like the, you know, what are the fields, the structure of the data, the schemas, if that's dependable. And, you know, that, that stuff, it was just utter chaos. And so, everything downstream would just break every night in some different way.

And, you know, my theory, so, so, and the traditional theory on how you do that is you just hire more and more kind of ETL engineers to fix and scrape data at a different system. Right. Exceptions. So, at one point in the budgeting process, you know, there was a column. So, if you wanted to fund a project, it would be like, okay, how many engineers do you need? You know, how many PMs do you need? How many designers do you need? Okay, fine. Then they added like these

data warehouse ETL people on the end. I was like, you know, Jesus, this is now like a tax on everything we do. We're going to have some number of full-time humans just to scrape data. I think this is not

my exacts. And so, there's got to be some more like infrastructure solution that takes some of the pain out of it because the, the data lake was like one destination, but there was, you know, a search system and a whole set of analytics applications and other real-time systems and just each one was its own little team that was kind of building integrations. You know, we had kind of one pipeline

per data type or up to five pipelines for a single data type depending. So, that, that was the idea was to unify it, but it was, you know, it was controversial, I think, in part because nobody wanted to work on data pipelines. It just seemed like, you know, not a very cool area to work on. So, you know, I think for some of the engineering management and so on, I was kind of like not it, right, was the attitude. And then, you know, I also cut across all these other technologies

that had full-time teams. So, it wasn't political. It was just like, you know, I just thought, practically, we'd be better off if we could unify some of it. So, yeah, anyhow, it was somewhat controversial. And people also thought that you couldn't build something that would work for all the different use cases. So, you know, kind of have strong guarantees and be scalable, but also be real-time. You know, there was some reason that we had a batch pipeline and we had

log aggregation and we had changed data capture and we had like three other things. So, the, yes, the coding on the vacation was the way through that. I was like, well, if, you know, if I show you we can do it, then you can't say we can't do it. Right. So, yeah. So, there was a, so we wrote like a very early version of it that was kind of a, you know, it was like a bunch of individual single-node instances, but very efficient. And that actually, you know, it ended up getting

released. It was probably a newsfeed early on, but it was quite buggy. You know, I think we, and then we got better after that as other people started working on it. Like you're saying this is like a huge undertaking and you mentioned manager. Well, yeah, was DJ Patil your manager at this point? No, he was on the, he was on the product side. So, yeah, it was a different group of, you know, there was a whole set of different people, you know, involved in different ways. I just think

nobody was very excited about data pipelines. Understandably so. How did you get Bians, like from the manager from management? No, the prototype, you know, prototype solves all problems, right? Like if you have a prototype, everybody's like, well, it could almost exist. So, we should finish it. So, and then I think, you know, it got rolled out for some of the, you know, user activity, kind of real-time logging stuff. And then from there, I think it spread over time to some other

use cases. But, yeah, it was, you know, it was a challenge. It was anything that cuts across the natural organizational structure in a company. It was always hard. You know, from a management point of view, that's what you're always trying to do is actually design the org structure to make the problem you want to be solved. Easy. Oh, yeah. Connerational. Connerational. Yeah, that's right. Whatever you do that, you make some other problem hard.

And so, you know, it's a little bit like the, like, you know, sharding or partitioning of data in a distributed system. Like some queries will be fast because they're all within one partition. So, of course, they're going to be slow because they cut across, right? So, it's a little bit same with the org design stuff. So, like, when you were working on this prototype on a couch tray, yeah, this was a question of partition. Sorry. Yeah, this is a, this is a, it was a Christmas

vacation. Plus, I think I, at one point, I flew out to see my grandma. No, no, I guess it was a wedding that was in Pittsburgh. And so, it was on the plane and staying with my grandmother. Um, my cousin was getting married or something. So, did, did some more fun. Eventually, eventually got to something that, you know, mostly worked. Oh, so I, I imagine, like, a lot of people would try to start something, right? When they are

feeling inspired by like a paper or something else, but then they inevitably gave up, right? Like, after they hit the second or the third role block, like for you, there must have been some kind of reward function, right? Where as you were going through it, you're like, oh, man, like learning so much or what, what was that reward function for you? Yeah, you know, I, I think it was a combination

of things like I, I like computer programming. So, you know, it was just fun to work on. Like, there was interesting problems to try and design the system and then, you know, once you're kind of working on it, you're in some rhythm, you know, it's interesting to work on. So I, I enjoyed doing it as kind of a spare time project. And that was my way of kind of learning and keeping

this skills sharp. Um, at various, like at the time I was doing Kafka, I think I had more of a, uh, I can't remember if I was an engineering manager or architect at that point, but, you know, basically I wasn't writing as much code. So this is a way of like not becoming totally a pointy haired, um, you know, keeping the language. Yeah. Yeah. So, um, you know, so I, I enjoyed it from

that point of view. And then, you know, I think I'm just extremely stubborn. So then once, you know, once I was convinced that this solved a big problem, you know, I was just going to keep pushing on it till, you know, till I hard failed, which, you know, hadn't happened yet. So, so I just kept pushing. You know, I think I, I, in retrospect, at that point in my career, I think I was a little, um, you know, I, I think I was a little controversial internally, because I was always

pushing on these things in different ways. Um, you know, I, I think internally people thought I was a little bit of a, a little bit of a cowboy. Um, but, uh, yeah, I was well-intentioned. So, uh, hopefully there's, hopefully there's no hard feeling. Oh, looking back, like, what do you change the way you approach some of this? Um, you know, I got better at communication over time. You know, I think, uh, probably like a lot of engineers early on, you know, I would just try and like, say

things that were true. Um, and I'll speak a little wrong way, but I agree with you. No, I think that's good. Like, you want to say things that are true. That's not wrong. Um, but that was pretty much the extent. So I would just be like, you know, wrong. And, um, then I realized, you know, I, I think managing people is a helpful exercise because you're trying to get people to do stuff. And you realize like, okay, you know, what do I need to do? I need to really understand

the world from their point of view. And then when I'm speaking, my, you know, it's not sufficient to just say things that are true. That's like necessary, but it's not enough. Uh, I need to also say things that are going to change what they think. And so if I want to change what they think, I have to really understand how they think. And then, um, you know, explain it in a way that makes sense in their world. It is likely to produce that outcome. And you see this a lot where you actually,

you know, it's common among engineers. They're, you know, they're, their communication is actively provoking the opposite of what they want to happen. And so, you know, the solution to that is not to become, you know, totally mealymouthed where you just don't say anything. But the, the solution is to, you know, think a little bit from other people's point of view. Um, you know,

what's going to change their mind and what's going to change how they think. And I think that's a very important skill, you know, for, you know, communication, I think it was one of the most powerful things. Broadly, you know, if you want to change, if you want to get people to adopt some new technology, if you want to change how an engineering team works, if you want to start a company, you know, ultimately this is some idea that you want to propagate. So you got to think a lot about

that. And so I think that was very, it was a valuable exercise for me. You know, it was definitely in the surety on my part, really in my career. And it was a good evolution. Now I don't think, I don't think what that means is just to become, you know, like I said, mealymouth, my joke internally is it, you know, engineers say there's a problem when there's a problem. And then, you know, at the director level, people say, oh, we have challenge. And senior director level people say, oh, we have

an opportunity. And then at the sea level, people say, we have a big problem. So it goes full circle, right? And, you know, I don't think you have to be mealymouth. I think you say there's a problem. But the, but I think being aware of what, you know, what is the impact I'm trying to have and getting good at that? That's a skill where you, that you develop over time. And, but only if you try. And then, you know, that's the only way you can have impact on trying to get a large group

of people to do something differently is by developing that. If you don't do that, you know, you're just kind of shouting into the wind. Yeah. Yeah. Totally agree with that. And this is something I noticed in a lot of senior engineers too, where takes a little more intentionality, takes a

little more patience than needed, or then people think it's necessary. And you see when some people when they go into, well, so called conflict where two teams are basically screaming into the wind, someone comes in and can provide that clarity and still achieve the desired result where the stuff that is wrong gets fixed, but there's a way to put it in words. And it's not a common skill

said, though I agree. It's one of the most important ones for engineers. Yeah. Yeah. Yeah. It's, you know, I think that kind of really building empathy is an important skill, I think, for, you know, management leadership and any of those things, which is, you know, ultimately just comes down to try to figure out where, where people are coming from. Yeah. One aspect of this is like, when you ask people who are really good communicators, one thing that people often say is like, it's practice,

you do more of it and then you get better at it and initially no one is perfect. So have there been resources apart from trying it a whole lot that have helped you in improving this? Yeah. I think that's accurate. I mean, I actually think the problem with communication is people don't like to work on it. And so, you know, there are different people think of different things as work. And people don't like to do things that they don't think are important work. And so you find this a

lot in technology. So if you want, you know, if you want a product or an idea to be adopted, you have to really think about how to communicate that idea. And, you know, it turns out for these kind of enterprise products, that has a word it's called product marketing. And, and yet a lot of engineers who actually care a great deal about building things that people use, they actually are quite allergic to product marketing. They think it's, you know, awful, right? There should be

no, you know, and yet it's actually quite hard to convey things. And, you know, if you think, I always think about it like, look, you know, people are very busy, you know, so they don't have a lot of time for new thoughts and ideas. So if you want to get something through, you have to really boil it down. You can think of it almost like a kind of compression. So it's very, you know, compression

is very CPU intensive. If you want to compress an idea, you have to really think about how to convey it. What is it that people understand that's like that thing? You know, how can I tell you about it in a way that's, um, brief and compelling, but also accurate. Like, I actually provoked the right understanding on your side. And if I can do that, that's actually quite valuable. And many of the biggest and most successful companies have built on an idea that's ultimately relatively short.

It's like actually easy to convey, like a picture of the world like that. And so if you can find it, you know, minimum easy, um, you know, I think it's a really valuable thing. I think it's a valuable thing if you work inside a company and you're trying to accomplish something, if you can boil down why that matters, that idea can spread a lot more effectively. Um, and yet for most people, they would not be willing to spend two hours thinking about that, uh, problem. Even though they

would agree that it's very important. Um, so I think some of it just comes down to what do we think is work? Like what are we willing to spend time on? A super fascinating, like, um, okay, make it product practical. Like for a microengineers, like what do you think are like the most underrated or like the work that has a lot of value, but people know, like don't perceive it as like real work.

For, for engineers, I typically think that communication side is, is lacking, you know, I, I typically think senior engineers can have a lot of impact just by, uh, explaining what they've done well and or what they want to do or what they want other people to do in a way that's, uh, actually very compelling. Um, you know, the same with speaking, et cetera. I, I, a lot of people develop those, but you know, the willingness to do that, I think is a big magnifier of impact.

Um, you know, it was interesting when I was at LinkedIn, I, um, you know, it was, uh, mentor to a number of people, right? So you get paired up with somebody and you try and, and one of the things that realizes like, oh, everybody, you know, you know, I actually really want mentoring, I actually just want to get promoted. Yeah. Yeah. Yeah. One, one question about, you know, good, I want out of my mentor. How do I get promoted? Yeah. I was like, yeah, I actually don't think it's that

hard, right? If you, if you just consistently do good work and then you explain to people what you've done in a way that they can understand and why it's valuable, you know, and you try and increase the scope of that value creation over time, eventually they're going to promote you. You know, maybe it's six months later than it should have been, but like, it doesn't really matter. It's always that line is going up is interesting. Like people don't think about that

underlying value nearly as much as they think about the, um, you know, the path to promotion. And so I was like, well, you know, you know, I thought the formula was kind of simple. It was like, do something valuable, you know, communicate effectively around it. That's kind of all you need to do. You know, let's spend the time on the valuable thing. Uh, and yet most people want to spend a lot of time on the mechanics and promotion, which, you know, it seems like is kind of an inevitability.

If you have somebody in an organization who's just consistently, uh, producing a ton of results, you know, well, the process is perfect or not. Typically they end up, you know, getting promoted over time just because what they're doing is bigger and bigger. What you're saying is like, some words kind of a byproduct of what you do not go in itself. Yeah, yeah. It's like a lagging, you know, it's a, it's a lagging measurement of value, not a leading, you know, and I think

many people see it the other way in engineering. There's like, oh, I should get promoted so I can do that. But, but your earlier point is kind of like the mechanism is the more interesting work, right versus the other artwork value creation. Yeah. Well, I think what people actually want is just the output data. Yeah. So many blog posts around how to get to this level. So, and so do X, Y, and Z. Um, anyway, so, uh, talking about Kafka itself, sir, you had this initial version.

By the way, what does, what did this prototype look like at the time? Like what did you have all the, yeah, the early prototype was, um, you know, it was more like a single server Kafka that you had to have like a load balancer in front of. And then we, you know, by the first kind of, like multi use case version and inside LinkedIn, we, we filled it out. At that time, uh, Jun and Neha joined the efforts of there was a, there was a full team of, you know, three working on it.

Uh, and, um, you know, at that point, it was kind of a, you know, quasi distributed system at least,

uh, that would kind of scale out and handle a bunch of use cases. So the, depending on when you count that, you know, first version, you know, it was, it was actually, you know, by that point, it was actually quite similar in terms of the overall style, what it enabled, not nearly as sophisticated or good as it is now, but it was, you know, the basic idea was there that kind of core abstraction was the same.

I see. Uh, and in this case, did you always intended for it to be open source or did that just happen by chance? Yeah, yeah, it was built to be open source from the beginning. Um, the, you know, that was kind of our strategy for those early infrastructure projects. It was, it's really hard to get people to work on internal infrastructure because it kind of, you know, it's some nameless layer inside the company that people only know about when it breaks. And so, you know,

this turned to these things into more products. And so my, my theory was, oh, you know, we can hire better people and, um, they'll do a better job because if you think about something like a product, then you're kind of like, well, it's my product good, right? Like, do people like it? You know, does it have a good interface? Is there documentation? So like, internal infrastructure, nobody will write any documentation. But if they open source, that it's like, oh, yeah, they'll document it.

So that was the theory. And, you know, it was most successful. Not all the things that we did got a ton of adoption. Um, you know, Kafka was early on extremely unsuccessful as an open source project into we kind of went and did some talks around it and tried to get some adoption in the rest of tech. But, but yeah, it was, it was intended to be open source early on because we thought, oh, this is, you know, this is, you know, I guess our feeling at the time was, hey, everything is

kind of moving to a world of these, you know, elastic distributed systems. So all the things that we had that were single server, you know, we'll kind of move to distributed, um, you know, but a lot of the work at that time was these databases, right? So it was different types of key value stores, data lake, whatever. It was some kind of storage system of some sort storage inquiry. And, you know, at least the feeling for me was, you know, hey, a more original and

interesting problem is the kind of movement or flow of data. So not like data storage, but, you know, kind of data, data in motion, data at rest. And, you know, so the idea was really, you know, trying to explore that and do that right. And, uh, having built one of, you know, whatever, 50 key value stores that all became open source at the same time, I thought, oh, you know, it's better to work in some area where nobody else is doing it because, um, you know, you know, uh,

you know, be more likely, uh, to, you know, kind of build something that people use. And, and I think that was true. Unfortunately, it was a little too true, uh, you know, being an area that nobody was interested in at the time, we initially didn't get any interest or adoption. But then, you know, I think as people realized, um, you know, how this could help with their kind of internal data

flow problems, it became much more popular. Uh, Kate, can you speak a little bit more about that in terms of, um, how you got more adoption, both like internally versus externally, like, uh, with the talks and things like mentioned? Yeah, we, you know, so the challenge for us was, um, you know, our vision was something that was a lot more than just like a simple message queue, but it was really hard to figure out how to explain it to people, right? So it's kind of back to that

product marketing thing. You know, I think we didn't really, we were like, well, what is it? At the end of the day, is it like a log? Is it something else? So the, the initial effort was, we kind of went around and did some talks. And so looking at it, I think I did a talk at like Airbnb and a few other places. And, um, those weren't well. People were excited about it. Um, as we were thinking, you know, we're kind of contemplating, oh, maybe there could be some kind of

company around this. And I wrote a blog post called the log. It was this really long thing. And the idea was, well, if we could get people excited about the bigger picture vision, you know, maybe there was some viability to this project. And if not, maybe not. So I spent a lot of time on it. Yeah, I probably spent like three full days just writing this long blog post that was like every thought in this area that collected in one place. And, and it was really popular. So then we were like,

okay, like people are excited about this. We just have to explain it to them. Even if we don't have a pithy one set and set explanation yet, at least we have a 20 page explanation. Uh, you know, they, they people can get interested in. And, um, you know, kind of following that, I think we decided, oh, you know, if we really want to take the project all the way, you know, it probably

makes sense to go start a company. Like at a certain point, LinkedIn only needs an internal infrastructure layer to be so good, you know, like maybe there's six people working on this, you know, maybe it goes to 10 people, but there's not, there's not a world where LinkedIn needs to build this the way all the other companies need it to be because it's just not there. It's not

their problem. And so if we kind of want to finish the job, you know, we're probably not going to be able to do that, you know, working in the, um, you know, in the side corner of a, you know, so, you know, professional network software thing. We got to go, you know, turn it into something with it directly goes out, figures out the problems that people have using this and adopting it and can start to build out some of the layers around it. They kind of complete the product.

And so that was the, that was the origin and that was, uh, that was when we kind of, um, you know, decided to go start company. Um, so how did it convince other shutters point, or did you and Neha, for example, all of you, all of you co-founder of the company, video all the same time, or did it require some, I think it was, yeah, I think we were all kind of thinking,

you know, I had definitely been thinking about it. I think Neha actually originally proposed the idea across the, you know, the, the three of us, um, but, but I think it was kind of in the air, you know, we were hearing from a lot of companies that they were interested in this, you know, I think at one point, um, I forget what it was, but it was, you know, some media or TV company was like trying to adopt the open source. They were like, okay, look, like we just need to pay you

to do these things. And we were like, well, you can't, you know, that's not really available. Like, we're, you know, like you can do the things in open source, but like we're not available to help you. So, uh, but we were like, well, okay, this is interesting. Like, at least, you know, it does look like this is applicable outside of tech companies. And so, yeah, that was a big question for us.

Um, you know, it was how broadly applicable is this? This is something where it's like, yeah, okay, all the parts of the tech company that has to happen in real time, but does that make sense elsewhere? Like with a bank work that way. Um, and it turns out, yeah, like these big, you know, other companies outside of tech are actually bigger, more sprawling, more business units, more systems that have been

built over more time, you know, more legacy. Yeah, yeah. So how it plugs together in a sense is even more valuable. Um, and so, so yeah, in fact, the need there was probably, you know, even better. And so, you know, as we saw that, we were like, oh, this is, you know, there's this fundamental transition from thinking of data as, you know, kind of almost static tables, right, where the fundamental thing in a database is how do I lock the data in place, run a query, and then kind of

unlock it, or, you know, some mechanism to pretend that things aren't changing. Uh, and then, you know, that's most exaggerated in a data warehouse where you're literally just like copying in a snapshot or something. And, um, you know, that, that, that's a very different model from the kind of flow,

you know, flow of data across, uh, across the company. And so, okay, there's a, you know, there's a very fundamental change that's going to happen because ultimately, you know, companies are, out there in reality doing business in some way, you know, and that's a very continuous activity. Um, you know, so like a LinkedIn, it's available all around the world. There's things happening

all the time. It doesn't make sense that it like, you know, we at 12 p.m. Pacific, we like, copy a bunch of data into some data lake and run our most sophisticated processing, like there's nothing magical about 12. That was just a convenient time to schedule it, right? Um, you know, the, the natural way of doing things would be some kind of more continuous stream processing. And so, you know, it seemed like, okay, inevitably that change has to happen because the problem in the world

doesn't match the underlying infrastructure layer. And there's some valuable initial step that's applicable to a bunch of companies today, right? That, that we have. So if you put those two things together, you kind of have a valuable first step with a really valuable next 10 steps, you know, that could be a business in some form. I don't think we knew exactly what form, but the, uh, you know, at least it seemed like a, you know, good direction to start walking. And, uh, I remember reading

that lock was the first time I learned about Kafka. And, uh, it was extremely valuable. And that was like, I was just starting out and engineering and, uh, just reading a bunch of things just went way over my head at the time. It's like, why stuff is this really a problem? And like, as time went on, I learned the value of it. Uh, but when you're starting this company and let's say you're pitching to investors, as you mentioned, what did the evolution of that pitch look like at that point? Or

what was that petty one line of two line statements that you could share? Yeah, yeah, that was, um, that was definitely the evolution of my understanding of how venture capital works. So the, the initial pitch was, um, you know, these are the trends in data and infrastructure. It was like, okay, look, look, there's more and more complicated specialized systems and applications, right? So

there's a ton of sprawl. And then the volume of data and what we consider data is growing. And then the needs from customers, like they want better experiences or the digital part of the business growing. So effectively, there's like more data. You have to use it in more sophisticated ways across the company. So all the parts of the company need to continue to kind of connect together. And there's going to be a need for infrastructure for that. And, um, then sort of at the end,

we were like, oh yeah, you know, and we're a team that has done this very large scale. And there's open source, which is like, you know, very broadly adopted that people are building, like big, important production systems to bound. And so then what I learned was people mostly, like, tuned out the whole first part of theoretically, why this would definitely happen. And kind of ignored, you know, the first, you know, whatever it was, 25 minutes of a 30 minute pitch.

And at the very end, they were like, oh, people are using this. And it's recognizable companies doing really important things. So then I was like, oh, huh, take the side about the traction, put that first. Then explain the theory. Then they're very interested in the theory. But the first thing is actually just like, yeah, there's that thing happening in the world that I can prove it. And that requires heavy involvement and investment from companies. And that means,

you know, there's a good chance they'll buy a product in that space. So, and indeed, it's actually a smart way of thinking. You know, it's very easy to come up with clever ideas. It's very hard to produce actual results. And you see that, you know, I think it's true inside of a company as well, you know, the kind of actual results speak a lot louder than theory.

And once you have the results, then you're very interested in the theory. But, you know, as long as it's purely theoretical, you know, we can all remain skeptical or have different ideas. Once you've kind of shown it, then it's a little different. You know, one of the results, it's the hardest to get is always, you know, something that people actually want. But, you know, it's very hard to take time out of your day to go adopt some new software platform

and build around it. Yeah, that's a big undertaking. And so, if you're choosing to do that, there must be something that's very valuable to you that you would take that risk, you know, invest that time. And so, that's actually a very good signal. Yeah. And Kafka, in this case, as an idea, was already validated in a way with like open-source adoption. And like, again, when you were building Kafka originally, you were trying to solve this

problem, like, instead of data at rest, putting data into motion. Business is not something that many need to think about very, at least organically. It doesn't come to many of us nationally. So, what did that look like in terms of developing a business around it? Like, how would you go about figuring out, this is how we can charge people for it. This is how we can pitch. This is how much we

should raise. Yeah. Yeah. It was an interesting thing, you know, as we were starting, you know, at that time, there were some open-source companies that weren't very successful or very well thought of. You know, Cladero was a company, but it, you know, it's kind of grown to size, but you know, had a number of challenges and kind of skeptics. And, you know, which actually increased over time, but you know, at the time we were pitching it, that was the state of things. So, that was kind of

the only model you could point to. There was some, you know, much older things like Red Hat that it didn't, it was really a replicable model. And so, you know, yeah, that was the question is, hey, you know, is there a product here that can be successful? And, you know, we felt like, yeah, there is. The dilemma for us early on was whether we wanted to start with a software product or

a cloud offering. And it was, indeed, it was a very big dilemma because, you know, at that time, if you looked at where money was spent, it's probably 90% on promise and maybe 10% in the cloud, right? And if you looked at the size of like the data streaming market, it was basically like $0.00. Approximately, rounds to zero. So, we were like, well, you know, on one hand, like, you know, cloud offering is a really great way to deliver this. But on the other hand, it's not really

clear that you can take a really early market that it's going to grow and just building 10%. Like, first of all, somebody else will do the other 90%. Second of all, it's just not enough to like get going and start a company. And so, you know, it was a difficult time in that respect because I felt at least for us, we had to do both early on. And that's very challenging to try and have,

you know, two models or two products. It ended up being very valuable because a lot of the companies now are now in the same situation where they're actually spread across their own data centers and cloud instances and so on. So, that ability to kind of span all of that ended up being a really big deal for customers that have that problem. But for the initial part of the

company was quite challenging to be able to build both products in the way that was good. But, yeah, that, you know, the initial challenge for Confluent was, yeah, go figure out how to sell it because we were actually kind of starting a little bit ahead. You know, we had at least an open source offering that was, you know, pretty successful building, you know, a very like product around that was not super hard. So, yeah, the first challenge for me was like, okay, go figure out

how to build an enterprise-good market, which I need nothing about. And, you know, I just had the the un-luckiness for that task to fall on me because, you know, the other two co-founders had proper technical jobs. And, I was like, oh, okay, you can do it. And, you know, in indeed, I was quite clueless, like, even as we were pitching the company, you know, I think some investor asked me, like, oh, what, you know, what's your strategy with services? And, you know, what they meant was

consulting. But I actually had no idea. I was like, I thought they meant web services. And I was like, yeah, you know, we'll definitely build some web services. But like, I don't think that's the first thing we're going to start with. And the guy looked at me like I was, you know, crazy. And, but I wasn't crazy. I was just completely ignorant. Like, I'd never worked in an enterprise company. I had no idea that, you know, there was a whole consulting services strategy that would

go along with adoption. And, you know, so that was just the state of things. But, you know, the good news was we had people who were using the open source that we could go talk to. And so, you know, the nice thing about enterprise products is you can actually just ask the customer, like, what would you buy? And, you know, you got to be thoughtful about, well, you know, everybody tells you nice things, but would they really do it? And so, yeah, just, you know, at that point, it was just kind of a

blunder through it. And so, we, you know, we hired some early sales reps. You know, I hired somebody who had, it was just kind of a generalist. And then in different roles in their enterprise companies and could help build that out around that part of the organization. And so, yeah, the,

yeah, the initial, you know, customers, I just kind of went and found. And then as we had sales reps that helped to actually get deals closed because it turned out I was much better at the sales pitch than the, you know, getting the final money out of them at the end, which turns out to be a skill in its own right. So, the, yeah, so, so, you know, I was actually very close to the kind of early

sales and marketing effort for that reason. And that's the best way to learn something, you know, I think in a large company that go to market ends up quite complex over time in terms of different segments and strategies and teams and how it all fits together. But in an early startup, I mean, it's just very simple. It's, you know, it's like if you want to learn how, you know, transportation works, you want to look at a bicycle, not a, you know, combustion engine automobile, right? Because

it's just one is so much simpler than the other. You're just like, oh, yeah, you turn the pedal and wheel goes right. And it's kind of like that. It's like, okay, like what, what, you know, what do we need to do? You know, we need to find a way to tell people about the thing we have and why it's good. And then we need to have a way to allow them to give us money to get the good thing and then

help them use the good thing. And so it's not, you know, it's not rocket science problem. You just have to, you know, figure out how you're going to accomplish that with a relatively small group of people in an early company and then how it will kind of grow and evolve over time. So yeah, it's, you know, I've been relatively close to that side of the business sense, but, you know, of course, we brought in really amazing people who did all the hard stuff. Once, once we passed this very early

stages. As you were picking up these like new skills in a very pressurized and like a rabbit manner, was there any sort of lessons that you've learned from engineering that you were able to apply to these new things that you were learning? Yeah, absolutely. I, you know, it's interesting. I think engineering is probably the least useful skill in a lot of ways for becoming a CEO. But there's some parts of it that are really good. So why is it the least useful? Well, it's like,

you actually don't need the CEO to write a lot of code. In fact, it's not that helpful if the CEO is trying to write a lot of code because they're usually not around to like do all the work through to production and support it right at best. They kind of fling something from some night, you know, coding session and hope somebody like fixes it and makes it work, which nobody really

wants to have. So, you know, so the course skill set is actually not a useful thing. You know, what a CEO's do, they do a lot of communication, they do, you know, you might instantly do some of that as an engineer, but of course you do some of that in every role in the company. So, but there are some things which are good. You know, one is engineers have to learn a lot of stuff. So you're kind of just on this treadmill of learning new stuff and usually engineers have high

confidence in their ability to learn something. And so if there's something new, they're just, you know, they'll just read about it and you know, there's a belief that you can do that. I think that's really important for a CEO because actually, you know, it's one of a couple of roles that sits across the CFOs this as well where you sit across every function. You have to kind of understand how they work and you probably won't understand it as well as somebody who's run that function for

their whole career. But you need to understand it well enough to know, you know, if that's going well, you know, if you've hired the right person to do it, you know, to have an opinion on what's needed there to kind of help it guide it and integrate it into the rest of the company. So you need to understand

it pretty well. And so if you're not, you know, if you're not going to spend a lot of your time kind of learning and thinking about things outside of what you already know, it's not going to go well. And then I think as a company gets bigger, you know, I do think that ultimately an organization is a kind of system. And so that kind of system thinking and design, you know, kind of modularity. How does this module, is this module overly coupled to that module?

You know, do we have a good API? You know, what's our observability story for knowing if this thing is working? You know, what, like, what's my feedback loop to like run this part of the business and know it's on track versus no, it's not on track. I think that style of thinking is, you know, extremely important in software engineering. And actually very much what you do for a larger company is definitely, you know, that those same kind of things, which is okay.

You know, you peer inside a module and try and get it in good shape and working. And then ideally you want to have, you know, what are the outputs of that that I can reason about. I know kind of what the input is, like what's the investment and what's coming back. If that's good and it seems like there's somebody we trust running it, then I can just kind of let it run. That's the observability story. And if not, then, you know, maybe I need to get in there and do some profiling and see what.

So yeah, I think that's, you know, that's good. And then I think this is true of engineers. Generally, it may be even particularly true of these distributed systems people. It's actually a relatively gritty profession where you spend most of your time on things that are not working. And I think that's true of management roles as well. You know, you spend most of your time fixing problems and trying to get stuff to work. And then once it works, it's like, okay, right, onto the next

thing. And so, you know, there's some analogy there as well. And so, you know, if you look at people who run tech companies, there's actually good percentage of engineers, even though the actual core skill set is not that useful. And I think maybe it's for those reasons. That's super fascinating. On the learning part, right, I remember the memes about front-end engineers having to learn new framework every two years. Yeah. But now I'm like, oh,

maybe that's not a bad thing. I mean, just keep your skill set of how to learn new things in check. So that's really powerful. Thanks. So it's not every day we speak with the CEO, a public company. And you took conference public, started with Kafka as a project to start up to now, a public company at this point. So great achievements all around. What does a typical day in your life look like? So I'll post two questions and you can take it

any other direction. So like that part, the other part is as you're describing all of this, having analogies for distributed systems to basically having well-functioning orgs, have you developed certain practices to stay productive and kind of get ahead of the problem a little bit and still keep some of your time free to think because I'm sure there are a million things wanting attention. Yeah. You know, the job for a CEO is there's different ways to do it.

Right. So I've seen kind of two models. One is people effectively manage at a relatively high level and just hold accountability across different organizations and then put a lot of effort into the public face of the company externally and the face of the company internally, the integration with different stakeholders, which tends to be the employees, customers, and the investors. So you know, one model is you kind of spread yourself across and that's the full-time part.

The other model I think is a little bit more t-shaped, which is okay, you try and take that horizontal bit and actually get that down to about 40% of the job so that you know, in steady state, your day would be about half free and then the other 50% is just very topical. So at any given point, there's some big problems or some big opportunity, right? So it can be something quite looking what's the next thing we're going to do or what's the horrific tragedy that

is unfolding that we need to solve. And then you dedicate about half the time to that. Usually it's like one or two things that can slide into that and you have to just be very disciplined. And so I definitely follow the second model, which is I ideally want to push down the steady state commitments to a reasonable amount. And then the remainder of time is relatively kind of flex. And then I'll have a focus area at any given time that I'm more deeply involved in

and that could be an area where I'm trying to hire a leader for the team. It could be an area that I don't think is on track or it could be an area where there's some big opportunity or unlock. They're like, oh, if we could get this right, you know, that would be great. And then I'll try and dive in actually at a relatively low level and just really understand what's going on there is best.com and try and help get it into good shape.

These like bigger these blocks, I do they just kind of come up or do you have like a system to kind of track them so that once you do have free time, you're like, sorry, flexible time, you start digging into one. Yeah, yeah, the, you know, I always feel like time management is just a process. You know, I feel like you're out in the jungle with like a machete and the overgrowth that grows and it just consumes everything, the vines and the weeds and whatever and then you kind of

hack it back. But the jungle just keeps growing and so, you know, so I think it's a little bit like that where effectively the day to day activity is just kind of eat up more and more because like things creep and there's always another one on one and another checking and another something that gets proposed and then you just kind of hack, hack it back and you're like, okay, you know,

got that done and then it grows back again. So I don't know if that's a methodology, but yeah, it's just calendar management like where, you know, look at the calendar, where, where am I actually spending time? Does that actually match my priorities and is all my time accounted for? If all my time is accounted for, then there's effectively no flex time for whatever the most important thing is, which seems wrong. It seems like there should be some time for the most important

thing. So, so then, then, you know, then the machete comes. Speaking of time, we are, we are reaching towards the end of our conversation while we have a bunch of questions to us, but we'll save them for another time. And thank you so much for taking the time from your busy schedule, but before we let you go, one last question. In terms of putting data in motion, I think you have been successful at that with the open source project with Confluent as a company, where Kafka

has become this de facto choice. People don't ask what should I use if I want to move data around. It's like, we're use Kafka, like that's just the option. What is something that you're looking forward to next? Yeah, I think it's still early in the journey, you know, that in what we wanted to achieve, we always felt like Kafka was kind of like step one, right? So, you have to have some stream that people agree on that everybody will, you know, write the data out to and read the

data from, but that's not the end of the stack, right? So, if you think about what are the needs for that, you want to be able to do really rich, easy, real-time processing of data, you know, that kind of stream processing and the reprocessing and so the, you know, the evolution of the ecosystem around that is at least as important, you know, in the product as that kind of core stream itself.

To some extent, you want that corbite to just be solved, like it's like a utility. It's like, you know, the wires in the house conduct the electricity, like if they don't, that's the big problem. But if they do, it's not something you spend all day thinking about, right? And so, a lot of what the electricity goes to, that's kind of the exciting thing. So, yeah, you know, so our goal is continue to build out that, you know, platform around data streaming and fill out those other

pieces. You know, so we've been putting a lot of work into our offering of fling. You know, this is a stream processing system, you know, typically for stream processing, it's been pretty ungainly to run these systems yourself. You know, it's relatively hard. The opportunity in the cloud is really take that away, like make something, it's just very elastic where you kind of write the code, you write the query, it goes and runs. You know, I think there's really an opportunity to take

stream processing and do it in a way that's kind of a generalization of the batch world. So, you know, I said, okay, a batch system, you kind of freeze the data and place inquiry at some point in time. You know, the opportunity in the stream processing system is, you know, you kind of process at a point in time, but as new data comes, the answer just keeps evolving, right? And that's a very powerful generalization that matches a lot of the activity and applications in a company and makes

it vastly easier to build around streaming data. So that kind of growth of the platform that makes it really easy to connect into all the systems, really easy to process the data, really easy to kind of govern it in the large. This is something a lot of the companies that build the most around this have kind of built in house, but our, you know, our goal and a lot of the work we're doing now is

really, you know, bring that out as part of the product. So you can just have something that's kind of, you know, very cost effective, very easy to scale, just kind of solves this problem for you to every company, you can have an awesome infrastructure that everything plugs into around streaming and make that like a really default way of working, you know, really obvious way that systems would interact and the applications would be built. And that's as important as the kind of database

world in companies. So that's, you know, that's the journey that's ahead. And you know, to do that, of course, as well beyond Kafka, it's really how all these parts come together into a, into a URVAD product. We look forward to our comes next. And Jay, thank you so much for joining us today. We had a great time speaking with you and learning from you. We have a ton of more questions to us. So we do hope there is a second time and we do get to speak with you again in the future.

But for today, thank you so much. My pleasure. Thank you guys. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremissadventures.com. You can also write to us at helloatsofffirmissadventures.com. We would love to hear from you. Until next time, take care.