#23 – Sujay Jayakar: Dropbox, Convex

There's another kind of interesting decision here on Dropbox by design was always like a sidecar. It's always something that just sits and it looks at your files. Your files are just regular files on the file system. And if Dropbox, the app isn't running, your files are there and they're safe, and it's something that you know, regular apps can just read and write to, and in some sense like Dropbox was unintentionally local-first from that perspective, right?

Because it's saying that no matter what happens, your data is just there and you own it. Welcome to the localfirst.fm podcast. I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and love the craft of software engineering. For the past few years, I've been on a journey to build a modern high quality music app using web technologies, and in doing so, I've been following down the rabbit hole of local-first software.

This podcast is your invitation to join me on that journey. In this episode, I'm speaking to Sujay Jayakar. Co-founder of Convex and Early Engineer at Dropbox. In this conversation, Sujay shares the story on how the Sync Engine powering Dropbox was built initially and later redesigned to address all sorts of distributed systems problems. Before getting started, also a big thank you to Jazz for supporting this podcast. And now my interview with Sujay. Hey, Sujay. So nice to have you on the show.

How are you doing? Doing great. Great. Really happy to be here. I'm super excited to have you on the show. I've been using your work really since over a decade at this point when I was really getting into using computers productively. And we just the other time had another really interesting guest, Seph Gentle on the podcast, who has worked on a really fascinating tool, called Google Wave back then that had a big impact on me.

And you've been working on another technology that had a big impact on me, which is Dropbox and still has a very positive impact on me. That was all the way back then over 10 years ago in 2014.

I don't think I need to explain to the audience what Dropbox is, but, I want to hear it from you, like what led you to join Dropbox, I think very early on and just hearing a little bit just embedded in your personal context when you joined it, and then we're gonna go dive really deep into all things syncing related, et cetera. How does that sound?

Yeah, that sounds great. It's actually a really funny story. my career here in technology started in 2012. I was actually studying mathematics. I was going to go work at the NSA doing cryptography, and I was born in India. but I'm a naturalized citizen for the United States, and, you have to be, have security clearance to go do these types of cryptography things.

And you know, my clearance kept on dragging on and on and on and they like interviewed my roommates and apparently just a very sketchy guy so I had an offer to go work there, but it kept on dragging on. And then my roommate, at the time was a computer science major who wanted some like someone to go with him to the career fair and, just started chatting with the Dropbox people and you know, it's about like a hundred people around that time.

And, chatting turned into hanging out at dinner, turned into interviewing and being a math person, I did my interviews all in Haskell and didn't know any real programming. and then yeah, that turned into doing an internship, dropping out of undergrad and. just following the dream. And so I worked on, at Dropbox, I worked on a bunch of things. I started off working on our, like, growth team. So I did a lot of like email system.

Like I did this, I worked on this thing called the space raise, like a promotion. Oh, I remember that. Yes. I think I've, I've earned quite a lot of like free storage, which I think over the time has like gone down. But that was a very smart and effective mechanism. I surely invited all my friends back then. I couldn't afford a premium plan being a broke student that worked. And then from there worked on the sync engine for some time.

And then right now I'm the co-founder and chief scientist of a startup called Convex and my three co-founders and I met working on this project called Magic Pocket, where Dropbox stores hundreds of petabytes now exabytes of files, for users. And we used to do that in S3. And so the three of us worked together on a team to build Amazon S3, but in-house and migrate all of the data over.

so we did that for a few years and then, Worked on rewriting the entirety of Dropbox, the sync engine, the thing that runs on all of our desktop computers. we rewrote it to be really correct and scalable and very flexible. and shipped that. after that left Dropbox in 2020 I was trying to decide if I wanted to get back to academics or not. So I did some research in networking and then decided to start Convex in 2021.

Certainly curious, which sort of research has had your interest the most in this sort of transitionary per period, but maybe we stash that for a moment and go back to the beginning when you joined Dropbox. you mentioned there were around a hundred people working there currently. how do I need to imagine the technology behind Dropbox at this point?

it clearly started all out with like, desktop focused like daemon project, like daemon process that's running on your machine somehow keeps track of the files on your system and then applies the magic. So explain to me how things worked back then and what was it like to work at Dropbox when there were around about a hundred people.

Yeah, I mean, it was pretty magical, right? Because the company had, I think gotten so many things right on the product side and then those showed up in technology. But just this feeling of like Dropbox being this product that just worked right? It was for everyone. It was not just for technologists, but anyone should be able, anyone who's comfortable using a computer should be able to install Dropbox and have a folder of theirs become magical.

And without understanding anything about how it works, they should just think of it as like an extension of what they know already. yeah. And so like the ways that that showed up I think were really interesting. At the time there was a very strong culture of like reverse engineering. So to have this daemon that runs locally. You know, there was one of the amazing early moments in Dropbox was that, if like you open up finder or explore and you have the overlays on it.

Like that used to be done by like attaching to the finder process and injecting code into it and to the point where, uh, when some folks had gone to talk to Apple at the time and about like working with the file system and everything like the, there were teams at Apple that asked Dropbox, how did you do that in Finder? So you wanted to offer the most native experience. There weren't the necessary APIs for that. And so you just made it happen. That's amazing.

Yeah. Yeah. And so that that idea of like, how do you create the best user experience, something that you know, for the purpose of making non-technical users feel very confident and feel very safe using it. That was another, I think, really deep like company value of like being worthy of trust and taking people's files very seriously.

You know, I like remember having a friend who was in residency at the time and he was telling me that he keeps all of his, like some of his non HIPAA stuff, but like his things that he looks at on Dropbox and you know, pulls him up and he's consulting 'em. And there's a part of me which is terrified by that, right? Like we think of software as something where like throwing a 500 error is fine every once in a while.

And a Dropbox that was there was a culture of making users feel like they could really trust us. And then that showed up for things like making sure that, like when we give feedback to users, if we put that green overlay in finder. They know that no matter what happens, they could throw their laptop in a pool. They could like they, anything could happen and their files are safe. Like if their house burns down, they don't have to worry about that thing.

And that's like all of that reverse engineering and all of the emphasis on correctness and durability. It was all in service of that feeling, which I think was really cool. so on the engineering side, at the time it was like in hyper growth mode. So they had a Python desktop client. Almost all of Dropbox was in Python at the time. And so there's a pre my py, like big rapidly changing desktop client that needed to support Mac, windows and Linux and all these different file systems.

and then on the server, it was like we had one big server called Meta Server, meta, I think was from metadata. and that like ran almost all of Dropbox. We stored the metadata in MySQL. The files were stored in S3, and then we had a separate notification server for managing pushes and things like that. And so it was like kind of classic architecture and like reach was starting to reach the limits of its scaling even at that time.

And, those were a lot of the things we worked on over the next 10 years. Wow. So was the server also written in Python? So it was all one big python shop. Yeah. And the server was all written in Python. we, had some pretty funny bugs that were due to it's kind of crazy to think about it now.

You know, we, you working in TypeScript and full time and to think of, like back in the day we just had these like hundreds of thousands, millions of lines of code with no type safety and with all types of crazy meta programming and decorators and meta classes and stuff. And yeah, so there was a, it was all in Python when I showed up. it was not all in Python and not all in one big monolithic service when I left.

So you mentioning joining when there were around a hundred people and you probably already at this point had like multitudes more in terms of users. Being in hypergrowth, it is sort of this race against time where you only have so much time to work on something, but growth may be outrun you already and things are already starting to break. Or You know like, okay, if things gonna grow like this, this system will break and it's gonna be pretty bad.

So tell me more about how you were dealing with like the constant r race against time to rebuild systems, redesign systems, putting out fires. What was that like?

Yeah, and I think there's like kind of an interesting place to take this on. I think like the normal things were on scale right there. Those were like. One, kinda class of problems of being able to handle the load. But I think one kind of really interesting, dimension of this that led to our decision to start rewriting all of the sync engine in 2016 was actually just like customer debugging load.

You know, we have we had hundreds of millions of active users and they were using Dropbox in all types of crazy ways. Like one of the stories is someone was using Dropbox with like, I think it was running on some, I don't know if it was like a raspberry pie or something, something on his tractor. Like the guy ran a farm and he was using Dropbox to sink like pads in text files to his tractor. And I might be getting some of the details wrong, but it's something like that.

And so people would just use Dropbox in all types of crazy ways on crazy file systems with kernel modules running that are messing things around or so I think, You know, in terms of getting ahead of scale, I think we found ourselves around 2015, 2016, in the place where for the syn engine on the desktop client, the entire team just spent all of its time debugging issues.

We had this principle of like anything that's possible, anything that a protocol allows, anything that some threading race condition that's theoretically possible will be possible. And then we would see it, right? Like users would write in saying, my files aren't sinking. And then we would look at it and we would spend months debugging each one of these issues and trying to read the tea leaves from traces and reports and reproductions.

And it'll be like, oh they mounted this file system over here and then this one and this one are in a different file system. So moving the file actually did a copy, but then the X adders were in, preserved this and that. You know, in terms of that theme of like getting ahead of scale, like I think there was first this realization that like the set of possible things that can happen in the system is just astronomically large. And all of them will happen if they're allowed to.

And we do not have, no matter how much like incremental time we put into debugging things, we will never be able to keep up. And the cost of doing that is that the entire team is working on maintenance like this. We couldn't build any new features.

So I think that was a motivation then for the rewrite to is can we find like points of leverage where if we just invest a little bit in technology upfront, like by architecting things a particular way, can we just eliminate a much bigger set of potential work from debugging and working with customers and stuff like that. So maybe this is a good time to take a step back and try to better understand what was Dropbox sync Engine actually back then?

So from just thinking about it through like a user's perspective, I have maybe two computers, and I have files over here. I. I want to make sure that I have the files synced over from here to here. So I could now think about this as sort of like a Git style, approach. Maybe there's other ways as well. walk me through sort of like through the solution space, how this could have been approached and how was it approached?

is there some sort of like diffing involved between different file states over time, those are being synced around. Do you sync around the actual file content itself? Help me to understand. Building a mental model, what does it mean back then for the sync engine to work?

Yeah. Yeah. It's a super interesting question, right? Because I think like you're saying, there's so many different paths one can take and it's, I think one of those things where like if someone asks, like design Dropbox in an interview question, there's like definitely not one right answer, right? It's like there are so many trade-offs and like different forks in the decision tree.

I think one of the first things is that, so you have your desktop A and you have your, maybe you have your desktop and your laptop, and one of the first decisions for Dropbox is that we would have a central server in the middle, that there would be a Dropbox file system in the middle that Dropbox, the company ran, and we did that from this trust perspective, we wanted to say that we will run this infallibly when you get that green check mark when it's there.

You know, even if an asteroid destroys the eastern side of the United States, like we will have things replicated in multiple data centers. And that you know, and then also it's accessible anywhere on the internet, right? You can go to the library. This is not so common these days but I remember when I was a student, like, go to the library, log into Dropbox and read all your things right? rather than having to bring a USB stick around.

And so I think that is the first decision, but it's not necessary, right? Like there were plenty of distributed, entirely peer to peer file syncing, designs, right? And so that was the first decision. And I think the kind of second decision was that if we imagine our desktop and our laptop and you have the server in the middle, the desktop might be on Windows, the laptop might be on Mac OS. So I think that decision to support multiple platforms. Is like another really interesting one.

This is like where I think Git and Dropbox can be a little bit different. And that Git is at the end of the day quite Linux centric. It's case sensitive for its file system. It deals with directories and it makes particular assumptions about how directories should behave. And that was something with Dropbox. We wanted to be consumer, we wanted to support everything and we wanted it to feel very automatic, right?

That like, someone shouldn't have to understand like what a, like unicode, normalization disagreement means. Right? Where in Git like in really bad settings, like you might have to understand that, that you're right, you with an accent differently on Mac and Windows. so I think that's the kind of like next, side. So then Dropbox has its design for a file system and it's a central, it's like the hub and all those folks are your phone, your. desktop, your laptop and whatnot.

and then so to kind of get down to the details a bit more. So then, yeah, we have a process that runs on your computer, that's the Dropbox app, and that watches all of the files on your file system, and then it looks at what's happened and then syncs them up to the Dropbox server. And then whenever changes happen on the Dropbox server, it syncs them down. there's another kind of interesting decision here on Dropbox by design was always like a sidecar.

It's always something that just sits and it looks at your files. Your files are just regular files on the file system. And if Dropbox, the app isn't running, your files are there and they're safe, and it's something that you know, regular apps can just read and write to, and in some sense like Dropbox was unintentionally local-first from that perspective, right? Because it's saying that no matter what happens, your data is just there and you own it.

and you know, there are other systems, right? Like if you use NFS a like a network file system, then if you unmount it or if you lose connection to the server. You might not be able to actually open any files that you have the metadata for. Right. And I remember from a user perspective, the local-first aspect, I really went through like all the stages where I had a computer that wasn't connected to the internet yet, and that at some point I had an internet connection.

But, files were always where like everything depended on files. Like if I didn't have a file, things wouldn't work. Everything depended on files. There were barely websites that where you could do meaningful things. certainly web apps weren't very common yet. And then Dropbox made everything seamlessly work together. And then when web apps and SaaS software more came along, I was a bit confused because I felt Okay.

I t gives me some collaboration, but seems to be a different kind of collaboration since I had collaboration before. But I also understood the limitations of, when I'm working on the same doc file, through Dropbox, which gets sort of like the first copy, second copy, third copy, and now I need to somehow manually reconcile that. And when I saw Google Docs for the first time. That was really like a revelation because, oh, now we can do this at the same time.

But at the same while I saw that, I still remember the feeling where, but where are my files? This is my stuff now. Where, where is it? And that trust that you've mentioned with Dropbox, I felt like I lost some, some control here and it required a lot of trust, in those tools that I started now step by step, embracing. And frankly, I think a lot of those tools didn't deserve my trust in hindsight.

I still feel like we've lost something by no longer being able to like call the foundation our own in a way. And I'm still hoping that we kind of find the best of both worlds where we get that seamless collaboration that we now take for granted. Something like that Figma gives us. but also the control and just being ready for whatever happens, that's something Dropbox gave us out of the box.

I just wanna share this sort of like anecdote and like almost emotional confusion as I walk through those different stages of how we work with software. Totally. And we've ended up in a place that's not great in a lot of ways. Right. And I think you know, I think part of the sad thing, and maybe from even like an operating systems design perspective is that I feel like files have lots of design decisions that are. Packaged up together.

You know, like one of the amazing things about files is that they're self-contained, right? Like on Google, I don't know what Google's backend looks like for Google Docs, but they probably have like all of the metadata and pieces of the data spread and different rows in a database and different things in an object store. And just even thinking about like the physical implementation of that data, it's like scattered around probably a bunch of servers, right? Maybe in different data centers.

And there's something really nice about a file where a file is just like a piece of state, right? That is just self-contained. And I think the thing that I think is one of the things I think is very unfortunate is like from a operating systems perspective is that that decision then has also been coupled with a very anemic API like with files, they're just sequences of bytes that can be read and written to and impended and there's no additional structure beyond that. And I think like.

Folks the way that things have evolved is that we've given up on, too have more structure, too make things like Google Docs, too be able to reconcile and have collaboration and interpret things more than just bites. We've also given up this ability to package things together. Mac os had like a very kind of baby step in this direction with I think they're called bundles. Like the things where like if you have like your.app, they're actually a zip file, right?

And there's all types of ways, all types of brain damage for how this like, doesn't actually work well. You know? But the idea is kind of interesting, right? It's like what if files had some more structure and what if you still considered something, an atomic unit, but then it had pieces of it that weren't just uninterpretable bites. And I think that's like, the path dependent, way that we've ended up where we are today. That makes sense.

So going back to the sync engine implementation did the Python process back in the day did that mostly index all of the files and then actually send across the actual bites probably in some chunks, across the wire? Or was there some more intelligent and diffing happening client side that you would only send kind of the changes across the wire and how do I need to think about what is a change when I'm dealing with like a ton of bites before and a ton of bites after?

Yeah. It's really, really good questions. I think maybe like the first starting point is that like files in Dropbox were stored, just broken up into four megabyte chunks. And that was just a decision at the very beginning to pick some size. And on the server, the way that those chunks were stored is that they, each four megabyte chunk was stored by key to by its shot to 56 hash. So we would assume that those are globally unique.

So then if you had the same copy of a bunch of file, or you had a file copied many times in your Dropbox, we would only store it once. And that would just happen organically because we would say like, okay, I looked at this file, it has three chunks A, B, and C. And then the client would ask the server, do you have A, B, and C?

Like the server would say, yes, I have B and C already, please send A, then we would upload A. so there was already like at the file level there was this like kind of very coarse grained Delta sync. at the four megabyte chunk layer. and then the kind of, it's funny, these things evolve, right?

Like then the next thing we layered on up top was that in that setting where you decided B and C were there already and you needed to upload a then with a, the desktop client could use rsync to know that there was previously a prime and do a patch between the two and then send just those contents.

the kind of thing that was pretty interesting is that a lot of the content on Dropbox was very incompressible stuff like video, images, so the benefits of deduplication both across users or even within a user. And the benefit of like rsync was not actually as much as one might think, at least from the like, terms of bandwidth going through the system. It wasn't that reductive because a lot of this content was just kind of unique and not getting updated in small patches.

And on your server side, blob store, now that you had those hashes for those four megabyte chunks, that also means that you could probably deduplicate some content across users, which makes me think of all sorts of other implications of that. When do you know it's safe to let go of a junk? do you also now know that, you could kind of go backwards and say like, oh, from this hash, we know this is sensitive content.

And have some further implications for, whatever we don't need to go too much into depth on that now, but, yeah. I'm curious like how you thought of those design decisions and the possible implications. Yeah. Yeah, for the first one yeah, like distributed garbage collection was a very hard problem for us. We called it vacuuming and in terms of making Dropbox economics work out of, like, when we couldn't afford to keep a lot of content that was deleted that we couldn't charge users for.

So that was you know, there's all additional complexity where different users would have like the ability to restore for different periods of time. So we would say like, anything that's deleted, it doesn't actually get deleted for 30 days or a year or whatnot based on their plan. so then, yeah, like having to do this like big distributed mark and sweep garbage collection algorithm across hundreds of petabytes, exabytes of content that was something that we had to get pretty good at.

And when we designed Magic Pocket, where we, implemented S3 in-house, we had specific primitives for making it a little bit easier to avoid race conditions where like, if a file was deleted. And we decided that no one needed it anymore. But then just at that point in time, someone uploads it again, making sure that we don't accidentally delete it. So that was like, yeah, definitely a very tricky problem. And I think in retrospect this is like an interesting design exercise, right?

And that if deduplication wasn't actually that valuable for us, we could have eliminated a lot of complexity for this garbage collection by not doing it right. I think for the second thing, yeah. So at the beginning when Dropbox started, if you had a file with A, B and C and you uploaded it, it would just check, does A, B and C exist anywhere in Dropbox? And, that got changed over time to be does do you as your user have access to A, B, and C?

And you know, 'cause otherwise you could use this for all types of purposes, right? To see if there exists some content anywhere in Dropbox. And, that was something where we would in the case where the user was uploading A, B, and C, say none of them were present in their account, we would actually force them to upload it, incur the bandwidth for doing so, and then discard it if B and C existed elsewhere. Yeah. Very interesting.

I mean, this would be an interesting rabbit hole just to go down just the kind of second order effects of that design decision, particularly at the scale and importance of Dropbox. But maybe we save that for another time. So going back to the sync engine, now that we have a better understanding of, how it worked in that shape and form back then. You've been already mentioning before, like as things as usage went through the roof, all sorts of different usage scenarios also expanded.

you had all sorts of more esoteric ways, how you didn't kind of even think before that it would be used this way. Now all of that came to light. I'm curious which sort of, helper systems you put in place that you could even have a grasp of what's going on since a part of the trust that Dropbox owned or that earned over time, was probably also related to privacy.

So you, you couldn't just like read everything that's going on in someone's system, so you're probably also relying to some degree on the help of a user that they like send something over. Yeah. Walk me through like the evolution of that and that you, like as an engineer, if there's a bug reproducing that bug is everything. So walk me through that process.

Yeah, and you know, like we had a very strict rule, right, where it just, we do not look at content, right? and so that was the thing when debugging issues, the saving grace is that for most of the issues we saw. They were more metadata issues around like sync, not converging or sync, getting to the client thinking it's in sync with the server, but them disagreeing. so we had a few pretty, yeah, like pretty interesting supporting algorithms for this.

So one of them was just simple like hang detection, like making sure, like if, when should a client reasonably expect that they are in sync? And if they're online and if they've downloaded all the recent versions and things are getting stuck, why are they getting stuck? So are they getting stuck because they can't read stuff from the server, either metadata or data? Are they getting stuck because they can't write to the file system and there's some permission errors?

So I think having very fine-grained classification of that and having the client do that in a way that's like not including any private information and sending that up for reports and then aggregating that over all of the clients and being able to classify was a big part of us being able to get a handle on it. And I think this is just generally very useful for these sync engines. the biggest return on investment we got was from consistency checkers.

So part of sync is that there's the same data duplicated in many places, right? Like, so we had the data that's on the user's local file system. We had all of the metadata that we stored in SQLite or we would store like what we think should be on the file system. We would store what the latest view from the server was. We would store things that were in progress, and then we have what's stored on the server.

And for each one of those like hops, we would have a consistency checker that would go and see if those two matched. And those would, that was like the highest return on investment we got. Because before we had that, people would write in and they would complain that Dropbox wasn't working. And until we had these consistency checkers, we had no idea the order of magnitude of how many issues were happening. And when we started doing it, we're like, wow. There's actually a lot.

So a consistency check in this regard was mostly like a hash over some packets that you're sending around. And with that you could verify, okay, up until like from A to B to C to D, we're all seeing the same hash, but suddenly on the hop from D to E, the hash changes. Ah-huh. Let's investigate. Exactly. And so, and to do that in a way that's respectful of the users, even like resources on their system.

Like we wouldn't just go and blast their CPU and their disc and their network to go and like turn through a bunch of things. So we would have like a sampling process where we like sample a random path in the tree and the client and do it the same on the server.

we would have stuff with like Merkle trees and then when things would diverge, we would try to see like, is there a way we can compare on the client and see like for example one of the kind of really important, goals for us as an operational team was to have like the power of zero. I think it might be from AWS or something. My co-founder James, has a really good talk on it. but we would want to have a metric of saying that the number of unexplained inconsistencies is zero and one 'cause.

Then the nice thing right, is that if it's a zero and it regresses, you know that it's a regression. If it's at like fluctuating at like 15 or like a hundred thousand and it kind of goes up by 5%, it's very hard to know when evaluating a new release, right? That like that's actually safe or not. so then that would mean that whenever we would have an inconsistency due to a bit flip, which we would see all the time on client devices, then we would have to categorize that and then bucket that out.

So we would have a baseline. Expectation of how many bit flips there are across all of the devices on Dropbox. And we would see that that's staying consistent or increasing or decreasing, and that the number of unexplained things was still at zero. now let's take those detours since you got me curious. Uh, what would cause bit flips on a local device?

I think a few, few causes, one of them is just that in the data center, most memory uses error correction and you have to pay more for it, usually have to pay more for a motherboard that supports it. at least back then. now like on client devices we don't have that. So this is a little bit above my pay grade for hardware cosmic rays or thermal noise or whatever. But memory is much more resilient in the data center. I think another is just that, storage devices are very greatly in quality.

Like your SSDs and your hard drives are much higher quality inside the data center than they are on local devices. And so. You know, there's that. it also could be like I had mentioned that people have all types of weird configurations. Like on Mac there are all these kernel extensions on Windows, there's all of these mini filter drivers. There are all these things that are interposing between Dropbox, the user space process and writing to the file system.

And if those have any memory safety issues where they're corrupting memory 'cause of the written in archaic C you know, or something that that's the way things can get corrupted. I mean, we've seen all types of things. We've seen network routers get having corrupting data, but usually that fails some checksum, right?

Or we've seen even registers on CPUs being bad where the memory gets replaced and the memory seems like it's fine, but then it just turns out the CPU has its own registers on CHIP that are busted. And so all of that stuff I think just can happen at scale. Right. that makes sense. And I'm happy to say that I've hadn't had yet to worry about flip bits, whether it's being for storage or other things, but huge respect to whoever had already to, tame those parts of the system.

So, you mentioning the consistency check as probably the biggest lever that you had to understand which health stage your sync engine is in the first place. was this the only kind of metric and proxy for understanding with how well the syn system is working or were there some other aspects that gave you visibility both macro and micro? Yeah, I mean, I think this yeah, the kind of hangs, so like knowing that something gets to a sync state and knowing the duration, right?

So the kind of performance of that was one of our top line metrics. And the other one was this consistency check. And then first specific like operations, right? Like uploading a file, like how much bandwidth are people able to use because for like, people wanted to use Dropbox, but, and upload lots, like huge data, like huge number of files where each file is really large. And then they might do it on in Australia or Japan where they're far away from a data center.

So latency is high, but bandwidth is very high too, right? So making sure that we could fully saturate their pipes and all types of stuff with debugging. Things in the internet, right? People having really bad routes to AWS and all that. so we would track things like that. I think other than that it was mostly just the usual quality stuff, like just exceptions and making sure that features all work. I think when we rewrote this system and we, designed it to be very correct.

We moved a lot of these things into testing before we would release. So we this is I think one of the, to jump ahead a little bit, we designed, decided to rewrite Dropbox's sync engine from this big Python code base into Rust. And one of the specific design decisions was to make things extremely testable. So we would have everything be deterministic on a single thread, have all of the reads and rights to the network and file system, be, through a virtualized API.

So then we could run all of these simulations of exploring what would happen if you uploaded a file here and deleted it concurrently and then had a network issue that forced you to retry. And so by simulating all of those in ci, we would be able to then have very strong in variance about them that knowing that like a file should never get deleted in this case, or that it should always converge, or things like the sharing that this file should never get exposed to this other viewer.

I think like the, having much, like having stronger guarantees was something that we only could really do effectively once we designed the system to make it easy to test those guarantees. Right. That makes a lot of sense. And I think we're seeing more and more systems, also in the database world, embrace this. I think TigerBeetle is, is quite popular for that. I think the folks at Torso are now also embracing this approach. I think it goes under the umbrella of simulation testing.

that sounds very interesting. Can you explain a little bit more how maybe in a much smaller program would this basically be Just that every assumption and any potential branch, any sort of side effect thing that might impact the execution of my program. Now I need to make explicit and it's almost like a parameter that I put into the arguments of my functions and now I call it under these circumstances, and I can therefore simulate, oh, if that file suddenly gives me an unexpected error.

Then this is how we're gonna handle it. Yeah, exactly. So it's like and there's techniques that like the TigerBeetle folks, like we, we do this at Convex in rust with the right, like abstractions, there's like techniques to make it not so awkward. But yeah, it is like this idea of like, can you pin all of the non-determinism in the system can, whether it's like reading from a random number generator, whether it's looking at time, whether it's reading and writing to files or the network.

Can that all be like pulled out so that in, production it's just using the random AP or the regular APIs for it. so there's like for any of these sync engines, there's a core of the system which represents all the sync rules, right? Like when I get a new file from the server, what do I do? You know, if there's a concurrent edit to this, what do I do? and that I. Core of the code is often the part that has the most bugs, right?

It has the, it doesn't think about some of the corner cases or if there are errors or needs retries or doesn't handle concurrency. It might have race conditions. So the kind of, I think the core idea for determination, determin deterministic simulation testing is to take that core and just kind of like pull out all of the non-determinism from it into an interface.

So time randomness, reading and writing to the network, reading and writing to the file system, and making it so that in production, those are just using the regular APIs. But in a testing situation, those can be using mocks. Like they could be using things that for a particular test and wants to test a scenario or setting it up in a specific way. Or it could be randomized, right?

Where it might be that reading from Like time, the test framework might decide pseudo randomly to advance it or to keep it at the current time or might serialize things differently. And that type of ability to have random search explore the state space of all the things that are possible is just one of those like unreasonably effective ideas, I think for testing. And then that like getting a system to pass that type of deterministic simulation testing.

It's not at the threshold of having formal verification, but in our experience it's pretty close and with a much, much, smaller amount of work. And you mentioning Haskell at the beginning? I still remember when I, after a a lot of time having spent writing unit tests in JavaScript and I, back then, in the other order, I first had JavaScript and then I learned Haskell, and then I found quick test and was quick test, Quick Check. which one was it? I think it was Quick check, right?

Well, right. So I found Quick Check and I could express sort of like, Hey, this is this type. It has sort of those aspects to it, those invariants and then would just go along and test all of those things. Like, wait, I never thought of that, but of course, yes. And then you combine those and you would get way too lazy to write unit tests for the combinatorial explosion of like all of your different things. And then you can say, sample it like that, and like, focus on this.

and so I actually also, started embracing this practice a lot more in the TypeScript work that I'm doing through a great project called Prop Check. and that is, picking up the same ideas and for particularly those sort of scenarios where, okay, Murphy's Law will come and haunt you. this is in distributed systems. That is typically the case. Building things in such a way where all the aspects can be, specifically injected and the, the sweet spot.

If you can do so still in an ergonomic way, I think that's the way to go. It's so, so valuable, right? And yeah. And yeah, the ability to, for prop tasks, for quick check for all of these to also minimize is just magical, right? Like it comes up with this crazy counter example and it might be like a list with 700 elements, but then is able to shrink it down to the, like, real core of the bug. It's magic, right? And you know, I mean, I think this is something like, you know.

A totally different theme, right? Like one thing at Convex we're exploring a lot is like coding has changed a lot in the past year with AI coding tools. And one of the things we've observed for getting coding tools to work very well with Convex is that these types of like very succinct tests that can be generated easily and have like a really high strength to weight or power to weight ratio are just really good for like autonomous coding, right?

Like, if you are gonna take like cursor agent and let it go wild, like what does it take to just let it operate without you doing anything? It takes something like a prop test because then it can just continuously make changes, run the test, and not know that it's done until that test passes. Yeah, that makes a lot of sense. So let's go back for a moment to the point where you were just transitioning from the previous Python based sync engine to the Rust based sync engine.

So you're embracing simulation testing to have a better sense of like all the different aspects that might influence the outcome here. walk me through like how you, went about. Deploying that new system. Were there any sort of big headaches associated with migrating from the previous system to the new system? since you, for everything, you had sort of a defacto source of truth, which are the files.

So could you maybe just forget everything the old system has done and you just treat it as like, oh, the, user would've just installed this fresh, walk me through like how you thought about that since migrating systems on such a big scale is typically, quite dread

Yeah, dreadsome is, yeah. appropriate word. I think one of the biggest challenges was that by design we had a very different data model for the old sync engine. We called it sync engine Classic. Affectionately. And then we had for Nucleus was a new one. Nucleus had a very different data model, and the motivation for that was that sync engine Classic just had a ton of possible states that were illegitimate.

It could, if you had like a, the server update a file and the client update a file, but then a shared folder gets mounted above it, things could get into all of these really weird states that were legal but would cause bugs. And then I think that was like one of the big guiding principles more than even just like Rust or Python, was just like designing what states should the system be allowed to be in and design away everything else, make illegal states unrepresentable.

And so that, what that then meant is once we had that. When we needed to migrate, we had a long tail of really weird starting positions. So where you basically realized, okay, this system is in this state A, how the heck did it ever get into that state? And B, what are we gonna do about it now where we can basically, it's like from a mapping function, this is like invalid input.

So can you explain a little bit of like, how you constrained the space of, and how you designed the space of, legitimate, valid states and what were some of the, if you think about this as like a big matrix of combinations, what are some of the more intuitive ones that were, not allowed that you saw quite a bit? Yeah, so I think part of the difficulty for Dropbox, like as syncing things from the file system is that file system APIs are really anemic. File system aPIs don't have transactions.

They don't things can get reordered in all types of ways. So we would just read and write to files from the local file system, and we would use file system events on Mac, we would use the equivalent on Windows and Linux to get, updates. But everything can be reordered and racy and everything. So one, like common invariant would be that if you have a directory you know, like files have to exist within directories. If a file exists, then it's parent directory exists.

And like simultaneously, if you delete a directory, it shouldn't have any files within it. And that invariant guarantees and that the file system is a tree. Right? And then we, it's very easy to come up with settings, with reads from the local file system where if you just naively take that and write it into your SQLite database, you will end up with data that does not form a tree. and then especially even with like I know it's being unique, right?

Like if I move a file from A to B, then I might observe the add for it at B way before the delete at B or I might observe it vice versa, where the file is transiently gone and disappeared and we definitely don't wanna sync that.

and then with directories, if I have like a, as a directory and then B as a directory, and then I move it's, I could observe a state where A moves into B, which then without doing the right bookkeeping, might introduce a cycle in the graph and a cycle for directories would be really bad news, right? so all of these invariants were things that the file system APIs, they don't respect, even though the file system internally has these invariants, right?

You cannot create a direct recycle on any file system. Definitely. I mean certainly without root And all of these invariants exist but are not observable through the APIs. And so then we sync Engine Classic would get into the state where it's like local SQLite file would have all types of violations like that. So then how do we read the tea leaves of like the database is in a really weird state we can't lose.

And to go back to, I think what you had talked about at the beginning of this was that we always had the nuclear option of dropping all of our local state and doing a full resync from the files themselves. But then the problem is that we would entirely lose user intent. So if, for example, I was offline for a month and I had a bunch of files, and then during that month other people in my team deleted those files.

If I came back online and didn't have my local database, we would have to recreate those files and people would complain about this all the time because. They would delete something and wanna delete it, and then Dropbox would just randomly decide to resurrect it. So those types of decisions we, we tried to avoid that as much as possible, but then that meant having to look at a potentially really confusing database and read what the user intent might have been. Right.

I wanna dig a little bit more into the topic of user intent. Since with Dropbox you've built a sync engine very specifically for the use case of file management, et cetera, where user intent has a particular meaning that might be very different from moving a cursor around in a Google Docs document. So can you explain a little bit, what are some of the, common scenarios of, and maybe subtle scenarios of user intent, when it comes to the Dropbox design space?

Yeah, totally. and I think the for regular things like say editing files. I think we saw that like people just generally did not, maybe because of the way the system was even its capabilities, people did not edit the same files all too often.

So maintaining user intent when file, when everyone is online, just kind of taking last writer wins Where I think user intent became very interesting is if someone went offline, like they're on an airplane before wifi and airplanes And they worked on their document and someone else worked on the same time. In that case, we observed that users always wanted to see the conflicted copy and that they wanted to get the opportunity to say, like, I did.

I put in a lot of effort into working on this when I was on the plane. Someone else, put in probably a similar amount of effort when they were online and you know, so last writer wins policies. There violated user expectations quite a lot because either a person had to win and then the person who lost would be really upset. so I think those were pretty interesting. I think with Moose, like with more metadata operations I think people were a little bit more permissive.

Like if I moved something from one folder to another, another person moved it to a different folder. having it just converged on something as long as it converges. We observed it being like people didn't worry about it too much. I think the place where user intent is really interesting with moves is with sharing. So I think thinking about this from like the distributed systems perspective on causality, there would be like someone might have like, I dunno, their HR folder, right?

And I don't know, like, let's say that someone is transferring to the HR team is they're getting added to the HR folder. But then say before they were on the team, they were on a performance improvement plan. So then the administrator for HR would delete that file, make sure it's deleted, and then add them to the folder. And so their user intent is express in a very specific sequencing of operations, right? That like this causally depended on this.

I would not have invited 'em to the folder unless the delete was stably synced. And that making sure that gets preserved throughout the system, even when people are going online and offline and everything is a very hard distributed systems problem. Right. and it was intimately related with the details of the product. Right. yeah. How did you capture that causality chain of events since you probably also couldn't quite trust the system clock? How did you go about that?

Yeah, this became even more difficult, right? Where file system metadata was partitioned across many shards in the database. So then we ended up using something like Lamport timestamp, where every single operation would get assigned a timestamp. And those timestamps were usually only reading and writing to their particular shard and for whatever timestamp the client had observed.

But then in these cases where there were potentially cross shard, they weren't transactions, but like causal dependencies, we would be able to say like, the operation to mount this or to add someone to the shared folder and there them mounting it within their file system has to have a higher timestamp than any right within that or. Rights including deletes.

so then that way when the client is syncing it would be able to know that when I am merging operation logs across all of the different shards, I need to assemble them in a causally consistent order. And that would then respect all of these particular invariants. Right. So you having thought through those different scenarios for Dropbox and made very intentional design decisions that, for example, in one scenario last writer wins is not desirable.

Since that might lead to a very sad person stepping off the plane because all of your data is suddenly gone, or the other person's data is gone. so you make very specific design trade-offs here when it comes to somehow squaring the circle of distributed systems. Which sort of advice would you have for application developers or people even who are sitting inside of a company and are now thinking about, oh, maybe we should have our own Dropbox style, linear style sync engine internally.

Which sort of advice would you give them when they Yeah. Start thinking this through to the detail.

Yeah, I'll talk through kind of how we structured things at Dropbox to be able to navigate these types of problems. And I think the patterns here, can be quite general. I think what we ended up with was that like thinking like distributed systems syncing is hard, right? So we would have the kind of base layer of the sync protocol and how state gets moved around between the clients and the servers and all the shards. We would have very strong consistency guarantees there.

So we would not use any of the knowledge of the product at that layer. So from a, like thinking of Dropbox in the file system as a CRDT. Dropbox allows, like moves to happen concurrently. It ha allows you to add something while another thing is happening. But at the protocol level, we kept things very strict. We kept them very close to being serializable that every view of the system was identified by a very small amount of state, like a timestamp.

And that would fully determine the state of the system and like the amount of entropy in that was very low. And then whenever you are modifying it, you would say, here's what I expect the data to be, and if it doesn't match exactly, it will reject the operation. And then by doing it, structuring things in that way, then we made it very easy for product teams and for even us working on sync to embed all of these like looser more product focused requirements.

They also may wanna change over time into the end points, like layered on top. So every time we wanted to change a policy on how like a delete reconciles with an. You know, add for a folder or something. We didn't have to solve any distributed systems problems to do that. So I think that like pattern of saying that, like is there a good abstraction?

Is there something that is like very powerful that could solve a large class of problems, doing that well at the lowest layer and then potentially weakening the consistency above it. I actually really like the Rocicorp folks have a really great description of their consistency model for Replicache of it being like session plus consistency.

And it's like a very similar idea where like when we build things on a platform, we may as our with our product hats on, like want users to not have to think about conflicts and merging and all that in a lot of cases. But those decisions might be very particular to our app. And that's something that holds for everything on the platform. And then there's always a way to embed those decisions onto, say. Session consistency and Replicache or serializability and other systems.

And so I think that's like that separation of concerns I think is something that can apply to a lot of systems. Right. So maybe we use this also as a transition to talk a bit more about what you're now designing and working on Convex. What were some of the key insights that you've taken with you from Dropbox that ultimately led to you co-founding Convex?

Yeah, when we first were starting Convex we were looking at how apps are getting built today, right? Like web apps are easier to build than ever. Even in 2021, it's incredible how much, like more productive that compared to 10 years before.

Right. It was, and I think we noticed that the hard part for so many discussions was managing state and like how state propagates I think it was from the Riffle paper right, on how like so many issues in app development are kind of database problems in disguise and that how techniques from databases might be able to help.

So with Convex we were saying like, well if we start with the idea of designing a database from first principles, can we apply some of those database solutions to things across the whole stack? So say for example, when I'm reading data from it within in my app, I have all of these React components that are all reading different pieces of data.

It'd be really nice if all of them just executed at the same timestamp and I never had to handle consistency issues where one component knows about a user or the other one doesn't. Similarly, like why isn't it possible to be that I just use query across all my components and they just all live update whenever I read anything, it's a automatically reactive. So those were some of the like the initial kind of thought experiments for what led to Convex.

I think the other one that was really motivated from our time at Dropbox and I think is like kind of a both a blessing and a curse. It's kind of like one of the key design decisions for Convex is that Convex is very opinionated about there being a separation between the client and the server. So we saw this at Dropbox where they were just different teams, right? And you know, as we've seen with like even the origin of GraphQL, right? Like that ability to decouple development between.

teams working on user facing features and the way that the data fetching is implemented on the backend, it's gonna be really powerful. And so kind of the kind of thought experiment with Convex is, can we maintain a very strong separation while still getting like live updating, while still getting a really good ergonomics for both consuming data on the client and like fetching it on the server. Right. So yeah, walk me through a little bit more through the evolution of Convex then.

And so, in, in terms of all the other options that are out there in terms of state management and I think most what applications are using is probably something that at least to some degree is somewhat customized and hand rolled and comes with its own huge set of trade-offs. Help me better understand sort of the, where you mentioned the, opinionated nature of Convex. What are the, benefits of that? What are the downsides of that and other implications?

Yeah, so when you write an app on Convex we can use maybe like a basic to do app, right? The linear clone, everyone does. you write endpoints like you might be used to, right? Where it's like list all the to-dos in a project like update a to-do in a project. and those get pushed as your API to your Convex server. the implementations of that API can then read and write to the database and Convex has like a, kinda like Mongo or Firebase, like API for doing so.

I think the main benefit then of Convex relative to more traditional architectures is that if you're on the client, the only thing you need to do is call the, like the use query hook. You're saying like, I am looking at a project I just do use like use query list tasks and project that will then talk to the server, run that query, but then also set up the subscription and then whenever any data that that query looked at changes, it will efficiently determine that and then push the update.

So part of what is like been nice with Convex is that you are getting a client that has a web socket protocol, it has a sync engine built in. You're getting infrastructure for running JavaScript at scale and for handling sandboxing and all of that. And then you're also getting a database, which is, you know. One, supporting transactions or reading and writing to it.

But then it also supports this efficient like being able to subscribe on, I ran this query, this query just ran a bunch of JavaScript. It looked at different rows and it ran some queries. the system will automatically efficiently determine if any right overlaps with that. So the combination of all of those things is like part of the benefit of Convex, you just write TypeScript and you write it in a way that's, feels very natural and everything just works.

And I think some of the like downsides is that it's it is a different set of APIs. it's not using sql, it's doing things a little bit differently than they've been done before. yeah, it's like kind of interesting even today to see like what you know. Talking about AI code gen, right? Like models have been trained, pre-trained on this huge corpus of stuff on the internet. And when are they good at adopting new technologies? Technologies that might be after their knowledge cutoff.

And when are they like it's better just to stick to things that they know already. Right. So what you've mentioned before where you say, Convex is rather opinionated for me. in let's say five years ago, I might've been much more of like, oh, but maybe there's a technology that's less opinionated and I can use it for everything. But the more experience I got, the more I realized no, actually. I want something that's very opinionated, but opinionated and I share those opinions.

Those are exactly for my use case. So I think that is much better. This is why we have different technologies and they are great for different scenarios, and I think the more a technology tries to say, no, we're, we're best for everything, I think the, less it's actually good at anything. And so I greatly appreciate you standing your ground and saying like, Hey, those are, our design, decisions that we've made.

And those are the use cases where, you'd be really well served building on top of something like Convex. And, I particularly like for now where TypeScript is really the, default language to build full stack applications. And it's also increasingly becoming the default for. ai, based applications as well. And AI based systems speak type script, just as well as English. And given that Convex makes that full stack super easy.

And also I think you can, when you build local-first apps, it can sometimes get really tricky because you empower the client so much. You give the client so much responsibility and therefore there's many, many things that can go wrong. And I think Convex therefore, takes a more conservative approach and says like, Hey, everything that happens on the server is like highly privileged and this is your safe environment.

And the client will try to give you the best user experience and developer experience out of the box. But the client could be in a more adversarial environment. And I think those are great design trade offs. So, I think that is a fantastic foundation for tons of different applications. Yeah. talking about some of these strong opinions being both blessings and curses, right?

Like over the past few months, one thing we've been working on is trying to bridge the gap between those two points in the spectrum, right? we wrote a blog post on it a few months ago of like working on what we're calling our like Object sync engine, trying to take a lot of the principles from more of a local-first type approach of having a data model that it is synced to the client and the only interaction between the server and the client is through the sync.

And the client then can always render its UI just looking at the local database and it can be offline. It's also fully describes the app stage so it can be exported and rehydrated or whatever. it's very interesting design exercise we've been on to say like, can you structure a protocol on a sync engine in a way such that the UI is still reading and writing to a local store that is authoritative.

But then that local store is like to kind of use like an electric SQL terminology is like that is a shape that is some mapping of a strongly separated server data model. So we still have a client data model and server data model, which might be owned by different teams and evolve independently and, we also have that strong separation where the implementation of the shape is privileged and running on the server and has authorization rules built in and get the best of both worlds.

And we've kind of, we have a like beta that we've not released publicly thought open, sourced out there, but kind of a thing where we, I think they're still figuring out like the DX for it. And I think we have something that like algorithmically works and it's like the protocol works, but it's like, it's kind of hard. Right. It kind of reminds me a lot of writing GraphQL resolvers of like saying How do I take the messages table from my chat app?

Then under the hood that might be joining stuff from many different tables and filtering rows, or might even be doing a full tech search query in another view or something. and coming up with the right ergonomics to make that feel great for a day one experience. I think something that's like still we're working on, still kinda like a research project, right?

Well, when it comes to data, there is no free lunch, but I'd much rather to have it be done in the order and sequencing that you're going through, which is having a solid foundation that I can trust and then figuring out the right ergonomics afterwards, since I think there's many, many tools that start with great ergonomics, but later realize that it's on a built, on a unsound foundation.

So when it comes to data, I want a trustworthy foundation, and I think you're going about in the right order. Hey, Sujay, I've been learning so much about one of my favorite products of all time, Dropbox. I've learned so much of like how the sausage was actually made, how it evolved over time and I'm really excited that you got to share the story today and many me included, got to, learn from it. Thank you so much for taking the time and sharing all of this. Thanks for having me.

This is super, super fun. Thank you for listening to the localfirst.fm podcast. If you've enjoyed this episode and haven't done so already, please subscribe and leave a review. Please also share this episode with your friends and colleagues. Spreading the word about the podcast is a great way to support it and to help me keep it going. A special thanks again to Jazz for supporting this podcast. I'll see you next time.