#4 – Martin Kleppmann: CRDTs, Automerge, generic syncing servers & Bluesky

So much work in building a web app goes into reinventing this backend infrastructure that every single company has to reinvent again. And so if we can make the data sync protocol and the data storage on the servers efficient, like loading and synchronization of large collections of documents, all of that can be generic.

So if one person is then building a graphics app and another person is building a spreadsheet and another person is building a document editor, they can all use the same syncing service as the backend. That I think is part of the economic value proposition of local-first software. Welcome to the localfirst.fm podcast. I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and love the craft of software engineering.

For the past few years, I've been on a journey to build a modern, high quality music app using web technologies. And in doing so, I've been falling down the rabbit hole of local-first software. This podcast is your invitation to join me on that journey. In this episode, I'm speaking to Martin Kleppmann, who is one of the authors of the original local-first essay.

Martin has been exploring local-first software in CRDTs for over 10 years, which has led to the creation of Automerge, which we discuss in depth in this episode. We are also exploring the ideas of a generic sync server and the impact this technology could have on local-first software in the future. I also have a very special announcement today as I'm co organizing the world's first local-first Conference. It will happen on May 30th in Berlin, and I would love to see you there in person.

Go ahead and grab your tickets on localfirstconf. com. Before getting started, also a big thank you to Expo and Crab Nebula for supporting this podcast. And now my interview with Martin. Hello, welcome, Martin. Thank you so much for coming to the podcast. Hi, Johannes. Thank you for having me.

I'm super excited to have you on the show. You're obviously no stranger in the local forest world. Um, but would you briefly mind introducing yourself? Uh, yeah, sure. So I'm just recently an associate professor at the university of Cambridge. I've been at Cambridge for quite a long time, but for a long time that was like on fixed term academic contracts. So this is my first permanent university position, which is nice. It means I can keep doing this stuff long term.

Um, yeah, previous to that, I did in some past life work on startups. sold a startup to LinkedIn back in 2012, but then shifted over to academia. Amazing. and so you're one of the coauthors of the local-first paper that was published on the Ink & Switch site. So, I think most people are also in the local-first space are familiar with that, but I'm very curious, what is your personal story behind you sort of finding your way to local-first?

Yeah, there is, it probably starts about 2013 or so, fairly shortly after we had sold the startup to LinkedIn and I was at LinkedIn, but our project got canceled. And so I was kind of looking around for new things and I came across this. paper from Mark Shapiro and colleagues on conflict free replicated data types or CRDTs. I can't remember how I come across it. Maybe somebody put it on Twitter or something like that.

And I read this thing and I was really intrigued by it because I felt that, you know, this seemed like a way of Making the software a bit less cloud dominated. I had got a bit frustrated with the whole startup world. You know, as I was doing web based stuff, social media stuff, it's all very much like centralized services, which put all of the user's data in one big database. And I was just a bit uncomfortable with that. I felt like, it's not really in the user's interest.

Obviously it's in the company's interests to try to collect as much data as they can and monetize it in whatever way they can. But for users, it's not really great. And so I was sort of trying to overcome my unease with this by looking at technological solutions that might help. And so then I came across these CRDTs, which seemed like it could be a part of the answer to the problem. I felt like it was a way how you could make software that would run on the users.

device and store the data locally on the user's device where nobody can take it away from you. And at the same time have all the conveniences of cloud software with like real time collaboration and sync across all of your devices and being able to easily share data with other users and so on. So that was kind of in core of the local-first idea there already, but it then took us several more years before we really were able to articulate it clearly enough ourselves. And so then.

I can't remember when the local-first paper came out. Was it 2019 or something like that? So yeah, about 2014 I left LinkedIn for a year. I spent on sabbatical writing my book and then 2015 I joined the university and started working on CRDTs myself and then started like gradually building up the technical foundations. It then still took us quite a long time before we had like really articulated it. but all that time was gradually working towards what we now call local-first.

I'm curious whether there were like any particular milestones you've reached during those early research years where they're like moments where you thought you hit some walls and you thought this was a dead end. I mean, with any kind of research, it's always like lots of little dead ends and then getting out of them and trying other things again. But I think that's just part of the normal process. So I'm not going to pretend it was smooth in any way.

Obviously, there's lots of things that didn't work along the way, but also most of them are sort of in retrospect, kind of don't matter too much. So like, you know, we have very detailed discussions about. How a particular merge behavior should work, for example. So if one user makes one change to a document, another user on a different device makes a different change, we need to merge those things together. You know, you can have hours and hours of debate about how precisely that should work.

But then in the end, once you've settled on an answer and That answer seems to be broadly okay, then, you know, then the question just becomes uninteresting and we move on to more interesting things. So yeah, there's, there's lots of that kind of things along the way. And like a lot of changes we made to the implementations of these things, like the software evolved a lot. Like my first CRDT implementation was in Ruby.

And then that later turned into a JavaScript implementation, which was the beginnings of what is now Automerge and then that's later got ported to Rust. And so now the Rust implementation is our primary one. So, you know, we've really gone through three languages there and God knows how many orders of magnitude improvement in performance, the early versions were extremely, extremely slow, but you know, it's, it gradually gets better as we keep working on it.

I'm really eager to dive in deeper on Automerge and hearing your side of the story on how Automerge came to where it is today. Before going into Automerge, Automerge is a library to deal with CRDTs, but not everyone might be super familiar with CRDTs. I don't think there's a better person to explain what CRDTs are than you. Could you give a quick summary, and introduction to CRDTs?

Yeah, so the basic idea is that you've got some data on multiple devices, the user on each device can independently update that data, possibly while the device is offline. And then at some point later, the devices sync their updates. And ideally, we just want them to merge their states together in some way. And CRDTs are just algorithms that perform this kind of merging, plus the data synchronization and so on. So the idea is that.

You know, often the changes made on two different devices will affect different parts of a document. One person is updating one item in the to do list and another person is updating a different item, and so it's fairly easy to merge those together. In principle, you can end up with conflict cases where, like, it's a graphic software, one user makes the rectangle red, another Person makes the same rectangle green. Well, what do you do?

Well, I mean, you probably just choose one of the two and then if the user doesn't like it, they can change the color again. So it's algorithms just for automating that kind of thing. Because what we don't want is for the user to be shown like a pop up saying, Hey, this file was changed on two devices. Please pick which one you want to keep and which one you want to throw away. I think that would be bad. And like previous versions of Apple's pages, also did that kind of thing, I guess.

I think if I remember correctly, but fortunately now we have better algorithms which, which just allow changes to be merged together with minimal ceremony. So that's really all CRDTs are about. A huge amount of research has gone into like figuring out how to make the merge behavior good. So that depending on what types of edits people make, the end result is hopefully something that was more or less what they expect, what the users expected and also in making these algorithms fast.

Because, , you can implement these algorithms in a very simple way, but the simple way tends to be very inefficient. And so making it so that it doesn't take too much disk space, doesn't take too much memory and it's generally fast, that actually requires quite a lot of sophistication on the algorithm. So that's where a lot of the investment has gone over the last few years, but yeah, but that's broadly what CRDTs are and Automerge is just a library that implements this stuff.

So there are other CRDT libraries out there, but, Automerge is the one that I've been closely involved with over the last few years.

Yeah, I think Automerge is probably one of the most advanced CRDT implementations right now. And as you've mentioned, you built your first versions, not in Rust as it is written today, but there were predecessors to this. So given that this is now such a. Such a long journey.

I think it's, if it's fair to say, , that you've been working on this for 10 years, I'd be very interested in hearing your reflections on the history and the process of taking Automerge from the beginnings to where it is today. Yeah. So when I started working on CRDTs, there was no CRDT for JSON data, for example. So there were existing data types for sets and maps and counters and registers and things like that.

So just these kind of little atomic data types, but nothing that really composed them together. Uh, oh, and lists as well. I mentioned that there were data types for lists. And so in a way, JSON is simple, you know, it's just, you can put maps inside lists and maps and lists inside maps and compose them arbitrarily. But there's still interesting questions you have to answer, which is like, for example, what if one user deletes an object while another user makes an update inside that object?

How do you merge those things? And so one of the first research papers I wrote was an algorithm for doing a CRDT for JSON data, which answered exactly this kind of questions.

And then Automerge started out sort of conceptually as an implementation of this paper, although we ended up actually choosing different behavior for Automerge than the paper chose, but you know, after examining a bunch of applications and what sort of behavior they would want, we came to the conclusion that a different behavior was better, but that was basically the genesis of the whole thing. So I can't remember which year that JSON CRDT came.

paper came out, but yeah, I was working on it like in 2015, 2016 ish. And then, I think it was about 2017, Peter van Hardenberg got in touch with me. So I knew Peter from back in my startup days because he was running the Heroku Postgres team at the time. And our company, which was called Reportive, was one of the bigger customers of, uh, Heroku Postgres at the time. And so We had, like, talked to Peter as part of, like, just scaling our database.

years later, I hear from Peter again, because he had read my JSON CRDT paper and went like, Hey, we want to try actually building some apps with this. Have you tried actually building some apps? And I went, Oh, no, no, no. I just do theory. You know, I just write a paper and I have this extremely janky Ruby implementation that actually only does half of what the, what was says in the paper. So then, , it got together with, , Peter and Ink & Switch.

And I think Ink & Switch was quite new still at the time. And we did, , this project together in which we essentially built a adjacent CRDT. That actually worked in JavaScript. In fact, Orion Henry wrote the first version of that and brought it to me. And I went like, yeah, nice API, but no, those algorithms are totally wrong. And so then we worked together to make the algorithms right as well. And it was a great collaboration because you know, the Ink & Switch folks were.

Just much better, like API design and also UI design and general app development than I was, whereas I sort of brought the like more mathematical style of thinking of analyzing the algorithms and making sure that they were correct, and that was just a great collaboration. So yeah, we've, we first wrote this library. We originally called it Tesseract, but then there was already a JavaScript library of that name. So we renamed it to Automerge and that name has stuck since.

So yeah, I think Automerge started around 2017. And then a few Ink & Switch projects used it, but it was very much research quality software. You know, it was extremely slow. It had bugs. The file format was extremely inefficient. So it was kind of impractical to use for most things. As a vehicle for doing research, it worked quite well. But then at some point, like it became clear that, okay, we actually want to start building more ambitious software on it.

And it's not really acceptable if it takes three minutes to load your document off disk. So, you know, okay, we have to make the. figure out a new file format to make the file smaller and, , figure out new algorithms to make the whole thing faster. And then also we decided that the Rust implementation would be better. Um, not so much because Rust is faster than JavaScript, but rather because it's more cross platform.

And so we can compile Rust to WebAssembly for the web, but we can also compile it to native libraries for iOS and Android, for example. And so Orion did a lot of work on the port to Rust. Uh, again, and a few others contributed to that, and Alex Good got involved with that too. But then at some point, two years ago or so, we then made the call to make the Rust implementation, the primary implementation of Automerge.

So all of that JavaScript, I had, I'd been maintaining the JavaScript implementation as this research code over the years, but we decided to just completely deprecate that, throw away all of my old code. And I've done, actually no work on the Rust code of the implementation. So that's all been done by Alex and Orion and other people now. And I've just moved into more of an advisory role, which suits me really well. You know, I'm very happy to be the one not writing the code.

Other people are much better at writing the code than I am, but I know I can think about the algorithms and the protocols and the data structures. And that's what I find fun. And so then, About a year ago or so, we then declared Automerge to be production ready. So at that point, then, you know, the Rust implementation was mature and fast.

and we got a sponsorship thing going with GitHub sponsors, which allowed people who were commercially using, or companies that were commercially using Automerge, to sponsor its development. And that is now supporting the work of Alex Goods, who's now professionally maintaining Automerge. And that is just such a good arrangement now.

I'm really pleased with how that's working because it means that we have high quality software that's being professionally maintained, but at the same time, you know, we haven't had to go out and raise venture capital, which we feared that that's, you know, might be at odds with the values of local-first. And so this way by Essentially bootstrapping it off of the sponsorship revenue. I think that aligns everybody's interests very well. And so that has allowed the project to do very well.

That is an incredible journey. And I mean, this is for an open source project. Particularly, I think most people use right now, Automerge still in a JavaScript context for a JavaScript library, where I think you're thinking more in terms of dog years, Automerge is really a monumental project and it has come incredibly far. So I'm super excited for that. So where's the project today? You've mentioned that it's reached production readiness around about last year.

Does that mean it's the APIs are final, the research behind it is concluded and now it's just performance optimizations or what is left to do? And I just, there's so much, so much we still want to do with it. So what we mean with production ready is like, there are no egregious bugs that we know about and the performance is good enough that.

You know, it's plausibly usable in real software, which some of the research code definitely was not, but it's got much, much better, but in terms of features, like it, I think we've only really just started. So what Automerge started with is a basic JSON model, so you can have maps. Where the keys are strings and the values can be either nested maps, or they can be nested lists, or arbitrary recursion of those things, or primitive values like strings and numbers and booleans. And that's it.

Then, okay, we, we added counters because actually counters are actually not very useful, but everyone seems to use them for demos. So we include the counters so that we can have the demo as well. Then, a big thing we added was rich text. So that's something that a lot of applications need is. text with formatting.

And the first version of that is released and implemented, though the first version only supported inline formatting, such as bold and italic but not block elements like headings or bullet points or things like that. And so there's an updated version of that coming soon, which adds support for block elements too. So this is now nice. You can put rich text anywhere inside a document.

So. You know, it's, if you want to make a Google Docs equivalent thing, you can do that, but you could also have, for example, a vector graphics software that has some rich text just inside the text boxes. And the rest is a drawing consisting of like arrows and lines and freeform, whatever you want. And so the JSON type document model has allowed extension in those directions very well, but there's so much more we still want to do.

So. Like an obvious missing thing is undo in collaborative software is actually quite subtle in terms of the behavior you want. And so in particular, it's not generally the case that you want to undo the most recent operation, the most recent change to the document, because the most recent change to document might've been made by somebody else in a part of the document that you're not looking at. And so.

Undoing somebody else's change in a completely different part of the document is definitely not what you intended when you hit command Z. So actually doing undo well requires, inspecting the editing history of the document, which we can do because Automerge keeps the editing history anyway, but actually surfacing that and making the right APIs the right underlying algorithms, that's still some work in progress.

Another thing that we've long Try to add as a move operation so that, for example, you could reorder items in lists or if you have a, say a file system tree, you could drag a directory from one location to another. That is also quite subtle to implement because you have to answer questions like, what happens if two users can currently move the same item to two different places? You don't want to duplicate it. In that case, you want to just. pick one of the destinations.

Or you get weird things where like you have A and B which are siblings and one user moves A to be a child of B while concurrently another user moves B to be a child of A and now if you're not careful you could end up with a loop between A and B and That would be a mess as well. So to move operation very carefully has to handle those kinds of cases.

You know, we wrote the research paper about it several years ago, but actually turning that into the kind of production quality code as part of Automerge is still ongoing project. And so those are kind of the near term things that we want to. Features, examples of features that we want to add to Automerge. Other stuff we want to do better are, for example, synchronizing large collections of documents. So at the moment, Automerge really just deals with one document at a time.

But in many apps, you know, you might have a collection of 100, 000 documents and most of them don't change most very much. So we need a protocol for efficiently figuring out which of those many documents have changed and then synchronize only those which have changed and have minimal overhead for those that have not changed, that kind of stuff.

So you mentioning, uh, collections and that right now Automerge is only working on sort of a single document level, but you want to go further into collections. So collections makes me think of databases. Can you contrast a little bit of how someone who thinks about data primarily in terms of databases, how your brain needs to change to think primarily in terms of Automerge and how. What in the future where someone uses Automerge, do they still use databases?

Do you think about the data that Automerge just manages sort of like as an implicit database? How should I think about that in the future? Yeah, I think there's, there's a lot of similarities between Automerge and the database. And we've sort of like internally joke that, you know, we're not writing a database because writing a database is a crazy thing to do that nobody should like try to write their own database, but it looks like we are writing a database. And shh, don't tell anybody.

So like, yeah, a collection of documents definitely starts smelling quite a lot like a document database. There's sort of differences in data model and sort of a usage pattern compared to like how. Mainstream databases are built, you know, you can take MongoDB or even the JSON support in Postgres and they give you a JSON data model. And so in that sense, it's similar ish, but they don't really have the conflict resolution aspect.

So they assume that all of your rights go to a single leader server and that server just serializes all of the updates. And therefore you never end up in a situation where you have to. merged to diverged versions of the document. Whereas in local-first, I mean, the whole point of local-first is that you have to data locally on your own device. So that means you inevitably end up in having to do this kind of conflict resolution.

So even though the data model is maybe on a high level, similar to something like MongoDB, the data synchronization and the conflict resolution aspects. is something that's very different from a server oriented database. So you could say it's a more client oriented database where it is intended to be embedded into client software. And that would get us quite close to, I think, what Automerge wants to be.

So we have the beginnings of something like that in a library called Automerge repo, which is it's sort of a wrapper library around Automerge. Automerge itself is basically just an in memory data structure library. It does. Nothing with disk or network, it's just purely an in memory data structure, but Automerge repo adds the IO layer to it. And so it provides adapters for like storing data, storing documents on disk and loading them again, and for synchronizing things over the network.

And it also manages a collection of documents. And so this is how the whole thing starts looking a bit more like a database. Another difference I would also note compared to something like MongoDB is that a lot of these server side databases assume that a single document actually doesn't get updated all that often. though you know, you might do 100 writes to a document over the lifetime of a document.

Whereas the types of Documents we're thinking about every keystroke when you're writing a text is a right to the document. And so you can easily accumulate hundreds of thousands of rights to a document over the lifetime of a document. And if you have that sort of high rate of updates, that forces entirely different data structures and data formats.

so I think if you try to use MongoDB or Postgres, And, you know, write a new version of a document on every keystroke, they would not perform very well because they actually write an entire new copy of the document to disk every time you update the document. And, you know, that's just not going to work if you're making hundreds of thousands of updates to a single document.

And so that's why Automerge then has got a whole bunch of clever data structures and file formats in order to deal with those very frequent, very frequent, but small updates to documents.

So I'm personally, as an app developer, I try to think about like, what is the best foundation to build an app? And so you've mentioned that the early prototypes that you've built for what later became Automerge, you built with Ruby, maybe you probably built apps with Rails in the past and Rails was really a great foundation to build a new app. I'm wondering. In the next five to 10 years, when you put on your local-first lens, like what is the rails equivalent for local-first?

Should I think about Automerge becoming more and more like a new kind of rails that's less of an app framework, but more of like a. A data framework that takes over more and more app framework responsibilities. Can you paint a bit of that picture for me? I think the analogy is really good, actually. I, yeah, I would think of Automerge maybe like as the active record component of Rails or so it's the data component.

it's not a whole app framework by itself, but you could definitely imagine building an app framework where it's an important part of it. And the rest of the app framework would have to do stuff like reactivity of updating the user interface in response to edits that have happened and figuring out how to handle user inputs, blah, blah, blah, all that sort of things.

So I think that framework doesn't exist yet, but I would really love to see somebody build the equivalent of rails, for local-first software. So what are the missing pieces for that? So you've mentioned that the way how you and Peter have met is through Peter's previous work on Heroku. So Heroku, I think, played a major role in making rails. So. Easy for developers since it's not just easy to work with it locally, but it's also easy to roll it out into production.

So what does it mean for me right now, if I'm building my first little app, my first little prototype with Automerge locally, what does it mean for me to roll that out, that I can share it with my friends and use it sort of in a, in a bigger scale? Yeah. So at the moment It still requires a fair amount of. The stuff you have to write yourself. So for example, you know, we provide as part of automated repos, some integrations with like React or.

It's failed to also as examples of how you can build use interfaces on top of Automerge, but you know, it's very just basic example code. I think it's not like an entire framework, but it's something that hopefully people can use to start building apps. Likewise for like the network synchronization. We have a sync server, it's open source and quite simple, and you can just deploy it yourself, but it lacks all of the features that you might want.

So there's no authentication, for example, which is something probably most apps will want. Really, we would like end to end encryption for the data synchronization for many applications as well, so the server doesn't have to store the plain text of your documents and a whole bunch of other things related to synchronization. So I think we will always want. The option for people to develop these things themselves and run it themselves if they want to.

But at the same time, I think there's a lot that could happen around having it's kind of packaged up in a nicer way where maybe there's a hosted cloud service that just provides a syncing service for local-first apps and if you. Choose a certain framework which might be Automerge based and a certain networking layer, then you can just use this synchronization service and you don't have to run your own servers. And that would be the sort of Heroku equivalent I would see of, of this world.

So I really hope somebody builds that. and a part of the vision of local-first is that, you know, we'll probably have to have cloud services involved in this data synchronization, but if we can make the synchronization protocols an open standard. Then hopefully there can be multiple different providers that can interoperate. And so if it decides that one particular provider has changed their pricing in a way that's too expensive or they're too unreliable or whatever.

You should be able to just point your app at a different provider and just continue working. And in some way, like Heroku had this as well, in that, you know, you didn't have to write custom, you have to use custom Heroku APIs to write your app, anything, you know, you just write a standard Rails app and you deploy it by pushing to a Git repo. And there was just a small amount of Heroku specific configuration.

And if you wanted to, you would always be able to take your app and run it on a different hosting provider as well. And so again, I think that's sort of Style I would like for local-first software too, that we have this interoperability and we have multiple companies, could be startups, could be big companies. I don't really mind providing this kind of cloud syncing services for local-first software in such a way that it can interop and you can easily switch from one provider to another.

I think that that would be really my dream for an ecosystem that's working well.

That sounds incredible. and I'd love to love to see that. It kind of makes me a bit reminiscent of the days of like torrenting, et cetera, peer to peer. We've talked to Peter in, uh, in a previous episode about peer to peer and there's some real technical challenges that we, that need to be overcome and maybe can't be overcome in the, in the shorter term, but I'm wondering. How that sort of more abstract syncing service would compare to some of the existing technologies.

I've mentioned peer to peer there because what was so interesting about it is like that, it's that you formed the sort of ad hoc network where people didn't, there was no server where it's something needed to be. deploy to, but things just started working together. So with that syncing service that you're mentioning, that could be kind of a platform agnostic, would that be similar to peer to peer in that regard?

Or would you still need to kind of deploy a quote unquote backend app to that syncing service that it actually does perform the work you want to have performed for your particular local-first step? I think. The best results for, you know, for user quality of software would be for it to use peer to peer when it's available and use a cloud service when not.

I think doing only peer to peer is really difficult because, for example, you can only talk to another peer while it's online at the same time. And if you've got two devices that are never online at the same time, then you can never synchronize data with between them. So. That sucks pretty badly because people do just close their laptop from time to time or turn off their smartphone or whatever. So I think pure peer to peer just doesn't work reliably enough.

Plus there's all of the problems with like NAT traversal and just the networking infrastructure doesn't work well enough. However, when peer to peer does work, it's amazing. And so if you've got two devices on the same network in the same building, it seems outrageous to send all of your data via AWS US East One in Virginia, if you could just send it via the local wifi from one device to another, right?

So then opportunistically using peer to peer when it happens to be available is an amazing thing. And it's, you know, it provides a lot of robustness and independence from the network. So that, for example, if you've got your laptop and your phone, and you're in some remote location where you don't have internet access, you can still sync data between the two of them.

And, you know, we have a sort of rudimentary version of that with say, AirDrop on Apple devices, but that's like one off file transfers really should be able to just do that for live synchronization as well. So I feel like the combination of Cloud and peer to peer just gives you capabilities that. only cloud or only peer to peer doesn't. And so that really seems to me like the most promising direction is to combine the two.

And the nice thing with CRDTs is that they just don't care what your networking layer is, right? All you need is some way of getting some bytes from one device to another. That's all they need. And whether that goes via a local network or peer to peer over the internet via a distributed hash table or via a cloud service or via multiple cloud services. CRDT doesn't care that any communication channel will do. That makes a, makes a lot of sense.

And this sort of hybrid nature where it optimistically uses the close peer connection, where if that works, then the experience is even better, but it kind of falls back to the cloud where it needs to. And it also will give you some benefits maybe such as backup, et cetera. So that future sounds amazing.

So one thing with these cloud services. Is that, you know, in the traditional way of building web apps, a lot of your application logic lives in the backend. You know, you have a backend database running on a server and then you wrap it with some server side code written using some server side web framework, and then you put it all behind a load balancer. And so you've got this, all this huge infrastructure on the backend. And one of the promises I see of local-firsts.

Is that actually because we've moved all of the interesting application logic to the client app, to the end user device, the server side that remains can be really simple and actually not contain any app specific code at all. so my vision for these syncing services for local-first software is that there's virtually no application code on the server. The server is just this generic piece of software where you just take it off the shelf and run it.

And, you know, you can just use a hosted cloud service. Maybe AWS will run a local-first backend service and charge you a few cents per gigabyte to use it. And that would be amazing. It can be, you know, this generic thing. So you don't have every single app reinventing its own backend service. You know, so much work in building a web app goes into reinventing this backend infrastructure that every single company has to reinvent again.

And so if we can make the data sync protocol and the data storage on the servers efficient, like loading and synchronization of large collections of documents, all of that can be generic. So if one person is then building a graphics app and another person is building a spreadsheet and another person is building a document editor, they can all use.

The same syncing service as the backend that I think is part of the economic value proposition of, local-first software is that actually, you know, we can just save ourselves a huge amount of software engineering work by making these backends generic. I couldn't agree more with that vision. I totally want that. Do you think Automerge will be the foundation for that? Is there something more generic, something more abstract of like an open syncing protocol, whatever that might be?

and Automerge would be one of multiple that implement compatibility with that. If someone is interested in that vision right now, is there anything that someone can take a look at and maybe deploy an early version of that already? Yeah, I think Automerge is trying to be a solution for that, and I would love for the Automerge protocols to be open standards one day.

I think, you know, we've thought about engaging with the IETF, for example, for standardization, although I think right now is just too early because it's all still very much work in progress and it hasn't settled enough yet to be ready for standardization. But in the long term, that's something we would definitely like. And we would like there to be multiple interoperable implementations that can all talk to each other and which are compatible with each other.

So yes, whether that ends up being exactly the Automerge wire protocol or something a bit more abstract, I I'm not entirely sure. I mean, other people are working on similar things. So one project that comes to mind is braid, for example, which they are engaging with the IETF and they're trying to build some standards or extensions to HTTP to enable data synchronization.

And they're trying to do it in a way which is not specific to any particular CRDT library or even using other approaches such as operational transformation. So they're trying to be generic. What I'm not sure yet is whether you can be generic and still get good enough performance. that's a trade off there. So in the automotive sync protocol, we're able to make a lot of optimizations.

because we know a lot about the types of data and how they're exchanged and we can control the data compression and the data formats and so on. Because we control the stack, we can do a lot of interesting optimizations there, which are more difficult if you have a generic protocol. So I think that waits to be, we'll have to wait and see how that develops in the future.

And I certainly believe some kind of protocol will become a widely used open standard for synchronous for data sync in local-first apps. It might be Automerge or it might be something else, but that's generally the direction we're heading. I'm really looking forward to that point. I mean, local-first already today. Is providing so much value, both to developers and to end users by simplifying the developer experience by making apps faster, giving you data ownership, et cetera.

But I think once we've reached that point where there's a more. General purpose, generic syncing service that works possibly also across apps that people can put a little node of that, for example, on a Raspberry Pi running next to their home router. I'm really looking forward to that. So I can't wait for that. Looking forward to maybe having you back in a year from now to hear some more progress update where things add in that regard, but I'm really looking forward to that.

Yeah, it's good to be very exciting to see what people build.

So besides your work on Automerge, you're also involved in the new project called Bluesky, which came out of Twitter or now called X as I think was sort of also like a research project inside of Twitter. And that was now took its own path. So, and you're involved there as an advisor. I'm wondering whether there's any connection to your interest in local-first as well, or whether those are separate paths. That is a sort of, um, high level connection.

I would say, you know, Bluesky is a social network it's decentralized and it aims to provide a bunch of features which just don't exist on like Twitter and Facebook and a centralized social network. So in particular, it's built on an open protocol and there are multiple different implementations, interoperable implementations of that protocol. And moreover, multiple hosting providers that can run different parts of the system.

And Bluesky is designed in such a way that it's very easy to move your account from one provider to another, for example. So for example, if you don't agree with one provider's moderation policies, it's fine, you can go to a different one, who's more aligned with you, or you could even run your own if you're technically, enthusiastic enough. So on a technical level, a lot of the implementation of. Bluesky looks quite different from something like Automerge.

There's no CRDTs in Bluesky, for example, but the sort of philosophy and the values that it embeds in the software are actually quite similar to local-first. This idea that users should control their own data, you know, you should always be able to have a copy of your own data that you can just take with you or move to a different provider. That concept is. Exists very much across both local-first and Bluesky in the case of Bluesky, of course, you know, it's a social network.

So the entire social network consists of the data from many different people, the posts, the likes, the follows and so on. But the way it works is that all of the data from a particular user goes into a repository, which you can think of a bit like as a git repository. And so every post that you make, every user you follow, every like you make.

Every user action of your own goes into your own repository, and that is your own, and you can download a copy of it, and on the server, it's literally just a SQLite database, there's a separate SQLite database for every single user, and you can just get a copy of it, and even if your provider just suddenly disappears, you can upload a copy of that. To a different provider, change your user ID to point to the new provider and everything just continues working.

And so that idea of having easy interoperability and easy migration paths from one provider to another, that's something that I think both Bluesky and local-first share. But then the, otherwise the implementations end up being different. Like it doesn't really make sense to have a local-first social network, because for example, working offline makes sense if you're talking about a document editor.

It doesn't really make sense in a social network because the whole point is communicating with other people so that the offline aspects, for example, don't really feature in Bluesky, but sort of the data ownership aspects do.

I agree that there is a big difference between a social network like Bluesky. And more like productivity or personal apps, I'm still curious, given that they share a bunch of similar values and some technical similarities to better understand what if you were to try to build Bluesky with a more local-first approach. There's a few technologies that leverage syncing behavior for SQLite or maybe replacing SQLite with Automerge just in theory.

I'd be very curious to understand, is there a certain impedance mismatch that you'd be running into by trying to build something like a social network with a local-first approach? I'd be curious to understand where you really run into troubles there. Yeah, so the data for one individual user, you could easily put in an Automerge document just as well as you put it in SQLite.

I think that that would make fairly little difference that you could certainly use Automerge to synchronize the data for a given user. What's different in a social network is that you have these global views, which are aggregated over everybody, which is just not something that exists in a document editor. So like in a social network, you know, want to know all of the likes on a particular post.

And if each user writes their like to their own repository, that means you have to index all of the repositories, look for all of the repositories that contain a like of a particular content, piece of content, and then add them up. And that gives you your number of likes. Or if you want to get all of the replies on a particular thread, again, you have to look at all of the posts that have been made by any user anywhere in the network and find all the sign reply to a particular piece of content.

That just requires this kind of global view of everything, if you want to do it properly, you can kind of do it in a somewhat local version, which is kind of what ActivityPub and Mastodon try to do. So there's no global index in with Mastodon.

There's, you know, no, nobody really maintains a copy of the entire network, but if user A replies to user B, then the User A's server sends a notification to user B's server, and therefore user B's server finds out about this reply, just adds it to its local database. But that way you can end up with a problem of different servers seeing different reply threads, because not every reply is notified to every server.

And so then you get Weird inconsistencies are depending on which server you're on. You see a different set of replies to a particular post, which is a bit strange, but that's just a part of the way that Mastodon works. And that's something we try to avoid in Bluesky by instead saying, okay, like the individual repos is just a single user's data.

And then in order to do something like a reply thread, Actually, we have a big indexing service that works a bit like a web search engine, which crawls the content of all of the individual user repositories and aggregates it all. And assembles the reply threads. And so that's something where there's no equivalent to that in local-first software, I think, because that's just something that like document editing style apps just don't need to do.

They just don't need to actually do aggregations across many apps. I would say that maybe an exception to that is if you want to do search across many documents, for example, in that case, you do need to build a search index. But it's still a search index containing only the documents for a particular user, or maybe all of the documents for a particular company, but it's not all of the documents in the entire world. That makes a lot of sense.

And I think it's sort of intuitive where like local-first starts out really dense about like your own documents, maybe the documents just on your other device or on the device of a friend of yours.

So the network, the suspending is like still pretty dense and this is what makes all of those technologies work almost trivially, but the more you go global with this to sort of like social network level, this is where that, uh, is really put, put to the test and it's probably not the best starting point that being said, I think this might still also be an interesting project for some, some folks who might want to rebuild an app in a local-first way, but

there might still be some more global nature to some parts of the data that maybe could be complimented in some way. Maybe there's some new architectural patterns that are emerging. for Overtone, for example, I'm trying to build the app in a local-first way, where really like all of the, your music metadata and actually your app.

Your music data is locally available if possible, but music as such has also a very global aspect to it right in the world of Spotify, you have practically like infinite amounts of music that you can't just like all. locally download there's too much and also other people have other kinds of music. So I'm also trying to explore sort of hybrid solutions there, which are really interesting design challenges. I'm eager to share more of that on a separate occasion.

And you've actually already provided me some great feedback and some personal conversations before. So, yeah, this is a really interesting case study in a. And I love exploring pushing local-first a little bit to its limits through various app use cases. So your involvement in Bluesky is a very interesting, at least theoretical case study at this point. So you've mentioning working offline for Bluesky. And that it might be not the primary use case.

I want to use this as a segue, as I see a little bit of confusion sometimes on Twitter, where people synonymously talk about local-first and offline first, and there is a difference and I want to share a little bit more broadly what that difference is, what is a offline-first app? What is a local-first app? Where are they different? So maybe you can share your perspective on that topic. Yeah, I would say that local-first includes offline first, but it tries to be a lot more than that as well.

So the term offline first existed long before local-first, and obviously we were aware of it. And in fact, we modeled the term local-first after offline first to some degree, because we thought it was a good term, and it captured something that we wanted, but it was not really sufficient. Because yes, having users being able to work offline is, it's Obviously a good idea.

It seems ridiculous if people can't work offline, but we wanted to also capture this idea of personal data ownership so that the data is yours and it can't be taken away from you. So in particular, for example, if there's some software that Stops working. If the company that made the software goes out of business, then I would argue that's not local-first. So it could be offline first.

So it could be that, you know, it's a nice Google Docs style document editor just take Google Docs as an example, like, okay, you know, it. It works fine. You can even, if you choose the right settings, make it work offline, and you can, you can edit your docs in whatever way you want.

But if Google decides to just discontinue the service, hypothetically, or if Google just decides to block your account because some automated system has flagged you as violating the terms of service, whether you did or not doesn't matter. You basically have no recourse. And at that point, you're just locked out and you lose all of your data. And so The fact that the app allowed you to work offline is kind of beside the point then because you still don't have ownership of the data.

And so it's that, this idea that you should not, never be locked out of your own data. That's really something that we wanted to capture in the idea of local-first. And so now if you can, can't be locked out of your data, that kind of implies that you must have the data on your own device. Which then also implies that you can probably edit it offline, because if you've got it locally anyway, then why not just enable offline editing?

But the kind of the chain of reasoning goes in a different direction. We would start with the data ownership and then offline editing follows from that as a consequence.

That makes a lot of sense, and I think that makes it really clear. I see a lot of people referring to offline first, almost synonymously as to some glorified version of aggressive caching, but the way how you lined it out here makes that a lot more clear. And I suppose this is not just having access.

To some form of the data that you can like download a CSV from all of your user data, but that the software is actually still fully functional or as functional as somehow possible, even in the worst case where the folks who are building the software are no longer able to work on it. And to really provide a better alternative to SaaS software X shuts down and the entire app is just. It's gone with probably all of your data. So I think that's a really clear alternative. Yeah, exactly.

Like I, you know, you do get this thing all the time when some SaaS, startup shuts down and they give you two weeks to download a zip file of JSON. You know, what can you do with that zip file of JSON? You can't re upload it into any other software. So basically it's just big fat middle finger to the users.

So really local-first is an attempt to overcome that in a way that,, you know, at the very least, you know, for example, if the software can operate peer to peer, that could mean then at least you have a peer to peer fallback. So even if all of the cloud services go away, it could still operate. Or if it uses a backend service that's interoperable, so you can switch it to a different provider.

That means then you could still use the software that, you know, maybe you purchase a license to the software in sort of the traditional non subscription type business model, and then you could use it in perpetuity, perhaps by pointing it at a different syncing backend, or in the worst case, running your own syncing backend, if you really must, but ideally just switching it over to a different provider.

And I'm hoping that's like the local-first term should , try to encapsulate and capture those types of values.

I fully agree. I'm curious now that you've been thinking about local-first now for more than 10 years, and we've come really far in that period of time when it comes to CRDTs and Automerge is production ready to use. At the same time, given the ambitions that you've outlined for, it feels like we're just getting started. I do think that already is a good time to really switch your default instead of going cloud first, go local-first for app use cases where it's possible.

But I think it's still very much the minority of developers. Who built this way. And given that you've seen such a broad spectrum of different data architectures that you've also outlined brilliantly in the book, Data Intensive Applications, I'm curious what you see as things that still hold back local-first to become more mainstream. Is it just a matter of time that there's more progress around Automerge around other technologies? Are there some other things that you would like to see?

Yeah, I mean, there's, it's such a big conceptual shift, I think, which is a challenge, you know, because there's a huge amount of say, educational materials on how to build web apps, you know, entire university courses are built around the idea of teaching people how to do this thing, coding boot camps, documentation for huge amount of software projects, books, videos, you name it, you know, everything is that there's just so much infrastructure on teaching people how to build it.

Apps in the centralized cloud way and local-first is just much newer. And so it hasn't had the benefits of decades of investment. Moreover, you know, there's the cloud providers have a strong commercial incentive to produce good quality documentation on how to use their services. So it's not surprising that there's good documentation available for those things.

And I'm hoping that at some point there will be big companies built on the local-first paradigm as well, which then are similarly able to. Fund the development of this sort of documentation and learning materials and so on, but it's just going to take a while. So I would see that as probably one of the biggest challenges. It's just a new way of thinking and people are not familiar with it. I think once people get it, then a lot of people seem to get excited about it and buy into it as well.

And, you know, sometimes there's. There's concerns that, you know, this is not for all apps. And I'm the first to acknowledge, yes, local-first is not for every single app. There's some apps which are best to build in a sort of centralized cloud way. That's totally fine. So I think part of it is also helping people understand for which types of apps would you pick a local-first approach versus for which do you pick a centralized approach.

And then of course, like just the general ecosystem needs, needs a lot more work. So, you know, the software libraries that we use, things like Automerge are They're pretty robust already, but it's still fairly new software compared to, you know, a web framework that has been around for 20 years or more. Uh, so one thing that I find encouraging is just within the last year or so, it seems that.

A whole bunch of startups have started using the local-first term just on their product marketing pages as just something they assume readers of the page will be familiar with. And that I find very encouraging. It's, it sort of shows that, you know, people are buying into the idea that enough that they are willing to, you know, have their product foundation on it and their marketing around it, explaining to users why it's valuable to have local-first. And I think this is the way it will succeed.

You know, it's the local-first will succeed only if many, many people in many, many different companies are able to use it to their advantage in order to provide a better experience to their users and their customers. And Build sustainable businesses on top of the idea and so on. So It has to work for everybody. And I think it will work for everybody because it's, you know, it's a win win. it's good for the app developers. It's good for the users.

I think questions still to be had about exactly what the business models look like, but I think that can probably also be figured out and then that way it works well across the board.

Yeah, I love that observation and I agree, I think some of the favorite tools that I'm using, they are all like, maybe not adhering to all seven local-first principles, but directionally, they are going in the direction of local-first, and it's almost like a quality badge that some products associate themselves with say like, Hey, we're trying to build this app local-first. And I, as a user know, Oh, this means it's probably one of the fastest app experiences that I get.

I feel much better about the data that I'm putting into it. So it's just, it gives me a much better baseline in terms of my expectations as a user. And I'm happy for the developers building it since they probably also have much more fun time. So, but you've also mentioned the question marks around the business model of local-first.

And I remember from like the good old days when you downloaded software and you needed to buy it, you needed a serial number, but then there were also a large group of people who would just crack software and use it illegally. And I think at that point, it was really seen as a solution that SaaS would just rent out your software on a monthly basis. And that sort of solved, the entire pirated software problem.

So I'm wondering, is local-first pointing in a direction to go back towards download software? Hopefully. Pay for that serial number, is there a best of both worlds, something that's not quite you rent your software AKA cloud and has all the problems, but maybe as a business, you do don't need to worry about pirated software and you get paid if you choose to have a paid plan as well. Do you have thoughts on what a business model in the local-first first world looks like?

Yeah, I personally wouldn't mind going back to the model of license keys and perpetual licenses. I personally quite liked it, but I do totally understand that for the companies making the software, like having recurring revenue is really, really nice. Even besides the piracy things you mentioned. And to some extent, I think there's no nothing stopping people just.

Doing subscription apps, if they're local-first as well, you know, just the fact that we've moved some of the logic from a server backend into the client doesn't stop you from being able to do a subscription. We can just tell people it's SaaS and sell it in the same way. And maybe that will work just fine.

I mean, It is true that because we have this idea of the user data ownership in local-first, you can't quite hold a gun to the user's head in the same way and saying like, if you don't pay your subscription, we will delete all your data, which is something that cloud software can very much do. And so it's possible that that means that then, you know, more people will drop off and stop paying the subscription. You know, you could make this.

the software simply not work anymore if the user hasn't paid their subscription. And of course, people could go in with a hex editor and change it so that it remove that check. But to be honest, not many people are going to do that probably. If they did, they would be in the same category as the people who did, who pirated licensed keys in the old software model. Like there's no way you can extract any money from them anyway.

Basically, it's probably not worth worrying about them too much and instead focus on those users. You can monetize who will pay their bills. And you know, as long as a reasonable percentage of the people pay, that's still fine. Peter van Hardenberg likes to say that back in the day of pirated software, people would worry that, you know, 95 percent of software is pirated and only 5 percent of users pay.

But actually with freemium software, A lot of starters would be very happy with a 5 percent conversion rates of free to paid. That's a really good conversion rate. So actually if you view it through that angle, you know, just. Not worrying too much about the people who are not going to pay anyway, and make sure that you provide a good experience for those customers who do want to pay. I think it's, it should be fine to build a solid businesses that way. I agree.

And I'm looking forward to see which sort of models do emerge. And if anything, I think the cloud has really rewarded. a very small number of like huge kind of monopoly like companies. And I'm kind of nostalgic about the days where you had a lot more smaller software vendors who really put a lot of care into for a particular audience might be a niche audience built the best possible software for them. And those are then probably also the people who would pay for software.

So I'm optimistic and I'm looking forward to see. Which sort of business models will emerge and yeah, can't wait to see where this is going.

Yeah. that's one of the things that makes me excited about local-first as well as hopefully it should just become a lot cheaper to build and run software because cloud software is just ridiculously expensive because like you need a backend team and the front end team and the backend team needs to be on call 24/7 in case the servers go down and then, you know, suddenly you've got a huge team and costs. A lot of money just to pay all those developers.

And then you have to have a mainstream app for a big audience in order to have a big enough market. And so that then cuts out all of this kind of indie software developers that you were talking about. And so we're hoping with local-first software, if we can just commoditize the whole backend so that app developers don't have to write their own backend. All you're doing is pulling some local-first framework off the shelf.

And writing your custom app logic in your front end, it just becomes a so much cheaper to develop the app. You don't have to worry about the whole 24/7 on call rotation. And then that makes it economically feasible again, to have these niche apps that are built by one or two people. And they only have a small customer base, but that's fine. You, all you need to do is provide a decent income for those two people. And then you can have these niche apps that.

Really perfectly serve a particular audience and just do that one thing really well. That's something I would, I would really like to see. And we're starting to see beginnings of this, for example, like one of one of our big contributors to Automerge works on an app for assistant directors of movie shoots. to plan their schedule of when they're going to shoot what and which actor they need for which scene on which set, with which props, et cetera.

And, you know, it's a super niche piece of software, but I really, really want him to succeed because I think it's just a great example of if we can make it easy for him to build this kind of software for his particular. Use case, then we can do the same thing for 10, 000 other niches as well. Yeah, I fully agree. This is something I'm super excited about. local-first as a whole is if he goes for life and realize how little in some ways software has penetrated our real.

Live where you interact with something and then you think about, wait, we have computers, we have technologies to solve this. Why hasn't it arrived in these parts of our life yet? Where would make life better? And I think the answer is typically incentive models of the cloud. If you build something for the cloud, you build it for like a, you need to build it for a huge audience, et cetera. Otherwise it's not worth it. Particularly if you go venture capital based.

So I think this is where local-first really completely flips the moth, allows people who are passionate about a particular use case, a particular niche to go for that niche and that you don't need to worry about reaching a giant audience if you don't want to. And I think local-first can really change the economics there. So I'm super excited about that. That's almost like a second order effect. And I'm sure there will be others that I can't really think about right now.

But I have a gut feeling that it will be a good one. So yeah, Martin, this has been a real pleasure to have you on the show today and sharing all of those anecdotes, the thoughts on the, where things are coming from, where things are going. So do you have anything else that you want to share with the audience before wrapping up? Not really. I'm, just very happy if people are interested in local-firsts.

So, I mean, thank you to you for running this podcast for helping, popularize the idea further. And thank you to everyone who's listening and for being interested in it. And I hope the community will continue growing further as we get more people. You know, just building it in the direction for what they want it to be.

So I think, you know, we, we can just provide a set of starting values and some technical tooling, but in the end, it'll all depend on what the community decides to build around it. And so I'm really excited to see what will come when, as people take these ideas and run with them.

Awesome. Yeah. Whenever we do our next show together, I'm sure there will be a lot more apps being built in local-first that we can already point to that did not exist today. So I'm really looking forward to that. Martin, thank you so much for coming on. Thank you, Johannes. It's been great. Thank you for listening to the localfirst.fm podcast. If you've enjoyed this episode and haven't done so already, please subscribe and leave a review wherever you're listening.

Please also tell your friends about it. If you think they could be interested in local-first, if you have feedback, questions or ideas for the podcast, please get in touch via hello at localfirst.fm or use the feedback form on our website, special thanks to Expo and Crab Nebula for supporting this podcast. See you next time.