#24 – Ben Holmes: Astro, Simple Sync Engine & Warp

What if I'm on my phone, I type a thing and then I go back to my computer and I try to see it? Is it gonna merge it together? Is it gonna break? I have no idea because. Client side stores don't help you with those problems. They're not designed for those problems. They're designed for like ephemeral state that can disappear at a moment's notice if it needs to for some reason.

so that led me down exploring local-first technologies and I found pretty quickly how capable SQLite is for these use cases, including in the browser. Welcome to the localfirst.fm podcast. I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and I love the craft of software engineering.

For the past few years, I've been on a journey to build a modern, high quality music app using web technologies, and in doing so, I've been falling down the rabbit hole of local-first software. This podcast is your invitation to join me on that journey. In this episode, I'm speaking to Ben Holmes. A senior web developer and educator known for his whiteboard videos after having spent most of his career building server centric applications.

Ben recently explored local-first software by building a simple sync engine from scratch before getting started. Also, a big thank you to Jazz for supporting this podcast, and now my interview with Ben. Hey Ben, so nice to have you on the show. How are you doing? Hey, I'm doing great. Yeah, how are you doing? I'm doing fantastic. Super excited to have you on the show.

I'm very certain that most of the audience already are familiar with who you are since you have quite a reach on, Twitter, x, other platforms, et cetera, where I think you're doing an excellent job of taking like novel concepts and breaking them down in a very simple and approachable way. And I think you've done the same for some local-first related topics recently. So maybe some of the audience are already familiar with your work in that regard.

But, for those who are new to you, would you mind giving a background who you are? Yeah, totally. I'd be shocked if everyone knows, but if you've seen a guy with a whiteboard around the internet, it might be me, especially if it's a vertical video. so I've been doing a lot of work in just the engineering space for a long time. So been at Astro for a few years and we've been building the framework for content sites.

And I know actually you have some experience working with Contentlayer and other things, but content sites are a nice place to get started because they're a nice, well-defined use case for web technologies and Astro was trying to spearhead being like the simplest way to do that. so was able to contribute to that for a long time and also produce a lot of videos on learnings in the process.

So it started with taking a rock band microphone and a whiteboard outta my closet and just recording stuff and just seeing where it went and been doing it for years now and kind of honing the craft. And we've covered all sorts of topics, including, different web frameworks, Tailwind tips, database providers using SQLite. Most recently putting SQLite in the browser, which is gonna be very relevant today, I'm sure.

and most recently I've made the jump to Warp Terminal, so I'm gonna be, well, I have joined their team as a product engineer and we're sort of spearheading the future of bringing AI and agent workflows to everything you do in the terminal. So if you forget a command, need to kill a port, need to go through a full get workflow or even edit files in your projects, you can just ask Warp to do that for you or fall back to all the features that make Warp just a really nice terminal.

It's a really great fit. Been working with them for a while and now get to do that full time. We'll continue to do all the videos that you see around the internet. That is awesome. I'm certainly coming back to Warp since I wanna learn more about that as well, but taking things one step at a time. you've been digging into local-first related, you've been mentioning SQLite running SQLite in the browser. Looking into that over the course of the last year or so.

I'm very curious, like what led you to explore that since with Astro, et cetera, you're probably, that's like a different part of building websites, really where you maybe like Astro is famous for introducing the island architecture where you go like very light on bringing JavaScript into the, website. And if it's almost like the other extreme where you want to bring a lot of JavaScript into the web app, deemphasize the server part, and also more on the spectrum from website to web app.

Certainly much heavier on the web app spectrum. So yeah. Can you share more what led you to, to this path?

well the past year has been kind of like an existential crisis of how are you supposed to build websites, and I think we all went through that as an industry, as things kind of shifted back from client side, heavy react apps towards things that are more server rendered and people are slowly trying to question that idea again and see what we can learn from storing more information in the client and sinking it back to your servers. So I also noticed a wave that's completely opposite.

Of local-first apps, which would be something like HTMX, where everything is purely server driven, state stateless. Everything is from like the REST API protocol. And there's a really nice simplicity to it where a state lives in one place, the server's a source of truth for everything, and you use different return values for each HTML sheet to decide what is going to render next.

And you accept that there's going to be network latency for most interactions on the site, except for very small dropdown toggles and the like. But anything that involves state is always going to go back to the server and map it back. That obviously has trade-offs that people have tried to get away from with client side architectures.

But the reason it's so nice is you don't have to think about, you have this server state, you have this client state, and you're constantly trying to keep them sort of melded together. And I think that's something that a lot of server side rendered applications have run into most recently react trying to add on, like use optimistic hooks and libraries like TRPC, letting you show data optimistically and then replace it with the server value when it comes in.

And these are important principles, but it's a lot of manual effort to first send a request to the server and also keep the client in some optimistic version of that to cut down on network latency. You're having to pull levers on both sides and you don't really know where the truth lives.

So, from my experience that's led to a lot of manual work, writing and rewriting local stores, maybe with Redux back in like the mid 2010s now using, query hooks, even GraphQL if you're racing for those kinds of tools. So it's, very messy using those things that are isomorphic, AKA, things that run both on the server and on the client, and trying to think of it in the same way. So one response to that is just, we don't need client side JavaScript. We're gonna do everything on the server.

And that's very easy to understand because it goes back to like how rest was designed in the eighties. And then there's the other reaction, which is the, to other side, the client is the source of truth for pretty much everything that's going on. And the server's just a broker to keep different clients in sync and to push changes so everyone can stay up to date or even doing decentralized servers if you go further. But it's that other side of the coin. Of we want one source of truth.

We don't want to think about, we have this server and this client and we constantly have to write logic to glue them together. No, you either store all the state on the server, kind of like an HTMX style, or you store all the state on the client using local-first technologies.

And having explored the first one for a long time, since Astro is meant to be like a static website or server driven website, I was excited to explore the other side of storing everything on the client, keeping it up to date, and then figuring out how synchronization happens. That's also been around for a long time. If you look back to just like the first calendar app on your phone or on your computer, this challenge has been around since probably SQLlite was created.

but it's only now that web devs are starting to get a lot of footholds to also apply this to websites if that's something that you need. I think you've summarized it really well. And, I would go even as far as saying there is an elephant in the room that most people are aware that there is an elephant, but they really don't have yet the, the right terminology. And I would say the term here of the elephant is distributed systems.

We have distributed states and distributed systems are really, like, that's a core discipline of computer science that doesn't really have just like the good answer, but it's a really, really hard problem alongside of like naming things and so on. But, that part, like everyone who's building a web app, anything that has state.

On the server, like even in the server, typically you have by definition also already a distributed system where you have states in your API throughout the request lifecycle, but then you also have it in the database. You might have like concurrent requests. So there you already have a distributed system. Typically there it's much less bad because they're like, you can just trust the server already less. So you just trust the database. So you basically push it all down to the single choke point.

But if you wanna trust the client even more now all the distributed parts are get the distance grows and therefore the divergence. And I have in university, when I studied computer science, I don't think I've actually taken a class on distributed systems and I've missed a memo where someone would've instilled it in me is like, Hey, everything that are you gonna build will suffer from distributed systems, make sure to understand this problem properly and then design the architecture around it.

And I think most web devs are not aware of that. And what you've just laid out, I think is exactly suffering from this problem. And this is what I think where some framework creators are making the very smart decision to empower the server as much as possible. Because if you live by that sort of maxim, at that point, the implied damage that you can cause by building it in a certain way is minimized.

And, my journey over the last five years or so has been almost like intentionally letting the pendulum swing to the most other extreme where all the state is governed by the client, since I'm pretty convinced that the middle ground is just pain and suffering. So I'm pretty convinced that you should really analyze the use case that you have. And if it's a daily driver productivity app, you probably wanna move as much state to the client as possible.

Where it happens, like you produce most of your state in your calendar app or in your Notion, or in your other notes app or whatever you want to have it declined. And, if it's something like New York Times, then you don't produce any of that date. So it should be on the server. And, I, feel like more people should, start with that assumption, then build an architecture around it. But, yeah, I think you summarized it super well. Yeah. And it is tough.

To prescribe either side because there are neat buckets like content sites and client side apps like Notion. But the classic example is, well what about e-commerce where things are server driven until you're in the add to cart flow and now it's client driven. So what do you do then? How do you architect it? We actually met this challenge building the Astro storefront front template, which was mostly server driven with some client side components sprinkled in.

I think the answer there was, it's still fine to leave things server side and not rely on optimistic updates too much, except for like small examples like increase and decrease quantity in your cart. Do you want that to feel instant and happen on a delay or a debounce but everything else is server driven. There is no perfect architecture, I guess is what I'm saying. It's more painful when you really try to blend it 50 50 and nail every use case.

But if you can keep it 80 20, where 80% is in one realm and 20% is business logic in the other realm, like 80% server driven, 20% client side complexity or less, then you're kind of minimizing the footprint of pain that you could run into since I've certainly noticed that with some of the newer patterns and older patterns with. Client side apps. but there was a second thing you mentioned about distributed systems that I, totally agree with.

I was reading the book Designing Data Intensive Applications, the big O'Reilly book, probably the first one you find looking up like CS principles. And I think there was one section about distributed systems where it walked you from like single leader replication. You got one database and if you want to have some caches to make reads faster, you just put all your rights to one database and then it'll replicate the reads out to everyone else. So you can't write to a bunch of different regions.

You can write data to one region and then it'll sort of. Push out all of the clones and allow eventual consistency to work. but there are cases where that doesn't work anymore. Like in Notion, I cannot wait for the server to update the document. It's just gonna update when I'm typing and when I add blocks and when people are invited to join the document. All that stuff matters. So you need some way to be able to write to maybe the nearest node.

Like if you go fancy with CloudFlare, you could have really localized, durable objects that are two feet away from your computer and that reduces latency in some sort of magical way.

but then it kind of explains well if you just put a data replica just on the client device, that's another version of multi leader replication, where instead of replicating in some really edge node, you're just replicating on the person's computer and everyone is like a mini server unto itself, where you just read and write data and they'll all report back to some centralized source of truth later on. So when you're working with, like, is it a distributed system? It is.

Even when it's running on your device, if there's some sort of synchronization layer, you're just moving it closer and closer and closer to the computer until it iSQLiterally in the computer. It's inside the house. I've, heard a, very interesting framing of sort of like that elephant in the room and like that problem that we just talked about. I think it's by Carl who worked on SQLSync and recently, released Graft, which is another fascinating project.

And he has written a blog post about, don't recall the exact name we're gonna put it in the, show notes, but it was basically along the lines of like, your application is a database and that's bad. I gotta look it up. what the, exactly the title was.

But it was basically, he's making the point that every app that is sufficiently getting more complicated will basically build its own version of a ad hoc database and if you're just using React useState or something else enough, and then you try to add persistence, et cetera.

What your app is basically becoming is a poor implementation of a database where like all the things that a database does a lot of like R&D great work to make it fast, to make it correct, to make it like nicely, like transactionally correct, et cetera. All of those things you're now trying to handle through, useState and useEffect and like when this thing changes, also change that thing, et cetera.

And, I just thought that framing was so elegant and, his conclusion, which I agree with is like, hey, if let's try to make the app about what the app tries to do and let's leverage a database so we can focus on the actual like features, et cetera we want to implement.

And I think that's another kinda articulation of the elephant in the room that we're accidentally and without being aware of it Building databases as that are sort of camouflaged as apps and, we should embrace it more if we're starting to see those signs of those problems.

So you've mentioned that you've gone down this path, a lot out of curiosity and just because you've, pretty exhaustively explored the more server centric paths, can you share more about like what were your frustrations and pain points that you felt where you wanted to go more, like embrace the client more, but still trying to do that with the more server centric architecture.

So you've mentioned optimistic state, et cetera, maybe can motivate this through a concrete app you wanted to build where you just felt that that pain and that impedance.

Yeah, I mean I've played with all sorts of side projects as we all have. and I was working on a few different things. One was like a localized note taking app, and I was also playing with local LLMs and other pieces to add vector search into a local environment because I was running up on the limits of using Notion and being very frustrated with loading spinners and offline warnings when I was trying to use the app.

I hear that's being changed and rectified as they build out their own, like SQLite replication. I know that's in there. There's some fascinating videos about that on the internet. but I still do see the value of just open up Apple Notes, you type things and it's just kind of there in a SQLlite store. You can find it on your file system. All your Apple Notes are just in a SQLlite store and it's great. so I thought, well, it seems like that's the answer.

If I were to go the traditional server route, I would sit here waiting for all these updates to persist, which just wouldn't work. Or I'd be sort of gluing together a bunch of useState calls and figuring out how do I update this server and what if someone else updates it? What if I'm on my phone, I type a thing and then I go back to my computer and I try to see it? Is it gonna merge it together? Is it gonna break? I have no idea because. Client side stores don't help you with those problems.

They're not designed for those problems. They're designed for like ephemeral state that can disappear at a moment's notice if it needs to for some reason. so that led me down exploring local-first technologies and I found pretty quickly how capable SQLite is for these use cases, including in the browser. you can load up SQLite with like a wasm build as you're very aware and use it even with. Like very nice SQLite libraries.

If you wanted to use Drizzle for example, which is a common SQL querying library and JavaScript, it matches onto the browser version of SQLite perfectly. So you can actually make declarative, find many notes and it'll just do the little SQL query. It'll join it up with all the authors of the post and it'll just give it back to you. Kind of like you're on the server, but you're on the client. And you can use client side ORMs or query builders, whatever your flavor of preference.

So that was very empowering to see. Yeah, you can bring all of these niceties you get from server side data querying and bring it into the client. And I tried to stretch it a little bit further by asking, what about SQL extensions? Could I add a vector search plugin, for example? The answer is yes. There actually is a vector search plugin that I think is developed by someone on the Mozilla team. So it is pretty battle tested in different languages.

I think they're sponsored, yeah I've had the chance to meet them in October when I was in LA super lovely person. And, they have some sponsorship from the Mozilla team currently. Nice. Yeah, and I was playing with the, rust flavor of that since, well now I'm working at Warp, and Warp is a rust powered terminal, so naturally I need to get up on the Rust knowledge. And also if you build apps with Tori, which is a native desktop application tool that uses Rust as well, so.

Side tangent, but it is nice to use tools that could work in JavaScript as well as rust in very efficient languages. so I reached for that. I put vector search in. I was also able to run an entire LLM across the data in the browser and just load up like the entire vector search setup thing. and I just said, all right, I'm gonna backport all of my markdown files into this thing. I'm gonna vectorize all of them in the browser. I'm gonna search them in the browser.

And I was able to get it working with like next key search. And it was absolutely mind blowing. Like this is like an actual AI powered search tool. And you can get like with every keystroke new results And is isn't that wild like that this machine that we have like sitting on our laps or like this machine here in my hands that is capable of all of those things and just the way how we kind of build web apps.

Over the last decade or so, we've kind of forgotten or denied the capabilities of our client devices. And we only just like trust the server and we've almost like, why did no one tell us that this is possible? And that's so magical when you see that this is working and just the stuff that's currently, in the works with like web, LLM, et cetera, where it can run like a full blown like Llama model locally in your browser. running on web GPU is absolutely wild to the capabilities that we have.

But I think what's holding us back is where does our data live and everything else. Kinda like, it's almost like a second order effect from where the data lives. And this is what, your anecdote really nicely highlights of like, you go with SQLite with your data and then you like bring in another superpower of like SQLite Vec with the vector embedding, et cetera. And I think it all starts with where the data is, how your application is being shaped. Yeah. And it is good to find that.

Just common data layer. SQLite is the easy answer. PG light is a more robust exploration of bringing Postgres to local devices. That's a bit earlier on. but I think we're entering a world where software is just so easy to spin up that you will very quickly have a web client, a mobile app client, a desktop client, and they're all talking to the same sync server. And when you're in that world, it's nice to just reach for SQLite. 'cause I can run SQLlite on my iPhone.

I can run SQLlite on my Android device, I can run it in the browser and I could run it in a desktop application. So as long as you just have this concept of clients write to SQL servers and those SQL servers have some way to talk to each other, then you can build these multi-platform applications very quickly, even across different languages. And just figure out what that sync layer looks like, which we can probably talk about.

that would've been my next question since, I think what you started with was probably without the sync part yet, where you can just locally in your browser web app, you successfully ran SQLite using the wasm build. Then you brought in SQLite Vec, you could, get the AI magic to work. You saw like how insanely fast everything feels. You write something, you reload the browser. Even if you're offline, it's all still there. Great. But now you're thinking, okay, it works on like local host 3000.

how do I get it deployed and how do I get it so that, if I accidentally open this in a cognitive tab and I close it, poof, everything is gone. and how do I get it show up on my, phone so this is kind of collaborating with yourself, but on also collaborating with others, which we maybe punt on that for a moment. But yeah. How did you go from, it works locally almost like if you use like just local storage to trying to share the goodness across your devices.

Well, I think we both have the privilege of just ask Andrew on the Rocicorp team and he'll point you to some resources. if you don't have that, I do recommend that. Well, the one approach that I used was just reading through the Replicache docs. That explain how their sync engine works on a very high level.

And Replicache is a batteries included library, well some batteries included library that sets up a simple key value store for you where you can write and read keys on the client and you can author code that will post those changes to a server that you own and pull changes back from the server whenever changes are detected. And you can implement that on a SQLite or anything else if you just have their simplified mental model, which is very inspired by just how Git works.

So in Git, if you had to change locally and you wanted to push it up to the main line, you would run Git push. And the same kind of thing happens with their sync engine service, where anytime you make a change locally and you can decide what that means, maybe it's a debounce as you're editing a document, maybe it's when you click on a status toggle in Linear and you wanna change that status.

Whenever the trigger is, you can use that to call push, which would probably call an endpoint written on your own server. That's simply a push endpoint and that can receive whatever event you ran on the client so that the server can replay it, back it up into its own logs and tell other clients about it whenever they try to pull those changes back down so you can run your own little push whenever you make a change.

And then other clients on an interval or a web socket or some other connection can pull for data whenever they want to get the most recent changes made by other people. Now the question would be, what am I pushing and what am I pulling? Like what data? Needs to be sent across. And there are a couple different methods. I know Replicache uses a data snapshot mechanism.

I don't fully know the intricacies of it, but I know that because they use like a key value storage, which is much simpler than like a full-blown database. They can take a snapshot of what the server looks like right now and then the client has its own copy of that server snapshot and it can just run a little diff the same way you'd run a get diff to see what code changes you have made. It can run that diff and you can see, all right, I added this key.

All right, updated this status flag, and then tell the server, this is the diff, this is the change that was made to the data. And the server can receive that request and say, okay, this is the change that I need to make against my copy. And then I will tell other clients to also apply that diff whenever they pull. it's a very Git inspired model, and it works if you're able to diff data easily.

If you are working with like a full blown SQLlite table, you run table migrations, tables, change shapes, that's a very hard thing to keep track of. So another option that I implemented for the, learning resource I created called Simple Sync Engine. If you find that on my GitHub, probably in the show notes, you can see that. But it was meant to be a very basic implementation of this pattern that uses event sourcing instead.

So rather than sending a diff of how the data should change, instead sends an event that describes the change that is made like a function call or an RPC, however you wanna think about it, where you would tell the server I updated the status to this value. And in our implementation we create these little like function helpers that can describe what those events are. So you might have like an add status event. So like the type of the event is add status and that accepts a few arguments.

It accepts the status you wanna change it to, and the ID of the record that has that status currently. So the server receives that event, it sees, okay, the type was set status. I see two arguments here. The ID of the record. And the new status that should be applied. I'm gonna go ahead and run that event as a database update. So it has that mapping understanding of, I received this event, I know that maps to this SQL query, so I'm gonna go ahead and make that change on my copy.

And then whenever people poll, you can send that event log out, or you can send whatever events the client hasn't received up until that point. You can kind of think of those like Git commits where you're pulling the latest commits on the branch and the server's able to tell you, here are the latest events that were run. Go ahead and run those on your replicas so both the server and client know what events. Actually mean, like this event means I need to make this SQL query.

And it's able to do that mapping. and you can have a bit of freedom there. If you had like a very specific kind of data store on the server, like MongoDB, you could customize it to say, whenever I receive this event, it means this MongoDB query and this call to the Century error logging system, or whatever middleware you wanna do.

As long as server and client agree on what events exist and what changes they make in the data stores, then everyone can be on the same page whenever they're syncing things up and down. you're very familiar with event sourcing as well. I'm curious if there's things that I've missed or important edge cases that we should probably talk about. I think you very elegantly, described how simple the foundation of this can be and hence the name of like Simple Sync Engine.

I think this has served as a great learning resource for you and I'm sure for many, many others. And once you like, start pulling more on that thread, you realize, oh shit like, okay, that thing I didn't think about. Oh, what about this? So you've mentioned already the reason why you preferred, event sourcing over the snapshot approach because like with SQLlite, What would you actually compare? This is where I would give a shout out again, to Carl's work with Graft.

this is what he's been working on. We should have him on the show highlighting this as well. But, where he's built a new kinda sync engine that, is all about that diffing of like blocks of storage. I think all focused around SQLite and that gives, that does the heavy work of like, diffing and then sending out the minimal amount of changes to make all of that efficient. Since there's this, famous saying of like make it work, make it right or correct and then make it fast.

And working on all of this has really given me a deep appreciation for all of this since like, and I'm sure you probably also went through those stages with the Simple Sync Engine, like with making something work in the first place was already quite the accomplishment. But then you also realize, ah, okay, in those cases this is not quite yet correct.

And then you could go back and like try to iterate and then you also realize, okay, so now it has worked for my little to-do app, but if I, now, depending on the architecture, if I roll this out and put in hack news and suddenly have like 5,000 people there on the same time, this thing will break apart. Will A not be correct in some ways you didn't anticipate and will also not be fast enough. So now making this fast. It is at the end of the day is like really reinventing.

It's like your app is becoming a little database and you want to like move as much of that database burden to the sync engine. This is why folks like Rocicorp, ElectricSQL, et cetera, they're doing a fantastic job trying to absorb as much of that complexity. But building something like this by yourself really gives you an understanding and an appreciation for what is going on.

I love the Git analogy that you've used, but just a, a couple of points just similarly to how, your sync engine works is actually very analogous to how the Livestore architecture on a high level works. But I've had to, before I arrived at that, I really wanted to think through a lot of like the more further down the road.

Like, what if situations since, one that I'm curious whether you've already run into, whether you resolved in some way or left for the future, is how would you impose a total order of your change events? So this is where When you have, like, let's say this to-do app or like a mini Linear, app. Let's say you create an issue and then you say you, complete the issue or you toggle the issue state, et cetera.

It can mean something very different, if one happens first and then the other, or the other way around. And for that, where you have your events of like, hey, the issue was created, the issue status was changed, this, the issue status was changed, that, the order really matters in which way in which order everything happened. And, this might be not so bad if you're the absolutely only person using your app and typically only on one device.

But when you then do stuff between multiple devices or multiple users, then it's no longer in your single user's control. That stuff happens concurrently. And then it really matters that everyone has the same order of the events. And this is where you need to impose what's called a total order. And I'm very curious whether you've already, hit that point, where you thought that through and whether you found a mechanism to impose it since there's many, many downstream consequences of that.

Right. And I definitely did hit it and. You will find in the resource, it's not addressing that issue right now. It's in a very naive state of when change is made, send, fetch, call to server. And there are a few problems with that, even if you're the only client.

Because first off, if you send one event per request, it's very possible that you send a lot of very quick secession events in like an order, and then they reach the server in a different order because maybe you pushed the first change on low network latency, the next one on high latency, or actually the reverse of that where the first change hits after the second change because that's just the speed of the network and that's what happened. And also you need to think about offline capabilities.

If you push changes when they happen, how do you queue them when you're not connected to the internet and then run through that queue once you're back online? That's another consideration you kind of have to think about. Could be solved with just like an in-memory event log and just kind of work with that. But you still have the order issue. I'm familiar with atomic clocks as a method to do this. There are even SQLite extensions that'll sort of enforce that, having not implemented atomic clocks.

Is it kind of this silver bullet to that problem or are there more considerations to think about than just reaching for something like that? Right. I suppose you're referring to vector clocks or logical clocks on a more higher level? Yeah. since the atomic clocks, at least my understanding is like that's actually what's, at least in some super high-end hardware is like an atomic clock that is, like that actually gives us like the wall clock. So Right, right now is like.

Uh, 6:30 PM on my time, but this clock might drift, and this is what makes it so difficult. So what you were referring to with logical clocks, this is where it basically, instead of saying like, Hey, it's 6:30 with this time zone, which makes everything even more complicated, I'm keeping track of my time is like 1, 2, 3. It like might just be a logical counter, like much simpler actually than wall clock time.

but this is easier to reason about and there might be no weird issues of like, Daylight saving where certainly like the, the clock is going backwards or someone tinkers with the time, this is why you need logical clocks. And, there, at least the mechanism that I've also landed on to implement, to impose a total order. But then it's also tricky, how do you exchange that? how does your client know what like three means in my client, et cetera?

And the answer that I found to this is to like that we all trust. A single, authority in the system. So this is where, and I think this is also what you're going for, and with the Git analogy, what we are trusting as authority in that system is GitHub or GitLab. And this is where we are basically, we could theoretically, you could send me your IP address and I could try to like pull directly from you. It would work, and that would also work with the system that you've built.

However, there might still be, they're called network petitions, where like the two of us have like, synced up, but some others haven't. So as long as we're all connected to the same, like main upstream node, that is the easiest way to, to model this. An alternative would be to go full on peer to peer, which makes everything a lot, lot, lot more complicated. And this is where like something, like an extension of logical clocks called vector clocks, can come in handy.

you've mentioned the, the book, designing dataset intensive application by Martin Kleppman had him on the show before. he's actually working on the version two of that book right now, but he's also done a fantastic free course about distributed systems where he is walking through all of that, with a whiteboard, I actually think so, I think does what, what the two of you have very much like you've both nailed the, craft of like showing with simple strokes, some very complicated matters.

so highly recommend to anyone who wants to learn more there. Like, learn it from, from Martin. He's, like an absolute master of explaining those difficult concepts in a simple way. But, yeah, a lot of things go kind of downstream from that total order. So just to, go together on like one little journey to understand like a downstream problem of this, let's say we have implemented the queuing of those events. So let's say you're currently on a plane ride and, you're like.

Writing your blog post, you're very happy with it. You have now like a thousand of events of like change events that captures your work. Your SQLite database is up to date. but you didn't just create this new blog post, but you maybe while you're still at the airport, like you created the initial version with it with like TBD in the body. And your coworker thought like, oh, actually I have a lot of thoughts on this. And they also started writing down some notes in there.

And now, the worlds have like, kind of drifted apart. Your coworker. Has written down some important things they don't want to lose, and you've written down some things you are not aware of the other ones neither are they, and at some point the semantic merge needs to happen. But how do you even make that happen in this sync engine thing here?

And this is where you need the total order, where you basically, in the worst case, this is what decides, like who, gets a say in this, who gets the last say, in which order those events have happened. The model that I've landed on, and I think that's similar to what Git does with rebasing, is basically that before you get to push your own stuff, you need to pull down the events first, and then you need to reconcile your kind of stash local changes.

On top of the work that whoever has gotten the, who got lucky enough to push first without being told to pull first. So in that case, it might have been your coworker because they've stayed online and kept pushing. And now it sort of like falls on you to reconcile that. And I've implemented a, like an actual rebase mechanism for this, where you now have this set of new events that your coworker has produced and you still have your set of events that, reflect your changes.

And now you need to reconcile this. So that is purely on the. Event log level, but given that we both, want to use SQLite now, we don't need to just think about going forward with SQLite, but we also now need to think about like, Hey, how do we go? Like in Git you have like, you have this stack of events, right? So you have like a commit, which has a parent of another commit, which has a parent of another commit.

It's very similar to how your events and this event log look like, except it's now no longer just one event log, but you also get this little branch from your coworker. So now you need to go to the last common ancestor. And from there you need to figure out like. How do I linearize this? I've opted for a model where everything that was pushed once cannot be overwritten, so there's no force push. So you basically just get to append stuff at the end.

But, in order to get there, you need to first roll back your own stuff, then play forward what you've gotten first. and then on top add those. And the rolling back with SQLite is a, thing that I've like put a lot of time into where I've been using another SQLite extension, called the SQLite Sessions extension, which allows you, per SQLite write, to basically, record what has the thing actually done. So instead of storing, insert. Into issues, blah, blah, blah.

when running that, you get a blob of let's say 30 bytes, and that has recorded on SQLite level, what has happened to the SQLite database. And I store that alongside of each change event, that sits in the event log. And the very cool thing about this is, I can use that to replay it on top of another database, but to kind of catch it up more quickly. But I can also invert it. So now I have basically this like, let's say 20 events.

And for each, I've recorded what has happened on SQLite level, and now I can basically say. When I need to roll back, I can revisit each of those, invert each of those change sets, apply them again on the SQLite database, and then I'll end up where I was before and that's how I've implemented rollback on top of SQL Lite. So this is as mentioned when you're going, down the, rabbit hole of like imposing a total order.

There's a lot of downstream things you need to do that makes this even more complicated. But, from what I can see, you're, on the right track if you wanna pursue this further. Yeah. And I do have a rebasing mechanism in place in mind that's more, just kind of a sledgehammer. I got two SQLite databases in mind.

in the same way that on Git you have like your local copy of the main line and your local copy of your work, there's always this local copy of Main, that's just whatever events have come from the server. So this is the source of truth that the server has told me about and that was something I forgot to mention earlier. Explaining all of this is the server is the source of truth. It has that main line of the order of all of the events, and that is what all the clients use to trust.

But yeah, it has like that local copy, and then when it pulls from the server, it'll update that copy. It'll look at all the events that are kind of ahead in the client, and then it'll say, okay, I'm gonna roll back my client copy of my branch to whatever the server is. And it's literally just a file right call. So it just overwrites. Your like client SQLlite file with a copy of the server one.

And then we look at the events that the server didn't acknowledge yet and then we replay those on top as a very basic way to pull and make sure, because it's very possible that you made some changes locally that the server hasn't acknowledged yet. Like you've pushed them up still in process and you pull down the latest changes and you don't see all of that stuff that you pushed up yet because of network latency.

So this sort of avoids that problem where you pull down from the server and now you need to replay whatever you did on the client that the server hasn't acknowledged yet. It hasn't received that network request. So that was a very basic need to have some rebasing, but it does get a lot more complicated when you have collaborators on a document. I've seen a few different versions of this. CRDTs is the fun, like magic wand. It does everything.

but there are also solutions from Figma, for example, where they say everything in Figma is kind of its own little data structure. Like you can put some text and that's its own little data field. You have rectangles. Those are a data field. And whenever you update a rectangle, like you update the pixel width of a rectangle, that's like an update event on some SQL table that stores all the rectangles for this document.

So whenever you make that update, it'll update the pixel value of whatever that row entry is, and then it'll push it up for other people to receive. And when you pull it down, it's last right wins. In other words, whoever the last person is in that order that the server decided on that total order. That's a new word I know about now.

Didn't know it was called total order, but yeah, that, once you pull it down, whatever the server said was the order of events, that's gonna be the final state of that rectangle on your device.

The only time it becomes a problem, and you may have experienced this, if you're ever working on like a fig jam together with a bunch of people, if you're all typing in the same text box, everyone's just like overriding each other and a text box glitches out and changes to whatever's on the other person's screen. You can't see people's cursors because you're fighting to update the exact same entry in the database and it can't reconcile those changes.

so it only works up to, like you're editing different things in the file and you're not really stepping on each other too much. As soon as you're stepping on each other trying to edit like the same text field, then you wanna reach for something that's very, very fancy, like CRDTs. Which will try to merge elegantly all of the changes that you're typing into the same database field.

It's maybe over-prescribed because of how powerful it is, but for those specific scenarios, it's really nice to reach for, and we can talk about them if you want. I only have a high level understanding of what CRDTs do, but it would be something to apply that kind of problem. my takeaway from where to apply, CRDTs versus where I would apply event sourcing is, CR DTs great for in two scenarios. One, if you don't quite know yet where you want to go.

And where in the past you might've reached for, let's say, Firebase to just like have a backend of service. You know, you might want to change it later, but you just, for now, you just want to get going and, you can, particularly if you don't have like a strict schema across your entire application. So you just try to like, not go off the rails too much, but at least the data is like, mostly, like across the applications in a good spot.

But as you roll this out in production, and, we are shipping an iOS app as well, that someone is, running an old version on. Now you don't quite know, oh, this document, this data document that has been synced around here, this might not yet have this field that the newer application version depends on.

So now you have, like, this is where time drifts in a more significant way and in the more traditional application architecture approach you would, this way you don't trust the client in the first place. Then you have like your API endpoint and the APIs, versioned, et cetera, and everything is governed through the, API. But now you also need to tame that problem somehow.

So at this point you're already, going a little bit beyond where I think CRDTs shine right now, which brings me to my next kind of more evergreen scenario for CRDTs, which are like very specific, tasks. And so text editing, particularly rich text editing. Is such a scenario where I think CRDTs are just like a very, very good, approach.

There's also like, you can also use ot, like operational transform, which is, somewhat related under the covers, works a bit differently, but the way how you would use it is pretty similarly. And, related to rich text editing is also when you have like complex list structures where you wanna move things within the list. So if you want to go for the, Figma scenario, let's say you change the order of like multiple rectangles, like where do they sit in that layer order?

how do you convey how you wanna change that? You could always, have like maybe an array of all the IDs that give you this perfect order, but if this kind of happens concurrently, then you need to reconcile that. So that's not great. And this is where CRDTs are also like a very, special purpose tool, which works super well. And so what I've landed on is use event sourcing for everything except where I need those special purpose tools, and this is where them reach for CRDTs or for something else.

That's kind of the conclusion I, took away if you like the event sourcing approach. But, I think ultimately it really comes down to what is the application that you're building and what are, like, what is the domain of what you're building and which sort of trade-offs does this require? So I think in Figma. The real timeness is really important and it is recognized that those different pieces that are floating around, they're like pretty, independent from each other.

So, and if they're independent, then you don't need that total order between that, which makes everything a lot easier in terms of scalability, in terms of correctness, and then you don't need to rebase as much. distributed systems is the ultimate case of it depends. and I think trying to build one like you did, I think is a very good way to like build a better understanding.

And also I think that opens your eyes of like, ah, now I understand why Figma has this shortcoming or Notion if we are trying to change the same line, change the same block as where last writers, applies. Whereas in Google Docs, for example, we could easily change the, same word even. And it would reconcile that in a, in a better way. But, maybe you have some advice for people like yourself when you're just getting started on that journey.

What would you tell people what they should do maybe shouldn't yet do? today 2025? There's more technologies out there now. What would you recommend to someone who's curious?

Depends on the type of learner you are. Sometimes some are very. outcome driven, like I need to see an app running in production for me to really get excited about this Tech. Other people are very first principles driven. Like I want to like screw in every nut and bolt myself to get excited about this thing. I tend to fall into the first camp where I think it is very useful to just look at the docs for something like Replicache and see how would you implement this kind of protocol step by step.

Like how would you set up the event sourcing? How would you put the SQLlite store in the browser in the first place? Like what capabilities are there? And then try to think through those edge cases. As you run into them trying to build something, I use Linear as my sort of learning example, but you could use pretty much anything you want. so that's definitely one approach.

Now there's just so many resources for how these things work under the hood that you can easily learn about the intricacies yourself. Another, resource is the talks given by who's the engineer at Linear. I think you've had him on the show. Tuomas? Yes. Yeah. He gave a few really helpful talks about how the Linear sync engine works on a high level, and that one's more opinionated. It reaches for technologies like MobX, which is a react specific state store, and also MongoDB for documents.

but you still get a high level of how they think about the problem, which is really nice. the other option, if you're really results driven, you wanna see a local-first step running. You can reach for all sorts of frameworks and libraries at this point. Zero is the one that I've played with most recently, and it is Alpha Software you'll run into, it holds your hands, plugging in every battery and setting up everything. But error codes could be very confusing.

but luckily their Discord is very welcoming and will answer any question that you have since their only goal is for everyone to get excited about the tech and use it in production. So I think Zero is a really great starting point as just, I wanna build an app. I'm gonna reach for a library. It will give you a query builder. So instead of writing raw SQL, it'll help you write SQL queries with some JavaScript functions.

And it also works you through very common problems that you do hit at some point. And the big one is data migrations and, well, not data migrations, schema migrations, because when you have a data store on the client and you have a source of truth on the server everyone has to agree on how that data is shaped if you're using a SQL model and not something that's Firebasey as you were mentioning.

So in those cases, you have to know like the four or five step process of update the server schema to add the new field, then update the client to add that new field.

And then if you're trying to delete an old field for some reason, you would need to execute those on client, then server in the correct order, and then manage a database version so that if a client tries to connect with really, really old application code, the server can say, sorry, I only accept people who are on version five of this SQL schema. You're on version three, so I'm just gonna hard refresh your webpage and get you up to the latest version.

all of these challenges are really interesting to think about and Zero helps you think through them out of the box and presents docs on all of these problems before you run into them. but I happen to be the type that wants to run into as many brick walls as possible without someone telling me what to worry about. I just wanna worry about it.

so I think the Simple Sync Engine resource is great just because it doesn't do very much and there's a lot left up to the reader to go off and try to run into those challenges. I'm sure splunking through like the LiveStore implementation, I would find 50 ways that I could improve what I'm doing to get to that next step of like resilience, schema, migrations. I literally didn't even touch schema migrations.

there's so much that you need to think about that just crawling through open source libraries is really, really helpful with, so that's my preferred learning approach. I just like going that way. I completely agree. And I also like, it's, it's sort of a bit of convincing yourself, is this entire thing worth it? And what I always appreciate if someone knows a little thing about me and then tells me, you know what? I don't think this is for you.

I wouldn't hold anything back for someone who wants to look into this. to say like, this might not be what you're looking for. If someone is very happy with like building web apps with Vite Astro NextJS, et cetera, and they're productive, they're building this, e-commerce platform, or they're building a more static website, I don't think there's anything really where local-first would change their work situation.

But if they're frustrated with like actual apps that they use day to day, when you're frustrated like yourself, when you're frustrated with Notion being too slow, et cetera, and you're building those more productivity daily driver apps yourself. For me that was like a music app. I got frustrated with Spotify and other music apps.

I think this is the, right scenario where like local-first has something to offer, but, and I think it has also the potential to become a lot simpler and easier over has already become a lot simpler and easier or the past couple of years, and it's gonna be even more so in the future. And there will be use cases where it's actually simpler. To use local-first to build something, then using Next for something. but that won't apply to all scenarios. And so it is not a silver bullet.

the closest thing you'll get to a silver bullet is the right architecture for the right application scenario, but by default there is no silver bullet. Neither is local-first. And I think someone should evaluate, Hey, is this even for me? that's, I think should be the starting point. Yeah, and a meta comment just because now I'm in the agent coding space, Warp is getting more capable by the day of actually editing files and scaffolding new applications for you, from the terminal.

I've found it's less valuable to know the syntax of how all of these libraries work and a lot more valuable to just know high level, what are they doing, what's the architecture and how would I debug it? because these agents are very good at spitting out the syntax, if you draw a very clear picture.

So if you go off and read designing data intensive applications, and you start diagramming to yourself how all of these systems are distributed, you could bring that diagram to Warp or just the cloud website if you want, and say, I wanna build this kind of app. Here's how the architecture's gonna work. This is gonna talk to this, and I know about this library.

I know LiveStore uses event sourcing, so I would like you to implement that and use React, but follow the handrails because I understand the architecture. It'll give you a way better application than if you were to just say, give me a local-first app with React. It would probably maybe not struggle in the beginning, but definitely struggle as you try to figure out what it is doing or debug whatever sort of system level issues you're having. I fully agree.

And given that the both of us are not just application developers but also tool creators, we spend a lot of time thinking about like, how do I leverage the degree of freedom that I have here in the API, the way how I design the API that is intuitive for someone that they like, ideally that this becomes like a pit of success where they intuitively use it the right way, but also if they use it the wrong way, how do they notice?

Do they notice like as early on as possible through type safety or only if they're already in production and they felt like, wait, no, this, like, this was a path that I've wrongly taken six months ago. so you want to design all of this in a way that you like learn as early as possible whether you're in the right track or not. And I think you can't get better than simplicity than going for simplicity. And this is why I love the path that you've taken with the Simple Sync Engine.

Through the push pull model because that's already, that is deeply familiar for developers and that is how we're using Git and that has really been proven. And there you can't really get much simpler than that. And I think simple is great for everyone. and once we have a simple foundation, we have a reliable foundation. We can build fast and nice things on top of it.

But particularly mentioning AI systems, I make a lot of design trade offs now differently, where I care less about how much effort it will be to write or to discover that thing. Since we have now LLMs, do TXT, et cetera, I care a lot more about like, how does, how will you even spot where like this doesn't seem right. The robot has given me something weird and just doesn't match my, like, primitive understanding of how this entire thing fits together.

And that should also help the robot to like not go in the wrong direction in, in the first place. So, yeah, I love that. Like, and going for a simple design decision, the simple like overall system architecture that's gonna help you as a future programmer, observing little robots, building things, a lot more to know what's going on. So maybe we use that as a last segue to hear a little bit more about what you're doing now at Warp in regards to agents, et cetera.

I've been using Warp for, a little bit. I still use it, for my standalone, terminal, but most of my terminal work is also happening within Cursor, which is integrated in like the agent thing and Cursor, et cetera. Maybe can, yeah, help me a little bit of like how I bring those two together and use them for what they're best at.

Yeah, and it's an interesting world of sort of agentic coding solutions. It feels like there's a new approach to it every other day. before joining Warp, I was also using Kline a lot, which is a VS Code extension that's fully open source that will, from my experience, give you a more quality agent output. Since it has these two phases, you can flip on a plan switch and it'll use a reasoning model to take whatever you tell it and turn it into like a step-by-step plan.

And you could walk through like, no, that architecture doesn't make sense, or Let me upload this six cal draw. I actually want to work like this. And you can go back and forth on like a design doc and then you can flip it to act mode and that engages Claude to go build that for you. And it's very hit or miss actually doing like the code edits. We're all struggling with that. but it was like this really nice mental model of, oh yeah, we're gonna plan it out together.

We're gonna design jam, how this thing's gonna work and then we're gonna go build it and I can just let it run and see how it ends up working. and Warp is doing something similar but not within the confines of VS code. And also with the addition of a voice button that I've been using a lot more recently because you can talk faster than you can type is generally what I found. So I can just speak into my terminal, here's how I want the app to work, this, that, and the other.

And then it will, depending on how complicated the question is, it will reach for the planning step. Otherwise it'll just give you an answer right away. So if it, see that's kind of complicated, let me plan it out. It'll give you that same kind of like document, here's how it's going to work. And then you can say, okay, do it. And then it will go off and tell Claude to make file edits and do other things on your machine, which is much further than Warp has gone in the past.

and I've really enjoyed using this to build Swift apps recently. Since I was just fascinated with like, how could I build a really slick desktop client like ChatGPT there's this desktop shortcut to like pull up a little chat bar. Like I want something that integrated. I don't wanna be confined to Google Chrome anymore. I wanna break out of it. but if you open up XCode, you're just met with this like decade old auto complete that doesn't have anything that you want in order to get stuff done.

but Warp is just a terminal. So I'm like, okay, I'll just open the Swift project in Warp and say implement this feature. And it doesn't it, it can literally just enter any directory that you have and just start doing things. I've also used it to like migrate my open source projects from a mono repo to a set of micro repos on my system and says, oh yeah, I'll just make a new directory. I'll move all those files over and I'll make the necessary file edits with Cloud. Very like hit or miss quality.

We're dialing it in. But this idea of you're not constrained to the IDE anymore, you can kind of just pull up your terminal and ask it to modify anything from the simplest request of, help me get revert, whatever the heck I just did. And it'll help you get to something more complicated, like, why isn't my Postgres server running? And then it'll check your Homebrew installation. And then you can take it one step further to, I actually want to fix this error I see in my dev server right now.

Because you're running the dev server in your terminal, it can say, all right, pause the server, debug, debug, debug, restart the server. And then if something fails again, I'll go back to debugging. So it's like watching your terminal session and figuring out how to help you do something.

It feels like it's this natural next step of let's go from you're in an editor typing code quickly to like, this is a general purpose tool on your machine to edit all the software that you're writing in any setting. so I'm just very excited about that kind of future. And we've been moving very, very quickly towards it. Just in the past month, it's gone from barely usable to, I'm actually using this a lot for projects in a language that I don't even know how to speak. Swift.

it's been kind of crazy how far you can get from zero to one without a lot of field knowledge. And intuitively this makes a lot of sense since like Eternal is kind of like the OG chat up in a way where like all the way back to like IRC, et cetera, but also now with, using ChatGPT a lot or, or other LLM chat products, like yes, you're, chatting with the thing, like you write a command, the command happens to be like plain English or another language, and you get something back.

But the, kinda like back and forth. And the interplay is very similar and it makes so much more, so much sense that you now bring that into the terminal as well, where you get the best of both worlds. You can like ride out things in a fuzzy way. The terminal helps you to like put that into proper computer speak.

but then you also get the efficiency and the correctness from what you can do in a terminal and with all of like just a top-notch craft that you get within Warp as a terminal with like blocks, et cetera. So yeah, highly recommend everyone to, give it a try. Yeah. And it's free to reach for all those things.

And anyone who is just bothered by AI in their terminal, you can turn it off and just use Warp as a really good terminal, which is how I started using it way back, probably like, 2021, I think is when it said I created my account. I just used it because I wanted something that looked nice and now it's going a lot deeper. And yeah, the, chat app analogy is perfect.

There's literally a toggle between typing out commands and asking it a question, and it'll even flip back and forth based on like natural language, which is fancy. I mean, I'll just hit the keyboard shortcut, but why not make it a little flashier?

That is awesome. Well, I've already seen, a bunch of your recent videos, about content related to that. I'm looking forward to many more of those. Is there anything else that you wanna share related to your local-first, explorations or otherwise? Yeah. so my profile is bholmesdev everywhere. So any learning resources I've put out like videos on local-first and conference talks, bholmesdev on YouTube and Twitter and Bluesky.

And also on GitHub, so these Simple Sync Engine project I mentioned, that's on my personal GitHub. Also under bholmesdev. You should see it as one of the star repos if you look up the profile. That's it. Perfect. I've also put it in the show notes, so everything, you'll find it there as well. But, I'm really, really excited you have put in the effort to create this project, because I think there's no better way to learn something than trying to do it.

And you've done that and you've allowed other people to follow your footsteps, and I think that's a fantastic learning resource. So thank you so much for doing that and for, yeah, helping others learn and, sharing what you've learned and for coming on the show today. Thank you. Yeah, thanks so much. This was like a really far reaching conversation. I hope it turns out good. Thank you for listening to the localfirst.fm podcast.

If you've enjoyed this episode and haven't done so already, please subscribe and leave a review. Please also share this episode with your friends and colleagues. Spreading the word about the podcast is a great way to support it and to help me keep it going. A special thanks again to Jazz for supporting this podcast. I'll see you next time.