#21 – Seph Gentle: Google Wave, eg-walker, creativity, AI

And you know, like I don't think economically we need local-first software, you know, like we can trudge along with the feudalism of Google cloud services for, you know, probably for a very long time. but I don't want that. I want beautiful software programs that work in harmony with who I am as a person and with in harmony with what kind of communities I want to be able to foster. Welcome to the local-first FM podcast.

I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and love the craft of software engineering. For the past few years, I've been on a journey to build a modern, high quality music app using web technologies. And in doing so, I've been falling down the rabbit hole of local-first software. This podcast is your invitation to join me on that journey.

In this episode, I'm speaking to Seph Gentle, a prolific software researcher who's behind projects such as the new AgWalker paper and JRJS, one of the oldest local-first open source projects. Before, Seph also co created Google Wave over 10 years ago, which we explore in depth in this episode. Before getting started, also a big thank you to Electric SQL for supporting this podcast. And now my interview with Seph. Hey, Seph. So nice to have you on the podcast. I'm a huge fan of your work.

I've learned so much from the paper you've recently written about Eg-walker. And the more I dug into your work, the more I realized, oh, wow, you've also been involved in this. So for example, you've been working, On Google Wave, like way back, but also on projects such as ShareDB ShareJS, et cetera. So welcome to the show. How are you doing? Thanks. I'm great. It's so wonderful to finally meet you as well.

I feel like you're one of these faces that pokes up everywhere and, yeah, it's lovely to meet in person and get to actually chat about all of this stuff. Awesome. Yeah. So I've already mentioned a few projects that you worked on, but maybe you want to give a background for the audience.

No, absolutely. So, I guess the story starts way back in, I think it was 2010, 2011. And I was invited onto the super secret at the time project of Google Wave, which was run out of Google. And it was, not many people know this, and I'm sure someone will get in trouble, get mad at me for saying so, but it was a secret even inside Google. Sorry. Google was thinking we've got lots of people who've worked in startups, but we're not a startup anymore. How do we get some of that innovation?

And so Google Wave started as this skunkworks project. They didn't even tell Google us about it, which was a wild thing at the time. and before long, you know, in the original vision to anyone who hasn't tried it out. Was if we reinvented email today, what would we want it to look like? What could it do?

And so, Wave is this almost cross between, email and Google Docs and something else where you can have, A conversation, which they called a Wave, was a series of messages, but messages could go down. So we could have a series of conversation items. You could also embed messages and embed entire threads inside a comment. So I could write something and you could start a comment thread inside that and any comment, anything that anyone wrote.

Anyone could come along and just collaboratively edit it and change it later. So, we sort of built this thing not really knowing how it was going to be used, which is a beautiful luxury. and the answer is that lots of people used it for, you know, say like, for playing D& D games. And they would put all of the information about the campaign itself at the top of a Wave. And then the actual session would be a series of messages and then they'd start a new Wave for the next session.

Or, say minutes. So you'd have the agenda for a meeting and then during the meeting anyone could, you know, put in little comment threads about what was actually discussed. And then your minutes for the meeting turn into the agenda. Sorry, your agenda turns into the minutes afterwards. Because we've now got this collaborative document. So, yeah, I fully remember when Google Wave just first came out. I was still in high school back then. I was 16 and, I was like using a few Google products.

I think mostly Gmail and maybe a bit of Google Docs to just keep track of my, own thoughts. But when Google Wave came out, I was just so curious and there, there wasn't really like a real use case I had for it back then, but I got immediately hooked by the novelty of it. And it was, I think it was really the first time where it really clicked for me, like, Oh, this is computers are not just for me, like looking in my single screen here and then sort of asynchronous.

Yes, I can send an email, et cetera, but we can in real time collaborate on the same stuff and that sort of same magic that happens if we're like sitting at the same table and do something together. Now that can sort of be replicated in there. And not only did you build something in a real time collaborative way, you build it so well. I think Google Wave really set the bar for a long, long, long time. What I really like.

Immersive, high quality web experience could look like and that was, yeah, you mentioned like in 2010. And so I think that had had a long lasting impact on my own, longing and motivation to go all in on the web. Well, that's, that's very kind of you to say. I feel like, I'm like, Oh, I can't take all the credit for Wave. I mean, at peak, there was 60 people working on it and I came onto the project reasonably late.

So, you know, it wasn't my grand vision, and something though for this audience, you might be fascinated by, we built the entire thing to be, it wasn't local-first, that term certainly didn't exist at the time, but it was fully federated. So it worked like email did with the idea that people could run their own Wave servers.

And, you know, and so you could collaborate within your organization on your Waves and then the document wouldn't leave your organization unless you explicitly added someone from outside of your organization. otherwise it would stay in the four walls of your company and your own servers, which was really, really cool and groundbreaking. I mean, like hearing that in retrospect doesn't sound very, like, that's not what I associate with Google these days, like Google typically builds like a very.

Google centric service, take it or leave it. It works really well integrated in the rest of Google services, but going in a more federated, like open protocol direction is not immediate. Maybe this was just a different time back then. Maybe. I think that Wave was also, it was an unusual, it was a weird project inside Google itself. It was run out of Australia in the Sydney office. so you know, it was, it was like this little, little project at first about what could we build.

And. Google's got a funny relationship with openness. where you've got, I mean, I agree with you about most of Google's product offerings, but then Google are one of the hugest proponents of the web and the web of course is still fully open. And you know, as much as people like to complain whenever. Google Chrome integrates with Google services. Google at their heart really still does believe in the open web.

And, you know, so the question is, could we use that knowledge and that understanding to be able to build more products that work like email does? And of course, again, Google, you know, with Gmail as a, you know, it's all built on open protocols. so yeah. but no, it was really ambitious and it was a really beautiful. product, product use, as you say, it introduced lots of people to actually things being real time and collaborative by default, which I really think is the right default.

I told my mom at one point I spent, you know, a couple of years of my life trying to make, we, we had a joke that we wanted to make two windows look exactly the same. and you know, so if you typed in one window, it would just appear on the other screen. And this is a really very difficult problem. and she kind of gave me this quizzical look and said, Doesn't it do that already anyway? And, you know, she just never tried and never noticed.

But it's really, really is, uh, Paul Graham has this question, he says, what feels like it would be obvious, it would be crazy for us not to have in a hundred years. And I think about that a lot, of, what technology would it be crazy for us not to have, in such that, if we could figure out what that technology is, could we build that now, and then just have it now? you know, I don't want to have to wait for someone else to invent all of this stuff.

Like, with Wave, it felt like it was right at the birth of collaborative editing. Google Docs was actually quite new at the time and, believe it or not, Google Docs had a bunch of correctness issues where, you know, if you type certain text and then bolded and then you unplugged your network cable at the right time and everything else, you'd end up Converging, you would see something and if you hit refresh, you'd see a different document on your screen.

these problems were still new and we were still figuring them out. But I feel really sad that there's not more software that takes advantage of this cool technology we've been working on. you know, like even things like Figma, and I love Figma for using CRDTs, but you can't run your own Figma. When I open Figma, it's, a centralized service. It just happens to be using CRDTs to do collaborative editing. but I feel like it's technology that we can use for so much stuff.

and that's, yeah, that was kind of part of my, like, jewel, joy and sadness in working on Wave. Right. Thought, ah. Yeah, I mean, let's not, let's not go to the sad part yet. Ultimately, Google Wave shut down. But I want to hear a bit more about the various aspects you've already mentioned. And I love the framing of Paul Graham in that regard. Like, what would be just. obvious, to build, and maybe we can also follow up which ideas feel obvious to you now.

And maybe that you want to work on, you don't have the time. Let's, let's shelve that for the time being. So not only was Google Wave just like an amazing product experience, like how fluid it felt, et cetera. And you have to remember this was like it's 2010, where web standards were not at all where they were today. And, I'm not sure how strict Google was back then about, like, browser compatibility. but I remember the experience just feeling phenomenal back then.

So what were some of the key challenges making this happen? I mean, that's very kind of you to say. That wasn't the universal experience and it depended very much on what browser you had. We, we had an internal Wave that was like scrolled for a long way of open browser bugs that we'd filed against all of the different web browsers in all sorts of different little edge cases. I mean, for context, we made Google Wave work on, I think it worked on IE8.

it certainly worked on IE9, which is the version of Internet Explorer at the time. but there was so much that was still, I mean, even still is not standardized, honestly. We had a rich text editing element, and we needed that to work with international characters and internationalization, and be able to, synchronize with any device on any platform with any browser. So, that was tremendously hard.

There was a team of about six or seven People just working on the input side and we had this insane test suite that would test and, you know, different keyboards in the office of, you know, for Korean characters and Chinese characters and, you know, European, various different forms, you know, different laptops running different versions of windows to make sure that we could get everything to synchronize correctly, and we used all of that to try and just make the browser

work correctly and work consistently, but from a technology standpoint, we forget now, but this is back in the day before we realized that JavaScript was actually a useful language and kind of nice to write in sometimes so Google Wave was all written in Java. And then compiled from Java to JavaScript via Google Web Toolkit. So, we had, I think that when Wave got shut down eventually, I checked the source tree. And internally we had 1. 1 million lines of Java code.

And obviously, that wasn't all the client side application. but then that compiled. To, it was like a 500 kilobyte JavaScript bundle, which these days doesn't seem so large, but at the time it brought down, you know, mighty web browsers, trying to run all of this stuff that we were, you know, trying to do written in, Java to be able to solve all of this like front end problem. and then on the back end. Operational transform was a new idea.

CRDTs were, like, I think that they were a gleam in some researchers eyes at the time that we were working, but they were incredibly inefficient. So we tried to build something that was federated and would let multiple servers, multiple users from different organizations all collaboratively edit a document, all built on top of a very old version of operational transform, which some very smart people on the team managed to get to work. But, it was incredibly new.

I learned a huge amount from working on that, which has kept me busy for the last decade and change honestly of trying to figure out how you can even solve a problem like that. there's another feature that I want to plug because I really hope that people who are listening to this podcast know about it and realize that you could build this into your own applications. We, we also had something called, Google Wave, gadgets.

We had gadgets and extensions and it doesn't matter the distinction, but you can have a Wave and inside of it. the internal data structure was an XML tree with annotations. So you could do rich text, but also have arbitrary XML nodes. you could make different little applications, and the application would be embedded inside the page, and the application would be given some small subsection of the XML tree to be able to collaboratively edit.

So as a result, we could build things and anyone in the community could build things that you could drop into a Wave and when you dropped it into a Wave, now you've got a small, you know, drop in application that you could collaboratively edit inside of. So, for example, we built a Google Maps extension that let you plan your holiday with a friend and, you know, with as many friends as you want, and you could all put down marker pins and draw little lines on the map.

we had a, the yes, no, maybe gadget was the most famous where someone would post in the office and say, Hey, is anyone up for seeing the new, you know, the new movie, whatever it is.

on Tuesday at 7pm, and in the Wave would be a little, like, three columns, and you could click yes, no, or maybe, and whichever column you clicked on, your face would appear in that, alongside everyone else that clicked on yes, no, or maybe, or you'd have little polls, and while you had the page open, you could see everyone's faces appear, all in real time, because that component, which isn't part of Wave itself, it was an extension, gets to be able to sit on top of the

collaboratively editing, infrastructure and, and work, which was so cool. That is so amazing. And I'm like both super excited just hearing about that again, like amplified by hell. This was like 2010, like this was like so. So much earlier, there was no WASM, there was no TypeScript. There was like, the web was like so, so, so early. And you didn't just innovate on those technologies, but you innovated so much on those.

Like you really try to, push to whole new frontiers where the product experience can be like that, that real time collaboration, like tied to a phase, et cetera. and I don't really think that we have that in a, as modular as a way, as you described it right now, like typically you have those sort of super modular ways, in a more standalone open, like niche open source thing, but not in a mainstream, product that, that everyone is using.

And, I think the closest to actually who's going after the same vision that comes to mind are the folks at Ink and Switch. Who are building Patchwork and, I get to see the in progress updates there once in a while. And it's so, so, so cool. And like, now I see the strong, strong parallels there. And I think they have a, similar vision there. They're just starting from a different foundation. They're building it on top of Automerge.

we're going to have, Peter von Hardenberg back on the show to talk about Patchwork as well as at the upcoming local-first conference. but there's many, many similarities there. so yeah, this is, this is super inspiring. Yeah. I couldn't agree more. I feel like that's sort of my, my dream software I want to have, in particular for creative work, I think like if I'm going to be. I don't know. I'm using IRC or some, you know, discord or something.

Fine. I understand that it makes the most sense to have some semi centralized system, but for creative work, for if I'm writing a, if I'm writing a book, if I'm collaboratively editing a video with some other people, I want to be able to have something that both has some data living inside some open platform that any application and anyone who's interested can grab a copy of the data and. listen to changes and make changes to the data model, you know, just as a raw JSON model or something.

And then also like that's sort of the Holy grail of then being able to have custom extensions and custom bits of software that can be plugged in. it's not a static platform. It's something that people can build cool UI on top of and try out different things. I've got this idea in my mind that I can't.

Escape, which is, it was someone in that broader community once described Google Wave as glorious data bus in the sky of, you know, this amazing data bus that just moves all of our data around between computers and you don't have to think about it and it's complicated and it doesn't matter. You just know that you've got some data and anytime you want it, it can be on your device and it can be up to date and it can be, you know, you can make.

Collaborative changes with anyone else and if you had something like a JSON data model or something like, the data model in Yjs or Automerge, then you could have a data model than any, you can have lots of different pieces of software interoperably interact with that same bit of data.

So, you know, if I've got, A whole bunch of server logs in there, and it's just in the, in the, in the cloud, in the, you know, not the cloud, centralized cloud, but in this data model, and I can subscribe to that, and I can have an application that listens to that, those logs, produces a report, and produces and updates that report in real time, and then puts that back in the set of data, and that's kind of a boring example, the one I often think about is

compilers, so I've got a bunch of source code files, and I could have all of that in this data model, so I've got a bunch of files, And I want to just be able to open any program on my computer to be able to make changes and not just on my computer, but on any of my devices, grab my iPad, grab my, you know, whatever, grab a Raspberry Pi and make changes. And then I want my compiler just to be able to listen to the stream of all of the changes and be live recompiling.

Based on the differences in the source code and putting back into another data model, a set of annotations describing where all the errors are and how the syntax highlighting should work and anything else the compiler wants. And then we can have different IDEs that don't have to understand the programming language at all because they can just listen to the compiler, you know, and then for example, I could have like a testing suite as well. And then the compiler is.

Keeping up to date and executable, a binary file that's just made of a bunch of small blocks, which are all like functions that are compiled in, linked in. And then I could have a testing suite that every 30 seconds, so long as there's been changes, it reruns the testing suite and keeps up to date a set of like, which tests are passing and failing. You know, I, like Unix model is, you know, like lots of small programs that can interoperate and talk to each other.

but then we, we threw a lot away and I, you know, to this day, I love using IntelliJ, right? Like, RustRover, that's it, yeah. Yeah, but these programs are these huge monolithic programs because the only way to get all of these small components of an IDE to talk to each other is by putting them inside one piece of software. the Unix model itself sort of fell apart because we don't have the right primitives to be able to plug programs into each other, you know?

I think this is where I also see stark parallels again to what the folks at Ink & Switch are going after where they're building this, big data bus with Automerge and then build sort of like systems around it. And I think Patchwork is both a huge exploration what that could be, but also drives even more, requirements for the ecosystem to form around.

It's sort of interesting how, the data bus is one aspect, but then the other, resurfacing idea is basically like a, stateful built system where you basically, like, you listen to something and you do a computation and that result, other things can listen to. So it becomes like this bit, this big tree that's ideally incrementally recomputes. And this is how most reliable systems that we use on a daily basis work. But unfortunately.

They're, concealed in a box and, typically don't interoperate beyond a certain system. And I think this is another common theme that I'd like to see explored more. How can, like within Google. So most things can talk to each other, but now everything is in that big Google box, but how can we go even beyond that? And this is where maybe coming back to, the initial ideas behind Google Wave, where you wanted to build your own protocol. Can you share more about that protocol and the goals for it?

Yeah, in my mind, the way that I would solve this problem if I were to start today is to pick some Data model. So for example, Automerge and then create a protocol where multiple peers, you know, that takes care of, peer discovery of authentication, all of that kind of stuff. so that way you can have some little package of software, some binary that you link to or a JavaScript package, you know, NPM library or something that will give you access to this, to this data collection.

And then you can make a program that can say, Oh, I would like to subscribe to this kind of data, please. And you can see it, for Wave. Like we started trying to solve this problem and at the time we used operational transform because there were no other options. Like this is, it's sort of 2011. I think the first CRDT paper might've come out around 2005, but performance wise, it, it performed terribly.

Like if you just typed a regular document of a few pages long, you would use almost a gigabyte of RAM in memory and a similar amount of disk space just to store the entire object tree that it created. It performed very badly. And that being client side or server side? both, because they'll, they'll be running all the same code. So, so that was the problem with CRDTs. So we used an operational transform based approach, which is based on the Jupyter, OT algorithm.

And at its heart, um, Jupyter, there's a few different ways to say this for people who aren't familiar with the technology. Jupyter uses it's only correct if there are exactly like either one or two participants, so it can't work at all if there's more than more than two participants. So three, there's a whole bunch of correctness problems, but you can solve that by having the server be one of the participants essentially.

So one way to think about it is that every user is collaboratively editing with us with their server and the server then. can collaboratively edit with every user individually. So you can have a real time collaborative editing system and OT based systems. Most of them work like this. So long as you have a centralized server, which gets to make all of the decisions. and really what that server is needed for is to order things and create a canonical ordering of all of the events.

So ultimately OT systems like that, and there's a lot of different OT systems, end up with a, You mentioned event sourcing earlier, you end up with an event sourcing style list of operations that can all be applied in order. And if you started an empty document, apply all the operations, you'll end up with the current document state. and really you just have one, you know, monotonically increasing version number, which is the version after, you know, N operations have been applied.

So that works great. It's pretty fast. It's what we use for Google Wave. But Google Wave ran into the problem where, We wanted to be able to have federations. So we wanted to be able to have an arbitrary set of servers work and for that, the network protocol, was horribly complicated and it didn't work reliably. I saw it work reliably. It felt like, you know, depended on the phases of the moon and everything else. where.

The servers needed to, between all of the servers, arrange themselves in a tree so that there was one server that was the most in charge of all of the servers. And unfortunately, if that server disappeared, then you'd have a net split. and all of this was just a, it was a, problem with the technology of the day. It was a problem with operational transform, ultimately. That we couldn't build what we wanted to build with OT. And that sort of got me thinking, like, how do we solve this problem?

so this is condensing about, 12, 13 years of my, my time, after just very briefly after Google Wave shut down in 2011, I was incredibly sad about it and I thought, oh, there's no way this technology is so cool. So JavaScript was, As a language that people like to write was very new. It was Node. js 0. 4. NPM hadn't been invented yet. if you can imagine it.

And I thought, Oh, I'm going to use JavaScript and just write a really, really simple small library that does operational transform in JavaScript. And I called it share. js and weirdly lots of people know about the project even though I've taken the website down and, you know, it's all, it's not used for much now. I had a call with someone, who works at WordPress the other day, and apparently Share.

js helped them land some giant deal with a company in Europe, which I'm not going to name, and was used as the back end for this, this whole news site for a long time, which I had no idea about because it just implemented Operational Transform well, which is all you want a lot of the time. so I made that and then, but I was still thinking about the question of how to make CRDTs fast.

There was a whole series of new papers that happened again and again and again, and I felt like if, if we had a good version of Operational Transform, we could make an open source version of Wave or something like it, but it still didn't quite scratch the itch, because I still didn't quite know how we could get all of these servers to collaboratively, you know, collaborate. Eventually, a few years ago, I ended up getting frustrated, and I thought I've been avoiding CRDTs for the longest time.

And I read some CRDT literature and I was like, no, no, what's the latest on CRDTs? How are they supposed to work? And I read some papers and I thought, This is ridiculous. You could just make, you could make an implementation of this that works fast. The only reason why CRDTs were slow at the time is because people hadn't spent enough time optimizing the algorithms themselves. And Kevin Yarns, the author of Yjs, he'd done a lot of work.

Martin Kleppmann, the author of Automerge, had done a lot of work. But at the time, Yjs was, performed okay. Automerge was incredibly slow and to work around that Automerge at the time separated all the code out into this front end and back end. So the back end did all the slow work and then it had message passing to pass message into the back end so that this whole slow part of it wouldn't interact, interrupt the user experience.

but I grabbed a whole bunch of ideas from both approaches and there's a YouTube video of me picking Kevin Yarns brain on YouTube if you want to, if anyone's ever interested watching for three hours, let's talk about CRDTs.

But it turns out that, you know, with the right application of some complex data structures, using B trees in the right ways, and, um, using some of the techniques that Martin Kleppmann, well, he was the first person I saw do it, where, in the file system, the way that Automerge saves stuff, saves data to disk, Most applications think about it, they call it an array of structs. So if I've got a bunch of rows in a table, every row is an object or a struct, and then I've got an array of them.

You know, I might have a hundred of them and each one represents a row. Most databases spin that on its head. So the database will instead store one file per column. So It's a struct of arrays, if you want to think about it that way. what Martin Kleppmann did is he said, well, each of these columns in something like Automerge, if it's describing all of the keystrokes that happened, we can store that as a string.

And if it's describing the positions inside a document that everything is stored, well, those numbers turn out to be sequential almost all of the time. You know, about, people usually type in runs of about 20 characters before they move their cursor. So if you just store run length encoded, that data ends up collapsing and becoming much smaller.

So I was saying before, CRDTs, you type a moderate document, it might take 800 megabytes of memory or a gigabyte of memory and disk space to store it in raw JSON. But if you run length encode everything, in these columns, you can save, store the same data in a megabyte or less. So. Martin Kleppman did that and did that in the file storage engine of, Automerge. Yjs copied that. Then I took that idea and did that. Oh, and Yjs also did some of this. I don't want to take credit for all of it.

It's all standing on the shoulders of giants here. but I made a library called Diamond Types where I was like, Oh, how fast could we make CRDTs go and use that internally? So all of the internal data is all run length encoded internally. And that's a. that's a whole tangent that I need to write up at some point because there's a whole bunch of clever engineering tricks to get, make that work. but as a result, you can make CRDTs incredibly fast, way faster than anyone could possibly imagine.

You know, we're talking in the order of millions of keystrokes a second in a text editor, which is, you know, more than anyone would want to type. and we can store it all very efficiently to the point that, I added, a, LZ4 compression. So. In Diamond Types, I store all of the text content of what you've typed and then a bunch of metadata that stores all this extra CRDT stuff, essentially.

turns out that if you LZ4, like, just use a very standard, very, very fast, low grade compression system on the text, then the amount of data that you save just by compressing the text lets you put all of that metadata in, and often you end up with documents that are smaller than the original text, even though you've got A keystroke by keystroke entire history stored in the same file, is really cool. so a few more decades or a few, you know, well, I guess one decade of work after, Google Wave.

And we collectively as an industry have figured out how to solve the problem of CRDTs. I mean, I'm, I've had several people come and talk to me over, over the years and say, Seph, you could remake Google Wave. And I'm like, I know, and we could remake it better and greater. Maybe Patchwork is trying to aim towards some of that. but we're in a much better place, from all of that work. So let's come back to share. js in a moment.

I want to learn more about those, various components, but maybe jumping ahead for a bit, what would it take today to rebuild Google Wave with the technology stack of your dreams?

honestly, the biggest thing would be some funding. So, Google Wave used Operational Transform, and now we've got great CRDTs available. So, I would use my own implementation of CRDTs, or use Automerge or Yjs. I think any of those libraries work great. And there's a bunch more people working on that would probably perform fine. So, we'd use the CRDT. Google Wave used, it actually People forget this.

We created a custom jabber, like XMPP extension, and the servers interoperated through that, which is complicated, and I probably wouldn't make that same choice. We'd need an identity system, if I were to do it today, and yes, I have been thinking about this.

I'd consider piggybacking on the BlueSky identity system, and, since that's actually a federated system even though they haven't built all of the tooling to make it easy, but, I think it's actually an excellent system and lets people, one of the freedoms that that system guarantees is that if you've got an identity on one computer or one domain name, so identities are associated with a domain name, you can migrate that identity to another domain name.

And anyone that knew your old identity will have their contact address books automatically update and all of your data moves across, which is great. So we need an identity system, maybe that, maybe something else. and then we need a web front end or a front end of some sort. And unfortunately, one of the, It's so embarrassing on behalf of our industry. one of the last bastions of unsolved problems is making a good rich text editor inside a web browser.

in the last Like, however many years it's been, it's 2015, so I guess it's been 14 years since Google Wave shut down in 2011. in all of that time, there's been multiple aborted attempts to try and make a web standard for being able to edit a rich text document, and none of them have succeeded. All of them have fallen apart, and, rich text editing is a massive mess. but there are a lot of JavaScript libraries that pave over a lot of the worst parts of that problem.

maybe lingering, on the last aspect for a bit, I have deliberately tried to avoid rigid text editing and all facets of it, just for my own sanity sake, me, working primarily on. Overtone and LiveStore. Overtone is a music app where I can gloss over rich text editing for the moment. This is where it's rather important that your music library is in sync, playbacks, playback history, playback state is in sync, etc. That's where I'm leaning on event sourcing.

But, Looking at rich text editing and real time collaboration on text, I'm aware of projects such as ProseMirror or CodeMirror as sort of the code twin project to it. I'm aware of the TipTap rich text editing framework. I'm also aware that Facebook has, I think, a pretty good one, which I'm blanking on the name right now. But yeah, where are those projects falling short?

I mean, ideally I'm with you, this would be a web primitive that is like standardized and implemented by all browsers, but what's so hard getting there and where do the existing solutions fall short?

honestly, they're probably mostly fine. They fall short in, for example, mobile editing, so I want to be able to have a native experience on my phone. if I open up Apple Notes, I can select text and I can bold it and highlight it and do lots of different things. a lot of the existing editors will, it's just really hard to hook into the native. Rich text editing through a web browser through mobile Safari. so they, they need that to work. And then they need it to work from Chrome with Android.

And then they need it to work with lots of different things. mean I still run into problems sometimes on Reddit. If you use the new Reddit interface, they've got a rich text editor. in the reply box and sometimes I've typed and then hit backspace. Sorry, hit undo and weird things reappear or disappear or the wrong text, you know, it gets jabbled and confused. so there's a bunch of correctness problems like that. That is, it's incredibly boring work.

but it's really important for this kind of thing. I have a longstanding rule. I don't know if this has a name, so maybe it can be Seph's rule if no one else has claimed it, but that essentially technology gets worse the further away from, mostly Bay Area software engineer problems that you are. So if you have the kind of problem that someone, a Bay Area technologist, thinks about a lot, there's going to be good libraries, good software, good tooling around solving that problem.

But. You, for example, speak a language that where characters are typed right to left, instead you're gonna run into problems. in Australia we have roundabouts, we, you know, which the Americans call traffic circles and Google Maps took a decade to be able to understand how to actually give you navigation directions around a roundabout in Australia, because they don't have them in America, they, I mean they have a few, but they, most Americans I know are terrified of them.

but I feel like Rich text editing is one of these problems because us software engineers mostly use plain text. So. There are so many great plain text editors that exist both on the web and on desktop. but yeah, rich text is just harder. It's further away from the problems that we regularly solve. I fully agree. And I love Steph's rule. maybe that it's going to become a thing.

and so in particular to you touching on mobile, I feel like mobile and the combination of mobile and web, that is indeed like one of the hardest modes that are still, largely untamed, and this is both I think particularly for web technologies, but, overall, it is just a, it had less time to kind of mature over time. And particularly for web mobile experiences, when, when it comes to text editing, et cetera, like dragging around a cursor, this is where things feel very choppy.

The, creator of React, Jordan Walk, He shares some, some things about that once in a while, where he's kind of particularly pointing out where mobile web is falling short, to be competitive with mobile native in terms of animations and like, just, native interactions. I mean, it's a very hard set of problems, but I think. Addressing those feels very worthwhile to make the web even a richer platform. But then you could also argue that sort of not necessarily in the interest of Apple.

And that's a whole different, kind of worms. I mean, yeah, it's easiest to think of Apple as a person, but they're actually, I can't remember what the 200, 000 people or something. And some of those people are really interested in, in open standards and some aren't. And some of them are great people and some of them are assholes. And that's true of every company that you could name, you know I agree.

And in apple's credits they have push safari and the capabilities way forward over the last couple of years It's really has come a long way Yeah, I think the safari team is very dedicated to making safari an excellent web browser and it shows but there's still some hard problems that haven't been tackled yet.

it's quite sad when I was young I started programming when I was about 10 years old and Every year there would be new things and new technology and, you know, like I got to see the birth of Windows 95 and, I thought, wow, programming is going to be so easy in the future.

You know, like I imagine that right at the time there was whole teams of people working on making something, but in the future, there'd be such good libraries and such good software tools available that one person could do the work of an entire team or entire you know, of a whole project because you could just grab in such good software and there'd be such good primitives available to be able to solve all the problems that you had. And I feel like that prediction of young me. I'm so sad.

I'm disappointed that it didn't come to pass. And instead we have, you know, it's like, Oh, back then, if you wanted to build a program, you want to write some software. Well, I mean, like, I'm so sorry to everybody else, but.

I would have just targeted Windows, you know, written a Windows application and there it is, and, you know, made some little command line tool in DOS or, you know, if you want to, if you're working in Linux, you would just make a command line tool because that was how most good software existed. But now it's like, oh, well, you know, I guess you're going to want a Linux and Windows and., Mac Os desktop version of your application. And then you need to make it work for iOS and Android as well.

So there's five different platforms right outta the gate and, you know, are you even trying, if it doesn't work on all five of those platforms, and it should work well. But I guess you could just use the web, but the web's not actually very good on, on, you know, on mobile platforms. So you really want to have a, native application. Oh, but all the programming languages that you use.

For building native applications on iOS are different from Android, different from the web, different from everything else. And, you know, and we keep on saying, well, we can kind of solve this with WebAssembly, for example, and that way we can have one, you know, target from all of these different bits of software. And, and that's true. If we had good UX libraries across every platform, which were consistent and work well, I mean, this is somewhere.

I'm very, I was very excited about react native when it first came out. And so many people were kind of like some random thing. I was like, please, please. I really want just one good, you know, application programming interface I can use to build real software for real users. it feels like a tragedy, a tragedy of the commons problem in my mind that there's so many companies doing really good work.

But each company, of course, is trying to solve the problem that their particular company will make money from. And there are the commons in software, the commons of the standard set of desktop applications, of user interface libraries. Say, if I wanted to write a desktop application in Rust. there's open source libraries which will interact from Rust to GTK and Rust from, to the Windows libraries and so on, but almost all of them are maintained by volunteers.

And it's a huge problem, like Google Chrome has probably had a billion dollars of investment put into it at this point. to be able to have a web browser that works across every platform and let you be able to run arbitrary software, like arbitrary web software across all the platforms Google Chrome runs. and it might take something around that order of magnitude to have a really good user interface library.

That works across every platform that exists today so that I can just build, like, write once, run everywhere, you know, like, write software once, write Patchwork, write something like that, and then just have Good primitives that mean that my program can work in every context that I want it to run, but as far as I can tell, no one's really putting that time and effort and investment in, you know, like, there's, there's things like Flutter and I'm very excited by them, but I've

burned by Google projects being canceled before, but it just feels like one of these things that like, I want that. And then, and then on the flip side of that, we've got, things like, SQL, which I have massive problems with, even though it's one of the most, one of the best, like think programs like Postgres are some of the highest quality software that exists today that we can, we use regularly. And yet its design is from the seventies.

there's a wonderful talk by Rich Hickey, where he talks about, value oriented programming and place oriented programming. And he says that. Back in the day, we didn't have much memory. Our computers had literally like, Some of the old Atari's and Nintendo Entertainment System has less RAM than is available in one tweet. You know, before they made tweets longer. It had like, it's like a hundred and twenty bytes of RAM or something ridiculous.

And they made Super Mario Brothers with these wild limitations. And so it makes sense, if you're storing data, that every time the value changes you want to Replace, you have one place where the value goes and you replace the value that's stored there. And now SQL is stuck in the past where we've got a table and we just edit the rows and you've got no idea of what's actually changed.

And that makes it incredibly difficult to build collaborative software because You need to know that kind of history. So people maybe layer collaborative things on top of Postgres, but then now Postgres, you're sort of fighting against the ergonomics of Postgres itself. And again, no one's solving this problem. And it really upsets me. Oh man, I have so many thoughts on this.

I 100 percent agree with like that assessment of like the tragedy of the commons and I'm both devastated about it and also hopeful because of some of the initiatives that you've mentioned. the TLDR of it is that I still think that the web is the best vehicle for us to, make it through this uncanny valley.

Since I think if you exclude some aspects, for example, mobile web or some, some other, or browser, strict browser compatibility to a certain limit, Firefox is lagging behind in a bunch of things. You can actually already live in the future. By building fully for the web, things like WebAssembly have come a long way. things like WebGPU, et cetera. And there's also new frameworks that are in the making, but not as mainstream yet that fully embrace those.

So where some lower level primitives could be taking advantage of WebAssembly or WebGPU. the thing that's, commonly strikes me is like the most comical unsolved problems is list rendering, where we have so many React list virtualization libraries, and most of them still suffer from the same underlying problems. And whenever you render a list, like some symptoms that appear typically comes along with frame drops.

But also like, it's rare that, you persist the position of a list and there's just like, if you start digging into, lists as unsolved problems, it becomes very comical, but the way how I, try to address that for myself is by rendering to a canvas. And that's a whole different approach altogether. We don't need to go into that, into that rabbit hole. But yeah, I'm, hopeful that things are coming along. And you've also been mentioning React Native. That has, that's a long, long, long project.

And in a way I have so much admiration and respect for the people who relentlessly keep pushing forward on that. There's been some really big. breakthroughs over the last couple of years, some of them you would think like, wait, that's kind of obvious. Why wasn't it like that from the beginning? But there, there are reasons for that, but it's really getting there. And, particularly for mobile, it is like, it's becoming a really attractive, platform to, to ship things.

And this is where you can, at least when your goal is at least code reuse. then you can get a long way if you target React and React Native. We're not yet at that point where you have the same set of like interactive components and they work in all platforms, no matter what, maybe at some point, but there's progress in that direction. yeah, I mean, while we're at it, I'm just going to throw out some more things that I hope other people fix. Well, someone fixes like it, it might end up being me.

I was saying earlier that, if I could live a thousand lives, I would just spend a lot of them rewriting bit by bit every piece of software that runs on my computer. So everything actually works well and consistently. I've been building a little helper tool web app just for the last couple days to be able to, annotate documents, to be able to, it doesn't really matter, it's some little helper web app.

And once again, I'm frustrated because I have a server, and my server has some data, and I've got a frontend, and my frontend needs to edit that data, and as I change the data, the changes need to be propagated to the server, and the server keeps a copy of it, which it stores on disk. And that problem, I feel like every single time I try and approach it, I have to re implement the wheel and I start from scratch. And it's not even this data that I'm editing isn't even collaborative.

It's just like, okay, great. You know, let's start climbing the ladder of editing once again. So the first step, you know, I've got one object and I send it to the browser and the browser sends back an entire copy of it. Next run. Okay. The server can do versioning where the browser can subscribe, it can get a certain version and find out what version it is.

Next run, Now I can subscribe to the browser, can subscribe from some version every time a new version gets sent, the server sends a new copy of the data. Okay, then now I can do differential updates. Now I can do collaboratively editing on top of the differential updates. Okay, now I can do like, you know, anyway, it's just a series of things over and over and over again, every single piece of software I end up writing.

And most of the time I'll cap out somewhere pretty low down that ladder, even though I'm probably one of the, you know, like, I know all of the tools, I know all of the ways that we can make this problem be good, but our platform isn't very good. And I think a lot of the problem with that is because like we still think in most software about messaging, it's a message oriented system, how we actually want to think for a lot of this isn't about messages. it's about updates.

it's, I've got some semantic idea of the state of the world. And instead of sending messages to the client, which is kind of a low level primitive, I instead want to be sending updates and describing how the document has changed over time. I was working with the, braid group, trying to build.

So we together wrote a proposal for the IETF on a primitive that we could add that's different from get and put and post and so on, but a different verb, which would be a subscribe verb, to say if there's some resource that lives at some URL, I want that resource to be able to change over time. And I don't want to have to re implant the wheel, reinvent the wheel every single time, describing how that resource changes.

In fact, it would be great if the web, like HTTP itself, had a primitive that described a resource that could change over time. And then, for example, NGINX, if NGINX understood that that was something that changed over time, instead of NGINX storing some cache invalidation time and then occasionally revalidating its cache, but in the meantime it's serving out stale copies of this resource.

Instead, NGINX could just subscribe to the underlying layer and say, Hey, I want, I'm interested in this document and tell me every time it changes. And then NGINX can be doing fan out to all the different clients that is interested in subscribing to that document. And you'd have a much better, much more performant way of being able to distribute data around. That is way easier to program, way faster from a performance perspective. And then it could be integrated into lots of things.

Like integrated into Postgres, integrated as a standard thing that we have access to in our web servers. Something we have access to in our web browsers and so on. And it's primitives like this in my mind that, you know, like, I'm like, Oh, please, please. Give me better things like this. This is the kind of thing that I feel like all computing, again, it's that lesson from Google Wave. All computing can work like this. You can just have the data update in real time.

And this is a primitive that almost every program I've ever worked on is needed in some form of, you know, In a web browser, in a mobile phone, in some monitoring application. I have some data that lives somewhere and I want my application to be kept up to date and told whenever changes happen, and then I wanna be able to edit that data from different clients and have something sensible happen.

And there's a simple version of that, which is a centralized version where a centralized server like has to authorize all changes. And you have a version number that goes up by one every time a change happens, which we should be able to just trivially have built into Postgres, in my mind, in SQL databases. And then there's the CRDT, the real time collaborative editing use case for that where I don't actually want a single version number, I want something kind of like Git.

but where I can collaboratively edit any kind of data and I can do it in real time rather than needing to hit commit every time I make a change. I want to push that out. and hopefully as well, I want to be able to do optimistic replication, optimistic concurrency control. So the Ink & Switch team is working on this, and having branches and so on for other kinds of applications, right? Again, it's, it's my rule. It's the, it's like.

We've built these tools for ourselves for code editing where we get to collaborate using Git. We get to have branches, we get to do merging, we get to look at all the diffs between versions. Everybody wants these tools. Everyone. You know, from people working on CRMs to people doing video editing to people, you know, I'm sure running, like wanting to talk to astronauts in space, this comes up all the time.

And for those kind of people, for these projects that want to build using these kind of primitives, it's so hard to have, like, we just, our primitives on modern software aren't there and.

No one like application teams aren't working on it because they've got products to ship so I've had a couple of folks Who are working on that on the podcast before so for example notably the folks working at Electric They're tackling specifically that problem for Postgres were exactly as you're describing that you can subscribe to or like you're you can basically just Express something they call a shape. It's sort of like we can oversimplify and it's like sort of a scoped query.

And then your client can subscribe to the result of those queries. And it basically always gets an up to date query result locally. Well, eventually consistent, but you already get those sort of one level up primitive, where you don't need to think about get, propose, delete, but you can just, subscribe to that.

well, it comes with its own challenges and if you're looking at the fundamental layer, that you describe as message oriented, I think that's closer to the metal and where you need to roll up your sleeves more, but it also allows you to implement things for different trade offs in mind. So one thing I'm, curious about how you're navigating trade offs when it comes to, to data. It's all about trade offs, like what for a given application, what can you get away with?

How much is fine that different parts of the application stage do they need to be fully transactional or is it fine that one part of the application Is maybe lagging slightly behind, and how do you address the cap theorem for your particular application? So I'm curious how you're thinking about navigating trade offs and like trade offs even go to a point, not just to build the initial app.

But also to keep in mind how you need to evolve the app over time, how certain are you about the data model that you've picked that this is the one you still have, like in five years, you're going to have it in five years, probably no matter what, but the question is, are you still happy with it in five years or, is this the worst decision you've made building the application.

So I'm curious how you're thinking about navigating trade offs when you design an application, the data layer for an application.

I mean, fundamentally as engineers, you navigate them one by one, you know, while sweating profusely. That's how that goes, I think. yeah, I feel like there's, there's a bunch of interesting problems in all of the things that you've said. I had a few, couple small conversations with PVH, talking about, he built this thing based around lenses of, of how data can evolve over time in a Cambria. That was it. Yeah. Um, in local-first software. And I have a bunch of my own thoughts about that.

And you know, like I've got designs in mind that I'd love to experiment and play with. So there's something there, which is, you know, like it's really interesting. HTML, has evolved beautifully over time. Like HTML has barely changed at all since it was first introduced. like we've introduced new tags, but almost like all of the tags still follow the same. Fundamentally XML style structure, and that's wonderful. It's wonderful. It's been able to evolve like that.

Email has evolved slightly less gracefully, but emails. I think email up email programs from the 70s probably still work today with most modern emails, which is incredible, even though. Email messages themselves are almost impossible to pass correctly. And, I've got a lot of thoughts about that as well. I've got some friends who work at FastMail and they've, pinned me to the wall with things about email that I had no idea about that are horrific.

but there's this open problem, which is how can data models change. So there's a, set of problems there, which I'd love to solve. There's a set of problems around, how is our data stored and sent. in my mind, what I would really like to have is. I feel like there's some kinds of programs where I actually really You know, I'm so sorry to the Ink and Switch folks, I'm quite happy for there to be a centralized server for Discord, or for IRC.

For something like that, when there's a place that we all go to, I'm quite happy for there to be one computer that's authoritative and decides, you know, all of the messages that are and aren't part of the log of messages. I feel the same way about banks. I'm more than happy for them to use a centralized transactional database. I'm not a Bitcoin absolutist by any means. I think decentralized databases work great.

but I'd really like some set of primitives where, The primitives that I use in my computer program, like what I embed into my program and use, are the same set of primitives that I could use either with a centralized server, which has transactions and various other ergonomics, or with something like Automerge or Yjs, where, I'm actually collaboratively editing a CRDT. And I feel like there's a lot of overlap there around what's the shape of the data. What does the change look like?

You know, subscriptions, which also you almost always want in a centralized use case. and then there's a bunch of questions around performance. so I wrote a CRDT library of my own called Diamond Types I haven't kept up with all of the demands that I've made of myself. busy and distracted, but I use that. That was the test bed in original place where I built Eg-walker, which is an algorithm that I've written a paper on, which is just for collaborative text editing.

and I wanted to make it really, really fast. And people kept saying, well, but don't you know that if you collaboratively edit a text document in something like Yjs or Automerge, it stores every single keystroke that's ever made to the document. And these documents are going to get huge, and we need some way to be able to prune history, and it's really important. So I said, well, Look, that might, I agree, I think that's a very interesting and important problem.

But let's first start, let's just start by trying to make it as fast as possible and as small as possible. So, you know, even if we prune it, it's going to be pruned to be even smaller regardless. this is always when I do a lot of performance work and people, a lot of times people reach for multi threading.

Let's, let's spawn it across lots of threads on your computer, which is a fine approach, but It's almost always better to start with trying to make it run as fast as you can on one thread before you run it across multiple threads. I mean, this is why the M1 series of, of Apple was like such a big deal because it showed like, okay, we can actually do a lot better even in a single threaded environment.

Exactly. Yeah, but It turns out that, the collaboratively editable text documents don't actually grow very fast. the amount of data, the number of bytes on disk is tiny. And, and yeah, like I think that it might be useful to build software to prune anyway. And I've got a bunch of, you know, like I've thought a lot about that problem because people keep talking about it. but from an engineering perspective, a lot of the time we can just keep all of the data and it's actually fine.

We just need to, for example, store it in some slightly more efficient way sometimes than raw JSON. And, you know, and like, that's the thing that we need to think about. modern computers are so fast. Like, yeah, I feel like similar to what you've mentioned before with quoting Paul Graham of like, what was the quote again? It's it's what would it be ridiculous to not have in a hundred years? Yeah, exactly. So you imagine a hundred years, it's like, we still don't have something. It's crazy.

Then it's, okay, well, if we didn't have that in 100 years, it would be crazy. Could we just build it today? Exactly.

So similar to that, we can also ask ourselves more often, like, Which things would actually be totally fine that historically we always like shied away from and that sort of like by now, there's sort of like this, picture of like a baby elephant kind of, pinned to a pole and it's like, it has learned it cannot escape and now it's, a full upgrown, elephant and still, pinned to this tiny pole.

I think there's like, a lot of similar stuff, the way, how we go about programming and we need to unlearn, forget about some things. And this is, I feel like also why some of the most brilliant, new technologies are by people, by, by newcomers, like new generations of programmers who are like blissfully unaware of some of like the old stigma. And this is how we get new ideas as well. yeah, I really agree with that.

there's something that it doesn't, I keep thinking about, there was a, there was the Unix room. So Unix was invented in a place by some people and they sat in a room together and they had one computer and they would write programs and show them to each other. And someone wrote spell, you know, and, and someone wrote like, they wrote these different little Unix programs and said, Hey, check it out. And then, some people, made the pipe operator and made piping work.

And they started to be able to connect these programs together to be able to make. You know, quite complex programs just as a series of small programs pipe together. This is a really beautiful primitive, and it's a beautiful thing that, someone created. And we have it on all of our modern terminals. I think there's even a version on Windows, but all Unixes have something like this.

I feel like at some point we gave up trying to invent new kinds of primitives like that, and I dunno why, but it feels like the sort of thing that we should be able to do. Obviously, you know, I sometimes think about, we've been talking about the web a bit. And, we have this beautiful idea, which is, so we've got these two different ideas right now. We've got a desktop application, and desktop applications have access to everything that the user has access to by default, which is interesting.

And Apple's trying to change this, but keeps getting in trouble for it. desktop applications are binaries. They're always native binaries on every platform. And they have access to some set of native, APIs and they can make sys calls directly.

And then we have another set of applications called web applications, which is sandboxed by a browser and, you know, historically only written in JavaScript and written in a way that they can operate on every, every computer, but only inside the web browser. And like. Web applications are really successful, I think, not because of the web browser, but because you don't have to install them.

And people sort of forget this, but they're kind of equivalent, except this one you have to install for some reason, and this one you don't. And because you don't have to install a web app, people spend a lot of time optimizing them for size. So you load up the New York Times and it'll load a lot of Crap, probably, but we're looking on the order of maybe half a meg of JavaScript. Maybe it's a couple megs of JavaScript and some images.

Um, whereas if I install, you know, I haven't actually tried this. I'm going to get in trouble by somebody who will actually go and look, but you look at most iPhone apps, for example, and they, there are hundreds of megabytes in size. I think the Uber app is like 300 megs or something crazy. Like the Facebook app is huge and it doesn't even have content. Like it's not even like the Uber app is loading. It's like installing databases. Who even knows what's in there?

But then web applications are written in slow languages and I just feel like it's like well, we could just make a new platform and we could give it a new kind of application where you don't have to install it, but also it gets access to some set of beautiful native primitives that might be even better than what the web provides today for applications themselves. Like the web obviously is designed as a document.

Viewing platform and then we've could have hacked on applications because it was so convenient, and we could make it so that data is just transparently available and accessible between programs beautifully using all of the tools and technologies we've been talking about.

but no one works on this stuff because we sort of take for granted, it's like, there's the ruins of old Rome, you know, and that's, that's like Unix or it's the, you know, the machine code and x86 assembly and then modern, you know, people say, oh, well, Then they build the web browser and then we build on top of the web browser react. And then you build an application on top of react, but then the application slows. So you do some other wacky thing on top of that.

And it's like, you know, underneath the city is this giant, you know, all of these ruins that are all full of code and all being used by modern programs. But that so many people, particularly young engineers, they have no idea about all of this crap. They just know that if they type the right thing into a react application, that they'll get something on the screen.

But all of that stuff, we can actually like, you know, burn it all if we want to, and then go and make beautiful things right from the very base of computers. Yeah. I mean, I think there's this sort of like innovators dilemma and I think this, you mentioning email as an example, I think this is one of the words, since you, you just emails and browsers, like you, you just cannot get away with like not supporting the old stuff.

And this is where, If you build a brand new thing, you don't have to, account for all of like the historic mess, and you can just start over. And, I think this is probably just having the, the courage or naivety to start over. I think that that's highly underrated. and also for the record, I've just checked, blank loading New York times. It's like currently five megabytes of just javaScript and, depending on, well, and 20 megs, of like other stuff that I've loaded, like with images and so on.

So, I think that this just proves the point, like even for New York times, like five megs of JavaScript, it's fine. I don't want to know how much of that is like ads related, et cetera, but, I just want to encourage people to rethink their, like their learned, thinking about trade offs. Like we're, it's at the same time, we're kind of indoctrinated with like, okay, we need to make the initial page load as quick as possible. We always have connectivity.

I'd like to alter that slightly and say like, when we're connected. We probably have pretty good connectivity. So let's embrace that, but we're, not going to be always connected for the moments where we don't have perfect connectivity. Let's, like piggyback on the times when we had good connectivity and we should just sync the deltas instead of like reloading everything every time. Also goes back to the build problem, the build system problem that we had before.

Since we don't build things in a principled enough way, so we don't trust the stuff that was there before, and so that leads to us reloading everything from scratch all the time. And, I think that's another big culprit of, systems problems. I read a great article a while ago, comparing the size of JavaScript bundles with War and Peace, saying like, you know, could you, how many copies of War and Peace could fit in your JavaScript bundle? and it's quite a lot.

and the other thing I sometimes think about is, how many bytes of data would it take if you took a screenshot of your website, even as a PNG, and then sent me the screenshot, would that be larger or smaller than it takes to load your website? And if the answer is that your screenshot would be smaller than all of the code that you send me, then maybe you're doing something wrong.

but even then, like, yeah, I really encourage people to also think about from the perspective of, the web browser, it's all code. You know, if you, if you start digging, it's code a really long way down. And all of that code can be changed. Like web browsers are open source. I've got some quite famously, infamously, I've got some source code, in Google Chrome, to be able to interact with Xbox game controllers.

I'm not sure if it's still there, but, I wrote, a user land, USB driver for Google Chrome. so that if you're on Mac OS in particular, so if you plug in an Xbox game controller to a Mac computer and you want to play video games, in browsers, then the Xbox controller will work. But there's so much stuff like that, that like, you know, and I think that's almost crazy that that, that is in Google Chrome.

it should be the operating system, but there's lots of perimeters that we could put in Google Chrome. That would mean that, you know, if there's some, if there are things that we want to be able to have access to as web developers, We can build them in as primitives, and everyone can get access to them. Like, a lot of the standards committees, you can just go to them, it turns out.

You know, like, there's open mailing lists of the people that build all of this stuff, and they're just nerds like us, you know? Like, they've just got all of their own crotchety, angry opinions that they'll tell you about if you give them half a chance. but, I don't know. I feel like there's not enough conversation around all this stuff, how it could be better.

Going back slightly to the moment where you reflected on that kind of the creation of primitives, what I found, at least looking back at my own creative work, what I found easier to than just forcing a primitive into existing out of thin air is rather embrace sort of like organic chaos, and then study it really hard and then see basically out of the existing mess. typically if you look hard enough, if you observe hard enough, you can, observe some things that later can emerge as primitives.

And, this is where almost paradoxically the more requirements I apply on a certain system. The more it becomes like simpler over time. And so I've basically just taken more kind of like inner peace by like, not knowing the primitives upfront right away. I'm basically just trying to solve the problem. Like I happily say yes to a few problems and I've had enough, past experience where out of that mess.

through lucky coincidences, I realized, okay, here's some useful, more general primitives and in hindsight, those primitives look very obvious and someone who then looks at that system and maybe builds a similar one, they can already embrace those primitive ideas and, build on top of them, the beginning, but I think for novel, new primitives, I think the most likely path for them to come into existence is out of emergence and not out of like forcing them into existence out of thin air.

I agree with everything and I'm a little bit dubious on that last point. And just to push back a little bit, I think that it's got to be this push and pull, you know, like they're both so important. I really like what you said though. Again, Rich Hickey, who, You should watch all of his talks.

Anyone who's listening to this, he's much more insightful than I will ever be, I think, but he gave a talk once and he said, he talked about hammock driven development as in software development driven by writing some code and then going outside and lying in a hammock and thinking really hard about it. And just, just letting it percolate, like letting the design that you've come up with percolate and say, how does this problem actually really want to be solved if I were to do a good job at it?

and instead of sort of half arsing a million things and having everyone across the industry half arse the same set of eight things, you know, lists and so on, I don't know, I can't help but think that if, if a few people spend some time.

Some serious time thinking a lot about lists, we could have some really nice lists in web browsers and all these different things and then it would just be solved for everybody, you know, maybe this is, this is why I've prefaced it with like, that's being for me, the easier path, like for me, step one is half arsing something. And then looking at it, but I need to have it be laid out wrong first for me to spot, oh, this is, this is how right.

Looks like some more brilliant people than me can probably skip step one. But for me, this one has been a more proven approach. I mean, it's the same for me. I used to teach programming and something, you know, my students would. Sort of sometimes come to me asking me how they should design their program before they even start writing any code. And I'm convinced that you know the least about your program, like you know the least about the program before you start writing it.

That's the time that you will know the least that you will ever know about this problem. So you're the least qualified you're ever going to be to design it correctly. You know, so you become qualified by throwing something on the screen and then taking the time to reflect on the knowledge that you've gained to figure out how it could be written in a better way. it's, yeah, I absolutely agree.

And I think this is also this underscore is like one of the most important things that I hold dearly in the craft of software engineering, which is the ease of iteration and the speed of iteration. And this is where I spend a obscene amount of time on just like making so that I have fun and iterating. That it's very cheap for me to get it wrong 10 times and it's fun to do every iteration. but if it's gruesome to do one iteration, then you just, run out of like joy and energy.

Until you get to the right point. So making the iteration cycles, it's like, it sounds very obvious, but I think very few people actually put in the work since it's also partially some pretty annoying work, like getting your, like your monorepo set up, all dialed in, like particular, like which JavaScript package manager should you use? Like all of like those stupid things, but getting all of that stuff dialed in, I think that the reward is, quite big. I mean.

I assume that you've read this, but, again, to plug more things that people should read if they haven't, there's that beautiful essay of Worse is Better, that talks about all of this. And I feel like, yeah, I think that life exists like in the intersection of thinking it through and just throwing something at the wall.

And I think all good software is going to have elements of both of these things where, you know, there's not too much upfront design and upfront thinking, but there's not too little of it either. You know, like there's got to be some amount of time, but after you've written a whole lot of code, you take a step back and you say, how should this have been? If I knew everything I knew now, how would I build it?

And I think everything beautiful and everything incredible has a lot of those feedback loops. Like I listened to an amazing talk at, GDC, the Game Developers Conference, once talking about this, and he said essentially all good games are made by having some ideas, writing them in code, trying them out, and then throwing out 80 percent of them because they weren't very good ideas, but 20 percent of them were really good ideas.

And he said in that talk that the quality of any program that you make is proportional to the number of iteration cycles you get through, not how long one cycle is. The number of iteration cycles, just the sheer amount of repetition that you go through in writing code, reflecting, changing the code, making it better, writing more code and so on. And I think it's, yeah, I think it's a beautiful idea.

I've been doing improv comedy classes lately and over the last few years and in improv we learn it's, you can't plan. There's no planning. You don't have a meeting with your, the people you share the stage with before the show starts. So everything is just asking the question of what comes next? Like, which I feel like is this life philosophy. Just at every moment, what comes next? And maybe what comes next is writing more code.

And maybe what comes next is stepping away and thinking about the code and refactoring it. But, like, I feel like that's the right question just to hold, you know, consciously. so you've given me two segues now to ask you about another topic that I think is very top of mind for you. So one is sort of this almost evolution, like, nature evolution of software and good products, et cetera, where it's all about the iteration cycles. That's like how biology works.

So this is why we as humans exist, et cetera, and why other good things exist. and then also asking the question, what is next? That is, that's the essence of how an LLM works. So I think you've been thinking a lot about, the topic of AI and what that means for our culture of programmers and software creatives. So I'm, I'm curious, what are some of the leading questions for you are when you're rethinking and like digging into the topic of AI?

I've told the story of me working on Google Wave and I started there in about 2010, but I've been thinking about AI a lot since about 2002. I think since I first around when I started uni. I've got somewhere a pile of notebooks full of handwritten notes, thinking through lots of different ideas and ways to build AI systems. and I started a PhD And I ended up leaving and I felt disappointed and I attacked myself because I couldn't get the things that I wanted to get working, working.

And instead of just chipping away at it, like I wish I had done in retrospect, I felt a whole lot of shame and guilt and then walked away from the whole industry and haven't really come back or it's been a while. you know, I haven't come back in a big way, for, you know, 20, 23 years or something now, which is crazy. more than half of my life, you know, running away from this dream that I had, but way back when I. Like, it's really interesting.

We're in this transition period at the moment where AI is a similar intelligence to humans. but something I think about a lot with this kind of software is, like I mentioned before that when I was young, I thought that one human should be able to, as our tools, let us become more productive. One person should be able to do the work of a whole team from a decade ago. And increasingly with AI think that's actually going to become the case.

And I don't really know what that's going to, how it's going to change.

A lot of things, but I can imagine as we go forward that if we had better primitives and had primitives that the AIs understood, that a lot of these kind of problems that actually just require a whole lot of code, for example, like building a really good UX, like a UI toolkit that would be cross platform, it's possible that we could start to build AI systems that could actually design good primitive tools, for computer systems and, I don't know what that means for us.

Like, I don't have any recommendations or advice or anything else, but it's really interesting to start thinking about it as, you know, if, if your productivity isn't limited by what you can personally code anymore, which is, I think, going to increasingly be the case, then what do you want to build?

And how can we start building, you know, like, utilizing AI tools that will increasingly, and obviously they're not quite yet, there yet, build large swathes of working in usable software., I feel like this is just really open question of what does it even mean to be a software developer, but also.

What could we do again, right, like that we can't do today because it would take the resources of Google to be able to build something that we might be able to do, you know, like if you wanted to build your own operating system and your own browser from scratch, you could just ask an AI to do it. How do we do that?

have you been able to derive sort of like a set of, invariants almost that are still going to be true, in the age where, AI will be a meaningful part of how software is created, et cetera. Like, which aspects do you think are still meaningful for humans to play a, meaningful role there? I mean, it's a great question.

I feel like there's this massive economic, you know, loom thing that looms over all of us of, If you can't use your mind to do things that an AI couldn't do, then why would you ever be hired? why would any human be hired for any job? And, there's a threshold that we need to pass. You know, obviously with computers and AI systems, we can become more productive than we ever are today and produce more abundance and more resources and more useful programs and everything else.

But if none of the humans are employed to do any of that, then we've got a huge economic problem. but The beautiful world on the other side of that, if we can possibly reach it, I think has to look like humans actually building the software and building like being able to be creative in all of the ways that we're born to be. I think that there's some element that's honestly our birthright in creative beings, being able to make stuff, you know, it feels so, so core to who we are.

And the more that I get in touch with myself, the more I just want to. Build and play piano and write stories and do this kind of stuff. So play as sort of like the most natural state of, who we are. Yeah, I did, again, this is a segue, that I'll, I'll try not to drag us too, too far down. I went to France a year and a half ago and, and attended a clowning intensive. So physical comedy and performance for a month. And it was horrific. The, it was quite abusive in its own way.

The guy who's was teaching it has been teaching clowning for 40 years. And, anyway, it was an incredible experience. But, he said, you know, he's this like grizzled old Frenchman. And he said that he thinks And he's trained Sacha Baron Cohen and all these different people, but he thinks that the most beautiful thing that exists in the world is watching a child play and, you know, so much of clowning and physical comedy.

And I think creativity in general is rediscovering as adults, if we need to, that spark that's inside all of us and that just wants to build things and wants to play and wants to try stuff out and so on. And I don't know, that's my, that's my hope for AI at least is that. If you were to play, if you wanted to make your own sandbox, you know, grow a garden, a software, whatever metaphor you want to think about it as, then what do you want to make? What, what are you called on to make?

You know, I live in a apartment building that overlooks a lot of the city and, there's this idea of Dharma that's talked about in, in Smithsonian philosophy, which is like, what is the duty that's your sacred duty? That's yours alone to reach. And that duty, by the way, might just be to like, be a really great parent. It doesn't have to be anything grand or grandiose in any way, but asking that question of what is it yours to do in the world?

That's yours uniquely that no one else will do if you don't do it. I think that's really the question that we want to be asking ourselves as creative workers. and from that perspective, I'm excited about AI because hoping I mean, I'm really hoping that it lets us answer the questions like that more.

And you know, like I don't think economically we need local-first software, you know, like we can trudge along with the feudalism of Google cloud services for, you know, probably for a very long time. but I don't want that. I want beautiful software programs that work in harmony with who I am as a person and with in harmony with what kind of communities I want to be able to foster.

And from that perspective, I think that's like why I care about local-first software and why I hope that more people care about it because I just think that. We deserve, I don't know, we can just have nice things, you know, like the cloud always feels like, feudal city states, you know, so back before we had democracy. And before we had countries, you have these towns and the town, it's not a democracy. There's one ruler, the noble, the local noble.

And if you upset the noble, there's no laws necessarily. That's a, recent invention, that the noble would say, no, not you. And you'd be either be exiled if you're lucky or like, you know, hung from the town square. If you did something, the noble didn't like, and no one owns property. Only the noble owns the whole town. That's the rule, right? And so if you have a house, you have a house because the noble has graciously allowed you to live there.

And that's the rule, you know, and if the noble wants to. Do whatever they want to do. There's no higher law than the noble, and that feels like the software ecosystem that we exist in today, where, you know, there's the Google city. And as much as I have endless respect for so many individuals that I know who I've worked with at Google. I think they're amazing people. The experience is Google is a feudal city state.

Where you walk into the gates and if the noble, with the local Lord doesn't like you, then you can get banished at any moment out of Google. There's no recourse. There's no laws. There's no rules. It's not a democracy. you don't have any privacy. Everything you do is monitored 24 seven by the Google cameras that exist everywhere. You know, and they're nice little plastic, you know, beautiful designed. whatever.

And if you don't like it, you can go down the street to the Apple town where everything is run by Apple and Apple has slightly different values from Google. And I happen to like Apple's philosophy on privacy more than Google's, but it's the same kind of world. And I personally believe in democracy and I like that my streets aren't owned by a company, call me crazy, but yeah. And I don't know how it interacts with AI, but I feel like we've got, I don't know.

It's like, we've got this opportunity to being more creative, to make more stuff, to not need billions of dollars of funding, and it's up to us what we want to do with that. And maybe it's a stretch and maybe I'm putting words in your mouth here, but, at least you've used the word democracy, and, maybe that is also like something that people see in local-first that is almost a more democratic approach to software. Yeah, I really agree.

it's like, I care about local-first software the most for creative work, for like, if I'm composing a song. It's my song. It's mine. You know, it's not Google's or whichever cloud service I want to host that. It's not theirs. I don't want them to be able to look in it or change it or monetize it or sell it on or anything else. It's my work. And if I want to work with you. On making that song. It should exist on both of our computers and be transmitted between our computers.

It's ours, you know, like there's some, I don't know, it feels like some principle. It just feels so obvious. And I don't know how I could possibly justify it. It feels axiomatic. I think a lot of people don't realize just how much, you know, Art and I feel like I'm probably preaching to the converted with, you know, with a crowd listening to this podcast, but we forget how much our computering experience is moderated by big corporations, like big companies.

And I think that's really sad because our computers are amazing. You know, like the computer sitting at your desk, like, I often think about this, that computers are like, like two to five gigahertz, two to five billion things every second, like, wow, you know, like, that's a lot of, that's a lot of steps. And you can make those steps do anything you want, like you can write any program, totally free in that regard. So we could make programs that.

To do whatever service, whatever needs we want that we'll be able to, you know, we've got all the technology, we've got CRDTs, and we've got access to low level networking primitives and all of this stuff. but I really want good software. so that if I want to, like, I don't know, I want to make a diagram for a paper that I've got good tools to do that with. And I want to just whatever it is, I've got a bunch of photos and I want to share, I've been getting into photography lately.

I want to share my photos with my friends. I want to be able to just have software that lets me do that easily. you know, like I really want to have. collaborative software. I want to be able to run a DND campaign on a, on a Wave or something that's Wave like where it's not like owned by Google's computers and we have to all agree to Google's terms of service to be able to continue having access to that. It should just be something that we can all access.

And if I want to write some new custom software to help me run my DND campaign, I want to be able to do that too and have it interact with the same data model as everything else. So yeah, I fully agree and I think often people who are not as comfortable yet in the local-first bubble as, as we are, I think a lot of people kind of think like, okay, why does local-first matter?

Maybe if, there's sort of the, this dystopian future where I can't trust the government, I can't trust the clouds, I'm like a persecuted minority, whatever it is, like, but it has people only assume local-first to be like meaningful, when you're kind of, when a dystopian future has already arrived. And I think you don't, sure, local-first is very meaningful in that part, but that's not what I'm like a I'm optimist. So I don't, anticipate a dystopian future. And so I'm not a prepper, et cetera.

I'm not a local 1st prepper. I'm a local 1st player. Like, I want to have more of, like, the local for the playful state of mind. Like, I. I'm so fortunate and privileged that I get to do what I love and, still, like, make a living from it. And, like, my ideal day that I, like, that I live each day is, like, that I had a, like, a day full of play, basically.

And for me, play here means, like, discovering things, like, losing sleep over a hard problem, but then getting the joy out of solving that and having sort of my own agency over what I'm doing. And, I think even like when a child plays, like a child is not told, like now you have to build this thing, but it's their own agency to do what they feel like and pursue their own goals. And I think local-first can.

And able that in a similar way, how the web has lowered the bar so much and therefore increase the agencies of individuals. Like, and I think the same thing is happening right now with like AI assist tools, where the barrier comes down even more for non programmers to build things. It gives them agency, gives them ways to. Playfully build things and I have the same hope for local-first that it brings down one of the big barriers, which is data management. And I'm very excited about that.

Yeah. Me too. Yeah. I feel like there's a bunch of ways that we could be building better software.

And, I just want all my programs to interoperate, you know, If I'm editing a text document, I want to be able to be typing something in a text editing environment on my laptop and then, you know, I run out the door and I'm on a train and I just opened up my phone and I can keep on editing it and I want all of my software to work like that and I want there to be And it's like on, on Unix, we have this sort of open file system.

There's any program can interact with the file system and edit files, which is great except that the file system makes it really hard for two programs to edit a file at the same time and have anything meaningful come out at all. So most programs don't really interact with each other around the data unless one program sort of saves something and another program opens it up later and then you do something in that program and save something else out and it has something else open it.

But with local-first software, I want. Sort of like an ecosystem of all the programs on my computer to be interacting. And then I also want to be able to have programs on my computer and your computer interacting. I feel like that should just be really easy to do. Like it It's something that should be out of the box, and if every programmer needs to re implement it and reinvent it from scratch, it's never going to happen, but it's something that could just be built into the system.

I like this way of thinking about it, since my primary goal for local-first right now is, like, step one is make it super, super easy, bring down the barrier to build something meaningful. For myself or a small group of people at all, but then step two, even wider vision is like, okay, let's bring together those experiences and bring them together in a way. That's not just sort of like a, a best effort kind of thing where like, oh, yeah, export JSON over here. import over there.

And like, now it's like, you only have people's last names or something, but at least there's some existence. But I think this is almost like a second chapter that will probably take a lot of effort and we need the solid foundations first, but the, yeah, the prize of local-first can, ultimately lead to data interop in the same ways.

Like our brain doesn't work in silos, but our brain, like when the conversation I'm having with you, I'm taking some ideas away from that and maybe bring them to another conversation I have. So my brain is already that big data bus. And that is a promise that local-first can fulfill. Yeah, I absolutely agree. And so I feel like there's that piece. And then the other piece is, there's things that we have as software engineers that everyone should have.

my girlfriend works, she uses a, there's a content management system that she uses at her work. And I have access to, because I'm a software engineer, I use Git. And with Git, I can have branches, and I can have pull requests, and I can see all the changes and history of changes. we've built those tools for ourselves, and then we haven't built them for anyone else in any other industry. And that, to me, is totally crazy.

It's like no one else, you know, like, we've kind of been really selfish as software engineers. It's that's rule in the real world. Yeah, exactly. Exactly. Right. Yeah, but with local-first software, I, you know, like with these, these primitives with these different ways of approaching software, I feel like we can make a set of, primitives that should allow any program to do that. Like it's, I'm really happy with the Ink and Switch guys and the work that they're doing, guys and gals.

but there's also, like, I feel like that should just be the standard set of primitives for most things I do, you know? I want to be able to open up Photoshop and be editing a document and have some commit type thing. And if you're using a different image editing program, I should be able to share it with you and you can just open up the document too.

And then if, I want to write some script that's going to interactively make changes to the document that I'm editing, like the image I'm editing, I should be able to have a, like a little program that just interacts with the same data model that both of these programs we're using can interact with. And it, it should let us, you know, and then also have the change history and we can look through the history of changes and see who did what. And we can have pull requests on a video editing session.

If we're editing a video together, I want to be able to have a pull request on a different way that the edit comes together. Like all of those kinds of things should just be built into the primitives of the software that we write. and the software that we use. And, you know, like I was talking to someone recently who was saying, he was like, Oh, I don't know how to explain it to the programmers that the people want all of these things.

And I said, I was like, Oh, well tell them, imagine if you didn't get to use Git and you went back to the world where it was like, you know, blah, final, final to final. No, actually the real final version. This is actually the world that most people who aren't software engineers. live in with most of the programs that they use today.

you know, in the best case, they've got, they've got something like Google Docs that, you know, makes a little bait and switch where they have exactly the change control that Google allows you to have. And by the way, last I checked, Google's API for Docs doesn't actually give you the change history of a document.

So, you know, once Google's got that change history, it's locked away in their servers and you can't do anything with it or interact with it or even download it for yourself, which is a real pity. but we can just have programs and files living on our hard drives and edit them and edit them with other people and see the change history and do whatever we want to. And I want everyone to have that capacity. So, yeah, like it's obvious, right?

yeah, that's another great way to, to kind of think about the future. Like, what have we as programmers figured out and how can we bring it? How can, how can we reduce the distance of Seph's rule here? And, how can we bring a lot of those things that for us are obvious? into other programs, but I think that the reality is, it's not that some product manager hasn't thought of it. product managers have like, it's almost like the peak of the iceberg meme.

only the stuff that's floating above the sea level, that is the stuff that was actually important enough and viable enough to build the stuff underneath it's not built yet, it was thought of, but it was just too hard and not meaningfully important enough to ship. And I think local-first can bring down the cost of that.

Yeah, I couldn't agree more, and it's not just, too hard, it's even if they do it, they've done it just for their application, you know, and so every application ends up with its own bespoke way to do change editing and change histories and everything else, and then none of them are collaborative, not like none of them are interoperable, like what we really need is we need something like the file system where every program that exists can interact with some standard set of primitives.

And then every program can take for granted that the file system exists and it sits underneath the program. And then, if you have something like that, not only is it really easy for, you know, product managers to build these features into their applications, which I promise you, I agree with you, I've talked to a lot of them, they'd love it. They really want collaborative editing. You know, everyone wants this.

so not only could they do it easily, but then all of the collaborative editing tools can all work together. You know, they can all interoperate, which would just be so obviously great. It's some people will like Vim and some people like Emacs and some people like something else. And, you know, why not have our documents able to be edited in all of those programs, you know?

Yeah. So I'm so grateful that we got to explore all of those ideas that are clearly on your mind. And I'm, Also so grateful for everyone who's making that a reality particular shout out goes there to to Ink and Switch anyone who's really like defying the strategy of the commons for like for software and I just want to see more of that and I think a big step of that is like to like step one is inspire people like open people's eyes of like, oh, we can do better.

and yeah, so with that, I just want to thank you so much for your time, like, sharing all that we've already, when quite extensive on this conversation, I feel like there is. A lot of stuff that we should probably explore in a follow up conversation. And we've already foreshadowed quite a bit of like what the folks at Ink and Switch are brewing with Patchwork, et cetera. So I'm really looking forward to learning more about that as well.

but, yeah, maybe the audience, if they want to see you, in person, we're also planning the next local-first conference where, if everything works out, you might be a speaker as well. So, happily invite everyone to come to Berlin at the end of May, for the conference and, to get a chance to talk to you in person as well. I think that would be marvelous. And I really want to echo that. Thank you for all of the people that are doing the hard work.

I feel like this is something that's complicated enough. We can't do it alone. And. You know, one of the beautiful things about software that is, lets you do collaborative editing is We can collaborate on all of that. Exactly. And it's even like a virtual cycle where people working on that makes it easier to collaborate. So yeah, that's a very optimistic and hopeful vision for the future with many challenges ahead. But yeah, I want to end on a positive note.

Seph, thank you so much for like, yeah, sharing all of your wisdom and thoughts with us here. Thank you. Thank you for inviting me. this is lovely and I hope we have a chance to chat more in the future. Thank you for listening to the localfirst.fm podcast. If you've enjoyed this episode and haven't done so already, please Please subscribe and leave a review. Please also share this episode with your friends and colleagues.

Spreading the word about the podcast is a great way to support it and to help me keep it going. A special thanks again to Convex and ElectricSQL for supporting this podcast. See you next time.