#26 – Adam Fish: Ditto, Realm

The customers of Ditto are businesses that have workers predominantly that are not at desks like us. They're out doing jobs in the real world. the internet is by no means guaranteed. And so being able to use smart software that leverages the network capabilities of the devices that workers are already holding is compelling. cause it kind of adds like another layer of, durability, to ensure their, application, their data, keeps working. Welcome to the Localfirst.fm Podcast.

I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and love the craft of software engineering. For the past few years, I've been on a journey to build a modern high quality music app using web technologies, and in doing so, I've been falling down the rabbit hole of local-first software. This podcast is your invitation to join me on that journey. In this episode, I'm speaking to Adam Fish, co-founder, and CEO of Ditto.

An end-to-end syncing platform with a focus on resilient connectivity. In this conversation, Adam shares the origin story of Ditto, his prior related work on Realm and the hard networking problems Ditto is solving. Before getting started also a big thank you to Jazz for sponsoring this podcast. And now my interview with Adam. Hey, welcome Adam. How are you doing? Very good. Excited to be here. Thank you. I'm very excited to have you on the show.

you and I have had a little chat before the show and just recollecting our members since the two of us had the pleasure to meet way back. I think this was at this point, like probably more than 10 years ago when you've been still at Realm. And we're gonna hear more about that, but it's so cool, like how, like in life our paths like overlap when we're in the same space. So would you mind introducing yourself? Yeah. Well, I'm excited to be here. Thank you for having me.

my name's Adam Fish. I'm the CEO co-founder of Ditto. And as you've mentioned, our paths crossed, I, it had to have been about a decade ago in San Francisco, where I was at Realm, previous to starting Ditto. And, at the time, were you possibly starting Prisma or it was like, yeah, it was around, that was the predecessor to Prisma at that point. Yeah. Yeah, so, Ditto itself is, you know, a member in sort of the local-first effort, but maybe not as as well known.

and so excited to share more about my story and, about Ditto itself. In short, it's an edge sync platform, and so we connect devices peer-to-peer and synchronize data, through a database, in addition up to the cloud. And so this is, something that, I think is very applicable to everyone who's really interested in local-first. And, you know, I have a long history in this, like you'd mentioned at Realm before it, and so excited to kind of share the story of how I got here.

Yeah, let's actually start with that. And I think there was already a stage before you joined Realm where you started poking at the space. So maybe we wanna start with that anecdote that part of your professional journey.

Yeah, so it's, I at this point, about a 15 year journey where I got my start as a developer coming out to San Francisco and actually working on a different startup. kind of went through a couple iterations of ideas. we were part of AngelPad, which was sort of like a Y Combinator, initiative Back in the day, it's not around, anymore. And, through that iteration, we ended up exiting the accelerator program focused on trying to build a mobile CRM application.

And so this was 2010, 2011, 12 timeframe where mobile was still very nascent, even with consumer use cases. But on the enterprise side, it was still, you know, no one had figured it out. And so we thought that was a big opportunity with that. and looking at the CRM markets, we thought, hey, that's a big market. We could go disrupt it. And so, we started building this application and the first thing we really wanted to do was build an incredible mobile experience. And so that naturally.

forced us into thinking how can we have local-first data? and so at first this was really about like user experience, just like instantaneous clicks. But then we were like, okay, working offline would make sense as well. and so we ended up building a very naive sync engine. and this was fairly abstracted because we were trying to plug into existing CRM systems.

And so we would pull in your data from Salesforce and then in a sort of like a server driven UI pattern, sync that down to the device, and then it would render the UI based off of the specific data, fields that you had in your CRM account. And so it was at the time, I would say fairly innovative for what was a, you know, relatively narrow application. And so, that was sort of the innovative technology behind the app.

And when we launched it, it was started getting meager success from a business standpoint. And, myself and my co-founder at the time looked at each other and we were like, you know, how did we end up building CRM software? Like, this is not something that I'm like particularly passionate about. and yeah, it was just like we had sort of like a, a moment, where we wanted to just like, take a look at ourselves in our lives and say, is this really what we wanna spend our professional careers on?

And so, for me in that moment of reflection, I was like, I'm really not interested in CRM, but this infrastructure that we built, that to me seems really interesting 'cause it solved the problem I was experiencing as a mobile developer wanting data locally and being able to create like great user experiences.

And so ultimately I have like actually a pitch deck that's like hilariously similar to stuff with Ditto from, 2013 timeframe where I was like, okay, I wanna pivot the company and basically, at the time sort of build something similar to Pars, and Firebase. and, you know, let's just become an infrastructure company, for good about the app itself. And so this basically was the start of the journey I've been on ever since.

and so I actually shared that pitch with the team at Realm because they had, you know, right around recently had launched the mobile. database and we were wanting to sort of rebuild a new version of the sync engine. And so they looked at that and said, Hey, that's very much what we wanna do, with our database, so why don't you come and, you know, help us, build it out. And so that at the time made a lot of sense. 'cause I was still not an expert in databases and sync systems.

This was all just, you know, we were learning on the job. And so it felt like being part of a bigger team that had an incredible set of experts was a better path to success than trying to do it on our own. so I ended up joining Realm and, was there through, through the acquisition with MongoDB. And ultimately this led to Ditto.

So it, truly has been just a, line all the way back from, you know, 2012 timeframe where I had that moment of, reflection of not wanting to spend my career, selling CRM software. That is awesome. And I see so many parallels for so many people in the local-first space, like whoever you're looking at who's building innovative things.

For example, like Aaron who is working on Xero, like, what they all have in common is that they wanted to build a me included, like, wanting to build a great app and then realizing, okay, what is holding us back to build this great app as we have it in our mind is having great, fast local data. And, and then you take like in your mind maybe short detour of like, okay, we're gonna build this data thing first and then we can build the great app. I'm still currently in a similar path.

This is also, like in a similar, like back and forth. This is what led me to build Prisma back then. Not yet with the intention of like having everything local, but to at least have a better experience working with data. But this is very much the path that I'm currently on where I'm trying to build Overtone. But for that, I also had to build, roll my own data layer. And a similar path is what, like Aaron has been on, and this has led to Zero. And the insight is always the same.

Like to build great app experiences, we need to have. great data experience, a great developer experience around data. And yeah, it's funny how like you're tackling the same problem over and over again. And every time you just like level up significantly and you're still saying true towards like the same insights and the same mission. So you've had now, basically three attempts and every time you're just like reaching higher and higher.

So maybe before we go into Ditto, maybe you can share a little bit more about when you joined Realm. Like what were your responsibilities and how did the technology that you've built in the first CRM app, how did that kind of translate or differ to what became Realm? And yeah. Maybe we can focus a bit more on the chapter on Realm.

Yeah. So, I was joking that I'm either crazy, you know, it's like that saying, if like you, if, you know, if you just keep doing something over and over, you know, either, either maybe you're gonna hit it right, or, or you're just crazy. And so I think the jury's still out on that.

and I do find, like everyone in sort of the local-first, community, all the different approaches with it, we sort of share that same sort of obsession of like, how could we just build this incredible infrastructure so that the ultimate apps that come out of it, are great. and so the, the transition from, the naive approach I would say that we had implemented, with the, that early startup, which was called Rubik.

it didn't really have anything innovative in terms of managing versions, is kind of the simple way to put it. Like we didn't really understand kind of anything about, like more novel approaches to managing consistency. And so, in addition to that, like. We were not database experts either. And so, coming to Realm was incredible because I was immediately surrounded by this fantastic technical team, including the founders that, had a deep understanding of how to build a great database.

And that's what drew me to it. Like Realm to me is a developer was incredible. I love like the object interface. It was highly performant. It was easier to use, you know, in terms of query, engine, compared to SQLite. And, I wanted to be part of the team that was building that.

And so, the approach that they ultimately settled on, which I kind of joined, right when they were in the early, early moments of designing the architecture around the sync engine was to adopt operational transformation, as like the algorithm, to sync. And so that was, at the time, you know, not something at all. I would. Feel comfortable trying to implement, on my own. And so, the Realm team had the smarts, across the board to implement it.

And so, we ultimately, that was kind of 2015 through 2018, you know, built that out. And the tough part about that, kind of in hindsight though, was that. I don't think, everyone understood the harsh edges around OT. And so it was very hard to implement in test and to really, you know, prove that it would be consistent in all manners. And so that just took several years and ultimately that had a big effect on the overall business.

and, you know, there were like a lot of lessons learned around that. And so, happy to kind of like dive in, on the technical side of what I may mean by that, but also, you know, the implications on the business, which ultimately meant like the company was sold, and MongoDB acquired it. And so those lessons both on tech and business is really what stuck in my mind, when starting Ditto.

Yeah, I'd be curious to hear on either both the technical as well as on the business side since Realm really, I remember it just being way ahead of its time. Really like And I think it's all started with this inside of like, hey, the mobile was just like, was fairly new back then and everything was about mobile, like the entire pars back in the service.

That became a thing driven through the momentum of mobile and then like to go to such a great extent to have like a local database that it syncs, et cetera, and then also going beyond mobile to go to multiple platforms that was really ahead of its time. but as the saying goes, like it takes 10 years to build a database, not just to build a database here from scratch, but also to make it sync. I'd be curious to, to hear some of those anecdotes. Yeah. Well, there's a couple things.

I mean, the part that Realm was incredible about was its interface as a database. And that was no small feat really benefited personally as a developer with the software being open source, like really being able to understand kind of how that was built. But they ultimately built their own storage engine and then had this really incredible API. But the, result of that from a, like a resourcing perspective meant it took a lot of effort, to maintain all of the different SDKs.

And so the Swift or objective CSDK had an API that, you know, wanted to look really good to those, you know, Apple Platform developers. And then likewise, there was a concerted effort to do the same .net or Java, you know, on, on Android. And so it meant that the outer layer of the database to create this sort of magical API, just took a significant investment. And so that was, to your point, like a hard problem in and of itself and being super popular, as an open source database.

It did, kind of command a lot of the team's attention and so to then add on and say, Hey, let's take OT, try to implement it in a generalized fashion to add this sync engine and produce a cloud service with it. That was a lot for a startup to pull off. And so ultimately that, meant really sort of never having the right amount of resources. and so to me that was one of the concerns is that you have to figure out the right balance of trade-offs.

because if you end up trying to perfect everything at once, and you know, and you're doing this as a business, like you might sort of just run out of time, in terms of building the business. And that was definitely part of Realms journey, which was difficult, you know, it's a venture-backed business, they've investors and so there's a clock, and managing how to use the engineering resources to solve the right problems that matter. Both for the product and customers, but for the business.

you know, there were kind of allocations, in hindsight that I think could have been done better. and a contributor to that is that just OT was so hard, to reason with, you know, I'm forgetting the, individual who, who wrote the implementation at Google, but I, you know, was sort of famous in saying, if I had to write it a second time, it wasn't gonna be any easier.

And so it was just a highly complex thing to reason about and, needed like property based randomized testing to get any confidence around. And so that just took a while and that was doubly complex because Realm had this complex data model, especially being an object database that had direct relationships as a first class citizen you had to deal with, how do deletes work in that scenario? and that no one had really done that before.

Like the implementation with, with Google Docs isn't needed for that. And that really speaks to the magnitude of the challenge here. Like I remember like experiencing Google Docs the first time and like how magical it felt, but like you say, that took like an absolutely brilliant team to, pull this off. And this was like for a very specific dedicated use case where you can make very specific trade-offs.

And now with Realm, you wanna build like a general purpose abstraction that allows for all kinds of different data models. So yeah, that, that's basically taking the same challenge and making it orders of magnitudes harder because you need to make sure it works for all those different scenarios. Yeah, absolutely. And that, was harder than anyone realized in hindsight, in my opinion. and the other side of it too was it just had technical challenges.

the actual algorithm itself, given that if you have a long history with devices that have been offline for a while, you're gonna have quadratic complexity to being able to transform the operations for the other devices.

And so that would manifest itself and it's always the worst times where you've got heavy load on the server or you've got this, this one errant device that shows back up with a long history and then suddenly, the server would itself run outta memory or, you know, um, basically, die in the process. And so, that was something that at the time we hadn't really fully solved. 'cause you have to then figure out how to prune histories to kind of prevent that scenario from happening.

But that's like a balance of, what does the application expect. and so I ultimately think those technical problems could have been solved, like there were solutions to it, but it was hard to do that under all the other constraints from managing the open source, success of the mobile database itself, plus just the general business concerns. And so that's ultimately, you know, what led to having to sell the company, and, you know, MongoDB acquiring it.

And so, but all of those lessons were sort of in the back of my mind, in terms of starting Ditto, because I felt really unfulfilled is the, you know, honest way to describe it. Loved Realm. The product felt like it really had the potential, but it got sort of cut short, in its life. And so the choice of joining Mongo and trying to, you know, fix and in, you know, continue to improve those things in a larger company seems somewhat difficult.

and that was part of the reason to just go start, Ditto was like, okay, like I've, done this once in a naive way, then start to really figure out how to do it in an advanced way, but didn't really get to pull it off, so could I go do that now? and so that was sort of the personal motivator, in starting Ditto was it felt like I had to just go do it on my own. Is the only way to really have the unconstrained ability to explore like new approaches. That is awesome.

And now you have like two rounds of practice rounds already under your belt, two notes just on the Realm side. it was sort of unfortunate news fairly recently that Mongo decided to sunset Realm, which I think led to another interesting business opportunity for Ditto, which we, maybe touched on later.

but another shout out is that we, at last year's Local-First Conf, we had actually one of the founders of, Realm give a talk Alexander, about like what made Realm so special, which what you've already, mentioned, which is like the surface area, that it feels native to whatever platform you're building for, whether you're building an iOS app, whether you're building a Mac app, an Android app, et cetera.

This is where Realm really had such a big edge and an advantage that felt native to whatever platform you're targeting. And I think that remains true to this day very much. And I think there's a lot to learn from there. Absolutely. I mean, I'm sort of like coming from Realm, like Ditto did to that trade off question. Like our interface mirrors much more of a traditional database, interface.

it's a document model and we have our own query engine called DQL, but it's, you know, more like a SQL interface and a sort of cringe a little bit because like it really isn't as elegant, as Realm, like Realms API was incredible. I mean that's, as a developer why I wanted to join the company is it's just like, wow, this is the product I want to use when building a mobile application. But the, but to that is, it was not without significant resources and effort, to make that, interface.

And in hindsight, that's where I sort of wonder from a business perspective, did that interface really give it, you know, drive the success of the business? at the time, it wasn't, it was very popular from an open source, but it wasn't really helping the business.

And so that was, The part that, I'm hoping eventually, you know, as Ditto continues to grow, we'd sort of be able to improve their API and stuff pull that, but we've had to sort of make the trade off that maybe that's not a day one problem, given the Realm experience. And I think this is really like what led to all of the success.

you're having so far already with Ditto where you're thinking correct about the trade-offs and like when you do things on day one and how you sequence those and you still have those ambitions to, do things eventually that you want to do. But I think, when looking at Realm, there's so many hard things like building the technology by itself. Like, let's even imagine we have more time, we have more resources.

Still super hard, but then to do it in this constrained environment where you need to build a business in the meanwhile, while you're, just like losing sleep over super hard, unsolved technical problems, that's just like really, really hard.

And I think having had the experience of seeing this journey once or twice, now really gave you a super unfair and to your credit, like a positive, unfair advantage to start Ditto as a business and to like solve as many technical problems as is currently, applicable and healthy to your business. So with that, I wanna learn more about Ditto, like, so you mentioned you started it in 2018. What was it?

Product wise, at that point, what was sort of like the first major milestones that, that you've reached and now since 2018, many years have passed and I think it's fair to say that you're probably one of the most advanced, biggest players in the local-first in the sync space. So there's plenty to learn.

Yeah, well, it's been a very interesting journey in the sense that a lot of early assumptions, actually were proven wrong. And so, excited to kinda share, those insights. And the first thing that sort of started the company was after that experience working with OT and just seeing all the challenges with it, I was immediately drawn to sort of the flexibility you could say with CRDTs, and kind of couple axes I guess for that.

one was it felt like just much simpler to build the actual merge semantics, like, you know, reading papers and stuff. It's like this is just not a lot of lines of code to get the same sort of predictable outcomes. And it was also just efficient, like these are simple operations to actually reason about how to merge data together with the trade off being that you have to increased the metadata of the system itself.

'cause it's like we are going to use more metadata to reason about, the different versions instead of using OT to me was always an algorithmic approach and that can be more efficient because you can send less data, but then the algorithm has to figure it out. But having seen all the challenges of having to scale, OT algorithmically, it felt like using more metadata was the right trade off, especially because there are other ways to sort of compress and manage that metadata.

So it, it felt like that problem probably is more logarithmic versus, quadratic or exponential where you had, with an algorithmic approach. And so that, that to me was just interesting coming from Realm. The other side of it though was like, Hey, this is a decentralized approach as well. this can work peer to peer. And that, in and of itself opened up new possibilities, that also built upon, experiences at Realm.

and so one of which is actually the part that Ditto is less known for, which is how to scale just the server side of a sync engine. because sync itself is stateful. You, can get into a scenario where client devices have trouble migrating to different, nodes in the backend. And that was something we faced at Realm was that it was very sticky, in sort of letting a device sort of reset. It's a state between the client and the backend. wasn't very straightforward.

but that's a lot easier to do with CRDTs because they're inherently, don't require like a central system. And so that was actually the main reason I was first drawn to it, was that stickiness problem. But the other side of it was like, well wait, if you can migrate around to different nodes in a server itself, why can't the nodes at the edge talk to each other? And that is, that aspect is really what Ditto is known for now.

but it, oddly enough came from looking at it from more of a server scaling issue, coming, coming from the experience at Realm. but the idea that you could connect devices directly. No one had done that before. And so that felt like, hey, if we could build that, that would really take this to the next level, and really be something I never attempted, you know, in the original startup or at Realm.

And so that, that was the part of where trying to do that at a larger company, felt really not pragmatic. probably the last thing someone who just buys a company wants to do is like, let, let some folks go and rebuild the system. So that was the genesis of like, this probably has to be its own independent company to kind of go and pull this off. And so in the first six months of the business, it was all around sort of proving is this even possible?

Because in my opinion, Ditto as a company or technology, it had a requirement on the edge, peer-to-peer aspect of whether or not the hardware had reached maturity to pull it off. like enabling devices to talk peer to peer is not a novel idea. That's, yeah. The internet was created because of that.

But enabling devices to do that wirelessly is very much a function of whether the hardware, was ready for that because like the original iPhone or early mobile devices were one underpowered, like batteries have gotten more dense in the years since, CPUs and memory have increased. and then the wireless protocols themselves, like Bluetooth is not the same protocol. It was even 10 years ago.

And so the early sort of six months of, prototyping with Ditto was very much a question of if this edge, peer-to-peer is gonna be like the key innovation that no one has pulled off. There was a part that we didn't really have control of, or which was. starting Ditto in 2018, like the right time, had the hardware actually, progressed to a point where this is, feasible.

and that's the, I would say perhaps serendipity aspect of it was that that was true like 10 years sort of from the original iPhone. Like, you know, 2018 was the iPhone 10. That was a supercomputer. and that was a trend line that we wanted to bet on was that those devices, even other form factors were gonna continue to become, really, really powerful and connecting them directly, would open up new forms of applications. And so if we. Pulled that off, like that felt like a really big opportunity.

So I just wanna linger on this point for a little bit since we're thinking about syncing your mind typically goes to getting data from A to B, and then you think about like, okay, that the data is correct, that it converges, how about like transactionality, et cetera. But we typically always kinda assume like getting the data from A to B of course. Like we have like a stable TCP connection and like it's all good. Maybe we're in fiber internet. but what you're realizing.

Okay. It is possible, like the data part seems to be like realistic, and, and feasible with CRDTs. This is where you still had sort of like your, more harsh experience with OT. CRDTs seemed very promising, and now you realize, okay, now from a network topology perspective, this is feasible that those things could exchange the data like directly. But I think what you've then realized is like, well, two devices don't just talk to each other.

That is, it's sort of like a miracle that the internet works at scale, the way how it does with like routers and network tables, et cetera. But now you want to kinda like jump beyond that and make the things talk to each other directly, which at the same time is, mind blowing that that's not the way, the default way how most things are that when I have here my iPhone.

And an iPad, like most realistically, when I wanna send something between those two devices, it's probably gonna leave my country. and this is basically the other hard problem you've kind of stumbled into, which is the networking. So if I'm thinking about like choose your hero for your new business, I was like, before you had like all the bars on like on data and algorithmic and now a new bar has popped up, which is like max out for network, for networking.

And that is, I think when I think about Ditto, this is, I think my mind immediately goes to, oh, this is the company who's like, has gone to all the extremes on networking and whatever networking way there might be. you probably have like an A plus implementation for that. Yeah, so that is ironic because, yeah, as I've mentioned, like coming from Realm, CRDTs were top of mind. And so this was summer of 2018. It was myself, my co-founder Max. we were just working on our own.

We hadn't raised money, didn't hire anyone. That came later at the beginning of 2019. And so we started working, with some, you know, third party, CRDT libraries starting to kind of build a prototype of a data store around it. And we were like, okay, you know, this is a hard problem in and of itself. but when, like we started to embark on getting the network layer to work. Wow. That was a very humbling experience.

so like Max sent was very focused on the data store in the APIs and I was like, oh, I'll get this Bluetooth thing working. no, no problem. And I just felt like, my whole job was just like clicking buttons. because you can't really simulate mesh networking, and that's, in hindsight, it was, I dunno, I guess maybe more obvious like networking is, you know, not predictable. and mesh networking itself is an optimization problem. So there is no like way to solve mesh networking.

and whereas solving data storage. There is an end state, it's like right, it predictably onto the storage device. don't corrupt the data. Like there is a solution at the end, but on the networking side it's all about balancing trade-offs and managing like the reactions to the environment that you don't have full control over. And so that was way, way harder than we anticipated. and our early version in 2018, you know, barely worked.

but it was enough that there were some customers that jumped out, specifically in the airline industry that had similarly internally tried to build something like this. 'cause they were trying to build apps that work in the plane where there's no internet. And they couldn't pull it off. And so even though our, initial prototype barely worked, the fact that it barely worked, was still enough to say, Hey, that this is a big deal.

you know, we would buy this product at, significant scale if you could actually make it work. And so that's what ultimately led to forming the company and hiring folks. But I mean, we had a prototype in like August and September of 2018. We didn't really have a solid functioning version until a year later. and that was really only on iOS. The Android one didn't come later, so I mean, it was practically two years.

And like 80% of that was all the replication protocol itself, the CRDT, the data store aspect of it became a backseat. We sort of said, okay, like, what are some kind of. basic semantics that people would want, like last right wins. And how, how do we implement that in a causally consistent manner? How do we provide an API where you can query and, you know, do, simple crud operations on this? But then it was like.

Put all of our energy into how do we make this work in these, wireless scenarios, specifically around Bluetooth and peer-to-peer wifi in mobile devices. So staying maybe at this timeframe for a little bit, if you consider the technology that you've had built out back then and today, like today, you probably have like orders of magnitude of more like capabilities and coverage for, for different like network protocols and topologies, et cetera, scalability.

and you probably already, to some extent at least kind of foresee those back then. You, probably, like in your head, you start to like sum up two plus two and you realize, oh, that means we should also do that and do that. But then it becomes like a scoping exercise and say like, Hey, with, how little can we get away? Building this out that is already valuable. so that's one thing that I'm curious like how you went about that.

And the other thing is going from something that like barely works and in most cases breaks to a year later, something where you feel pretty confident about, how did you go about building that confidence and how good, like, did you know it's gonna work? Or have you basically Yeah. How did you build that confidence?

Well, uh, no, there was still like moments in those first couple years where I still wondered like, maybe this actually really isn't going to be possible to the point on the hardware side. like this, well this is sort of changing now, but, to date still the only cross-platform way that devices can communicate, peer-to-peer would be Bluetooth or if they're on the same wifi network.

same wifi networks a little bit easier 'cause you know, you've got a, higher bandwidth stable network to a degree that's actually very much not true in practice, but, compared to Bluetooth, way more stable. But Bluetooth was the sort of the magical thing about Ditto. Like whenever we showed people the early demos, there'd be two phones on airplane mode and you'd click and instantaneously changes would happen. even non-technical people look at this and say like, what?

Like, I didn't, I didn't know this was possible. Like, this feels like magic. Have you done something to the phone? And so we ultimately put a ton of effort into trying to figure out how to make that work reliably. And so that to your point on like early design decisions, that choice to really figure out how to make Bluetooth a functioning protocol that you could sort of build enterprise type applications around. which is what, was the customer demand.

was a pretty tough constraint to work under because it is very unreliable. Every platform has bugs and errors in practice that are undocumented. Like, 'cause there are just like improper implementations of Bluetooth at the firmware level. and, you know, scouring Stack overflow or Apple forms or whatever. you'd literally find stuff that's never mentioned on the internet, in encountering it. And the only way to encounter that was to use physical devices.

You cannot run the simulator and find all this. So that really was why it was so hard, like designing a protocol around it that was optimized for the low bandwidth. That was challenging 'cause we needed to be hyper efficient, but at least that's something you can kind of like whiteboard out. Then going and proving to yourself, wow, this actually works in the real world.

Meant just a lot of tedious manual effort to the point that, I really just felt like a monkey clicking devices and sort of joked I should buy a, like a robot that would just click phones for me, as a way to test the software because there was no automated way to do it. And so that's the reason it took so long was to gain that confidence that the choices that we had made, would work in a reliable manner.

And it was just sort of, I don't know, sort of like hacking your way through a forest, with a machete. Like you didn't really know where you were going. You just had to just keep pushing, pushing onward with it. it was way harder than than expected. That is incredible. And like as a web developer, we kind of like, we don't know how good we have it with like tools like Playwright, which gives you like browser testing with like top-notch APIs, et cetera, works for multiple browsers.

We don't ever need to leave like the comfort of our chair and like we can just like use our primary device and then run things even headless. And you develop some pretty good confidence that stuff is gonna work. When you're building native mobile apps, then you're rather like you're flashing things on your phone. And then maybe you also have, like, if you're primarily using iOS, you might have an Android phone and maybe you have multiple Android phones because it's much more heterogeneous there.

And then you are already getting into that mode of like having a few test devices, but you're going way further than that because you need to like look for all the edge cases. So that brings me to a kinda funny question. I'm curious whether you have like a chart of like the number of physical test devices that you've kind of allotted over the years? Yeah, I mean we have so many devices, that we've built up over the years.

I mean, to the point that, I guess almost two years back now, we, we invested in actually building out. A, what you would call a semi anechoic chamber, to try to put a bunch of phones in a room, and run through simulations of them connecting up at this. It was up to a hundred devices. And so, Like there are, physical device testing companies that will run your app on a device, but they don't specialize in actually getting the devices in their, in their labs to talk to each other.

And so we basically set out to, build that. we had an intern actually who, was a mechanical engineer who actually designed the room and stuff. And so that's been sort of the fun part about this is like, I wouldn't have expected 15, you know, years later after being a software developer that this local-first obsession, would pull me into such a physical manifestation of it. but it was all from the, desire to, work with these peer-to-peer, like true peer-to-peer, wireless protocols.

and just the plethora, of different devices. And to be clear, like this is still true in iOS. Like you would think that Apple devices, their whole pitch is. We designed the hardware and the software, and so you would think across versions that a Bluetooth firmware would be the same and you wouldn't encounter weird behavior, but that's not true. It still exists. It's definitely way worse when you get into, you know, Android, windows, Linux, like the, broader plethora of devices.

But, even in the platform, you'd sort of expect commonality across hardware. you'd still encounter these, weird scenarios. and so we have like logic in Ditto that at this point sort of built out through that trial and error that it's not really novel like concepts, but it's just knowing when to smartly sort of reset the state of the system. which might even mean turning the actual network interface off and on at, specific, points in time.

and so people ask like, what's like the secret sauce of Ditto? And I, it's like, there's not one thing actually. It's, a bunch of like good ideas, but sort of layered on top of each other where each one like benefits the other.

And so like the choice of CRDTs was important because that opened the door to peer-to-peer, serverless, approach to datas sync, which then, you know, meant we had to do other things to optimize, the data transfer because we were working with Bluetooth and so we adopted, CRDTs, where you can send just the differences. so versus like an operations based approach. and so it's like that was a decision that affected then subsequent aspects of the system.

And so it's just a bunch of those smart trade-offs, which is sort of the reason why I call out the experience of Realm in seeing how then, these trade-offs, are really important both to implement this solution, a system in the right technical way, but also to manage the constraints as well with, business and, investors. Right.

And just like you said that you wouldn't have, believed 15 years ago, what this would lead you to today were, you're like, being exposed to so many different devices, so many different, like use case applications. You've mentioned the airline industry. You probably now see the world differently where it's like, aha, I know there's a device and nobody knows about this device, but this is critical and needs to talk to those other things.

I'm curious, like when you build all of this, like typically you don't get it right. like, it goes wrong, and then you need to debug it. When you build a web app, when you build a mobile app, you can at least like the road is like pretty well paved, so you get like log messages. You might get things like distributed traces, et cetera, metrics.

But when you're talking to, like, when you're talking about like literal physical devices where the network is part of the problem, so you can't just like SSH into it, how do you even get debugging data and visibility into what's going on?

Yeah, it's a great question 'cause I would say it's a topical problem that we're still, working through is as we deploy Ditto into bigger use cases, it's become sort of the second day problem of great that works, but how do you, know it really works? And the key value proposition of Edge being able to do things at the edge is you can build more resilient software. And so the customers of Ditto are businesses that have workers predominantly that are not at desks like us.

They're out doing jobs in the real world. And so in the real world, the internet is, by no means guaranteed. Cellular systems are unreliable. wifi systems are actually very difficult to manage at scale. And so being able to use smart software that leverages the network capabilities of the devices that workers are already holding is compelling. 'cause it kind of adds like another layer of, durability, to ensure their, you know, their application, their data, keeps working.

But that means if the whole value prop is to make the system more resilient, then Ditto itself has to work really, really well. And so how do you, ensure that? And so we've, we've had to build more of that out. and so it's sort of funny. There's, a, a feature we call the Presence Viewer. It's simple js application, that just shows all the nodes that are connected.

because every time Ditto, you start it up, there's a database interface, but behind the scenes it starts to take over Bluetooth, wifi, peer-to-peer wifi interfaces on the device. And it creates what you would say is an overlay network. And so the device itself will create the links it can to other nearby devices, but the system itself learns of the whole mesh because every device, shares, information actually through CRDTs about it's presence. So like who it's connected to.

And this eventually propagates around. And so we show this visually in this super simple application. It's just, you know, a bunch of nodes with colored lines showing the different connections. my co-founder built that, probably 2020, like it was just like a weekend project to throw together. But it ends up being like one of the most crucial tools customers use because they're like, well, how can I see what's going on?

Because if data, like if they're clicking their device and data isn't moving, I. The first thought is like, am I even connected? And so, that was sort of the first start in our realization that as great as the database itself needs to be to handle application level data, to make this really worthwhile as a developer platform, we have to provide observability capabilities as well. so that you can fix issues when they arise.

And so that includes now, almost like the sort of morphing into like a mobile device management platform where when you use our cloud service, you can observe, you know, the data in the database, but you can also request from the web portal logs from specific devices and they will transmit it up. And then you can, you know, go through them.

You can also see what are the settings and the configuration of the SDK remotely and even, like issue commands down to change stuff so that an administrator could say, Hey, I don't know what's going on. Let me turn Bluetooth off on a device. You can do that remotely.

And so those were definitely not our initial features that we were thinking about, but have come out of, ourselves having to debug this alongside our customers of like, yeah, this is a totally different way of building applications that doesn't rely on a server. And so you gotta have some way to actually go and get the, information on these devices to figure out what's going wrong there's a bug. That makes sense.

So in terms of platforms that you're targeting, we've been talking a lot about mobile devices at this point. Is this sort of the majority of deployed applications or are there other notable platforms?

Yeah, so all the core of Ditto, I mean, of all the different bets we made, one of the technical bet was betting on Rust. the 2018 was I guess, sort of a bet at that point in time. Now doesn't seem as as crazy, but, we picked that, for a couple reasons. One is, you know, we wanted to build something that could work across different hardware platforms and so Realm was in C++. So in a lot of ways, you know, all of the Rust benefits as a language felt like a, better choice.

But the other piece of it was. an annoyance that Realm at the time, which may maybe now would be more possible with WebAssembly, but, at the time they never could implement the mobile database in the browser. And so we were like, ah, let's not do that. Let's build this in one common code base and compile it to WebAssembly. And at the time, you know, Rust really was the language that given its history, was closely tied with WebAssembly.

And so, you know, Ditto itself is a Rust shop, and so all the core of the code base is in Rust and we compile it for different platforms. And then similar to Realm provide, a language specific wrapper around, the Rust Code through the FFI. And so, our customers though right now predominantly use just the mobile versions. So, swift, Kotlin, flutter, which, you know, we've recently added, in and React native, but you can also run in embedded scenarios. so like C# and the JS version of Ditto.

You can also run on a Raspberry Pi or you know, more IoT kind of scenarios. It just hasn't been as much of a focus, from a business standpoint. So those are more nascently used, compared to the, mobile versions. So going very hand in hand with the deployed platforms, you've been mentioning the airline use case before. Is this your most dominant use case or which kind of use cases do you see most application for Ditto at this point.

It's where we got started was the airlines. So, you know, I mentioned kind of that sort of regret feeling, when Ditto was started of not feeling like Realm really, was as successful as, I believed it could be. And so that, that led to, my background both engineering, both product.

And so in the early days when it was myself and Max in 2018, I told him, I was like, Hey, if we can't get someone to wanna buy this, like, I don't want to waste more of my like, prime professional career, like hoping this thing will be successful. 'cause I don't want it to end up where it sort of fizzles out. And that honestly is my, like, number one advice I would say in like the local-first effort the part that makes me most proud is something that like, could be long lasting.

just like, how can I leave my mark on the world and. My talent is in software. So like, how could I build something that could have a really long impact? And so to do that as a business means you gotta actually like, make money. And so in the early years, I was like, let's start prototyping, building this out. let's just reach out to anyone who might have any interest in sort of any aspect of the benefits of this.

And so we were looking at like all different to like, use cases of where this, platform could apply. And coincidentally, I had like a family connection to one of the major US airlines. so sent this email to, it was a fairly high up individual and he was like, well, I don't know who this is. so he basically pushed it down to, an intern, and was like, Hey, take a call with these two guys. Like maybe there's something here. It's a fantastic story.

'cause we, we put this pitch deck together and we were like, we're building Ditto. And it's the CRDT database and it's got all these amazing capabilities. And she was totally bored by like 80% of the presentation. And then at the end, because we had mentioned peer to peer, she was like, well, could it connect like just two devices together? And we're like, yeah, yeah, they can totally do that. Which was not true. Like, and actually do that at the time. Theoretically. Yes. It, could.

and she was like, well that's amazing. Like, we need that. Like, that will be amazing for our flight attendants when they're in planes. They could share information without having to rely on the wifi of the plane. And so they're like, come to our headquarters, like, we wanna see a demo of this. And so we. We're like, oh, well our schedule's looking really busy the next several weeks, so like, we're, we're gonna need to like plan this like a month later.

and ultimately then we feverishly went and, you know, got the peer-to-peer aspect working, as I mentioned with like the Bluetooth and stuff, but it barely worked. but what was really cool about that is like, we were at their headquarters and we were put in this like, nondescript room and, the woman came in, the intern, and a couple other folks and they were like, wow, this is really cool. Like, sit here, we're gonna go get some people. And they just left.

And we were just sitting there for like 30 minutes thinking like, what is going on here? And then lo and behold, like. The more senior executives of the company come marching into this conference room and they were like, let me see this. And everyone started poking all the phones that we had. We had this very simple app that you could, change the inventory values and it wasn't even working, like phones were crashing and there was this one woman being like, that one crashed. That one crashed.

But the whole time everyone was like, this is amazing. like, I didn't think this was possible. And so that was sort of the genesis of knowing that there was real customer demand for this. we obviously had to get it to work. but that's where we ended up saying, okay, let's build a company. let's go raise some money. was after that and ultimately led to selling to the airlines as, as original customers.

And so Delta, Alaska Airlines, Lufthansa, Japan Airlines, ANA, like major airlines around the world have adopted Ditto in that same use case where they have installed it as part of the mobile app that their flight attendants predominantly, are using. And so in a lot of ways, it's actually kind of like a CRM app.

this is an app that, you know, they have to log into to start their job, which is, you know, like their hours worked start when they, when the, you know, the plane is, is getting ready and it's how they, it's like their slack, for them. It's where everything they need to do, it's all the passenger information, it's meals that are loaded onto the plane. It might have safety issues and things. And so it's just their lifeline to their job.

But previous to Ditto, those apps were effectively single player games. Like they would download the initial data locally and then they would go into the plane and they could not actually share. And so they would make notes for themselves, but it was all private. And so with Ditto, suddenly now, like they have what we take for granted with a lot of the software we use at desk base, which it's collaborative.

And so, you know, they could message each other and you know, do other things that ultimately mean that they could do their jobs, more efficiently. And so this, is still a major use case for us, but you know, since then we've expanded into other things. But yeah, it was sort of a weird. vertical to start with. Given that like a year and a half into starting the company, we had the pandemic and the airline industry was, heavily affected by that.

but, yeah, it's, it, we were able to weather through that. Thankfully. That is an incredible story.

And just like, as a little random side note here, if you're looking at your LinkedIn history, it also mentions, I, suppose, in between some of your work chapters, you've been also helping out the TV show, Silicon Valley, and the story you've just presented about you, like walking into the airline and presenting those in front of the senior executives, that, like my mind immediately went to sort of like a scene from Silicon Valley. So this is, like coming full circle in so many ways.

That is incredible. Well, there is some similarities to the storyline that the individual who hired me at Realm was, ended up, him and another friend of his were involved in kind of more directly in, the writing. And so he. He asked me in like the third or fourth season, I forget at this point, which one it was, to be a technical advisor, which meant just making sure like the software code on the whiteboards and screens was semi accurate.

so it was, fun, but it was, is it ironic because I look at that show and there are definitely aspects of, especially the new internet that like, is definitely not like. Like totally disconnected from Realm. Like there were times where we said that internally at Realm, and even to this day, like Ditto is sort of a manifestation of that. So that's, it's not by accident. There was, there was some connect. Well, if you ever have to rebrand to Piper, we know where it's coming from. Exactly.

Yeah. So, well, I have so many more questions about Ditto, but I think we need to like split this up into a future episode at some point. And, also you will be giving a talk at Local-First Conf where I think we'll hear a lot more. I wanna just put the focus for a few more minutes on Ditto as a platform that I just get a better feeling for like what it would mean to build something with Ditto since when I'm looking at the website. Beautiful website by the way.

When I'm looking at the website, there's kind of two sync approaches. One is device sync and one is peer-to-peer sync. can you delineate those a little bit more and maybe anchor them in canonical use cases?

Yeah, so in a lot of ways, like Ditto is no different than Firebase or Realm. Um, like any sort of, embedded database synchronization, like we have SDKs, that are an embedded document database. unlike Realm, like one of the trade-offs we did was we leveraged an existing storage engine. So at at the end of the day, we're just storing data in a SQLite database on disk.

But we designed the API to be a document database so that you don't have to, at the database level manage schemas, which in our experience with Realm, which was very strict with schemas became challenging working across, teams that have like. Different iOS and Android teams, like sometimes big companies, they don't really coordinate as well. And so like Realm ended up being this like odd thing that forced coordination, amongst teams.

And so we thought, hey, schemaless document API would, be easier. And so if you try it out as a developer, you would see, hey, this looks a lot like in a lot of ways like Firebase. and so you read and write data locally, and then behind the scenes we replicate that data. 'cause at the end of the day, data stored as CRDTs and we replicate it to other nodes. And so the device sync system is basically the exact parallel to any server based synchronization system.

There is another version of Ditto, the server which we run as a cloud service that syncs data through the server. And so. That's not particularly super novel. I guess some of the CRDTs and things that we do with it, you know, has, benefits in terms of scale, but, the aspect that is Ditto is known for is that data can also move. You could say like. Vertically is to the cloud horizontally.

And so that's where devices, when you're using it in the peer-to-peer mode will connect through other transports to nearby devices. So in a mobile device that's, you know, Bluetooth like I was talking about, or peer-to-peer wifi, or the wifi infrastructure itself, it actually can use other radio networks as well if you're in device, if you are in a device that has some other custom radio.

and so we encounter that with some use cases where like a, a radio is plugged into the USB port of a mobile device, or in like an embedded scenario. And that's where all the smarts are. to your point on the networking side where we build this overlay network and devices can sync data, peer-to-peer. And so all of that, again, is, driven by the fact that. There's these, diff based CRDTs that are being transmitted around in a causally consistent manner.

and similarly to Firebase or, you know, other sync engine approaches, it's all query based. So what data you want is still defined by queries on the database. And so if you're, you know, building an application, you know, for like, the airline, you might have a query that say, I need data for flight 1, 2, 3. You subscribe to that query, which will. Give you a callback to let you know whenever data changes the database that matches that.

But that query is then transmitted to the other nodes, both to the cloud and peer to peer, and all the other nodes will continue to evaluate that query, with data that they have locally and push it to you. And so, that will happen again, you know, over Bluetooth. if a nearby device generates data, like a flight attendant adding a message, for, for a given flight, it'll get out to all the other devices, regardless of connectivity with the cloud.

and so that's where we've tried to make Ditto feel familiar in the way that people have experienced sync, systems before, from client devices to the server or cloud, and basically try to make that just map identically to that working in a peer-to-peer manner.

That makes a lot of sense. And, just as a funny side note, I'm, I must imagine that when thinking so much about the, airspace use case with flight attendants, et cetera, and devices talking to each other and like that you don't have connectivity to the cloud yet you are in the clouds. It's probably a funny situation. So, Adam, there's so many more questions that I would love to ask and I will ask them at some point, but I think. That for that we'll have to bring you back to a second episode.

I wanna say thank you for sharing all of this with us. this has been incredibly insightful, super entertaining, all of like those anecdotes that you've like experienced and gathered over like 15 years at this point. Really impressive. Really impressive. Like how far you've brought Ditto from the early beginnings, you and Max to now. Like, there's probably most like airlines are probably on a way to using Ditto at this point and probably many daily experiences that I, that I have.

You mentioned point of sales in a previous conversation. Super impressive. Thank you for sharing all of this with us. Thank you. Yeah, I'm really looking forward to the conference, and sharing more about the journey. 'cause yeah, like I said, this is something I 'm very passionate about and like, excited to be, you know, with others in the community and to the degree that Ditto can inspire others, you know, that, that makes me really excited.

'cause like I said, this is, this is all driven by that early desire of wanting this tool as a developer. And so, I'm hopeful that we can continue to push like the application development kind of standards forward. So that local-first is, more just the default scenario to building, client applications versus this, you know, historical RPC, you know, client server model. So anyway, I appreciate having me and I'm, you know, happy to, find feature time to continue to tell the story. Awesome.

Thank you so much. Thank you. Thank you for listening to the localfirst.fm podcast. If you've enjoyed this episode and haven't done so already, please subscribe and leave a review. Please also share this episode with your friends and colleagues. Spreading the word about the podcast is a great way to support it and to help me keep it going. A special thanks again to Jazz for supporting this podcast. I'll see you next time.