#12 – James Pearce: Tinybase

Well, can I just say that, you know, from the point of view of exhaustiveness, there are many, many unsolved problems in the local-first space. And I have a horrible feeling that it's easy to think that synchronization is the problem. And then once you've solved synchronization, you're good. Well, I don't think that's true. I think we all got hung up on, oh, once we crack CRDTs, we'll be good. no, no, no, no, that's the easy part.

It's not state management, it's the state of the state management in a way. It's like, should I be synchronizing this data? Because this person isn't logged in, or is logged in, or is online, or is offline, or has shared this document, or hasn't shared this document, and that is actually unsolved, I think. Welcome to the localfirst.fm podcast. I'm your host, Johannes Schickling, and I'm a web developer, a startup founder, and love the craft of software engineering.

For the past few years, I've been on a journey to build a modern, high quality music app using web technologies. And in doing so, I've been falling down the rabbit hole of local-first software. This podcast is your invitation to join me on that journey. In this episode, I'm speaking to James Pearce, the author of TinyBase, a reactive datastore library for local-first apps.

In this conversation, we explore how TinyBase works, including its custom query system, the various persistence and syncing integrations, as well as James plans for the future. Before getting started, also a big thank you to Rosicorp and Expo for supporting this podcast. And now my interview with James. Hey, welcome James. Nice to meet you and nice to have you on the show. How are you doing? I'm doing great. Thank you so much for having me.

Huge privilege to have a chance to come along and chat about all things local-first. yeah, very excited to be here.

It's safe to say that you're the first guest so far in the podcast, who's calling in from a boat, which I think is very on brand for what local-first software should enable. and I think Maybe there's a sort of romantic pitch for why you're doing local-first, or, or maybe not so much. I think we'll get to that in a moment, but the first time, I've took notice of you is like when you've been at Facebook back then taking care of overseeing a whole bunch of open source projects.

So would you mind taking a step back, briefly introducing yourself and sharing more about, your time at Facebook? Sure. Certainly. So, I'm James. I did have a career before Facebook slash Meta, fairly long time ago. I'm from the UK originally. I've been in tech nearly all of my professional life. Moved to California in about, 2010, and started work at Facebook and worked on a variety of things during my time there.

I was at the company for just over 10 years, but, you know, one of the highlights for sure was working on. The open source program, as I'm sure most of the listeners are aware Meta does have quite a large investment in open source whether it's on the client JavaScript technology side or on DevTools or on data infrastructure and of course now on AI, ML and LLM technologies. And so yeah, it was a, an awesome period of time for me. It was between about 2014 and 2017.

So I helped out with a bunch of the launches of things like React and PyTorch and HHVM and various build tools and a whole bunch of things that people are probably aware of. And yeah, it was a super exciting time, just in terms of helping to develop how the company thought about open source and what open source could do for the company. Wasn't the only thing I did whilst I was at Facebook. Also worked on Portal, which was the smart speaker device that the company had.

, sadly now discontinued, but, it was, that was pretty awesome working on, on a hardware project for a bit. And also spent some time in the data infrastructure team, working on some of the large data warehouse technologies, that also Meta is fairly well known for. So, yeah, one of the great things, maybe underappreciated, features of being an employee at Facebook is you, you do get to, move around a lot.

There are lots of mobility opportunities, and I took the greatest advantage of that, that I could, and worked on a wide, wide range of different things. but yeah, enjoyed, every minute of it and then in about 2022. So yeah, just coming up on two years ago, figured it was a, a good opportunity to try something completely different. And so, Moved on, moved on board Scout, which is the boat that you see me on now. And, sold up, my house and life in Silicon Valley.

And I've been sailing on the ocean waves ever since writing open source software, so that's where we are. That sounds quite incredible. And I mean, Facebook is certainly. One of the companies that comes to mind is like how to have a positive sum vision on how to do open source at scale. , I can't really think of another company that has had such an, such a positive impact on our web ecosystem. and in particular With React, with GraphQL.

I'm not as deep on the AI side, but I mean, the recent releases, et cetera, around Llama, et cetera, this is all, I think a great example of how a large company can have a very positive impact on the, the ecosystem beyond its own benefits. So it's great to hear that you must've had like a very positive impact on that.

Yeah. It's a lot of this. took time to develop. I mean you could argue we stumbled into some of these benefits inadvertently. I think early on it just seemed like the right thing to do. The company had been built on other people's open source when, you know, Mark Zuckerberg started the site. PHP, running on Apache with MySQL. And so there was always this sense of standing on the shoulders of other giants.

And so there was a, an almost cultural obligation to share back things like Memcached and, you know, some of the early things that the company was working on. but then it began to become apparent that it was more than just that obligation. It was also, starting to help lead the industry because as Facebook grew, it was experiencing problems that other apps or other sites weren't, just because of its scale.

And so it was like the advanced guard of finding out what happens when you build really complicated web apps. and so that was what gave birth to React, you know, at the time that React was launched. You know, most web apps were still MVC that was kind of the prevailing pattern and it was still, still super easy to have inconsistencies between state across a big complicated app.

And so, you know, React's innovation was, you know, how do we make sure that the, the user interface is always a pure function of data that's gone into it. And, you know, at the time it seemed heretical, but pretty quickly other people started to hit the same. scaling issues with, with MVC apps and, and, and realized that React was maybe a way out.

And that's the pattern I think at Facebook over and over again, whether it's data infrastructure or or even ML, you know, the, the company is experiencing problems that others will face, but haven't yet. And so it has a chance to maybe share back some of its solutions from the future, so to speak, so that others then benefit from those as they experience those challenges. Um, so that was, that was one of the big drivers, I think. Obviously, it helps with recruiting, right?

Because you're demonstrating the kinds of problems that you're solving and that gets people excited to come and help solve those problems. and it also kind of opens the windows onto the kind of code that people are building at a company like Facebook. And obviously, there are lots of world class engineers there, and there's no better way of learning that than seeing the kind of code that they're writing.

, and so it kind of gives you that little bit of visibility into the, the caliber of the, the engineers that are working there. And yeah, so there, there were benefits coming back to the company and just in terms of engineering brand and recruiting, that over time became, that became so obvious that it was like, okay, now we need to just keep investing in all of these things. Um, and it's now more than just a thing that we're doing for fun.

It's now something that's actually bringing real value to the business. Exactly. And I think there's no coincidence that now some of the most prolific, open source and tool builders, actually used to work at Facebook, like thinking of the people who invented GraphQL, now building other technologies or Matt Wonlaw, who was just on the show, was also at Facebook for, for a while. So I think really like this deep culture.

around great developer experience and very principled thinking, like marrying those two together. I think this is what, what makes tools really successful and sticky. And I think this is what the internal, Facebook engineering culture has sort of like brought to light. I will say it's not as easy as it sounds. there are lots of things you have to do behind the scenes and as someone that was responsible.

for the the overall program rather than individual projects, you know, I needed to think about how we I guess sell this concept internally, right?

Because if you're an engineering manager and you're running a team and they've decided they want to spend 20 percent of their time supporting their project on github You're like wait, but now i'm down 20 percent in terms of like my internal goals So a lot of it was a lot of the challenges were internally Making sure that everybody was on board with the, the macro global benefits, even if at a local level, it seemed like it was a cost. but we got around that in a variety of different ways.

, one of the ways that, you know, perhaps isn't obvious, but I will state it because I think it was, it was perhaps even innovative at the time, is make sure you're using exactly the same code from GitHub internally and I see lots of companies that, you know, are doing open source, but it's a fork that they've thrown over the wall from a couple of years ago. And now they've diverged and they're doing something completely different internally.

and that never works because of course that always gets forgotten about relative to the thing that's being used internally. But I think it's pretty awesome that you go to GitHub and you look at slash react, and then you go to facebook. com and you view source. And you will see exactly the same code, because the, the internal version of React and the external version are being synchronized commit by commit. and so that means that you never get out of sync.

It means that the engineers who are working internally get to see their value immediately going to the community and vice versa. The community gets to see their value, appearing on the world's largest website. So, those things are hard, like, infrastructurally to get working, but those are the kinds of things that, that then help the whole open source machine work inside a large company like that.

So in 2022, you mentioned you sold your house, you, like, made a drastic change to your life, and you moved onto your boat. how Did that lead you to local-first? It seems to coincide a bit timing wise to, to also, start like working on TinyBase as your primary project. Can you share more about that? Yeah, I guess with the benefit of hindsight, you can rework the origin story a little bit and say, Oh yeah, it was all intentional. it wasn't, of course. I had.

Some good milestones in my life you know, my, my kids were leaving home and had reached my 10 year anniversary at Facebook and the boat that we had been dreaming about, you know, was, was ready. and so it seemed like it was a good idea to do all these, these things at the same time in terms of my, my life choices.

but as regards local-first and TinyBase, which is the project that I've been working on that I'm sure we'll, we'll talk about, that was somewhat Coincidental, that was something that I had been, you know, thinking about for a long time. but I think, yeah, then once you move on a boat, the, a number of the benefits of, of building local-first apps suddenly become acutely clear. It's like, wait, Oh, sometimes I really just don't have a cloud. And sometimes even if I do, my network is pretty bad.

And it really throws into perspective the difference between the apps that I native apps, right, that do have much better local support and the web apps, which just now fail to load. And it really threw into stark relief the difference between them at the same time, as I said, I was working on TinyBase and maybe I can reverse the local local-first principle into into the origin story of TinyBase because it does seem to help support those kinds of things.

and then like almost within a month, I then read the Ink and Switch article and I saw the essay about Rffle. I'm like, okay, all my, all these things are converging and , and it all suddenly made sense. So there was a lot of serendipity involved in kind of weaving these narratives together. But yeah, certainly life on a boat does, does make you think about local-first a lot more than living in some, you know, high density, high network connectivity lifestyle.

So, yeah, no, it definitely helped inform my thinking about this and has encouraged me to, to keep working on, on TinyBase and other, other projects like it.

So you mentioned keep working on TinyBase. So something must've led you to start working on TinyBase to begin with, and later you oriented more around local-first, what was the impetus to start working on TinyBase? So, well, maybe I should just like rewind a little bit in case it's not blindingly obvious. I love building stuff and all of my life, I guess, I've been, you know, surrounded by computers.

I was programming when I was a kid and, you know, grew up always trying to create things with code and I guess the lifestyle of a manager or a director at Meta is that you tend to get a little further and further away from the code base, which I found sad because even though I love managing teams and people and helping grow teams, Organizations and so forth, like some part of me, like a primeval part of me still needs to be building something of my own.

And so, that, that was really the origin of just doing something, anything. I just needed to be able to keep building even if it was just a small project for myself. And then obviously once I left the company and I started to find I had more time, I was able to start investing a little bit more effort into it.

And yeah, the, the original kind of story for, for building TinyBase was that I had an idea in my mind of an app that would be really cool, but I knew that in order to build that app, I was going to need to have something like TinyBase. So, you know, without going into too much detail, you know, when we were at when I was at Facebook, I, I, I needed a way to manage the state of hundreds of open source projects, right?

And so we built some internal tools that allowed us to, Pull from GitHub, you know, the state of all of the repos, how many pull requests, issues, and so forth. And um, That tooling really made it easy for us to look at this huge portfolio and figure out what was going well and what wasn't.

But I felt like the world itself could benefit from something like that So I thought maybe maybe there's a there's an app out there somewhere which would be , a GitHub at scale kind of interface that would let you manage your open source portfolio. And so anyway, I thought, well, if I needed an app like that, and I'd need to put it in a browser, and I'd need to have some kind of way of storing the tables of repos and issues and pull requests, you know, how would I do that?

Like, what is a good tabular data store that runs in the browser? And there isn't, well, back then anyway, 2021 2022, there wasn't really an obvious way to do that. And so I thought, well, rather than building the app, I'll start off by building a state management library that allows me to store tabular data that in the future will store this data that I need in this app that I haven't built yet. And yeah, that, that, that's where TinyBase originally came from.

I knew it needed sort of database semantics, but didn't actually need to be a database, right? It could still be considered an in memory data store. and I'm sure everyone's familiar with the idea of a, you know, kind of a store in a React app. But I wanted it to kind of look and feel a little bit like a database. I wanted it to feel like it had tables and rows and cells.

I wanted to do queries against it, and I wanted to be able to do aggregates of the data so I could count up the number of stars or commits in a repo and all those kinds of things. so yeah, that was the original, idea of TinyBase, you know, something that wasn't a database, but sort of had database conceptual semantics. and that's where it popped up. And then of course I stumbled into the Rffle And you know, Ink and Switch material.

And I obviously spotted that you were doing something similar with, with a SQLite approach, which is obviously literally a database. So you literally get the database semantics. but yeah, by that point I'd kind of committed myself to doing an in memory approach. So I thought, well, let's, let's try both. , and I, and so I kept pushing, pushing forward with TinyBase. Now the story has obviously evolved a lot since then. TinyBase is now a lot better about integrating with real databases.

We can come on to talk about that. Um, but yeah, that was, that was the origin story. It was just something I needed for myself. I thought the world needed, and I went and built it. And then once I found I had more time, I guess I kept building. that is amazing. I think there's a lot of similarities in terms of your motivations, et cetera.

Maybe a few different technical decisions along the way, but for me, starting to work on Overtone would have not been feasible without having a very substantial State, local state management foundation to build on top of. And this is how I then like got to know Jeffrey, et cetera, and started getting involved with Rffle.

So it seems like there is a clear parallel to you wanting to build your own app and realizing, okay, that's going to be terrible working on that app without those primitives that you have in mind. So obviously you got to start building your, your own state management system first. so, and kudos to you. I think TinyBase is one of the most polished and, sophisticated solutions that is already out there where I think a lot of technologies are, maybe have really great ideas, but are still.

pretty early in terms of being approachable and having the supporting resources around it. So, for example, LiveStore is not yet open source. It's available to GitHub sponsors, but it doesn't have documentation yet. It doesn't have guides yet. It doesn't have, like, all of those, supporting material around it. Whereas TinyBase has a gorgeous website. Really polished docs, lots of examples, lots of guides.

you keep, releasing regularly, like new versions, integrations with different other technologies. So really, really exciting how far you've come. I'm now looking forward to trying to understand better the different details. Right. Well, just firstly, a bit, a little bit of a meta point on the, on the kind of the polish. so a side kind of motivation for this was to see what it was like to build software where I didn't have to make any major compromises.

and I'm sure you and, and, and the people listening or watching, you know, have, have written software where you're under a deadline or there's some set of constraints and you just start cutting corners, you know, ah, you know, this bit of code is really crappy or, well, we didn't have time to finish the test coverage or, no, just, we've got to ship it, right? Cause that's, and actually that's fine. Right. I mean that, that, that's life. I always feel like some.

Sort of perfectionist part of my brain is always like, ah, like next time, next time, I'm going to make it perfect. Next time, you know, the architecture is going to be perfect. Next time the test coverage is going to be a hundred percent. And so given that I now had this project that only I was working on, I was really only my only customer. I had a bunch of spare time. I figured.

, if I'm ever going to have a chance to build, quote, what I thought was perfect software, and of course that's subjective, it's just my view that it's perfect then this was going to be it. So, can I build a piece of software that has, you know, 100 percent test coverage, is refactored, like, so iteratively that eventually it can't be refactored anymore. , it's got all the documentation you could ever need. It's got the best website that me and my crappy web design can manage.

Um, and I, you know, I don't have a deadline, right? I don't have somebody breathing down my neck. I don't have some investors I got to keep happy. It's like, no, it's like, I'm going to polish this thing. It's, and it's a beautiful, feeling of kind of craftsmanship, which I have rarely had before in my professional life. You know, often you feel like you're a tradesman. You're just like knocking stuff out for a customer or for a deadline.

But suddenly I could feel like, sounds pretentious, but almost like an artist, right? You know, artisanal software that's like perfectly polished to my, to my specification. And that was an interesting journey in its own right. By the way, it's also a curse because you could just end up polishing and polishing and polishing and then forgetting to actually share it with you, share it with the rest of the world. So, you know, there are downsides to this as well.

But no, I always wanted to invest in test coverage, and documentation. And by the way, and this is an interesting side story we may not have time for, but like, as a result, you build lots of tools to help you do that too, right? So I've now got tools that help me generate the documentation, tools that help me generate test cases. The documentation, as you identified, has lots of examples and sample code. Each piece of sample code is also a unit test, right?

It'll actually take the code out of the documentation and run it which is obviously stuff that I had to go and build myself. but that was all me, you know, thinking, well, I don't want to, I don't want a piece of code in the documentation that doesn't actually run. Like, you know, what happens if I refactor the code and I forget that it was in the docs? I know exactly what you're talking about. There's a little thing that comes to mind.

I'm not sure where I've seen it, but I think it was along the lines of like a yuck yuck tree, like the, the concept of like yuck shaving and like a tree of dependencies or like how you would try to do one thing. And to do that, you had to like diverge into a different side project and then your side project has a side project and you see like a dependency tree of Yuck Shaving. So I know exactly what you're talking about. That's right. That's right.

And each one of those tools for the tools for the tools, each one of them has to be perfect as well. Right? So, anyway, it's probably a fractal thing. , but yeah, that, that, that itself has been a really interesting journey trying to build the, the software that I really always yearned to be able to, to build even as it scales, which is, you know, tricky to be honest.

And obviously once people start using it too, because then now you've got opinions coming from people outside saying, Oh, well, you know, you might think it's perfect that you do this, but we don't like it like that. You're like, ah, now I've got this dilemma. Like, am I going to break my perfect thing to do the thing that actually a customer wants? Well, of course, you know, because if no one's using it, then then what was the point?

So there is a a whole set of interesting trade offs that you then start, start making. But no, just again, true to its name, I would say the most interesting part of that is how did, how do you make it as small as possible? and I, maybe that's not so important these days in the world of, faster internet connections, et cetera, and good bundlers. But like, I wanted to see, like, could I make this like as almost, you know, infinitely as small as I could make it?

and so I spent a lot of time thinking about minification, a lot of time thinking about. you know, how to refactor the code in a way that, you know, made it compress really small and, you know, it's still, it's still pretty small for what it does. so I'm pretty pleased with it.

That is awesome. I highly encourage everyone who's watching this to also check out your beautiful website and the documentation, the examples, et cetera. I think you've laid it out very clearly. but given this is a podcast where most people listen to, would you mind in words, trying to describe what TinyBase is, like, for example, in the context of a React app. Okay. Well, firstly, TinyBase. org will do a better job of explaining it than I will do now, so definitely go, go check that out.

, it's very nice that you say it's beautiful. I'm no graphic designer and I actually think the website could do with a bit of love. So if anybody would like to come along and make it look a little nicer, they are very welcome to file some. CSS pull requests on me, I would be very happy to take those. But, that all said yeah, look, the basic idea or the strap line is that TinyBase is a reactive data store for local-first apps. And I'm very careful to say data store, not database.

I don't want anyone thinking that it literally is a database. it's a data store, but it just has characteristics that are perhaps familiar to people who have thought about relational databases in the past. I also say that it's for local-first apps because I'm riding on the back of the cool local-first movement that's happening right now. I guess there's no obligation to only use it on local-first apps.

I think, you know, even a permanently online app that wants some tabular data store in the browser would still be able to use TinyBase. But yeah, that's the principle. It's in memory. So the data is either loaded up from some local you know, to the browser, some local storage, or session storage, or IndexedDB, and then it's loaded into memory, and then you build your app on top of it in a way that is fully reactive to the changes that happen to that data store.

So, in a way, you have delegated all of the data provisioning, loading, synchronization, whatever, to TinyBase, and you just build your app as a pure function of the TinyBase contents, and I will take care of getting all that data updated in the background for you. Whether I'm pulling it from the web, or I'm synchronizing it with CRDTs, or I'm pulling it out of a local store, or I'm firing it over PartyKit WebSockets to your friends you know, I'll do all of that.

All you have to do is build a UI that hooks into tables or rows or cells or key value pairs in, in the TinyBase data. and your app will always be reacting to, to those changes. So that's the basic principle. It doesn't have to use React, incidentally. Like, there's nothing to stop you just, you know, listening to changes in the data directly, and then just, I don't know, document dot writing out stuff if you want to.

But React does work well because, obviously, that also has a sort of reactive principle. And so you can use hooks to attach your components to any of the elements of the, the, the store. And by the way, you can do that at any granularity. So you could have a component that was just watching one cell of one row of one table, or you could have a component that was watching a whole set of tables, or one set of key value pairs, what have you. Lots of different ways to listen.

And you know, TinyBase will make the, or do the job of, you know, alerting you when just that thing that you've looked, you're looking at has, has changed. And the same then goes for queries against the database, by the way. So if you want to do something more interesting than just watching raw data, you also want to look at aggregates or you know, relationships between data, or in fact, just queries. construct arbitrary queries. You can then attach your app to listen to the results of those.

And again, I'll take care of all the reactivity behind the scenes and your app will update when, when when those results do. So I guess when I explain it like that, it sounds like TinyBase is doing quite a lot. and that's probably why, you know, we're two years in and it's taken me quite a long time to get to this point. But yeah, that's the basic principle.

And I would say, I think this is kind of an interesting approach to apps, whereby You know, how the data gets into the memory of the app is kind of, like, delegated away. And you, as an app builder, now don't need to worry about things like going on or offline, or things like synchronizing, or things like storing. Like, you should expect your data storage layer to kind of handle that for you, and you are then just painting your app out of the palette of data that is provided to you.

because As I'm sure you know, and I'm sure many of the listeners know, the acts of synchronization and storage and dealing with all these network conditions is just horrible. And rather a small number of people deal with those problems than have everybody trying to build a local-first app having to re solve those problems is probably the long way to go. and so I, I, I guess that's what I'm hoping happens.

And it's not just TinyBase, of course, there are lots of other people in this space, yourself included, but, you know, many other vendors at this point who are trying to crack these problems. and I think it's because it's hard and we know that we have to get this layer of infrastructure going before the, oh the local-first ecosystem, you know, really takes off. I suspect that at the moment we may have more vendors building solutions than we actually have apps. That's.

Bit of an offhand assessment there, but, you know, there are a lot of people trying to solve these problems. And I think it's because it's genuinely hard. And what I really hope is that we can make it easy enough collectively that that people can build these apps much more simply. And you get the benefits of local-first without having to have a degree in computer science for synchronization. That's what we're all trying to get.

Totally. And, and I think we're getting really close to, to that where I think right now the de facto, solution for state management on the client might rather look something like MobX or some people might still like use Redux or Redux toolkit, et cetera, or, or like React query. and I think it's hard enough to build an app that works in the right way. And most developers don't, dare yet to think about persistence and working offline, trying to make it work, just like that.

And I think a technology like TinyBase can really, raise the, the floor quite a lot in terms of what a state management solution can give you out of the box. It gives you all of the things something like MobX does, but also on top of it gives you persistence, gives you offline functionality, gives you syncing. So I think we just, yeah, raised the floor.

So in terms of the different responsibilities, I'm curious to hear a little bit more, and I think what stands out about TinyBase here is is that you really went above and beyond to make things flexible and pluggable. So for the persistence path, I think you do allow a whole different range of different options to persist, whether it's IndexedDB or, or other options as well as for, for the syncing.

So would you mind sharing a little bit more about the details and the thoughts you have around that?

Yeah, certainly. So in the very first instance, I was just jSON serializing the content of the store and writing it to local storage, or to session storage, or to file, because you can obviously run TinyBase on a node server as well.

And, that's great, and in 2022 that seemed like the Basically, you know, when you're thinking about small stores anyway that seems like a fine array of ways to do it, but Obviously people start to push the boundaries of what you can store in those media and just serialize JSON isn't necessarily the way to go So I think IndexedDB was probably the next one that I tackled was not as easy. , just Just as a small side note, IndexedDB is not the most enjoyable API to work with.

Most notably, it lacks any form of reactivity whatsoever that I know of. I think there were some proposals that never went anywhere. But there's no observability. So it's kind of Crappy to have to poll for changes, but , yeah, I do now have that as well. So you can either save your in memory store to IndexedDB or load it or automatically save to it when changes happen or automatically load from it with a polling mechanism.

so that, that, that, that's there as well and and then around the same time, in fact, it was Matt Wonlow that I know it was on your show a little while back he was working on his CR SQLite solution and I started to see that SQLite was starting to emerge. as as a browser opportunity. And so I figured, well, you know, there'll be people who want to store it in the browser on, on a SQLite database as well. So I went down the path of supporting that.

So the tables that were in your memory store in TinyBase can now be literally written out to tables in a SQLite instance. Or read in, and again, reactivity is not perfect. , some APIs are better than others, but you know, do what I can to, to keep that as synchronized as possible. And that then opened the door to support things like PowerSync and, uh Turso and ElectricSQL. , I also did.

PartyKit along the way, which is not a SQLite store, but it's a kind of a WebSocket based way of storing data up in a durable object, and so it supported that also. So yeah, there was a period, probably over the last year or so, that the version 4 series of TinyBase, if you like, where each major release was me adding support for one of these new storage technologies or synchronization platforms.

The other two that are very worthy of note are AutoMerge and YJS, so those are more classic CRDT technologies, but you can store TinyBase into a Yjs document, or load it from a Yjs document, or into an automerge document. And then, of course, you've got the benefit of those two platforms synchronizing in whatever way they want to. So, yeah, I guess you could say I, am providing a lot of options. Maybe too many.

I, Suspect that when you go to the website and you see all these baffling options, you're like, Oh yeah, but which do I actually choose? so what I'm hoping to do actually fairly soon is write a guide for, you know, this is the kind of app I'm building. This is the kind of platform I should use, or this is how I should synchronize. Do I do the synchronization myself or do I rely on some third party to do it?

And at the same time, obviously we've seen companies like PowerSync and ElectricSQL, you know coming forward with their, their solutions here. And so, yeah, if people are using TinyBase and they want to Provide a a gateway to those systems, then that's that option as well. So yeah, I have tried to be as flexible as possible.

One of the privileges, I guess, of working on this as a hobby is that, you know, I can be open to working with as many of these partners as possible, whether they're commercial startups or just open source projects, right? I'm not in competition with anybody. And so if I'm providing a way for developers to onboard onto these different things, then that's great. I definitely have a view that the, the tide is rising up.

I'm saying that on a boat but the local tide is rising, local-first tide is rising, and I'm happy to see all boats go with it even in the long term. , even if in the long term TinyBase becomes not needed because, you know, all of these other platforms have, have matured to the point where they're offering what I already do.

But I think in the meantime, I'm hopefully providing a nice, disambiguation layer or you know, something like that to help people decide what their approach is going to be without having to rewrite their app dramatically. So yeah, today you could store your TinyBase data to local storage and tomorrow you could then sync it to ElectricSQL. Why not, right?

So in terms of the data that is persisted, let's say to IndexedDB or OPFS, SQLite, or synced to one of the syncing providers, how do you bring the data into memory? And do you typically, let's say I have a, let's say I'm using Turso and have my SQLite database somewhere, Is TinyBase automatically, hydrating all of that data from the database into the in memory version?

Or is there some sort of scoped version where maybe the database is two gigabytes and you only can constrain yourself to 200 megabytes in memory? how are you handling that? Right. So for all of the database based persisters, I call them, there's a configuration where you say which tables you want and, you know, whether you want to just load from those tables or whether you also want to be able to save back to them.

the worst thing I could think of would be somebody with some large production database and they connect TinyBase to it and the next thing you know, it's tried to load the entire database into memory or it's tried to write No, back to the whole database, and then, you know, that's the end of the world. So no, it's all very configurable on a table by table basis. And the one thing I have not done, but I'm pretty sure I need to do, is also provide some kind of pagination or filtering on that view.

because if you've got a database that has, let's say you, you, you shard it on user or something, and you only want to be able to see just the data from that user, which, you know, it's gonna be a pretty common use case, then make sure that persistence is not for the whole table, but just for the relevant rows, of that table. I think it's going to be harder than I think it is. I suspect there are lots of gotchas to doing that.

, and so for now, there's my, my solution or my suggestion will be that people doing this should probably have a per user database, which is kind of an interesting approach that I know some people have started taking anyway, for, for edge based, databases. Um, so that, you know, you're, you're in no danger of, accidentally loading data from a different user into someone's browser. But no, that's definitely something I need to tackle.

So it's scoped, to answer your question, it's scoped to individual tables as to whether it's load or save or both. But I think over time we need to add more pagination. So that you are just looking at, you know, the top 10 records, 100 records, or what have you. The reactivity gets really tricky at that point because I'm sure you know, you know, SQLite's reactivity out of the box is pretty weak. At best, you're going to know that a table changed. And what are you going to do?

Query the whole table to find out which row it was that changed. And so knowing that it was row 47 out of 2 billion that was the one that changed. is not currently something that the database platforms provide by default. Um, So, that's one thing where I am really having to work around those limitations right now.

, and I'm hoping, if I can't make it clearer to you, I'm hoping that people working on SQLite are going to crack this and provide, you know technologies that make it easier to synchronize at a much more granular level, or get reactivity at a much more granular level, so that TinyBase isn't having to poll everything in a way that doesn't scale.

Yeah, I mean, I, I think, Ideally for SQLite, the most principled and foundational approach would be to rebuild SQLite from the ground up to be incremental and to be reactive. And I think actually a former coworker of yours, who is um, Julian, who has also created, like HipHopVM at, at Facebook and, and HackLang.

he's actually trying to, to do that with a new startup called, Skip that is a very ambitious, or even audacious goal to build the whole new database from scratch to be reactive, from the ground up. Which I think is very, very interesting, but short of that, I think you, you're left by imposing, reactivity, primitives on top of SQLite and the best you can do there is to do the minimal amount of work, you need at a given situation.

So instead of pulling the entire table that, you know, okay, row 47 has changed. And that requires quite a bit of like a reactivity system on top of it. And so luckily with, a signal like system, that is a, it's a pretty good, primitive to put on top of it, to implement that reactivity. But, yeah, it's not as efficient as you, if you'd build it from the, from the ground up. That being said, SQLite is so fast that you get a lot of performance benefits out of the box and you can get away with it.

Yeah, that's true. you know, certainly on the client. Um, yeah, I, I, without, you know, sharing too much, internal stuff, there was a big push, at Facebook. I'm thinking six, seven years ago, you know, to think about reactivity for the entire Facebook stack, because this concept of, you know, taking the ideas of React that were out on facebook.

com and bringing them all the way back through the web servers, which are HHVM, all the way through to the data stores that were being used at the company, you know, like it's a utopian vision, but like, if you can make that push based all the way through. That's spectacular. and you know, Julian and I were probably in many of the same, uh discussion groups and, you know, internal Facebook groups where a lot of these ideas were being bounced around.

And, and so yeah, when he left and went off and did, Skip and you know, I guess maybe some of this informed my thinking about TinyBase too, you know, that dream is still very much alive. and yeah, it's coming from, ironically, it's like it's coming from React. That idea is coming from React. It's just like, well, okay, can we bring that idea a little bit further back, and then a little bit further back, and then a little bit further back? Oh, like, and then we end up on the database.

Like, why can't we get the database to be, you know, telling us about these changes? And, yeah, I guess that would be the vision. Unfortunately, I haven't checked in recently, but I think SkipDB wasn't Wasn't at a level where I could start working with it, when I last looked. Um, so yeah. I think it's not yet fully production ready, but as far as I know, they're just still working on it.

So in terms of the query layer for TinyBase, you have, not implemented SQL as a query language on top of your own data store. Um, so yeah. But you've provided, given that JavaScript or TypeScript is the primary way how you interact with it, you've just embedded a DSL, as a query language directly into TinyBase. So would you mind briefly describing that and, how that describes a user experience or the developer experience? Right, right.

So the, the journey here for me was that first I knew there were going to be some very simple query like primitives that I wanted to have. So one is, like a metric, which is some kind of, you know, number derived from the content in a table. And I knew that that was going to be the first thing I wanted to have if I was building this hypothetical GitHub app.

I was going to want to have a number in the top that said number of repos, and I was going to want to have a number that said number of commits. And so I actually baked in. a kind of a metrics oriented DSL where you can define a metric which is, you know, basic aggregates, counts, averages, sums, those sorts of things, which by default is normally just counting the rows in a table, but that is then a reactive entity.

So again, you can say, I just want this span in the top right hand corner of the window to just always show whatever that metric was, and then you can forget about it. And if that table is updated, You know that number's going to change, TinyBase will take care of it. another thing that I knew I was going to need to do was relationships between tables. So I built a very specific thing for just one local table, one remote foreign table, and then like keys between them.

So the value of one column in one table is used as the identifier for another. And so, yeah, you want to see the issues just for this repo, or you want to see the repos just for this user, then you'll be able to do those kinds of things. So those are, were actually baked in pretty early to TinyBase, just because they seemed Very important, but I, I guess I knew eventually people were going to want to push that on and start doing things like arbitrary queries.

And I toyed with the idea of, you know, supporting SQL in some form. And I should say, I love SQL. I have been writing SQL most of my professional career, which you know, goes back a fair way. And my most recent role at Facebook was, you know, the data analytics team in the data infrastructure org. So you know, we were working with SQL all the time.

Perhaps despite that, or because of that, I kind of got nervous about doing an arbitrary SQL parser, evaluator, executor, and then being able to make the results reactive, which was like, that's the, that's That's the non deniable requirement, like, it has to be, reactive. So, I didn't really have a lot of choice, but, oh, and by the way, you know, a full SQL parser with all the dialects you'd want is, like, it's gonna quadruple the size of TinyBase. It's just, it's code based.

So, no longer tiny. Yes. So, I wanted to come at it from a slightly different direction. I know I'm not the only person to have thought of doing more like a DSL approach to this, but I, I wanted to see whether I could capture as much of the, the valuable parts of SQL without turning it into a place where people could build stupid cartesian joins and kill their app. so I wanted to make it kind of a little bit on rails. And yeah, it was the idea of TinyQL was born, which is kind of a stupid name.

But it's, as you said, a DSL that allows you to, you know, select, where, group, limit. , actually, no, I don't have a limit in there. I limit a later part in the pipeline. But, yeah, basically allows you to join tables together. and then where them, there's havings and groups and those kinds of things too, which like really is what 98 percent of people want to do with SQL anyway. And haven't really come across too many queries that I can't express with these five or six quote keywords.

Yeah, the cool thing about SQL is you can do anything. The awful thing about SQL is you can do anything. And, I have worked with enough people who thought they knew what SQL did and didn't quite know what it did to, you know, see the trouble you can get into. So I really, yeah, wanted to make it a little bit more constrained than that. And it's, it's been, it was a real journey. That was a very tricky, for me, anyway.

, that was a very tricky thing to build, especially making sure that all the results are reactive. Because, by the way, you know, any of those tables could change. Any of the rows in those tables could change. In fact, you don't want to react to the changes of the results if some of the irrelevant rows change. You only want to know if it was the relevant rows that changed. So you have to set up listeners on all the things that were relevant and not the things that were irrelevant.

so that ended up being a little harder than I thought, but yeah, pretty proud with how it turned out. And as a result, and I would urge people to go check this out, You can build kind of interesting analytics apps, right, with this, in the browser. So if you go to the TinyBase. org demos, you'll see there's a demo. I think it's called car analytics or something like that. And it takes a data set of car data, different models, different makes, miles per gallon, years.

So there's a bunch of numerical measures and there's a bunch of dimensions. And in the browser, it loads all the data in pretty quick. It's not that much data, but you know, loads it in pretty quickly. And then you can do groups, sorts, filters, across all of this data in the browser. And it's like, it's completely instant. It's like, great, love it.

and look, this is not going to be great for querying the Facebook data warehouse, however many hundreds of petabytes that is these days, but it is going to be fine if you've got 10 or 100, 000 rows in your browser. And Don't be scared, people. Like, getting a hundred thousand rows of data into a browser is not hard. The browser is perfectly capable of doing that. You know, most even phones are very happy to run that kind of sized data in, in memory.

so yeah, for a lot of even analytics use cases, you can do, you could, you could build interactive dashboards with this kind of technology and have a sort of constrained set of queries all running off TinyBase. And it's super feasible, and it's been. It's been fun to build that. Whether or not people are using it in the wild, I don't know. I should say, I don't put any instrumentation into this product. So who knows what people are using it for, or how they're using it.

But I hope that those people who use it do, do find the value of being able to run queries, without, shooting themselves in the foot with SQL. Did you follow any sort of prior art in regards to how you built that reactive query system? So, as I was working on a similar part as, the work on Rffle, there, there was, some prior art in regards to Adapton and mini Adapton.

And so this was, some inspiration, that we followed for the reactivity system and we've like over time learned that, it's very similar to how now like signals is all the rage these days. It's like very similar in terms of the ideas. So did you follow some, some similar paths? No. And I wish I had. I think it's one of my weaknesses. One of my weaknesses is that I look at a problem. You know, it's like the famous Hacker News thing. It's like, I could have built that in a weekend.

I, I look at a reactive query system. I think, yeah, I'm sure I could build that. And then like two months later, I was like, I wish I'd read a bit more. I am not good at prior art and reading scientific papers or even just, you know, looking at other projects to see how they're done. I think, you know, I know what the result should be and I'm going to try to build it myself. And then I get into trouble. Fortunately, I worked my way out of trouble.

If it makes you feel any better, even with knowledge of the prior art, it still took easily more than two months just to get anything working. So, Okay, that's just, that's just my, my weakness that fortunately I was able to work around this time. But yeah, I think the other thing is that I obviously knew in intense detail how the reactivity of the underlying tables, rows, and cells were right.

And so I knew how I was gonna have to cascade the listener trees into all the relevant parts of the database. And, That was, yeah, that was just something I had to figure out for myself. The other thing I alluded to earlier is that TinyBase is just draped with test cases. Like, I am exhaustive, overly exhaustive. I'm completely OCD when it comes to test cases. It's not quite TDD. I don't always do the tests first, but, you know, I do eventually try to make it look like I did TDD.

So I built every possible query I can think of, and I build all these demo apps, and I build all these sample, you know, fragments of code, and they all get tested exhaustively, and so I figure out, well, once, once, once those work, I'm probably there. So that, that was my benchmark, and I got there. It's probably, I'm sure it's still possible to break it, but, you know, I think, I think I'm about where it needs to be.

So one of my favorite problems to, to think about, and it's a love hate relationship in terms of technical problems, which are schema migrations and schema evolution, which is something I thought a lot about, through the work on Prisma. And I tried for the work on Overtone, ideally having to deal with as little as possible, or at least have a very principled foundation for that.

And I feel this is a bit of a underdeveloped area in some of the available open source technologies right now, or local-first technologies right now. So I'm curious how you have been thinking about schema migrations for TinyBase. and if you've already built some foundation there. Okay. Well, before we talk about schema migrations, we should talk about schemas.

So. even having a schema was kind of a late thought for me, TinyBase outta the box by default, when, and especially when I first built, it, didn't have the concept of a schema at all. And, you know, any row could have any set of cells. I mean, a, a a row of cells was basically just a map. So it was possible for every row of a table to have completely different cells in it, which is a bit.

Scary, and contrarian especially for someone like me who'd just come from working on the world's largest data warehouse. the idea of doing this was, was a little daunting. But,, I felt like, you know, the word tiny meant I was allowed to do things like that and take away some constraints and just let people put arbitrary data into this basically a map of map of maps, right? Tables of rows of cells.

Now that obviously is fun for little demo apps, but at some point you need to start tidying things up. It's all very good when it's just one person building it too, right? Because I can remember all the fields that are supposed to be in this table. but when I've got colleagues who maybe want to know what the schema is, then should do something different. So I did, yeah, we do have schema support. I did add schema support, but it's optional, right?

You, you, you have to explicitly add the schema onto your store. but even then the schema is, I'm afraid, sorry, very basic. So you can say whether a cell is a, you know, string, a boolean or a number. And you can say whether it was required or not, and what the default is. That's it. So that seems a little daunting. I should say, by the way, that TinyBase only supports those data types anyway. So you can't put anything other than a string, a number, and a Boolean into a field anyway.

So I don't have support for arrays, or objects, or even null. so that kind of makes you realize this is a relatively simple thing to model anyway. But yeah, that, that, that's, that's where we are right now with schemas. And once you've added a schema, it will try to massage the data into that schema, and it'll drop anything that doesn't match. that's pretty basic, I will admit. But there are two, I think, two main things I need to do from here with this, because clearly that's not sufficient.

It's great for fun, simple apps. It's actually, I will say, it does allow you to do more than you might think. But anyway, people do want to do arrays, and That lets you do many to many joins, and they do want to do objects, so fine. We should probably get to that at some point. So, that's, yeah, that's my big push, I think, is making time based support richer data types inside a cell. I think arrays will be the first.

I'll probably stick to having arrays of one, type, like it's going to be all numbers or all strings. I don't think I'm going to have mixed arrays. That could get too tricky. I still want to have some kind of validation. so we'll do that. , that knocks off a many to many requirement that a bunch of people have had on GitHub that I can't currently do very easily. And so we'll, we'll do that. Secondly, I would love to do more with the schemas themselves.

So the, schema dialect that I put in right now is just, like I said, a fairly simple description of what the rows are, but There's Zod in the world, and there's Effect, and a bunch of other ways that people like to express schemas, and if they've already built their schema for some other part of the app, or they've built it for some, you know, some other part of the system, maybe up on the server, like, how can they then turn that into the schema that resides in the app itself?

So I have no doubt that I will be building Zod to TinyBase Converter, and Effect to TinyBase Converter or some, some, something to map, map those schemas in. Still doesn't solve your problem of migration, and I will admit that I am a bit daunted by that. In general, I am daunted by the whole space of anything that requires build time code mods and, you know, running generated code.

Like, I just hate that because immediately you've got a team of 10 people and all of their artifacts get out of sync. I've seen that at Facebook, right? Whether it's GraphQL or what have you. So I, always gonna be trying to look for solutions that don't require you to run a build step, to generate some magic code that you then have to remember to link in, how to actually avoid that step and still do schema migration. is not quite in my head yet. So I share your sentiment.

These things are underdeveloped, but I don't have an answer for you. Right. I can share my thoughts on it, which is, kind of like an architectural one, that has quite a lot of implications, but those implications are appealing to me, I would separate the read path from the write path and have the write path be event driven. And that allows me to do sort of event sourcing. So similar to how Redux works, but, persisted and distributed Redux, if you, if you want.

And, then like the way, how you interpret your Redux events. you can interpret them however you want, and you can very easily change that over time. So you basically just go through the entire event source, the entire event log of events, the entire history, and your app database that you read from, that is just, a result. By going through all the events and every event, you might come across like a user signed up event. , you then translate into a insert into users with all the values.

And you can change the user table at any moment, you can, instead of having one user table, you can have a customer's table and an employee's table and just reinterpret the events. And so this way, you don't have schema migrations at all. You just change sort of the . Interpretation of your event. it requires a bit of a different architectural approach and I'm building LiveStore all around this idea, which is very different from, from other approaches.

But it's the most principled approach that I can currently think of in terms of schema migrations, where you don't do like my schema migrations, basically just I design a new schema. And I target that and there's no, like, up or down scripts. and so that is very appealing, but it has other downsides as well. But this is certainly an idea that I, that I think is worth exploring more. No, I like that.

And it also shares, , Similarities with what I'm separately having to do for the, native CRDT synchronization, as of TinyBase version 5, which at the time we're speaking isn't quite released, but by the time this podcast is out might have been released. So, hey everybody, go download version 5. but no, it's, it's got a native CRDT support and I do that by sending, you know, basically little, little messages, but those are the data. level, right?

It's, you know, here's, here's how this cell changed, and I wonder whether like there's some higher abstraction for those messages that is then sort of schema agnostic or that can be mapped into whatever the schema is. Because by the way, the minute you start synchronizing things, the schemas getting out of date is going to become an issue very quickly, because maybe you've updated the app on your phone, but you haven't updated the app on your desktop, but now they're jogging.

You know, WebSockets to each other and all hell breaks loose because the schemas weren't quite the same. It's all questions at the moment. Yeah. I think there's a whole episode just to be done around that. Happy to form a little working group around this, but I think that goes beyond the scope of, of this podcast. but yeah, I think you've come really, really far with TinyBase and this is, I think, a very common area of the local-first technologies part that is not yet as fully explored.

and I've just changed my priorities to go double down on that from the beginning, since I know like how gnarly this is to deal with. Well, can I just say that, you know, from the point of view of exhaustiveness, there are many, many unsolved problems in the local-first space. And I have a horrible feeling that it's easy to think that synchronization is the problem. And then once you've solved synchronization, you're good. Well, I don't think that's true.

And the reason I think that is that I've tried building some local-first apps that, you know, have things like anonymous versus authenticated access, or that have private data versus shared data, or that have online, offline, and, you basically multiply this state machine together and you've got like, you know, two to the power of four combinations of state that your clients can be in.

And. It's all very well being able to synchronize data back and forwards, but, like, you don't want to synchronize a non shared store with another user, or you don't want to share an authenticated, you know, you brought a copy of an authenticated database to be local, but now you log out and now that's got to go, like, the state machine of what it actually takes to build a real app like Figma or I don't know. It's like, wow, there are lots of crazy, crazy transitions that can change here.

And I don't think anyone's really thinking about all of those. I think we all got hung up on, oh, once we crack CRDTs, we'll be good. No, no, no, no, that's the easy part. It's not state management, it's the state of the state management in a way. It's like, should I be synchronizing this data? Because this person isn't logged in, or is logged in, or is online, or is offline, or has shared this document, or hasn't shared this document, and that is actually unsolved, I think.

So I built a little sample app called Tiny Rooms a way back, and like, the app itself is stupid, it's just dragging rectangles around, but like, I tried to model all of these different states, and it was really hard. Like, I was, oh god. I would hate to be actually trying to build a production grade local-first app at the moment because those problems are really hard. Yeah, I think the, the benefits are that if you're building linear or if you're building Figma or if you're building, Tiny rooms.

the advantage for you is that you have very specific trade offs, very specific constraints that work for your app. And so you can take tons of shortcuts that for a more general solution are not viable. But for your app works. And so, given that, well, there, there's many trade offs that Figma can make that would not work for Linear and, and vice versa. But I think that is where you can still build things in a local-first way, that is like competitively, beneficial.

but I agree having the best of all worlds, is where we're a few days away from that, at least. Yeah. So I think that's one of the curses of being a, a tools builder like, like you, right. you know, I feel that I'm building shovels for people who are going to go off digging for gold. but I have to build a shovel that'll work in all different types of terrain to be successful.

Um, and so like I said, you know, is it going to be online, offline, auth, not auth, stored locally, stored remotely shared or not shared? Like, okay, now 16 combinations, got to support all 16. Whereas yes, if you're Figma, you can say, well, no, you can't ever use it anonymously. You've always got to be logged in. It's like, oh, great. Okay, fine. Like, I've just reduced that problem to eight. Yeah. Pick your trade offs very carefully.

And so this is why I'm also, why I'm working on Overtone and LiveStore at the same time. So if your app is sort of like Overtone esque shaped, then LiveStore will work really well for you. If it looks rather like Facebook, like a social network, or has like some very different shapes and trade offs, then it's not as good of a fit.

But I think this is also what's so nice about having all those different local-first tool vendors is they have all their different starting points and all their different thoughts on trade offs. And, I think this is where we're going to have a great shovel for many different terrains. Yeah. I think that's exactly right. You know, it's a dilemma. Do you build a tool that fits one niche perfectly?

And, yeah, of course, that really informs, and you'll build a, you know, brilliant tool for the one job you have in mind. but at the same time, you may have limited your market. Now, fortunately, I don't have to think about those things because, you know, it's not a commercial, commercial business. and I'm, but I'm trying to keep as many People are happy as possible.

And if I'm just listening to the feedback that's coming in on, on GitHub and people are saying, Oh, I want it to go off in this direction. And, Oh, someone else wants to go off in that direction. It's like, okay I'm going to try to, maybe I'm going to try to keep everybody happy. but then the complexity falls onto me. And that's where being a tools builder starts getting hard work. You know, this. Right.

I mean, for, for what it's worth, I think I'm trying to actually, write down the cases where LiveStore is not a great fit for, and I'm trying to be like, very explicit about the trade offs that I'm making. So for example, LiveStore right now expects you to load all of your data into a single SQLite in memory database. Luckily, SQLite is very efficient, like based on my benchmarks and testing, much more efficient than like JSON in memory.

It's much more efficient to keep like hundreds of thousands of rows in, in memory SQLite as you keep them in just JavaScript objects in, in memory. you need to just, deserialize the, just a few ones that you actually want to like currently keep in memory for, for your query. And so, but that's a major trade off. If you have your database, if that's necessarily five gigabytes. Then LiveStore is probably not a great fit for you.

Or if you don't want to follow the more like event sourced, nature of it, then it's probably also not a good fit for it. But I think this is what makes it more specific for when it's a good fit for someone, whereas if it's not. So actually this brings us onto an interesting thing that I think this community needs, which is. kind of a, a community view of what all these pros and cons are of the different solutions and what the decision tree should be if you want to build an app of a certain type.

You know, am I going to be online or am I going to be offline? Am I going to be authed or not authed? You know, how do I work my way down to what is going to be the best solution? Because of course, you talk to a vendor, they're going to say, oh, well, we can do everything. but something that allows people to know when, you know, when should I use. In something like the more mature front end world, people know when they should be using React versus Svelte or whatever, I guess.

There are plenty of places you can go and read up about what the pros and the cons are of these different solutions. But right now, if you're looking at Database Sync Technology A and Database Sync Technology B, they're both local-first, apparently, right? But one's going to be better for one thing versus another. Who's going providing that commentary.

And I think we haven't got that critical mass of users yet to be able to say, well, I tried it and this didn't scale, but I went to this one and this one worked. Someone else saying the opposite. And then you can like have that debate. and so. Actually, this is something I didn't mention, but, you know, one of, one of the parts of my local-first journey was, was putting together a directory of these solutions that I was finding. And so I launched that as the localfirstweb.dev site.

And that was partly me just trying to enumerate what these different solutions were. And obviously a community is built up around that. And now we obviously have the conference as well. And what I'm really hoping is that we see more and more people building real apps, learning What trade offs they're having to make, and which of these solutions they're then converging on. given those and, and I think it's fine to have lots of different types of tools.

Sometimes you need a pickaxe, sometimes you need a shovel, sometimes you need a pan to get the gold out of the river, right? But it's not always obvious which tool you should take until you've actually gone out into the hills and started looking for gold. Sorry to stretch the analogy, maybe it's because I lived in California for a while, but you know, I think we're not at that point yet, right?

We haven't had enough, we've not had enough miners going up into them, there hills to go look for the gold. so we haven't formed opinions about what the right trade offs are and which are the right tool chains to use.

We will get there. So speaking of real apps, it seems like I've recently nerd sniped on Twitter. I've recently nerd sniped you into building another app on top of TinyBase, which is a local-first GitHub client. And you've made quite a lot of progress on that. Do you, do you mind sharing more? Sure. So that was a, that was a clinical nerd snipe. That was, that was one of the finest even though I was the victim.

So as I mentioned, you know, one of my original ideas for an app that would use something like TinyBase was an open source dashboard that would let you see large numbers of repos and, and, and, and study them and understand them at scale. And I just never got around to building it. And then of course you said something on Twitter like, Oh, wouldn't it be awesome if there was a local-first GitHub client? I'm like, No! Damn it! That's what I was supposed to build! Now I remember!

So I took that and I figured that I would maybe go and put my, my code where my mouth was. So I started a little repo called TinyHub, which is in the same , parent org as TinyBase. So I guess you have show notes. There'll be a link in the show notes and, people can go check that out. And yes, it's a GitHub client for running locally in your browser. And I wanted to try and build it as quickly as I could, just to hopefully prove to myself that building apps with TinyBase would be fun.

And what it does is it, there's a quick bit of GitHub OAuth right at the beginning, and it then will pull repos that are either starred or that you, have personally, or any orgs that you're a member of if you accept those OAuth permissions. Anyway, long story short, it pulls the GitHub data via the API down into a local TinyBase instance. In fact, a number of instances.

It actually has, there's a TinyBase instance per repo, because if you pull React, like, okay, now we've got 20, 000 pull requests or something. So it will pull data locally, stores it in local storage. Yes which seems to be fine, actually. , just, it's a, the first hundred records which is the basic pagination for, for GitHub.

So it's, it's not, you know, a comprehensive open source dashboard, but it at least gives you A sample of, you know, 100 repos from 100 orgs and up to 100 pull requests and issues from each. And yeah, then just paint a React UI out of that data. And if you go check the repo, you'll see I put a little video on the front page of the readme. And of course, the beauty of this is that now all of this GitHub data is sitting in your browser.

And, it's in memory, but you can flip through it at 16 milliseconds a frame, to your heart's content. So, you know, you press the, you press the down cursor and it whizzes down through the pull requests and it's pffft. Views them all instantly. Not a spinner in sight. It's pretty awesome. I'm not sure it's a particularly useful tool at this point because it's read only. Um, and it obviously has to do some work to pull the data down at the beginning.

But it certainly, I hope, emphasizes the fact that if you co locate your data with the app, which I think is kind of like what this, this, this local-first thing should be all about at some point, it's amazing. Like, the UX is spectacular. Um, and you can just paginate through this stuff at the speed of thought. Which is really amazing.

I will just add a little side note here, is my philosophy around local-first, or my motivation for local-first, apart from being on a boat, is the user experience. I'm just tired of looking at spinners, because I'm waiting for something to pull down and cache locally and then get thrown away. Like, if I can get everything into the browser as soon as possible, and then just go through it at 16 milliseconds That's, that's amazing. So hopefully that has emphasized that point of local-first.

Yes, it's about data governance. Yes. It's about, you know, owning your own data. Yes. It's about you know, all the other things that are laid out in many, many other essays other than mine, but like having a UX like that is really, for me, what it's all about, that's the focus. , and yeah, so I'd encourage people to go check out TinyHub and that's me building what I think is a state of the art TinyBase app.

I would still say that to make it fully functional, making it read write is a little more tricky because whilst GitHub has some pretty awesome bulk read APIs the write APIs are a little different and making changes to a local datastore and then going back online and reconciling that back up to GitHub is not as easy as it sounds. I'm sure it's Events are probably the answer to that, is what you'll say. Um, but but I'm not there yet.

Yeah, again, to my earlier joke that we have more tools than, apps at this point, I really want to just add to the pool of local-first demos, right, but add to the number of apps that show what could be done, and really emphasize what the benefits are of this whole movement rather than just building shovels. I'm actually going to go out and look for some gold myself as well, just to make sure the shovel's okay.

I think that's a wonderful place to, to wrap it here. James, thank you so much for coming on the show today, sharing all of your wisdom on TinyBase, the path that led you here. Thank you so much. It's been a huge pleasure and I wish the podcast the greatest success. I'm so excited that you've set it up and honored to be a small part of it. And thank you to everybody that has made it this far and, followed the TinyBase journey. It's been a privilege to work on this and share it with you all.

Thank you for listening to the local-first FM podcast. If you've enjoyed this episode and haven't done so already, please subscribe and leave a review wherever you're listening. Please also share this episode with others. Spreading the word about the podcast is a great way to support it and to keep it going. A special thanks again to Rocicorp and Expo for supporting this podcast. See you next time.