Hey everybody, Welcome back to another episode of the Ruby Rogues podcast. This week, on our panel we have yush Nuwatya.
Hello.
No, I'm Charles max Wood from top End Devs. And this week we have a special guest. We have Mohammed Hassan. Mohammad, you built light stack, which is pretty cool. What else doul people know about you?
Well, I've been doing Ruby for for a long one now, I guess, I guess two thousand and five or something. I've been been in this industry for for a long while. I go by the handle old Mo for a reason. I think, yeah, it's it's literally old Mo, and I've been I've been doing computers since nineteen ninety something, so yeah, I have have a lot of work that has been
done in the Ruby land, especially in concurrency. I've released the library long ago if anyone remembers, was called never block once fibers started, and uh yeah, I got fascinated by by this idea of like high concurrency and and doing high performance stuff in Ruby. And then afterwards I also got into got in love with with embedded databases Siquo light to be specific. So basically I'm marrying the two.
So like high performance Ruby, high performance embedded database siqual Light in particular, and I've been I've been doing that with my own work in my own startups and looking forward now to actually deliver something to the community that will you know, spread this, uh this value to everyone that is using both Ruby and siqual Light.
Cool. So I looked at it and it and you can reckon me where I'm wrong. I didn't have a chance to try it because my life is absolutely nuts. You think the kids get out of school, your life gets lets the nuts, It doesn't. So yeah, I'm looking at it, and I'm sitting here thinking well, because I remember back in the day, sequel Light was kind of the not serious database. You know, you use it development and then you'd use a real database like post ris ql.
But with light stack it looks like you provide the database, which is kind of obvious, but then you're also pulling in stuff like cashing and jobs, and yeah, do you want to just talk about all of this because yes, you pull in just one gem and it does all this stuff.
Yes, I think. I think one one the biggest issue here was was the amount of dependencies you need when you start a new rails app like you need, you need like a database naturally, yeah, and then you need some some cash sover and and and then things go a little bit crazy and you make a cluster of those cash servers. And then you need a job processor, and you need a queue for that job processor. So you have read this for example, and then uh a
process for sidekick or something like that. And then you need full tech search, so you use something like elastic or open search. And then you need like uh cable uh capabilities, so you need a pop sub server and you have usually read this for that as well, or postographs and and and you end up with so many
dependencies and so many moving parts in your application. And I was I was thinking of like, yeah, like making making Siquo light your your database is nice, but like limiting it to just database means it's only limited to smallish applications that only require database. Actually, sequel Light, due to its latency characteristics, is capable of a lot more. And I went out to try and see if I can actually prove that it is capable of a lot more.
And so yeah, database, cash, pops up, que, job processor, full tex search, everything you'd need for not everything, but almost everything. You'd need for your application, and currently in the rapport but not released is like ID which is greatest basically reported to sequel light, so you won't need for your key data another separate process running somewhere in your infrastructure.
Very cool, I have to say, like I switched over to what is it the caching and queueing that? Yeah, solid cash and solid cute that came out with. Sorry, folks, I'm a little slow. I am under the weather big time,
but this looks so cool. I just wanted to talk through it, and so I was super excited to get rid of rettis right, because I haven't been using action Cable a ton, so yeah, I kind of like the idea of And I've also been using Melee search and I've been trying to move that into my Postgress database as well, just use Postgress full tech search. So so this appeals to me a lot, because I, you know, my stack gets a lot simpler.
Yes, definitely. I think I think there are two dimensions here. One of them is simplifying the stack, and one of them is actually realizing the use potential in having an embitted database. So one way to simplify the stack is to move everything to posgress posts is very capable, it's very solid, and it's been there since forever and you can rely on it. But what I'm my thing is there is a new dimension of performance when you go embedded,
like a complete a completely different thing. I think in Brighton, ruge I presented what it takes for a query to run if you go through like client server stack versus an embedded stack, and what do that is? Basically you're you're doing for most treats, you're doing a memory call and and and that's that's a huge advancement in performance
that you'll only appreciate it if you see it. Once you see that your Rail's application is able to able to deal with like potentially thousands of concurrent connections and then responding to them very quickly, you'll you'll, you'll you'll have to see it to believe it.
Basically very cool. Are usually you played with this?
I haven't actually had a chance to play with it. Sequel Light and Rails has been something that's been on my do list for a while, but well, life gets in the way. I think my next project is almost definitely going to be my next side project is definitely going to be a sequel light rather than postgress. But yes, I just I just want to get very clear in
my head about on the gotcha's with sequel light. And let's start with just what light stock actually does in certain things like with light job and sorry, with light cable and like like Katie he said, like like like cable used like pub sub it's like a drop in replacement, right and like Katie replicates credits that are these building on like native sequel light features off like pub sub in a key value store or.
Is a little blend then than that, so it doesn't have like something like notify in in postgress. And I like I had to build this functionality into Syqul light on top of Syqul itself, but on top of native sequel light. So by the way, the solid family of products like Solid cash, slid C they have they do have sequel light support, but the problem is they do deliver that on through active record, which is like an order of man nude more abstracted then going native with
sycuolight like directly and hence the siqal light performance. If if you go directly to the metal, it's it's a much much faster and the nice thing about about what we have in in light stack is that we have the seql code completely the coupled from the from the Ruby logic, so you can optimize that independently and work on ensuring gas every call is through an index. You're not working, uh, you're not walking tables or doing any table scans things of that sort. So it's uh, I guess,
I guess it's. It's it's implemented, but the functionality is not that hard. I guess it's it's really straightforward getting a cable and getting it to perform the problem that your core issue would be to get it to perform correctly under like concurrent operation, which is which is tricky and because you don't know whether the application would run
in in threads, run fibers, multiple processes or not. And this is I think the core contribution of light stack, which it abstracts all this and and make sure any component can be brought up very quickly, like I built
like cable in one hour. Oh wow, okay, because because all the machinery there for for like making sure that like uh, pulling of connections, threads don't go into these conditions and and proceeds are not affecting each other that was there already, so you only get to implement the logic of the higher level of course one hour with bugs. But but then I had a working for the type
and and and unlikely you was not much harder. It was done in two days, so and then it was mostly the functionality, a lot of functionality, a lot of logic on top of the connection. So I guess, I guess that's that's what light stack is able to do, bring up those features quickly, and I implement as much of them as possible in sequel land rather than in rugby land, for for performance reasons from one side, and and also for portability and separation of concerns as well.
Yeah, I think that that's a great approach. I've kind of been uh, blissfully unaware of sequel up to a point because Actor record kind of handles it for me. But for the client I'm working with at the moment, we've had to do some really advanced stuff and I have been getting my hands dirty with seql and the power that it has has blown me away a little bit.
So yeah, I completely get your approach there. So yeah, just for personal curiosity, could you talk about how you how you did pops up with just like SEQL and database features without having something like notify at your disposal.
Yes. So the idea is when when you do a database calls it in the traditional sense it goes through the network to the server and back, but in in in the sequel like sense, it's just a mess of coal, a mess a call to a local memory address, especially if you if you're looking at some page that is definitely in the cash because it is like access, it's hot in the hot section of your data. So basically it's a pulling it. It's pulling happening from from Ruby.
Like each thread that is that is subscribed to the pops up would constantly pull the database and ask it for for data. And and in benchmarking like four proceeds constantly writing and constantly pulling. I'm able to get around like one hundred and twenty thousand messages retired per second. So yeah, this is this and that's not on very
high end hardware, so it is capable. And now I was even able to do faster than some some IPC cross process solutions that are tried at first as an optimization, but then it ended up like siquolites faster.
Still, Yeah, I think when you eliminate the network from the equation of the performance speed up you get is just hard to comprehend because you're any AUTHO removing so much complexity from the call. Right, You're not opening a network connection, you're not sending anything over a wire.
You're not data from from buffers to like the network stack in the OS. There are so many, so many things that are eliminated, and your your life becomes a lot easier.
I'm in some cases, I think I kind of feel this, you know, because I've used some of the cloud databases, right, so you know, even if I'm pulling a you know, a database on Linode and a server on Linode or Digital Ocean or whatever, you know, pick a poison like those.
I feel like I felt that more than when I have you know, to virtual servers that I've set up in the same network, you know, on Digital Otion and stuff, because all of that stuff is it's I think, it's all software and it's fast, right, and.
It is fast, but it's not free.
No, that's fair, and it will never be free. You don't get that for free. But yeah, I think I think there are different levels to this, right, I don't know that, but completely convinced that you know, it's gonna be that noticeable depending on what your setup is. Yeah, you should be under the circumstances.
Yeah, yeah, definitely you should. You should try actually to go through the Like I'm all the benchmarks that they have for light stack. By the way, on on the GitHub ripple, these are run against postglass on the same machine.
Okay, so you put mostgress on the same machine and run not even on a versus I think.
Okay, Like all these are run through against postgress on the same very same machine, on the same space, same everything.
Well that changes my argument a little bit.
Yeah, so I like the point is how how much of the overhead? How much is the overhead compared to the actual work that you're doing, and it there is a spectrum here, not not all queries would be much faster and sequal lighte Like, if you're doing heavy io, if you hit the desk, then they will both be both be a lot slower. Then in that case, the savings that you had by not going through the network stack will be very un noticeable. They will not exist.
Basically because reading a single block from disk is much much more expensive, like orders of magnitude more expensive. So like, if you're talking about quis that are fulfilled from the page cash, then the difference would be really not If you have complex queries that will require disk access, then the difference will be less. But in no situation will siquo light bill will be slower than post aggress in reed scenarios, given, of course, the quity plan that was
provided by both databases. Both have very advanced quity planners, and they should produce for most in the most case, similar plans for the quities. But in case they're producing the similar plan, definitely Sequolite will be either as fast or much faster than like in a spectrum between fast and as fast and much faster than prosrogress. That's three equities, right, It is a completely different story.
Right. So my question then is, because yeah, I'm looking at these benchmarks and it's pretty impressive, what if you have to reach it over the network? Right, So, what if you've got enough traffic to where you've spread your application servers out over multiple servers, and then I'll have to hit the same source of truth, right, the same database that way, How does that change the equation.
So siquolite is not designed for this scenario like scaling out and connecting from multiple application servers to the same node database node is not what sequal light is designed for. Sickle Light is designed generally for virtical scaling rather than horizontal scaling, so you get a better, bigger machine or a VM basically, and in that case you can serve more from the same resources. But there's many attempts to
actually enable enable this level of scaling. But but then the point would be a little bit moved, like why not use postal glass. I'm working myself on a solution that that is midway, so it only transmits rights, but all reads are local, so you have multiple nodes, each one has a complete copy of the database and rights are distributed. I've written about that on my blog and still didn't open source it yet. But I'm not surely if I'm going to open source it, or maybe you
just try to build the service around it. But but you get a copy of the database of full copy of database locally and you can do all reads locally, so you get those benefits for reds and you get slight extralatency for rights.
Yeah, but yeah, I mean I have to admit most of the time when I'm scaling, Yeah, I just go into lin ode or Digital Lotion or wherever I'm hosting, and I say I want more RAM and a little more disk space, and it goes okay, and then my website's down for a minute. So what you're talking about that makes sense to me.
Yeah, yeah, because it simplifies yourself, like because once you go out of that, you you have to consider, okay, what about like a little balancing, what if it note fails, what happens here, what happened there? Then then you have so many moving parts and they keep just growing exponentially as as you add complexity to resist them.
Right.
I think in twenty twenty four, the amount of traffic you can serve with one single beefy sell. Yeah, it's just bloody unreal. I think horizontal scaling, If you need horizontal scaling, you will have enough money to pay people to get you off sequel light and on the postgress exactly.
That's I think. I think that the point is, or or the point that I'm trying to to spread, is it is unwise to scale more than what you need initially in your small or starting projects. You can get a lot more a lot more value in like eliminating overheads and increasing performance and eliminating like moving parts by moving to an embedded solution. And sycolitis is one heck of an embedded solution. It's an amazing solution. It just works.
And Yeah, my advice would be, don't focus on your infrastructure, don't have DevOps, focus on your application, and it will carry you a long way. And once you're in a position where you need to, you really need to move out of this exactly what you said, Like, you have the money and the funds to move out of this. Either either you're getting a lot of money or you have enough external funding to get out of this given that you have that amount of user activity.
Yeah, DHH actually said I'm trying to find the tweet, but he said in a tweet that if you bought like I think it was thirty two or sixty four gigabyte Hetzner level thing, that you at this point could have run the entirety of base camp a few years ago. Right, And so yeah, yeah, I'm really kind of yeah, I'm liking the way that this looks.
And then look at at things now like this camp now is a beast. It's a huge application and so many subscriptions. But and they had to go through a lot of pain to scale that. But if you're starting something like that, now you've got like at least five to six years of growth within a single box.
Yeah, I think, I said when they first started back in like the mid two thousands or whatever, for a number of years they were running on one physical server. Yeah, not even virtualization back then, because this is two thousand and four. So yeah, I'm fully on board with this. Just rent the VM approach.
Yeah, he said, you're right here. He said, you can rent a dedicated forty eight cores two hundred and fifty six gig two terror by AMD epic server from Heatzner at two hundred and thirty six dollars a month or undred and thirty six euros a month. Yeah. I mean, I've I've never had an application that got so much traffic that that wasn't enough.
Yeah, and those four the eight courses like translate to ninety six course in cloud speak. So yeah, that's that's a lot more than many many applications and and a lot cheaper, h Yeah, than than what what deployments we see on the cloud.
Yeah, it's crazy how much money you can save when you cut out complexity. Well, let's let's dig into a little bit just about sequel Light and its use and in a web application. So putting light stack and all the amazing stuff that does, let's put that to one side. And if I wanted to use sequel light directly with rails, I know there are some gotchas backups being one main thing is how do I back up and not lose
my file? And the second thing is I've been looking through the campfire a code base which is obviously sequel light ones product from Petty Sound Signals, And the only thing I could find in there that was like specifically to do with sequel light in terms of like initialization and set up with something called a busy handler. So could you talk what that is and why that's necessary.
Okay, So, originally sequal Light is configured for like really really embedded use cases, So it is for usage on a phone and probably between the three of us, like we have like maybe at least one hundred sequel Light instances running at this very moment on our phones and computers. But but this use cases is completely different from the web application that requires like multiple users accessing it at the same time as much as possibly as fast as possible.
So there are a few caveats and a few knobs that he that should be set. What light stack does it It completely does that by default when it's publishing its own like Siicula driver to active record and what there are other gems like fact un minded with Stephen Mark, he goes, yeah, he has a lot of effort done in that in that area, and and I think I think he is helping a lot like the Vanilla's equal light experience for rails. By the way, one one thing
about let's stack light Steck is not just trailed. It's basically you can use light stack with Anami and use it with with Sinatra whatever bring more cute like. But let us get back to the configuration. So basically what you need is the following. You need to ensure that you can do rights while you can do reads. The default configuration can only either do rights or reads. So you change the journal in mode from the default which
is delete, to well which is right ahead log. So in that case you're you can have a writer and either at the same time, actually an unlimited number of readers and a writer at the same time. That's the first configuration. The second thing is which is a little bit uh tricky, like when when when when connections are trying to capture the lock on the database, they will either succeed in getting the lock or fail because another
process capture the lock. And if it fails, then it will return to you, like ah, siqual Light busy error, and hence we need to have a busy handler, and that busy handler what will will do It will capture that error and not propagated to the application and it will try again later to capture the lock, which is basically what what a central server like postrogress would do in in locking. Of course, its locks are a lot more fine grained than siqual Light, which is like a
single lock for the all database. And and then you also have something in the in the transaction type. Sickle Light has multi transaction types. It has like the the default which is deferred transactions, and the there's the immediate and the exclusive transaction. What we do also for the driver in to run a web application properly is to make all transactions immediates. What that means is a deferred transaction is a re transaction that you can midway upgrade
to a right transaction. And and the problem with that is that if you do this and another transaction has if you attempt to upgrade the lock and another transaction already has written to the database, this operation will fail with a busy like sickle Light busy error. But if you start with an immediate transaction, then you will have a right lock right from the beginning of the transaction, and then you will not have this error again anymore.
So.
So basically the for configuring Siquelight, I think there are like four main things to do. Change the journey motu al, set a busy handler, and also configure your memory map to have like a shared cash across all processes rather than a cash pair connection. And yeah, and and and that's it and and set your transactions to medio. These are the four things that need to be done. Let's
Tack does that automatically. There is also the sequel light JAM sycal light adapter JAM from Steven also does that. And I think these some of these are being merged into into the upstream sickle Light driver for active records.
How do you make these changes? Do you do it? At Ruby level to run some sequel or how how's this gon figuration done?
Like in lightstack, it's it's done by by overriding the sequal light driver and applying these at initialization, making sure all connections share these attributes.
So this is on the the rails end or the SEQL end or whatever, not on the sequel light end.
It's it's on the sequel light adapter end or sickle light driver end.
Basically, so in database to do anything different, you're telling the application it has to manage its connection this way.
More or less. Yeah, but but at the end of the day, Raid itself is not aware of that. It's not it doesn't see that this is happening, that the changes that are we're going to go upstream will will bring that knowledge to the rails driver.
Mm hm. That makes sense.
And regarding backups, like is light stream power light stock or is that no?
Light stream is not part of lightstack, you can use light stream aside from using lightstack, light Stream is a very nice tool you can you can basically back up the database like incrementally and put that on S three.
Like I personally use a different method which I'm trying to spread as not just possible, but like in in in in most situations when when you have a VPS or like a VM on on something like line Ode or or Digital Ocean or anything, my accommendation would be to put the database on a replicated volume rather than
on the VM directly. And in that case you have you have the whole database living in in on the network basically, and and then if you have a file system that supports copy on write UH semantics like like better S or XS, then you can have a backup in of the whole database in under too many seconds
by just copying the file. And you can have as many backups of these as you want, like you can have like a chron that is doing a backup every second, or all you need to do is open a like read transactions, take a backup, and it will be consistent. I have also written about that. I have a blog post about backup strategies, and I explain some of these, including light stream, but I don't go deep in there,
but I also explain some of the other options. If you have like a COW file system, so like if I was if.
I didn't really care about having like every second of data like backup, if I wrote like a crown job to back up the entire database file like every hour or something like that and upload it to our S three or some remote storage. That's basically a viable back cup solution because it's just.
Yes, yeah, but but like let's swim will be even better here because it does just an incremental update, so not every second. It just sends the the changes or as as I as I mentioned on a volume on applicated volume. Then you have your data applicated across the data center. You don't need even S three. You just make a copy a copy CP of the file and it will be a copy on right and it will it will, it will. If you have gigs of information, it will take like two to four milli seconds to copy.
Wow, transactions have completed.
No no, no, It then unrelated to transactions because you open you open a read transaction and you copy, you don't get you get snapped. Okay, you get a valid snapshot of the database.
Okay.
I'm planning to add some some functionality in lightstack that would do that for you, like open the transaction and issue the right copy command. But it will not work if you don't have it will not work efficiently if you don't have like an XS or better f s or zfs.
Are there any like other gotchas that someone coming from a postgraph background. If I'm writing a rails app or any any web ap, let's not any web app with sequel Light as a database. Is there anything else I need to be aware of?
Definitely? And and And that's the biggest concern when when you're dealing with a database that is not I like to call it a decentralized embittered database because it doesn't have a server to coordinate actions, which is basically right performance and specifically things like creating an index on launch table and things like that. Because of its nature, Sequelight will will always take like a lock on the whole database whenever it's doing the right operation, so only one
writer at a time. Keeping keeping your rights small and untidy will mean that you get very fast rights like you can do like tens of thousands of rights per second and on a syquo light database, and it will beat any any other database. But if you have really large rights, any other database on a on a like a reasonable configuration on the same configuration like a single note.
But if you have really large right transactions that do a lot of things, especially if you're trying to do things within the transaction like going to a server or something, then you're you're hurting the syco light performance a lot, and and that would while this will degrade other solutions as well, including postrogression by sequel, they will be a lot more graceful at providing a chance for other transactions to run at the same time. Siquo Light will not
be able to do so. So if if you have like threally large transactions that are trying to do a lot in writing, you should hopefully avoid that and break it down, or you will suffer from like queueing right requests because only one writer at the time. That's one thing. The other thing is some some some of these are unescapable,
like creating an index. When you create an index, you have to wait until the index is finished, and it has to have like a snapshot like a correct state of the database well being created, so it cannot allow other writers to write at the same time, even to other tables. So in that case, creating a creating an index on the large table, really large table, can actually take your to debase flying briefly, and is I.
Think I don't know if I am completely making this up or if I read it somewhere, but is like N plus one query is a feature rather than a bug when it comes to sequel.
Life, Yeah, it's it's it's not a feature, per see. Basically, here's the thing. The overhead of running equity on sequel light is way way less than running a call across the network stack, whether locally or remotely. So if you have en plus ones, they're not as expensive. Of course, having one running one query instead of like n plus one queries is cheaper, but the difference is not as big as with client server database. It's a lot less, like it could be like fifty percent more or something
depending on end. So in that case, if doing an en plus one would actually make your call nicer, just go for it because because it's not as penalizing to your system as it is with something like uh, something like my sequel or postrogress. The authors of sequel Light, they use M plus one intentionally in some cases on their website and on the Fossil sem system because it makes things a lot easier and and it makes encapsulation a lot.
Like I had that beaten out of me. I swear.
Yeah, like I think about it, it's it's an optimization like eliminating M plus one is an optimization. It's not something that would make your code look better, right, Actually it's another way out.
Well, and and it still works, right, it's just fast.
Yes, the the idea in siquolizes is not that big deal. It's not an issue like of mice for for your code organization and installation better right.
Yeah, just a different mental model, isn't it, Because yeah, suddenly database is local.
Imagine if you cannot do partials in in in views, like you'll have to have those read and do all the partials inside in line. It's it's exactly the same. Bushis are more expensive, but but you don't go there because because then it's a nightmare. Good vice.
True. Yeah, So let's let's pick up on something we're actually briefly discussing before the show started. But so I have done a fair amount of work on search and postgrass the postgas native full tack search, and for the client I'm working with at the moment, I've altered on quite elastic search. What what is the search story on sequel light and does light stock have anything kind of building upon that.
Yeah. So like syco Light has its own native search implementation which is called FTS. There are multiple versions of that three, four, and five. Five is the currently the most maintained version. So cycle light does have an FDS story. The problem with the FTS story in Sequo light is that it is slightly rigid, so you have you have to have a table that maps to another table in the database, and the mapping has to be static. If you change a column you want to add it to
the index, you have to rebuild the whole index. Things of that sort. In light Stack, we have a light search component which builds on top off the FTS five but it also brings a lot of dynamism to it, so it actually implements like a dynamic layer. It's a lot. The interface is a lot like mini search, so if you if you use that, you do the same, you do define your index almost the same. But then all this is going down into an FTS five database and
FTI is five module. I'm sorry, so there is I guess there is a strong search story there and the performance is amazing. Like you have to see I have that blog comparing Mini Search to and then you have that in the benchmarks as well on the rippot Meli search to FTS five and I kept repeating the benchmarks
because they couldn't couldn't believe the results. Like uh, Emali searches has a lot more functionality, I would say, so Vanilla FTS five doesn't have, for example, like something like type of tolerance, and and you need to add some modules in order to support multiple languages. But aside from multiple languages and type of tolerance, the performance difference is like it has to be seen to be.
Believed, okay, uh And is that like just in like performance as in time, the time it takes. How the performance with regards accuracy accuracy.
That's that's a different thing. Yeah, yeah, that's that's a lot harder to measure. But but at the same time, it's it's an own quantity. Like working in l P is has like this. The industry has been has been done that for long and then the tools to to produce those are already there. So while while type of toolerance is not there by default, you can actually implement
that there is. There is stemming using the traditional stemmers from snowballs like porter and others, and and these are mostly the stammers that are used by almost all implementations. If we're talking about Latin based languages and Latin alphabets, basically and and and and and then aside aside from that, it's just typeotolerance that's the difference. So you get you get, you get near search, you get race search, you get like you can mix and match and or and and
things like that. You have perfect search all all the all the known suspects basically, So from a search quality perspective, I believe it's a The biggest difference is type, type of tolerance and fuzzy support, which is basically are basically the same.
This is not there.
But aside from that, you get you get, you get everything that you would get with the typical search engine like plastic Mealy or swings or things like that.
That's a pretty solid story, to be honest, I wasn't expecting it to be that solid.
It is. It is a very it's strange like the sique Light team, all two of them, they produce very very solid software and components, and they build so many things from scratch it's it's mind boggling. And yeah, I would love more people to actually use these.
So what I'm curious about is and and It's funny because you mentioned you've been doing this since two thousand and five, and I got into Ruby and Rails in two thousand and six, so I've kind of seen a lot of the same stuff come and go. What changed, right, because back then, sequel light, yeah, it was the default when you ran Rails without any flag for the database, but it was essentially understood that, yeah, you're probably going
to replace this with another database. So what changed to the point where now people are realistically looking at this and going, huh, I could run this in production?
Yeah, I guess. I guess this impression that sequel light is just like something to start with or even to use just for testing was the norm, and it was understandable because many of the features that we're talking about, they happened over the years. So, right, headlog mode was not there. I'm not sure if it was there on two thousand and five. I guess it was, but many of the other features around it were not. And now you see how how those features came together and made
Siquolide that rich. By the way, it also when you build it, it comes with a geopoly database and like you can you can do like distance calculations, and stuff like that. And this is based indexes and many other features like it's it's it has grown a lot and is now capable a lot, and I think people are starting to realize that, yeah, I might not need a lot more than that, especially with the advancement in hardware that is happening in parallel. So you get these two.
Now you can have like a single box that can do a lot and a piece of software that is capable of like managing all your data and your data requirements. And and what I'm trying to do with light stack is to build that layer between both, like your data layer and your application that is friction free, like no configuration required, you don't need to do anything.
Yeah, that makes sense, and it's it's kind of exciting just to see that, Yeah, somebody could really reliably come in and and do a lot of this stuff. It's it's funny too because and then I'm working on a different layer than you are. But I've been working on pulling together rails Composer, which I actually got the domain from Daniel Keho, and I've been working on pulling this together.
It's the same idea, right, It's I need these areas of functionality in my application, and so right, I pull in one gem and then I can just run generators and have it right, Yes, exactly, And so you know with this it's the same deal where it's I pull in light Stack and then I immediately have the capabilities I need to run a modern Ruby on rails app.
Yeah, almost almost all your data requirements are already fulfilled. Google a great application. And don't bother about the infrastructure, don't pay for DevOps like eliminate all these points of like pain, headache and and and mon uh spending.
So are there people out there currently running their production apps in with light stack?
Some some I know, yes, uh, and and some are are running light stack on no staging like I know of a large, large Ruby app that uh their environment staging environment was switched to light stack overnight without even the developers knowing. And they they didn't know except after pondering why everything is much faster.
That is a sales prige and a half.
I was gonna say, you're making me feel stupid for not trying this. Yeah, I think.
I think the idea is if if you're if you're if you're building something and starting, it's really unwise to add to to go on a more complex path, you should you should go with the least resistance path until it starts resisting you. And hopefully it will never resist you. We'll never show you that fiction.
Yeah, you're speaking my language here. I mean, this is where and I picked some of this up from reading Ayush's book The Rails and hot Wire Codex, where essentially, right he walks you through I mean the first what third, almost half of the book is you know, building the authentication piece. But you know, I got through it and I was just like, this, this is not as hard as I thought. And right, I kind of get all of the niceties that I want out of it because
I understand it. And you know, yeah, anyway it appeals to me in that same way because I'm looking at it and going, Okay, you know the underlying engine for this thing, which is essentially rack and the database, you know, that's stuff that I don't want to fiddle with, right, Yeah, and I can just rest everything on top.
Of it, exactly pushing that complexity away from you and hiding it completely versus exposing every complexity. Like if you're trying to deal with the traditional app you'll have to think of many things, including like where should I put my cash, where should I put my Do I have a single radius incense for all of these or do I have to split them? And how now that I'm scaling anyways, how do I scale these components as well? Because you can add application servers, but then your database
will need to scale. So the nice thing with with vertically scaling that you scale both together. You don't scale everything, but but with with like a client server architecture, you have to separately scale each one.
Does so again coming from my postgrap postgrass brain, like that's what my background is in. So yeah, two questions again, trying to find equivalent for postgress. Is there is that like a Jason Bee column and sequel light or any equivalent.
Yes, but it's a different formats. It's equal its own format. So you cannot get a jacon b record from here and just put it there. You have to convert it to jason first. But but yeah, it has it has.
J befo serialize it and then serialize it in the other database.
But but but almost all the jacent functions that you'd expect are there for both the adjacent text format and the JCNB format.
Oh wow, so you can like query for stuff that's inside and you.
Can create an index on an a adjacent field and you can Yeah.
Nice. And another thing I use quite often, especially to like just like filter stuff in a UI is UH like and I like queries, and I use the program program indexes and UH in postgress speed those up. So yeah, we spoke about full texa. But like, is trigram a thing in one One of.
The one of the UH tokenizers available by default is the trigram togonizer in FTS five. So you can have a trigram.
Okay, and that will help it like with like, and I like es exactly. Why don't I you equal light again?
So yeah, here's another Oh go ahead.
Yeah, I wanted to mention that I also have an extension for sequel light that uses roaring bit maps, so you can create facets for your search results. Like you have like a search index, and you want to generate
distributions of results. So you have like you get your social books, for example, and you want to see see in that search how many of those books are in watch contigorties and this this, this can be done using maps in like a blink of an eye, like under under ten milli seconds, you get get so many statistics stuff like you see on Amazon, like those things and how many of those are red, and how many are blue? How many are at that price, so many of that price.
All these those things can be done like almost matterally in sequel light very cool.
So I'm wondering. You mentioned that there was a company that switched over to light stack overnight, and I'm wondering what that process looks like. I mean, I would imagine that, you know, kind of the queueing and things like that, but you know, are probably pretty easy, right because the job's either been run or it's not, so you know,
pull the job from something else. But I'm looking at like my data on some of my apps, and some of it's kind of hairy, and so I'm wondering, Okay, you know, what would it take for me to go in and do you mean my.
Sequel in table like like table.
Yeah, I guess it's the state of my models that I'm not happy with, not necessarily the state of the data.
Yeah, that that's one one level above. Yeah, the problem
we're dealing with. It's completely opaque for the models. So you just you that's an option, yes, somehow, and you can do a sequel lump and and and just copy that over, or you can do a What what I'm trying to work on is to do an import export at the active record level so that you can easily pointed to that database and that detabaate and then the data gets migrated with active record semantics, so any any potential compatibility issues are just washed away.
Oh wow, that would be cool.
Yeah, like like active record does have this machinery already, or just need to do it together, right, or you can dump. But the way you can dump in active record h format basically huh, and and the store after you change the driver.
Yeah. The only other thing that I'm thinking about with my setup is so I've been using camal to deploy and it puts the traffic load balancer the way I haven't set up. I have a traffic load balancer in front of two application nodes, so effectively I would have to get rid of one of those. I wouldn't need traffic anymore.
Right, yes, but but if if they are on the same machine, then you can have a volume.
Right you volume for the doctor containers.
Yeah, and they can share it but it's it's much much more straightforward to guys you just have one.
Yeah.
The nice thing about having two is it makes like zero down to deployment is easier, but it is double still with one, and that's something we Yeah, that's a topic that I want to add ass at one point.
You can address it now if you want.
No, I mean there's I mean condition yes, yeah, yeah, yeah, like maybe maybe a light deploy of something.
That'd be interesting because.
Almost almost almost all modern applications of support like zero down time deployments, right and zero downtime resorts. So you definitely lose some of the web socket connections, but that's it. Even other connections can can.
Remain not well, especially if you're using something like camal with doctor base. Right, you just wait till the other the other I wanted the same machine the other container is running, yeah, and then it's okay, now everything new goes to this one and when the other one's done, you just kill it.
And I think I think that's one of the nice things about this setup that yeah, many many of the issues that people thought they're gonna face with with similar setups are actually addressed. And yeah, I would definitely recommend giving giving that smallish stack a go and then see if if it works for you. And if you're starting something new, just just don't just don't bother, don't bother with the complex that is.
There is there a use case where you would say, don't start it's equel light or light stack.
Let me think, so, maybe if there is a use case where there's something that is someone that is writing huge records in a database, uh and and like we're talking gigabytes, maybe some people do that, some people and and and that's like the core of what they're doing, then definitely you need something that can spread that data over a larger surface in terms of files. If you're doing like a network or even a distributed database eventually, but syco light will not be the database for you.
This is not.
Or not necessarily gigabytes or you're like treading in that area, like eight hundred megabyte filees still very large. Other than that, I think, I think it it could be a case by case basis. But if you also have an application where so many people are writing at the same time something big relatively that or doing like complex right transaction, syco light is a NOG But if you have so many people doing smallest transactions. Cycle light is smare than enough.
It would be even faster than the typical solutions. So yeah, I would. I would. I would say, like you'd need to have a strong case against using Syqule light to not use Siquo light.
So do you see this becoming kind of the thing that people reach for in the future by default?
I hope that would be the case, Like that's the that's the future. I see that. Yeah, Like why why go through the complex route if if I have something simpler? And I think it just makes sense to simplify things, like this industry has suffered a lot from breeding complexity, Like we've been doing really really complex setups, especially in in the front end. Then infrastructure domains just because people
wanted to play with with different stuff. And I think it's it's about time we and it's it's it's helping that the VC money is drying, So I guess I guess it's it's it's about time to to realize that we need build applications, not not build resumes, and focus on on what we're trying to deliver the value that we're trying to deliver that rather than this is my stack, this is my front end like stack and things like that. This is not this is not what what people should
be bothered with. And if your engineering team is actually focused on what type of stack they're building of consulted with a lot of teams, and I've seen teams that would would talk to me about the technologies they are using rather than what value they're they're delivering or building. Then then you're you're in the wrong place.
Yeah, I've been there.
Yeah, Yeah, it's it's it's it's not fun like seeing seeing really complex stacks and you ask how many users do you have and they tell you we have two hundred and and then why bother? Like why all this? Then? Why not focus on actually getting the business off the
ground and then and running with it. And once once you have ten million users, maybe you can talk like I I hosted millions of users on the databases, and like I had a situation when a friend was asking me about their they're hosting costs because they ballooned they were paying at the time it was a lot like two thousand, five hundred a month at AWS and they had almost half our user base or even less, and they were asking it was gaming gaming sort of both
of us. And they were asking how much we pay every month and I was really really like embarrassed to tell them that we're paying seventy five dollars. So, yeah, people need to realize they can they can get a lot of value by by removing complexity and removing redundancy and moving abstractions and indirection.
Right, it's pretty mad that computers are like an order of magnitude faster than they were twenty years ago. But yeah, software seems slower than it's ever been.
Yeah, it's it's it's stuck with it. Where like it because because computers are being faster, softlow is getting faster naturally. But it's it's it's not trying to do things differently. It's not No one has come up and said, like, let's make use of the capacity now that we have in a single machine.
Yeah, I was going to say, and and this was kind of came to me when DHH was doing his keynote at Rails World last year and he points out, you know, we're we have solid Q and solid cash because disc speeds have you know, grown in this way, right, And he was basically pointing out, yeah, you know we're
going to take advantage of this. And yeah, it drove home to me that the the limiting factor isn't the technology, it's the mindset, right, because I'm still stuck in you know when I started programming in two thousand and well, I started programming in high school in the nineties, but when I started doing rails apps into in two thousand and six, Yeah, you know, I'm looking at it and going, oh, I've got to account for all this stuff, and so I have to go wrangle all of these things to
make it work. And and so the limitation was the way I thought about applications being built, not not the hardware, the technology that was available to me. And then I think that's what you're saying, Mohammad.
Yes, exactly exactly like people, people like we have we have been conditioned to do things a certain way, like you need you need cash, you put in this inostance, you need job, you put the incense and the job runner and stuff like that, and people think this is this is the best they can get. And I would challenge anyone, anyone. I want to see a faster read cash reet performance from any any adapter for UH, for
for rails UH then then light stacks cash. Of course solid cash is is fast as well, but it's not as fast because it relies on active record. But once you hit sequel light, there is nothing faster than this. Like you're reading literally from memory. It's an order of magnitude faster than a radios at.
Oh hang on, hang on, time out. You're you're saying that you built the light cash and it is the queuing this way as well. Yes, so you're not relying on active record, You're you're just going straight to the database and going, huh, what's in the cash?
Exactly exactly, because because the nice thing about RAILS is that it has these really nice interfaces for active job and for active source cash and for a light cable, for active action cable and sorry, and and what you do is you provide an adapter that does the job rather than going one level up and and relying on active records that you you you you just delivered the functionality directly. It ends up being very fast, like really
really very fast. Not much faster than of course, having the active record abstraction top of you and actual records buys you a lot of things like like distributed databases and like you know, having multiple database for the same like the same application, talking to multiple databases and shouting and stuff like that. You don't need that to sique light. Yeah, it's it's one database on the same machine, so you don't need all of this.
You're making me.
Think, Yeah, it's really hard to kind of change your mental models once I get set. Like, like, my background is in uh was in mobile development when I started working as a professional software developer, started off as an iOS and Android developer, so was twenty fourteen and I only switched to RAILS in twenty twenty. And I remember, like, when you're in a mobile app, going to the disc
is considered slow. So when you launch the app, you basically load up whatever you can in memory, and if you're going to go to the disk, you have to kind of really think about the coreer writing because you don't want to lock up the UI. So I have this drilled in me. Going to the discus slow. Going to the disc is slow. So when I came to RAILS and I was trying to understand Russian doll caching in RAILS, I couldn't understand why there was a benefit because we still had to go to the database to
get the updated app value of the record. I couldn't understand that in a web context, rendering the HTML is the slow part. It took me months and I'm not exaggerating months to change that mental model. So I completely get that people have been doing this for decades, have this models set about how you structure and appy the database hour and stuff, and now they're just like, it's really hard to just change that.
Yeah, I'm glad, I'm glad you brought that up. Because one of the really nice things about switching to light stack or not nice actually the bad things about switching to light stack that it mhm suddenly highlights how slow your front end is, like the rendering path, like the
view layer. Basically it's really highlighted because you eliminate a lot of the buttonnecks at the database layer, and now your bottleneck, your core bottle neck is rendering whether you're rending the Jason or rending like htmail in your app.
So it just highlights it very very fast, and you now have to do with this level if you want to keep up with the with how fast back and this has become I think I think the other the other thing here is like rains has a lot of or Ruby itself had a lot of progress lately in the Jet compiler. So Ruby code is is gonna run faster, and it's getting faster and faster, which is really nice. We're looking at like fifty percent plus yeah, speed.
Improvement depending on what your app is doing, yeh, but but yeah.
I'm looking at something like Lobster for example. It's almost fifty percent faster in the benchmarks that the Shop of five team is sharing, which is really nice. It's it's a full fledged trade application. Now imagine this is one botle neck computation. Now the other bottle neck is IO. Now imagine if you also get even an er boost in that that regard. So, actually, by the way, moving to light stack doesn't just improve performance like I think.
Think about your situation where you have two vms talking to a single database. You'd probably end up with one VM the same size and getting almost the same performance from your application, especially if you don't have a huge bottleneck in the view layer. If you do, then yeah, you need basically as computational power as you had in the previous setup. But still you're saving a lot of cycles that you otherwise would have wasted in the communication
over the network stack. You'd save that for other competition needs.
Very cool. We're getting toward the end of our time. Is there anything else you want to add or maybe just summarize where we're at with sequal light and light Stack.
Yeah, so I would say there is a big update coming soon, hopefully like the next version of light Stack, which will have the light kid officially released. And I'm hoping also to bring one other major advancement, which is concurrent queries in Ruby. One of the issues of siquilight that it's a Sea extension and see extensions. Basically, as you're running Sea code, it takes over the process. The Ruby process doesn't allow other things to happen why it
is running. So that results if you're doing a red query or quity, whatever the type of the query you're doing, if it's a long one, it will lock the process
until it's finished. And the thing that I'm trying to to bring forth in the current next version is the ability to run multiple queries even in a single threaded application if you're losing fibers, or in different threads if you're dosing threads, so they will run concurrently next to each other and allow the application to be a lot more responsive even if you hit a daily slow query.
Cool, very cool.
Yeah, and I'm hoping I'm hoping to also make the other aspects of the story solid deployment backups and if if that, yeah, well once we're there. I I really think it makes a lot more sense to go with the simpler solution.
Mm hmm.
Regardless of what you're doing, it makes a lot more sense to go with the simpler solution. And hopefully this will be the simpler solution.
Nice, all right, Okay, on.
One final thing I want to I want to say, like, I'm really I'm offering basically a free consultation for anyone who wants to migrate to light Stack. So just ping me, and I'm happy to to help and and and and help you like gain the performance benefits and and and an opportunity for me to learn also about what could happen the wild in the process.
Nice.
How do people reach it for that?
I'm available on on X so you can d M me at any time and also you can you can send me a emails. I'm not sure if I can share my email here or not. Yeah, basically it's old mo same handle like x at Gmail.
Look what were you saying, Ayush?
I was asking the exact same question as you.
Okay, good deal, All right, well let's go ahead and do our picks are usual? You want to start us off?
I didn't thought about. Does it completely forgotten me?
Do these?
Do you want to go first?
While I'm thinking sure, I'll throw a few out. So it's been a few weeks since we recorded. Just because life is insane. Yeah, So I played a game with some of my friends that I hadn't played before. It's a really really simple game. It's called six nim. It's got a bunch of other names too, so you might have played it. I think it's also called like Take five. Not sure, let me look it up on Board game Geek. Yeah, it's called Take five or Category five or anyway. So
what you do is you have numbered cards. It's super simple. Board game Geek has a weight of one point one nine, which means that it's a casual game that is pretty simple, and they say that eight and older can play at night as probably fair. It'll play up to ten players. Two to ten players. I think it took us a half hour to play a full round. Anyway, the cards
are numbered zero or one. I can't remember. Start at one, but it goes up to one hundred, No, it goes past one hundred, but anyway, So what you do is everybody plays a card out face down. You flip the card over, and then the lowest card gets put out on top of the stack that it most closely matches to. Right.
So if the top card on any of the four stacks you've got in front of you is seven, and you put out ten, unless the eight or nine are on top of any of the other stacks, you're going to put it on the seven, right, and then the next one goes and so on. You take the stack if you play the sixth card on the stack, and then the rest of the game is is let's say that the lowest one is a seven and you play
a three. Then you get to pick which stack you take, and you're trying to get the fewest points possible, and different cards different numbers are worth different number of points. Some of the cards are worth one, some are worth two, some are worth three, and some worth five, And so you just play it til all the cards are gone, and then you tally it to score. That's the whole game. It was really fun, just kind of a you know,
an interesting filler game. And so my first pick is going to be six nim I'll put a board game geek link and an Amazon affiliate link into the chat or into the comments a couple of other picks. My son and I went and saw a quiet place day one last night, and it was good. I've only seen a quiet place. I haven't seen a quiet place too, but still it was good. It wasn't as good as a quiet place, but it was it was definitely, you know,
something that was worth seeing in theater. And yeah, I guess the only other thing that I'm just gonna let people know about if you want to follow me on Twitter or Instagram or something like that, I'm gonna be posting videos about doing AI. So I've been pulling a
lot more AI stuff into top end devs. I got really inspired by the episode we did with Obie Fernandez about building AI agents, and so this is this is not on the level of Hey, I'm going to train all this data into a model and then use my model. It's it's a level above that where it's okay, I've got a large language model. You know what can I make it do for me?
Yeah, I'd love to. I'm sorry, go ahead, no, go ahead. Yeah. I wanted just to have a shout out for Obi, like he's he's been one of my hoovie heroes for yeah, since so long.
Yeah, yeah, And I guess he's getting ready to release the Rails eight way whenever that comes out, Rails eight does, which should be in a couple of months, but yeah, I.
Think I think Rails it's going to be early next year because I think seven point two is going to come out like now, or maybe it's just out.
Yeah. I know that they were looking to try and get it out around Rails World, which is gonna be in September, but I don't know what the timelines look like now because I haven't talked to David in a while. But anyway, yeah, so he got me excited about that. I've kind of been looking at Olympia AI which is the company that he put together, but I'm looking at building an ai I assistant or set of assistants that
effectively are focused on podcasters. Right, So it's you know, it does a lot of the work of well, I want so many of these kind of episodes and so many of those kind of episodes, or hey, go invite such and such a person, you know, go invite Mohammed to come talk about light stack. Right, so it is smart enough to go make some web calls, go scrape the internet, go find your email address and then send you an email, right kind of thing. Or hey, we didn't record this week, put out an episode that we
haven't you know, re released within the last year. Right, And so then it can go and it can say, well, these are the top ones, and right, it's smart enough to figure all that stuff out, you know, or even to the level of, you know, I usually said this, you know, and then he he kind of thought better
of it afterwards. Right, it's like I shouldn't have said that because my client doesn't want me to reveal these certain things about Right, so I can go tell the bot, hey, go take out when I usually talked about this, this and this, Right, and then it'll go in and actually have the tools to go and extract it, right, you know, and my editor does a terrific job. But it'd be interesting just to see how far we can take it
with that stuff. Right where it's cleaning up that kind of a thing, right, just tell it, Hey, we recorded a new episode, so it logs into stream yard and downloads the files and kind of does some preliminary cleanup on them. Right. So anyway, it's it's that level of stuff. I think it'd be way fun. So I'm working on that. And then I'm pulling together because I know a lot
of other people want to learn it. I'm looking to do a boot camp where we use a lot of these tools, right down to some of the lms like the Lama three LM or something right where it's okay, I've got a baseline thing that I have to train some or maybe I am just going straight to GPT four or Gemini or some other thing right and just having it do the work for me because it is capable.
And so just just working all that in and you know, just over three months, just help people go from not knowing anything about AI all the way up to Okay, now I can add these features to my application, So keep an eye out for that, mostly on social media. And yeah, I'm also looking at putting together an AI summit in September.
So that's all anyway.
Yeah, and then the last pick I have, and I think I'm going to connect it to a GoFundMe or something like it is. I would very much Now that next year is going to be the last year for Rails Rails Comp, I would like to put something together like Rails Comp. And I know we have Rails World
and they kind of move it around. I guess they're doing North America and then not North America, and so I would like to alternate opposite them, right, so that there's a conference in North America every year, and then there's a conference somewhere else out in the world, right, and just see if we can get more access. But there is no way I can afford to cover things like an ex bo hal or things on my own.
And so if you all are interested in something like that and would like to contribute, I mean, obviously we're we're gonna you know, at certain levels of you know, of helping us out, you know, give you tickets or shout outs or you know, any number of other things. You know, and I'm thinking, you know, kind of base level, maybe you get a ticket, a little higher level, you get lifetime tickets, you know, another level, you get sponsorship at a certain level. So anyway, if you're interested, I
did buy the domain. I think it was railsxpo dot com. So go to railsxbo dot com in like a week or two after we recorded this, and I should have something up where it tells you what we're looking at for contributions and things like that. So anyway, that's all the stuff. I kind of wanted to get those out there just because those are things that I want to put together for the community that I think people are interested in. Are USh, what are your picks?
So I've got to both non technical picks. This week, I reread a book from my childhood a few weeks ago. It's called Havoka Rasa by Matthew Riley, and yeah, it's just one of those books that it kind of shaped my teens a little bit. Like it has a quote that I absolutely love and I loved at the time as well, which is two air is human to make
the same mistake twice as stupid. And I've pret I've pretty much reread the book just to try and remember what the context of that quote was, because I remember the quote, but I couldn't remember any of the context around it. But yes, I enjoyed reading it again. It's it's a book aimed more at a like probably a teen audience, but even now in my thirties, I still quite enjoyed it, despite it being quite far fetched at times.
But yeah, that's one pick. The other pick is a musical pick, which I don't think I've done for a while. It's a band called Solstice, who have been a little bit obsessed with, unhealthily obsessed with for the last couple of months. I basically had their two albums, the last two albums on repeat throughout my work day since about June, like start of June, which is just a bit mad
and I haven't got bored of them yet. So yeah, if you're into any kind of rock music at all, I'd highly recommend checking out Solstice the last two albums, All Light Up and s S. I A, yeah, those are my two picks.
Cool, how about you, Mohammad? You pick for us?
Not not not really, I'm not sure if that is interesting enough. But as we were like arranging to to get into this discussion, I was working on concurrent index creation for sequal Light, which will allow indexes to be
run in the background. I have I have now like a set of like features that are not open source yet, and I'm hoping i'll be able to build a service around that soonersh and allow you to host your Ruby siqual Light powered applications for a fraction of the current costs at a much higher performance than people are accustomed to without having to deal with with like Docker images, kemal or things like that, even even simpler more like hero gon Stroits very cool.
All right, well we'll go ahead and wrap it up. Thanks for coming. People find you on socials as old mo. Just remind people of that, and yeah, until next time, max out. Everybody