Ah, Hi everybody, welcome to another episode of the Ruby Rogues Podcast. I am your host today, Valentino Soul, and I'm joined by a very special guest today. Hey, I am you, Guinny. Yes, yeah, sorry, Yeah, welcome. I appreciate you taking the time to come on today. I'm gonna just introduce yourself and tell everybody why you've gotten so famous here on the show today. I'm not sure about famous, but I'm very happy to be here. And yes, I'm your Guinny. Longtime Ruby
Rogues podcast listener, thank you for reminding me. I'm a software developer who worked for many years using Ruby as my primary programming language and created some open source libraries and gems like buch loader for exam. Recently, I started my own company called Demi, and this company helps other companies track data changes in their databases. And I recently wrote a blog post about different audit trail approaches that can be used in Ruby. And yeah, today I'll be happy to
discuss it and share my knowledge. Awesome, thank you so much. And yeah, I appreciate bashloader. I definitely use that every day. So very cool. Yeah, So, I mean audologging such a popular topic to be honest, like one of those like kind of things that you over time will just end up using, right, it's something that you want to just also drop in and kind of hope that it works right, like and just like
don't have to think about it again. I think for the most part, what kind of you started down this path of like you know, really really diving into this. Yeah, it's funny that. Yeah, you mentioned that everyone is familiar with this problem or maybe at some point heard about gems like paper trail and things like that. But I noticed that it's mostly in the
Rugby community. When you step aside and for example, go into like JavaScript world or Golang world, like people have not heard about tools like that, they would use completely different approaches. And because Ruby has rails and active record as a like main or library, there are some lots of unified tools. And how I first encountered like, first of all, I used all these gems maybe for many years. For example, paper trail is one of the
most famous ones. What it does is basically, at a high level, just connects to like your model. In your model, you have some callback and these callbacks record changes in a separate table. And recently I worked at a company called Angelist, angelust Venture and angelist us Ruby, and one of the main problems that I tried to solve there was related to tracking. And there are many different reasons why people would want to track their data changes.
At this company, specifically, it was related to compliance and regulation, so the company operated in the financial space. It was important. There are some certain compliance requirements for example, soque socks and many different weird acronyms similar to SoC two which is mostly about security, but those are related to like financially reporting and things like that. And it was super critical to be able to record what happens with data in the company, as well as be able to
audit it and trace back like why this change was made. But I'm noticing that the full disclaim I'm working at a company called Demi that we try to build a productized solution something like that. And there are many different other reasons, like we're noticing why people care about audit trail in general. The most maybe straightforward one is rollbacks, like maybe you made some changes by accident or maybe by design, and you want to allow people to revert the changes they
made recently. That's one of the use cases troubleshooting is another one. So there are lots of companies that want to understand what happened. Maybe some companies implement on calls for their engineers and they do something manually, go in and do something for their customers, and that needs to be tracked as well. Regulation as I mentioned, in certain like financial sectors, banking and things like that, it's kind of like required. Some companies build activity feed as a
security feature or table stay features. For example, if you use like notion, maybe enterprise plant would have like a FTIVT feed for oddmans to see what happens within the organization. And people can also use these data for fraud detection analytics and things like that. Yeah, you make a couple of great kind of crucial points here, primarily like there's activity right like that people like the
trap, which I feel like often is like separated. And it's funny that you mentioned that this is like mostly like rails and Ruby specific, and I think maybe the other realms like go toward like the log capture approach. We're just like blogs are getting blasted and you try and like resolve all of the the time stamps as as they come, which I think is crazy Yeah, there are many different views on how to use, for example, data and
interact with the database. In their Ruby community, like convention over configuration, we have active record and many other ms data mappers for example. If you take a look at in the Go community, it comes from simplicity of the language. That is why many people try to stick with like a low level adapters and write sco queries themselves, and that is why it's even like hard
to build a unified solution like that. In the Jazz community, for example, there are lots of many ORMs that I don't know, work on edge and like work in serverless environments and things like that, So there are lots of lots of crazy technologies, which is also hard to build a unified solution. But yeah, communities like Ruby, Python, I'm noticing that. Yeah, they're more mature and they have already lots of great tools around that.
Yeah, that's that's really interesting. I mean definitely we could take some learnings
there. But I would like to just think about, like, you know, why people even like you mentioned a lot of like you know, mostly it seems like reporting and like trying to you know, align with you know, requirements, regulatory requirements, or things like that, even internal you know regulation where you want to be able to confirm that things are happening, why they changed, right, And I feel circling back to this like notion of
like activity is happening on your app and you want to like be able to isolate how the activity affected your data. Is that kind of like the approach that you're taking with themi of trying to visualize how those two things match up or you know, where do you focus on? Yeah, we take a slightly different approach. So there are many different approaches. So for example,
I wrote this blog post article that describe some of the approaches. At a high level, there are I call them callback approaches, so paper trail Audited, Mangoid History and gems like that. What they do they basically add callbacks and when you use ORM they would write history of these changes in the separate table. There are some tigger based approaches, so in a database you can write code, which is dangerous, could be dangerous, especially if you're trying
to implement complex business logic, but for simple triggers it works. So you can try to implement triggers when, for example, a record is being updated, inserted, or deleted, please do something. And there is one gem called like ITZA. It's very hard to pronounce when you see it how it's spelled, but this gem implements this approach with triggers, and I think that for majority of the companies, especially small ones, these approaches are great.
There is like low overhead. You just add them, maybe at them even just in case maybe you would do something dangerous and delete something by accident, or your logic was not tested well and it performed something crazy and you would want to roll back. That's kind of like good insurance. I would say. At DEMI we take slightly different approach. I can go more into details, but at a high level we use replication locks as a solution. There
is a pattern called change data capture. There weird like birding, but it's a data pattern used by data engineers very often. What it does is basically, yeah, capturing data changes from the database and moving these data somewhere. The most popular use case is probably ETL or solutions like that, when you would use some kind of like solution that connects to your database for example my squol, Postbose or whatever, and it tracks all changes and moves these changes
to warehouse or something like that. So we use this similar approach and change data capture CDC in short can be implemented in many different ways. There's like query based when you basically query by timestamp for example, give me all records that were updated from the last like hour when we checked. It could be implemented by using triggers again and beme, we use log based solution. So
posgress for example, has vowve ride ahead log. My sequel has been log mangodib for example, it has uplog like a skew light has vowalve as well ride ahead log. So all these locks they again briefly describe like why they exist in the world. So these right ahead locks, they're designed to be able to quickly record changes in the database in append only log before writing all the chain just to the disc with data indexes and things like that. That
are heavier operations and these locks they are crucial for data recovery. For example, if your instant crashes, it would try to recover from the last state, replace certain let's say changes from the wall and get to the later state. These replication logs can also be used for like cluster replication for example, if you have like primary and secondary, Secondary would connect to the ride ahead lock or replication log in a primary and try to read it and get and
seek the state. We use basically the same solution at Theme. We are sort of connecting like a read replica, but we also decoded. It's called logical decoding. We decode all these changes so it's you can understand like it's
not on a binary like file blog format. It comes from like decoded on a roll level, and you resee with all the changes and we simply stream and store them as a kind of like yeah in the separate table and people can use it and read it and people sometimes yeah, as I described, there are many different approaches like or use cases why you would use it, but at a high level, what we do we simply write a log of
all changes that people want to track. They can specify like what table to subscribe to, and there are certain technologies on the postulous level that were recently released that may help with performance as well. Plus we have some nice UIs that people can use to see like what happened to these records and things like that. This is really interesting. I love the article that you have that kind of breaks down all of this on. I've never heard of change data
capture before. It's kind of wild. There's a lot of pieces to the process here, which I think just comes honestly with a lot of the Avenger and kind of architectures. So I just want to take a step back, Like, let's let's walk through kind of like you know, why would somebody you know, aside from like, let's say, like what what are the primary use cases I guess for having your for having a need for that level
of like you know, orchestration and like granular rollbacks. Right, Like what you're talking about is having like I guess, a massive amount of data that you want to granularize what you're auditing and be able to recover from those in case anything should happen. Is that your primary focus, like focusing on that recovery when you're trying to do the audit trail or like where do you see a lot of the you know, the most common use cases of having the
audit trail and where like just having paper trail like starts to fail. Yeah, great question. Uh, we are very like we are noticing that it's a very horizontal solution and people use it for different purposes. That is why it's great when you give a solution for developers and they build something on top of it. We have maybe three main use cases. One use case is
just for troubleshooting purposes. We have, for example, there is one logistics company that has thirty or forty customer support reps and they're in the logistics business, so there's lots of real time data from analog world, lots of unstructured data, and they leverage AI and things like that to help with it. But they need to see what's happening, maybe help customers, their customers and cargo companies and things like that, help them understand what happened to this shipment,
for example, and they have lots of different inputs. It could be like constructured just an email. It could be a dog sign or something that changed the status of the shipment and they would need to troubleshoot and understand what's happening. So yeah, I previously like as developers, one solution is just going to a rails console or something like that. And if you use any of these solutions, you would just try to see like Querier record and get
all recent changes that happen to you. When you need to understand like why is it in this weird state, why it's not in like different state, what happened there, like what API in point for example changed it, and things like that, some additional metadata. The second use case is related to just yeah rollbacks. For example, we have there is like one company related to carbon removal tech for example, and they use POSTS, which is like
an extension for geolocation data on top of prosgress. So they do lots of changes in budgets and in certain cases when there is an error or something, they want to roll back to the previous state. And it's possible to just track certain selected tables and work with any extensions and things like that. And the third one is for many it's like table stakes, whether it's because of
regulation or for example, there are in healthcare it's very popular. There are like clinical trial management systems for example, they need to track what happened to patients data. And many of these solutions. Like one fun story like at one company, we use paper trail and we had so many locks accumulated in the versions table that we ran out of integers and we had to Like the
solution was simple. We basically okay, said start recording all changes in versions V two table, so we would keep the old one it will like cold archival, but all new changes please record in a separate table because we don't have any integers there and player IDs, So that is why it's good to use UIDs when possible. But one issue with for example pay Petrail and solutions like that is reliability. Maybe like people, maybe you heard that it's possible
to even with rails, to execute certain queries that bypass the callbacks. For example, if you execute like delete all or update all or update column or something, it won't trigger the callback, does it won't record the history, so you will not have the complete history. So that is where like log based solutions are critical and could be very useful because even if someone goes directly to a database executes an SCO query, it will be recorded because it happens
on their database level. And that's one of the kind of like main reasons. In terms of plus, there are some additional benefits like no performance penalty, so you're in the run time, you don't insert any additional records. It happens asynchroneously, but that's at a high level. Yeah, you made me think about because a lot of a lot of what we've been talking about
has been surrounding like just auditing very specific processes around the data. Right, Like we get a you know, going back to the logistics approach, right, like somebody's submitted an order and they're tracking the process of that and the order you know, I imagine has a table of orders and various aspects of the order that are being tracked with it, and that gets updated in the database kind of the same way, and it gets tracked, you know,
as it progresses through whatever business logic that we have. What I feel like is kind of missing from what we're talking about here is like that that overarching activity like almost it makes me think of open telemetry, right where we have like distributed systems right where they don't have to be necessarily you know, technologically right, but they are distributed events that happen throughout the business that fire off
and some activity happens and that has effect on the database. What do we do for like cases like that, Like are there like tools available to audit that process and how that's affecting the individual audit trails that are happening, and like where how do you approach that aspect of it. Yeah, you're definitely right. There are some like tools for example data Dog and things like that
that help with some visibility, observability, and tracing. Many of these tools they even implement a like just logging, so they would log what happens and for example, if you use like postcris, there is an extension popular one called pg audit. What it does it basically all queries that were executed. So if for security reasons, someone for example, gained access, you'll be able to, like during like cyber forensics operations or something, we will be
able to detect what happened and who executed what. So there are some solutions that do just log in off activity that happens, and you can do it manually as well. There are some gems like public Activity or ahoy gems that maybe used like Google Analytics on the front end or something like that, or in the browser it's similar but on the back end, when you can manually trigger certain events that you want to track and they will be stored in structured
format, not in locks, but in a database for example. There are also solutions that help with just performance, observability, open tracing and things like that. But there are not a lot of solutions that do something on the data level, and it's a little bit weird and funny because yeah, most of the products they use databases and that's the source of all data, and yet we don't track it like developers use geed for tracking and version in their
code, but there is nothing for like databases. And to be fair, there are some solutions that people sometimes use. For example, there are maybe two technologies that may help you solve this problem. One is temporal tables and another one is event sourcing pattern that some companies and people use, so temporal tables. I'll try to summarize it in the simple words. Basically, it's
within the SKAL standard since two thy eleven. What this feature does basically what I described it kind of like has a version of all records, So you have your record, plus you have a range and a timestamp basically a range
of time stamps during which period this record was there. So for example, if you're updating a record two times, there will be two records in a temporal table, and the first record will have like an end time range when the second update happened, and second update will have like infinite time range starting with the time when this update happened. So with the functionality, and that's
very powerful. It's usually helpful when you do for example, reporting I don't know, you need to generate some files or what happened last year, tax or something like that. That's very like powerful feature. Although many databases including postgars unfortunately don't have it implemented, so you would need to install some extension or build it yourself. So that's one option. And another one for just
all dating changes is rethinking how you make data changes is event sourcing. Event sourcing in simple words, it's kind of like maybe the best analogy is geit. So when we use GIT, we basically use an event source system. So GIT records all events, commits and all the changes that happened and then and that's the source of truth. So committs become the main kind of like unit of data. And from these commitits, the powerful feature is that you
can travel in time. So for example, if you want to roll back or see what was the state of like three commits back, you can do that by traveling and basically undoing the changes of each commit. I would say from my perspective, like in software engineering, event sourcing is like a unicorn idea. Everyone is talking how cool that is, but very rarely people implemented or do it successfully because it's a very challenging problem because you deal with mostly
with immutability of these events. So every change basically needs to be mutable. But what if you need to change the structure, then you need to implement some kind of like versioning or replay the events and append some additional data or like change the things. Plus then you need to somehow get to the actual state. So you'd record all events. For example, shopping card, you'd record an event that somebody added an item to a shopping cart, somebody clicked
on purchase, somebody purchased and it was paid or something like that. All these events would be recorded, but at the end of the day you need to understand what was purchased, for example, when showing the whole order history. To do that, people, you use projections sqres like for like separating reason rights, and lots of lots of other patterns from the book called the
Main Driven Design DDITY, and that becomes very complex. I would say that event sourcing makes sense in certain regulated industries, and there are lots of great patterns or like how people do it for example double entry book keeping system in banks or like yeah, accounting and things like that. This is standard where you would probably try to use this approach from the get go. Yeah,
you make a good point. I mean all of my even rooven design architecture related stuff as always, you know, the complexity raises as you start to spread things out and you try and like capture what you know, collect all of the things and merge you know, the state of things. You know, which pieces do you pick? Right? Like anybody that's had to merge conflict and get you know, can understand the issue here, right, like,
well, which which changes do you use? Choose right? And that definitely becomes more of a problem as you start to spread things out with you know, your event driven architecture. Yeah, exactly. Yeah, it's much simpler to use, for example, like crowd operations for example, just update
the record and that's transactional, right and do it or not. But it does make me kind of think like okay, well you know, as you know, it makes me wonder, like I guess if you have a need for right, like the things happening at different times, but really they're all the result of like one thing happening and having a simple audit trail, like it makes sense to just throw a gem in and then just have it handle
all of the things. But when it's like when it's something like you know, a time series database or something like, are we just solving the wrong
problem right? Like why not just have something like in a time series database that keeps track of the data as it happens versus spreading it out like what's the where do you see the benefit of having it be distributed like that over having it be more transactional and having things like Okay, well it happened in this series, Like you know, is it worth it what I'm saying, you know, to have that distributed nature versus just like taking the time and
doing things in series. Yeah, that's a good question, and I think the best answer is favorite developers answer is it all depends. But I think if you need strong guarantees and you have scale, then it totally makes sense to use this system. The question is then whether you buy or do it
yourself. If you do it yourself, then you would need to start thinking about all these like technologies school technologies like Kafka and maybe storing it in like yeah, time series databases and columnar databases and things like that or warehouses and use it somehow. If you need to use it from the app, then you need to somehow connect to I don't know snowflake to read it if you
need it for like transactional or like operations. Basically I think like back to this like anecdote with large number of paper trail records, like anecdotically, like from talking to other companies, I saw that for many of them, just managing the database becomes much more expensive when you record it everything in the same database. You would notice that, like this version stable becomes your the most
biggest table in your entire database, and that affects everything. It affects like backups, the storage, how fast your database would run, how difficult it is to upgrade it at scale, how fast it will start, how much traffic is there plus activity and things like that, and at some like that's probably like totally valid approach at a certain scale, but at some point you
would probably notice, like why am I storing it in my database? That is being killed by this performance penalty because you're essentially doubling the throughput on every For example, insert you would insert an additional record, or it could be even worse. If you're inserting one hundred records in one query, it will be one to one one hundred plus extra one hundred queries. For each record
that was inserted, it will be one hundred additional skill statements. And at some point it makes sense to somehow extract these data and it could be still in I don't know the same type of database that you use, and there are some benefits to it. For example, if your main transactional database is postgress, for example, what we do we store these changes and we call it destination database where the changes are stored from the source database. We store
it still in postgress. The reason why is because one is easier to consume. You just use the same adapter, same or m For example, in rails, you can specify a couple of connections for example, for these models go to these database, but for this model go to the other database. So you're on the application layer. If you want to consume those, we don't need to change anything. And two, especially in postgress, there are lots of cool tech technologies that help you with scalability. One is maybe yeah
time series. We mentioned, there's like time scale extension or super extension because it dramatically changes the way positives works and it allows you to scale, compress data, and record events in chronological order that it is designed for that with all these charding, partitioning and things like that. Yeah, that's really interesting. You know, that's definitely something I've seen, you know, scaling as
things go. You you you know, you want to set it and forget it, but then all of a sudden you're like, well, why is the database crashing when when we want to like you know, audit the audit trailing. Yeah, in affects not just the database, even like your run time. So your application is slower as a result because it waits for database,
but also tests. For example, if you don't disable for example versioning for your models in tests, Uh, imagine like eating a large like database or lots of things and testing things, it will just yeah, kill your test performance as well. That's a great point. So I mean, what
what can we do to like avoid these pitfalls? Right like aside from like, like you said, you like using some of these other extensions, Like, are there like best practices just like out of the gate that you can do to like maybe avoid some of uh, you know the performance issues upfront without too much complexity. Yeah, I think there is no like silver bullet.
Unfortunately, at smaller scale. For example, if you're a startup or something and you don't have lots of people or like customers or users, then it's probably not a problem and you can just buy yourself time in terms of velocity by building features and not thinking about optimizing things. But yeah, as you scale, unfortunately, companies hit this kind of like moment when they realize
they need something and they already build it. Usually like cod basic becomes giant, there are already lots of like team members and engineers on your team, and now you have to migrate everything, and that becomes very difficult to do if you want to like change things dramatically and these migrations. Yeah, I talk to some like people who build similar systems at I don't know, Airbnb, Gusto and other companies. It's all like very difficult when you need reliability,
performance and things like that. Yeah, I would say, yeah, I think it depends on your business. If you're starting and you think you would need all these kind of like tracking functionality and it's business critical. It makes you a business better. You can help, it can help you win
customers and things like that. Then I would think from day one, like how to potentially designed this system either use like event sourcing, or use these CDC change data culture or like some products that out there something like that. But if it's just like secondary, then you can either buy at later stages if you're successful some solutions or just it's not like the main problem and it's
probably will be deprioritized for many many years. You know. Part of me always wish that there was like a log rotate version for database tables or things like that, right where okay, after so many so much time has passed, you know, just take whatever's there and like stash it somewhere else. Yeah, almost like an auto database charter. Yeah right, yeah, yeah, Unfortunately it doesn't exist, but yeah, you need to build it yourself.
Yeah, and we we plan to do that. We call it like retention period, so how long people need to retain their data and changes and audit Because storing everything in the hot database that can be quered at any time is expensive. That is why people sometimes move it to like some other cold storages and archival or something like that. Yeah, it's three buckets and that's
awesome. I mean it's definitely needed. Uh. I wish so many times I had just like an option to buy that, right, Yeah, rather than have to build it and manage it and worry about performance of everything. Yeah, for sure. So where are you taking this from here? Like,
what what are the next problems for you to solve? I mean we've talked about quite a lot of different so I mean we haven't dove in quite to maybe some of the specifics, like you know, doing various layers of logging, right, Like we've you've talked about like HTTP logging versus data based specific auditing, right and console log in which we haven't talked about, but really interesting stuff from like thirty seven signals right with Console eighty four and audits
for doing that in the reals console. Yeah. There are many layers, Yeah, where so many layers? Yeah, many layers are covered and some are just there are huge gaps. So for example, you mentioned, yeah, HTTP logging, it's all done. For example, you can application execution, it's done. There are services data maybe it's solvable, and now we're trying to solve the dot on the database level console access. It's also I
would say a little bit specific to the Ruby community. For example, in the JavaScript community, there is there are not usually good like repal consoles or that come with frameworks and things like that. And it's good and bad. I'm noticing that people. It's good that people don't go to this console to do crazy things there, but sometimes they go to scale console directly or connect to and scale and perform changes on the database level. I don't know which
one is better. Yeah, with a review, Yeah, it's luckily we have great console experience. We have like gems like Pride that allow you to i don't know, go and jump into like classes and expect objects and things like that. And yeah, there are a couple of gems. One is Console nineteen eighty four and another one Audience nineteen eighty four. These are interesting gems. I personally haven't used them, but they seem to solve like very
popular problem, especially among B to B companies. Probably when you have customer data or customer requests. You need to do something manually, but you want to audit what changes an engineer did. For example, what these gems do. They simply you install them in the rails project for example, when your law to your rails console. They'll prompt you and ask like why are you
here, like what do you want to do? And engineers can specify like a ticket number and specify some context, I'm solving this customer problem blah blah blah, And you can type all the commands, and the commands may be locked in a separate database table. And the audience Jam nineteen eighty four allows you to inspect all these things. You can implement some additional approvals or like
retroactive approvals. But these solutions are like and I don't know whether there is any better solution, but these solutions are not bulletproof in terms of reliability accuracy. Because it's Ruby. You can do whatever you want in a console. I can overwrite everything and then after that do crazy things that no one would see and notice. Of course, you're hoping that your colleagues or you won't do that breaking all the rules. But it requires discipline, and yeah,
it's funny. I tried to implement something similar once at another company when we had a little bit more complex set up with SSOs to a WS accounts. These accounts would run on kumbernities certain process with the Rails console with some pre defined en viireables and these kind of like sets certain context for for example pay per trail or tools like that, so you can see who is making the
changes. And yeah, I haven't shipped this feature or it was not like released, but it's still I would think that it's a problem for many So if you're listening to this podcast, yeah, that's one opportunity to improve how to improve log in and auditing of their console sessions because many companies just leave there like developers leave there troubleshooting purposes, yeah, and things like that.
Yeah. Yeah, I remember first saying console nineteen eighty four and the audits nineteen eight for a kind of funny naming, but yeah, really cool. I've used them in a couple of projects. And it does have like you have to use like what is it the active record encryption stuff dataly so if you use maybe lock Box or something else, you're kind of like sol. Yeah, but it's really interesting, Like the interface is definitely pretty smooth,
and I do like it. But you're right, like it doesn't really prevent anything. It's more just like all right, well, as long as you you know, and you know the goal is to hire people that you trust anyway, So as long as you have like at least a process in place to audit it, it definitely makes things much easier to track that. And
from your experience, what did you do in these consoles? Usually, Like from my experience, it's usually either checking the data what's there because you don't have like fool UI for everything or something like that, or sometimes editing some stuff there as well. Some curious whether it's a common Yeah, I hate,
I hate going in production console. So for me, it's mostly querying if anything, right, Like, you have a bunch of related objects and it's like almost not impossible, but like much worse to like go into the you know, a sequel dump or something like that and sift through the queries
for it. But sometimes like you know, you need to, you know, just update some timestamp somewhere, and it's just like, you know, rather than having to generate a rag task that goes and runs, right, if you just have like one record that you're trying to get to that's problematic. That's like you know, failing a ton of repetitive workers or something like that, and you just want to stop it from doing that, you know,
but you want to track that. It's that that that you were the one that like updated it, right, It's like it's very rare that you need to do that. But there are those cases. Yeah, in these cases when people need to update data. Yeah, I think there are many also approaches how you do that, like one one off, Yeah, you can write a rake task. Some people run it in migrations, some people
execute stuff in rails, console or things like that. They think there is also like opportunity to improve things because writing, like it's good when it goes through the code, but the cycle can be in feedback cycle can be still like long and you don't know like whether you wrote everything perfectly or not. You need to test it and things like that. And yeah, I mean I would say in general like probably don't update things, yeah in a production
box and general like no writing. I feel like that's a good rule of thumb. Many just ben yeah, access like only for example one person, or like certain like deblop steam can access or something for troubistioning or like certain
emergencies. I know, I know some companies will do like you know only uh you know the console access only gets read replicas, so then it just prevents it in general, right, So then you just you can only read from the database in general anyway with your connected to console, which is really interesting and definitely like prevents it. Yeah yeah, but definitely not as flexible.
Yeah, Ideally you'd need to tightly isolate everything, like there are certain other distributed systems like readis cash man cash people can also use and drop things by accident. Ideally would also want to isolate those, maybe certain HTTP requests, And we also like logging each TTP requests that are going from your application because I can use all environment viiables and send I don't know, stripe request
to send myself like one million or something. Ideally it should be also traced like who made these HTTP request and or maybe ideally prevented from doing it in the console. So, I mean you bring up a good point, like I feel like we haven't touched much on like the recovery aspect of all of this, right, Like the whole point about and logging is in case you need to recover something, right like, or you need like next to reporting, right like, so reporting you can gather Okay, well we're doing a
you know, audit specifically of everything that changed this particular thing. But like most of the times I feel like the reason for that is there's a recovery aspect following that up, right, Like, so where is that in this whole process? I know, like some of these jumps we've talked about have some of this, but I feel like it's still kind of missing, Like what is what is your like approach there, and like how are you working to like resolve this kind of like stuff. Yeah, recovery is very tricky
problem in general. One issue, for example, is what if you somehow backed up some data or certain rows. You have the record historical version, but now your schema has changed. If you're using a scale like databases, Like, how do you resolve this conflict? What if you added the column you prepopulated it, but in your history you don't have it. Now you have to prepopulate it because the column may not like contain like shouldn't contain null
values or something like that. So if that becomes a problem. In general, you have gems like paper Trail. They allow you to query the version find the right one. There is like a helper method called Rafi. You call it and it basically instantiates an active record record with the version, and then you can call say again and the tool overwrite and store the historical changes. As I mentioned, Yeah, there are some schema changes that they kind
of like up to people to decide how to resolve these conflicts. There are some other different approaches that on the database layer people solve. So the one is very straightforward to snapshot or take back ups daily nightly. Ideally you maybe test your disaster recovery procedures and test whether you indeed can recover because very often
no one tests those and just things that they're safe. And there are also new popular techniques maybe heard about point in time recovery, So that's popular feature that many database hosting providers advertise what they do at a like in terms of implementation, how it works, Okay, how would you use it? You would use it by simply selecting any time you want to recover data, and it will either create a new database instance with the state as of this time
stamp or it will replace your database state. The way it works it uses these ride ahead locks that I mentioned previously. Because ride aheadlocks are like append
only logs, these solutions they continuous to archive them. So some companies just take these valves and world records and they move them to S three and then if you need to roll back, they give you like either the timestamp when, for example, a certain blob was archived, or they may also do some binary search within a certain file to find the exact timestamp you want to try to recover to this data point. And there are some other like cool
tag built on top of this idea. If you have like fowl and it's kind of like stored somewhere reliably, then you can do branching. So there are some serverless lots of server less hosting providers. What they allow you to do is to use basically copy on write copies for for example, your branch.
When you want to test something on staging, what they allow you to do is basically create a database that will simply point to the state from a shared voeve kind of like log So instead of copying all data again and paying for all x ray resources, it will copy from the main let's say history, but any additional changes that you'll make, it will branch off and it will just incrementally record some additional stuff. So some super cool technologies that out
there that help with branching as well as data recovery. Yeah, I've definitely seen the branching. It's pretty interesting, like is that a you know, it can just either wipe it off or like integrate it as as it merges down if you want it to, which is pretty interesting. Like, but in reality though, like what what have you seen? Kind of like things evolved too, like you know, what are people actually using? Are they
using copy or on? Right? Like it seems like a very like complicated process to integrate with, like what what are the where the trend's going? Yeah? I think in general people prefer cloud managed databases usually for these reasons because they can manage backups reliably. So if you run something on RDS or like I don't know, render SAPA, base, HEROC or whatever, you can trust that the backups they take they're probably fine and you'll be able to
roll back if necessary. That's one first line of defense. It's just relying on your hosting provider and any additional ones. Yeah, related to using additional tools, like the way I think about what we do. For example,
at BEMI, we provide selective point in time recovery. What it means instead of like it's very hard and dangerous to roll back the whole database state, rather than like rolling back certain data records that were affected, and with tools like that, when you can filter out changes and select the ones that you want and just apply those, that's very useful because usually, yeah, I rarely saw when people would recover from like full database back up, because it
means you would lose your history what happened after that, if certain other clients used it, and things like that, So it becomes very problematic and not very practical. So I have to ask this mostly because it's my interest, But like, do you see like AI fitting anywhere in this process us? Or is it kind of just like you know, there's this it's two sensitive
of a topic to integrate with. Yeah, I think there are some, and I would share one idea that the problem we try to solve with AI, maybe we'll solve it, But it's okay if you have a desire to do it, to go for it, because it will be great if you can solve it. But one maybe obvious solution is like AI for SKL. There are many non technical people who want to use like query data in a like business intelligent tool, metabase or something else, but they don't have like
the knowledge how to stitch things together for complex scenarios that's one idea. Another idea is we mentioned the audit trail and log and for example we have some customers, even non technical people use our UI for troubleshooting. But what could additionally help is data mapping. So usually database structure, it doesn't very well translate to real world language. You'd have some different ideas, maybe especially in
Mongo. I remember like all days in Mango, it's like document based like database, and it doesn't have a structure, so the column names you may have different columns sort of columns. It was recommended for performance reasons to shorten your column names to like one or two letters or something like that because it would take less space. And imagine seeing like an audit trail with weird like data changes and with m is equal these timestan b is equal weird string.
People would not be able to understand it. So where AI could feed it here is basically taking the structure and humanize it. Understand the data structure, understand maybe some business logic from the code base and muppets or anyone even non technical like customer support ors like the CEO could understand what's happening if they don't have like knowledge about how data is structured, and that's in general related to
these ubiquitous language DVD concepts. Ideally everything should be the same, but that releively happens. That's one idea. Another one is like some companies they they maybe you heard about techniques like RAG for providing some additional context with custom data to your like AI system, so it can use internal data. Maybe if you've use perplexity. Perplexity uses other resources that maybe even like updated in real time just an hour ago. Plus they have all these citation links pointing to
the source, so red techniques can help you to do that. But companies in general, when they deal with internal data, audit can also help. So you want to like have clean data, especially if you operate in the space when there are lots of documents and structured things maybe files, maybe emails. So I'm not sure, like it's for me, I don't know like
how to productize it or what's the exact like solution. But the problem is that, yeah, people want to take this data, but they also want to understand like what happened, for example, why it was like that, why these AI systems aid it like that, maybe we can find unit and you'll see that with like, yeah, certain conflict changes your response rate and accuracy has improved and things like that, so it's important to audit the results
of the AI queries as well. Yeah, super interesting takeaways there. Personally, I'm looking forward to, like, you know, just just getting some observation aspects of it out, so you know, like kind of just automate, automatically getting like the you know, bullet gem or something like that that just tells you, ay, like these things are not performing well, but without having to actually capture them. You just look at it sideways. Yeah,
I hope. Yeah, things will improve and things will be implemented. Yeah, I use the topical pilot and yeah it's some people hate it, some people love it. I like it. When it changes your mind. You have to know and understand how to use it as well to better leverage it. Sometimes I would write a comment explaining what I'm trying to do and do it then so just me some code that more or less much is but then I still need to check sometimes and things like that. But yeah,
there are lots of opportunities for AI. Very cool. Well, we've talked about so much here. Is there any other aspects you wanted to cover before we move into picks here. No, I think we covered a lot of ground. I think in general, Yeah, that's like tracking changes. Auditing is a big problem. There are tons of tons of ways how to solve this problem. Yeah, and I would recommend people try to evaliate them all instead of trying to just going with one by default because it's the hold this
one or something like that. See what works best for you. Yeah, well, for sure people should check out bemy and your blog has a lot of great content around audit logging and tracking various activities and the database things you can add. I would definitely recommend people check that out for sure, because I'm already going to dive into quite a few articles on there. You've got a lot of great stuff there. All right, Well, let's dive in the picks. Do you have anything you want to share or I can go
first if you want some time to think about it. I have two picks.
One is non technical and one is technical. So the non technical one is maybe if you're old enough, you remember there was a technology called RSS and now I'm trying to use it on a day to day basis the way I'm using it, So there are for example, lots of great technical channels or content out there, and instead of these systems being pushed to me or me checking I don't know subreddits or like hacker news and things like that, I try to aggregate all my content as a kind of like funnel into an
RSS feed. Some of them I use feedther, but there are many others and there's similar some of them they have help you get email address, so you can also subscribe to substack without polluting your in box. So I'm just organizing it and different subfolders I don't know tech related, I don't know, some engineering related and other topics, and I aggregate everything like stopstacks, top
subreddits within a day or something. With some ural arguments, you can get to these feeds as well, tech blocks from different companies, top hockering news within a day or something, digest. So lots lots of resources that can be accumulated there, and I have just a queue that I'm going through and reading when I have time on mobile or on a desktop. And my second technical peak is Nix. I'm not sure if people heard about this system,
but it's basically package manager and system configuration tool. It's sort of like our band r VM, SDF and all these things. But it's more powerful on the almost like operating system level. So it's not like just limited to programming languages for example, but the way we use it, for example, we have lots of projects in different languages. There is a full disclaimer. Nicks has, from my experience, has horrible devicts. It's very hard to use,
very hard to understand what it does. So there are some wrappers around it, for example death box, but there are many others we use death box. So essentially, what it allows you to do is to bring any tax stack without Docker to your local machine, and it's isolated to your project, meaning it will create a separate folder for your postgress for your readis whatever, for your like jilascript engine or not Jazz or Rugby and things like that.
You can specify all these dependencies so when you go to your project, you can keep using your I don't know favorite at a Sage orgh or whatever shell uh, and it just automatically enables all these additional dependencies within this project. Yeah, that these are my peeks for today. That's super cool. I'm gonna have to play around with that. Yeah, Nix is definitely a terrible, So thank you for sharing death box because that's pretty cool. So
I only have one pick today. Uh. I my awesome co worker Anna, She wrote this incredible article on the Toroximity blog recently on evaluating GPD outputs and how we do it for medical professionals, uh and using golden data sets and programmatic emails and some human emails as well. Uh, to just make sure that uh, you know, that output you get from large language models is accurate, uh, at least following you know, a certain set of
standards. So I recommend checking that out. It's really awesome and just like goes through a ton of stuff that leave them working out, which is really fun. That's great. Yeah, I hope that all healthcare providers not just blindly used AI without really too. Uh. Yeah, it's been great talking to you. I love to see all the work that you're doing, uh to help make auditing easier for a lot of businesses. And uh, you know, thank you for all the documentation and coming on the show today was
really great. H you know, chatting with you today. If people want to get in contact with you, how can they reach out to you or find you on the web? Yes, I mostly and rarely use Twitter or x it's cold nowadays. I have like medium githop everywhere. My handle is exaspark. You can find me there. And yeah, thanks for having having me here Valentina today. It was nice chatting with you. Yeah, totally.
Yeah. I mean until next time, everybody, I hope you learn a lot and can make use of a lot of this stuff because I know I will. All right, bye now, bye bye