Hey folks, welcome back to another Ruby Rogues. This week on our panel we have Ayush Nwatiya. Hello, hello. We also have Valentino Stoll. And now... I'm Charles Maxwood from Top End Devs. And this week we have a special guest, and that's Emile Kamp. Hey, everybody. So, Emil, I'm just going to clear my throat because, wow. Okay, let me try that again. So, Emil, I found you. I've been... corresponding with people, some on the Discord for Kamal.
which is the deployment technology that DHH put out. I think he made kind of a deal about it at Rails World last year, and then a whole bunch of exciting stuff this year. Kamal Proxy and stuff like that. I mean, I switched over from Kamal to Kamal 2 with a whole bunch of help from you and others on there.
I have to say, switching over was a little rough. Having switched over, it's a Anyway, so beyond kind of what I know about you from that, what should well so I think First off, I am a mountaineer, and in order to facilitate that, I live in a very flat country so I do a lot of running and then I need to pay for my travels so I do also engineering. And I've made a career of doing sort of critical infrastructure or things that have real world impact and real world.
consequences if something goes wrong right so both in terms of security and robustness and things like that and that's everything from the FinTech or the payment industry to civil engineering to online voting. Wow. Yeah, it sounds like kind of what our topic is here, which is building critical infrastructure. So you've done some stuff. I kind of need to work, I guess. I mean, so banking obviously needs to work. People get sort of pissed if they don't get their money. And then...
For civil engineering, I worked on tools that the engineers would use to build the airplane. which, and run tests on them, right? So both test benches and design benches for airplanes. And I mean, that has a real world consequence because they fall out of the sky, right? If you mess it up. And then, yeah, online voting. That's a whole thing. And you guys in the U.S. just went through one. One of these messy events, and they're messy everywhere, right?
And recently I moved to compliance and sort of making, trying to make it trying to make that easier and a better experience to become compliant and to obtain a certification. Right. Makes sense. So let's just dive in. I mean, I'm not I'm trying to think if I've ever worked on anything that was, you know, kind of at the level of, hey, if this break. really bad things happen. I mean, if it breaks, I've had them come to me and be upset like really bad things. It doesn't affect society at large.
what's the difference between working on those apps and working on just kind of your, you know, you mentioned not e-commerce or not, you know, hey, I've got to manage my social media or, you know, get podcasts out on time. Yeah, I mean, there's nothing, just to be clear, there's absolutely nothing wrong with that, right? That's just, I started out in that area as many developers do, right?
like some ad system when i was very young and then moved to some you know sort of author platform and then i sort of realized that it was it was just inconsequential in a way right I looked at people. You know, I have people around me. My father is a civil engineer, right? And so I have people in my family, my near family, that was building things in the real world. And I was like...
I want to do that, but I do like this computer stuff. So can we sort of maybe find a way where I can build things in the real world just using my computer? And so I slowly started to shift over into something that... founded more in civil engineering and sort of industry rather than commerce. Hmm. And then I started in a fintech company where I was part of building out and continuing to build on a system where you can... Basically, it's like direct debits, bank transfers, but worldwide. Okay.
So, and there, it doesn't necessarily mean a lot to you and me. And it's the same with, there's a bunch of them now, right? Like these send money home types of things. It doesn't mean a lot to you and me, but it does mean a ton to those who is on the receiving end of that, right? Because that's like half a year's pay or something that comes in, right? So it does matter for those people.
so so yeah that was sort of where i ended up and we are moving a lot of money like it it it seems it's one of those where little like it's it's a lot of tiny things that There's like millions and millions of dollars being moved around for that purpose.
And you don't want to mishandle those money, right? So you are hooking into the banking infrastructure, into Swift and into the IBAN network and the whole sort of... um pc compliance and everything with that right so you need to make sure that the software is robust, well tested. Testing has been a main thing of mine for a long time because of that.
and also document it well, right? So that's where I sort of started out with being required to document what I was doing and how it was supposed to work rather than just shipping fast and breaking things. I was required to test and document then test then stage it, then test some more, then ship and not break things. So that was a different paradigm. Yeah, so that's sort of where I first met that, right? And that was a pretty great experience.
so I was like oh can I have more of this can I maybe even get even closer right Yeah, so that's where all that started. And then I was hooked on the whole... Can we test it? Can we build something that runs Ruby on Rails, but makes a difference to people's lives, like in real world, right? So this was basically a payment situation just building Ruby on Rails.
And back then, many of the banks didn't have a proper API, so we ended up scraping a lot, and that's against terms of policy and terms of service and all of that, and they were really mad, and we went through that whole shebang of getting... you know sort of scabbled it and fought for that for a while Yeah, so it was a bit of a wild west back then, but it's definitely gotten better.
I think there's a lot of compliance work and legal work that went into the EU and did a lot to make sure that banks would have... not standardized APIs, but a minimum level of information that you could retrieve with an API, which made all of that a lot easier. It also means that I can technically build my own banking app from my own bank account because I have API access to it. It may not be obvious. It may be poorly documented, but it is there. So I think that's pretty cool.
Yeah. I'm curious if Valentino or Ayush have experience doing this stuff. Nothing that critical now. The closest I kind of came to anywhere near that world was when I worked for TransferWise, but I was an iOS developer, so... I didn't really have to do any of the hard stuff. I just made API calls, but I know they were also doing quite a lot of the scraping and stuff. There's a lot of selenium going on behind there to get into different bank accounts. It's all a bit scary when you think about it.
Yeah, it really is because your whole banging experience is tied up on some scraper working correctly and not messing around with the thousands of delimiters and the... The sent commas are different in different regions, right? So you don't want to mess around with that. I did transfer some money incorrectly. Luckily, the backbone banking system is pretty robust as well, so we were able to basically not move the money anyways, right? I did move far too many money at the wrong place.
Oh, never mind. That was a back-breaking day. I'll bet. So, I'm a little curious as we get into this. Like you mentioned testing. What does your testing setup look like? I mean, I'm sorry when I write my test, it's like, okay, it's good. Like, I'm reasonably confident, right? But I'm not, like, ironclad confident, like, banking-level confident that my code works right.
Yeah, that's a good question. I tend to not be too dogmatic. I know there's a lot of like do this or do that paradigm or do test first or whatever. Bye. I tend to just play around with it and make myself certain first that this is going to work whatever I'm sitting with. And then when I know that it works, it does what I need it to do, then it is worth testing, right? I try and test on the lowest possible level first, which means I will start with something like unit tests.
For that, I want to mute everything, right? You want to make sure that every argument... So you basically start with the least amount of arguments for a method and then you add in arguments slowly in I use Aspect. So it's in every context block or in every described block, right? So you're certain that every version of every argument is tested and has been permuted.
variable has been tested on its own, so you know how it behaves both with and without that particular variable. And you also know how it behaves if the variable has the wrong type. So because Ruby isn't quite where we have type safety yet, there's a lot of work going on with that. But we aren't there yet. So you want to make sure that we can handle strings.
numbers, not strings, trues, booleans, whatever. There's like, I have a little thing that basically says take this and just cycle it through all of the different types it can be and just put in garbage so that I know it doesn't break when a number is presented rather than a string or something stupid like that. and then you slowly move up. So you're confident that this method works, and then you move up to have a call it, and then you...
What I do is then I go through and test that. But I think... Opposed to what DHH preaches, or at least he has this idea that you would go through the entire stack, and I believe that their tests do that. I tend to... I tend to sort of stop testing at each level, so model tests are their own thing. and then when you test the controller you basically verify that the model is being called with the expected parameters because you know you've already dealt with the parameters on a lower level.
So it's not mocking per se, it's just that I only need to test that. model was created or was called, I don't really need to test that it was the correct argument, right? Because I'm certain about the correctness of the arguments on a lower level than and then just sort of slowly move up which means that I find that that means that the individual testing level And it gives you a little bit more freedom to sort of move a model around because it's coupled somewhat.
on a test level, but not tremendously, right? I only coupled the controller to the model in the sense that I checked that the model was called, not that it was called with a particular set of arguments. So it makes it a little bit easier to also restructure your tests when your code change. But it's just a lot of work on lower. I'm a big fan of Sorbet, but it's not.
It's not quite there yet. I think it works reasonably well, but it's very hard for developers to use. So I think there needs to be a lot. of work done in terms of making it nice and sexy to use so that we can get people to use it right. I'd love to dig in a little bit more on the disability aspect. Because one of the biggest things I see missed is maybe like... skipping over like maybe some lower level implementations and then like a system update break something and it just wasn't covered.
How do you go about like, you know, weighing whether or not that it's worth it to like actually run like a system command or an external API call or something? Yeah, so system commands, I usually check just that the signature of the call is correct. And probably that it returned the right thing, right? So that if I hit the OS, I'm basically testing that I gave it these parameters and I expect that. something back like a string or whatever it is, right?
because it's so hard to know what it was supposed to be, because sometimes the system commands will change their signature, but not necessarily the return value or the return value, but not the signature. So that's... very hard. I try to not go there. I try really hard not to go there because it's out of my control, right?
on the other end when I call other APIs. I definitely test that. I use something like VCR or something. I capture all the network traffic and I make sure to do the same kinds of... sort of making sure that if I call it with these arguments that I have control over I get this particular response. And if I do something else, then it breaks, right? Or it gives me the incorrect response.
And I also found that, for example, with banking, that one of the cool things we did was that we basically took all of the historical records that we had for all the bank accounts and we basically used it. Because then you can train on that. You can basically say, if I feed it all of this information that we know happened and it happened correctly because it happened, you know, a year ago or two or whatever. It should result in a particular set of... It should result in a particular...
value on the account, right? So you can actually use historical data as a testbed. And that means that you have a Factual. way to check if it's correct So yeah, I tend to use that as well. I also use that when I did software for the... for this for the sort of civil engineering and building flights and testing that right we also used a lot of existing information and to make sure that we had input output
so we understood that given these inputs this is the output that we would expect right and we could basically take old designs and try and run them through and you can see if does it break it down like you would expect does it run does it sort of fit to the engineering standard design standard that we were implementing against. They also have test value that you can use, right? Or test data. So it's also about going out and finding
big test data sets that you can use to also make a statistical sort of approach, right? I'm not making any statistical claims, but I am running. a lot of tests on a lot of data that you can be reasonably certain that if there were errors in your system, it would have been caught by this because it's so much. effective production data, right? Yeah, I imagine that working with the public sector that you would have a lot of access to that data, right? I guess depending where you live.
Interestingly enough, none of this is for the public sector. which is a little bit weird. You would think that it was, but it actually isn't. The regulation of the aerospace is public sector, right? But the actual building of airplanes is private commerce, right? private industry and likewise with banking the regulation of it is public but the actual banks are private. Do you find that difficult for the data access? Is that data proprietary then for doing large swaths of tests or finding it?
Yes, but most of the times the companies are interested in providing it, right? Like if you go out to pick... a big airplane or an aerospace company and you need to design a test system for them.
something to help them build right they are interested in giving you that kind of data so it wasn't that it was hard it's just that it is also quite narrow in scope like it's more or less only for that one right so in banking it would be a lot of records but only for one bank so you would only know how it looks in one type of account system, right? And then you would run all the tests on that. And now... Two seconds. There we go.
Hello. Hi. So it's not that it's hard to get, it's that it's quite narrow in terms of sort of like what... yeah it doesn't have a lot of breadth you get it for like one bank or one airplane company or you get it for one company who runs elections right public elections is a bit different definition you don't have any test data because all of the records are sort of you don't know who voted so you can't know how to test that right so you have to do a different testing paradigm there.
Yeah, that makes sense. I mean, here in the US, we have... anonymous ballot as well, right? And I don't know how they do it in other states. I mean, I'm involved in the process here in Utah. Yeah, you send in your envelope with your ballot in it, and they validate that. and stuff off the envelope, but then they pull your ballot out of the envelope and put it in the pile to be counted, and they don't track who you voted for.
Correct. And that is also the case for the digital solutions. A colleague of mine, Stefan, who is a cryptographer, a previous colleague of mine, he did a lot of work of sort of... gathering together a lot of the academic work so that we could exactly do that in a cryptographically safe way.
and you could also do it in a way that it was you know you could track it and you could ensure that this was in fact the case like you could give the the log data and the trace data to mathematicians and they could verify that mathematically it could only ever have worked if these rules were followed. which was like you know you anonymized and you did you did mixing and you did a lot of these things right
Right. So you protect the identity of the voter, but you also ensure that only people who are legally permitted to vote. vote and that they only cast one ballot and things like that. Yeah, exactly. And the test of the build idea for that and the testing regime for that is significantly different, right? Because this is a public space. To your point earlier, Valentino, this is a public domain, right?
You are a bit more scrutinized. You're also publicly scrutinized by everybody, right? Because we all know how it went with the Dominion voting machine case, right? Real fast, well downhill. So, well, and you bring up the Dominion voting machine case, and whether you believe that they worked or didn't, there's... If there's enough population that doesn't trust it, then it causes problems, right?
And that's the same everywhere, right? And I found that sort of on a completely different but somehow related topic, I really found that the... Aerospace, the airplane industry was really good because the, what's the American one? TSB transportation safety board is that tsa no tsb right yeah those are the ones that that do crash audit that sort of like go through oh yeah the ntsb yeah Both them and their European counterparts have a really really cool way of going about it.
They don't ever go and say, oh, it was that person's fault. If the person hadn't slept enough, if the person couldn't read the dial. It is the fault of the plan that made him not have enough hours to sleep, and it is the fault of the dial for not being visible. right so it's not they don't ever cast personal blame or they very rarely cast personal blame and I really thought that was a super cool way of going
Also just from a sort of like evaluating if something works well, right? Just like from a human perspective and when you do leadership in tech or if you... guide or lead others that I found that really valuable. Sort of that whole way of thinking about it. And they also, they collect a ton of evidence, right? They have a very structured way of collecting that before they then conclude that it works or didn't work or that this was the fault or that.
But they also, I think I learned a lot also in terms of like having redundancies, like what happens if this thing breaks? What happens if your job runner doesn't run? How are you alert?
What happens if your email doesn't send? How are you alerted? Do you have alternative options? Do you have... an alternative email provider that you can switch to easily do you have alternate job runners that you can switch to easily do you have any of this right um so that redundancy idea ideology was pretty cool um And for like large scale systems, it pays off to have extras of these, even though, you know, you use somebody like SendGrid or you use somebody like...
Mailchimp or something and you know they say that they scale and they do right but then ever so often it's down and it's pretty nice to have a backup even if it's a tiny backup and even if it's just a you know up and coming fatted cheap one. It will do the job for like a day. Right. And you will actually still send stuff out or you will still process jobs or you will still whatever it is. Right.
Also running like on the big scale, running something like multi-cloud or running something like an open stack where you can actually move to somewhere else. There was that case with that in the Australian case where the... where the bank got taken down because someone at Google decided to delete their account, right, for like 160 million US worth of servers. Just got new. So for those cases, it's pretty nice to have a second provider backup and some way of getting there, right?
That is more a DevOps problem. It's not really a problem that I've ever solved. It's a problem that I have, you know, sort of like assisted with and talked about and verbalized a lot in companies. to a high degree a DevOps solution that needs to go in there and not one that I am gonna build. So you're talking a lot about like people's lives and, you know, their livelihood, their money, like things that sustain life. Right.
And so like, obviously these infrastructures are critical, like what tools you're using are critical. So I'm curious, like what about like Ruby and Rails? You know, specifically feature-wise, like, do you see as being like, oh, this is like part of that critical process? I think The main part for me is that
It can be built. You can write it so that it is absolutely readable and understandable what's going on. So you can reason about... what's going to happen even without necessarily understanding the language fully or even without understanding what this ActiveRecord method is going to do underneath because stuff is named well. And because it's ordered and organized well, you can reason about what's gonna happen. You can have an expectation and it makes it a lot easier to build complicated things.
like you can go and test it afterwards right but just getting something that's complicated up like just getting it up on its leg It's really nice to have something that is structured well and that you can reason about and that you can talk about sort of what's the consequence of this method or that method. Yeah. Special guest co-host. Yeah, exactly. But I think and also I find that.
I mean, I've done a few frameworks throughout my career, right? I've always come back to Rails and Ruby, and I also started there, so maybe I'm biased. I haven't found any of the other frameworks that provide
the same sort of consistent experience through every part of the application. Whether or not you're doing low-level active record stuff or you're doing high-level... system tests or whatever you're doing it's consistently the same experience everywhere which means that it's fairly easy for a developer to switch gears and do something else right uh in in the stack
So like you can have a person who is sitting and doing the Swift transfer one week, right? And then he can go and he can look at the web scraper the next week. Because it's consistently the same experience and you can write it in a way, if you have a good style guide and you have good code review, you can get it to a point where it's the same experience every time. And you can get abstraction levels and abstraction layers that feel the same. And that makes it easy to build something that...
where you can also easily plug out something and put something else in, right? Like you can switch out one database provider for the other without much... Or you can do whatever it is that you feel like you need to change. It can be quite easy and quite straightforward if you've set it up. to be I'm not necessarily talking about modularity in the Seidwerk level of aggressiveness that...
that Shopify has, right? But you can definitely do something along those lines with just talking about it, organizing it. organizing the code well and just sort of thinking about how how is this going to be tested I find that that's also part of it right it's like how are we going to test this because if we are going to If we're going to test it in this particular way or that particular way, then that informs the API and sort of the interface for it. A database you tend to test the same way.
If it's file uploading or downloading, you tend to test that the same way, which makes it easy to switch out one.
component for the other if your test paradigm is the same right for for that component so i feel that's that's some of the reasons that i've chosen this uh again and again right and also like jay jay said on his keynote this is a one like it can be a one-man army right you can be pretty fast and pretty dangerous with it which means you actually need fairly low headcount to get somewhere really
And you can actually get something that's really secure, solid, safe, robust, and easy on the eyes and nice to use with only a handful of people. You can get somewhere with just one person, but you can definitely get a long way with a handful of people. E aí You are so sweet. This is why I'm here. Yeah. So one thing that I'm wondering about with all of this is, you know, it sounds like depending on what the concerns are in the app, right? So whether it's banking or both.
You may have different concerns about uptime or stability or resumability if Google deletes all 1,800 of your servers or whatever. So how do you begin to start looking at your application and saying, well, for example, in banking, we have to make sure that we don't have any of these kinds of... Issues, bugs.
show up right you know make sure our accounting is super robust but then you're talking about the voting and it's hey we've got to protect user data and make sure that you're right so that the areas of concern are different so how do you start to address that and then from there Where do you start to dive in and make sure that you've got a sound strategy for making sure that you're meeting those requirements? Yeah, that's a good question.
Let's just take voting because it's up and hot and a topic these days, right? So let's just start there. So one of the things there is that you definitely want. user privacy there's a few right there's like the whole user privacy and then there's the vote secrecy and then there is the it needs to be receipt free and like there's a bunch of these academic properties of a vote and an election, right?
You go through each of those and you just sort of sit down and you start at one and then you try and figure out how to do that, right? This is really the job of the cryptographer. Like, how do we get this? organized so that we match all of the... And then once we have a viable protocol that will do this, then the developers come in and we look at the protocol and we go like, okay, this thing is about encrypting the vote.
That needs to happen as early as we can possibly get it to happen, which means it needs to happen in the voter's device, right? So we need to generate, we need to pick a pick up a cryptography curve and then we need to basically generate key pairs in the device and we need to encrypt it as early as possible. And so you go through sort of like rigorously and slowly and just sort of reason about when will this part of the process happen and where does it need to happen? Sort of more on a timeline.
And then we need to transmit the vote. We transmit the vote to the server and then the server needs to basically allow... If you need to allow tracking or if you need to allow receipts or whatever is that property you need to have, then you need to figure out a way to provide that at this point. And then you go on to something like... anonymization, which is effectively where you take one envelope out of the other envelope to do the analogy. That needs to happen.
in a way that you can prove that it happened. So you need to have a good trace here, you need to have good logs and traces that you can publicly show. And then, you know, so you sort of move through the chain, sort of like the event of a thing, right? It's the same. And so you sort of need to reason about where is it going to happen. That gives you some ideas of should we need, do we need a web app? Do we need a mobile device, mobile app that we need?
Do we need to run cryptography? Do we need to run something advanced on the user's device or is the user's device just a portal? And that gives you sort of ideas on or some insight into how this should be organized, how this should be tested, how this should be planned out, right? But most importantly for voting is basically making sure that everything is tracked.
part of the concept of blockchain, which is hash chain, right? To basically hash everything together so that you can't move stuff in the middle without everybody being able to know, right? Things like that. Hvad må jeg tøre? Det du har lyst til. Hvorfor er det så slik? for banking right you want to you want to make sure that see the point is to get money from A to B so you want to reason about well first off how am I gonna get
the number that they intend to send, right? Like where does that come from? Where does the sort of outgoing amount of money come from? This is the second one. Have a good day. It's hard to be home alone with kids. So you want to basically say, okay, so where does the money come from? And then you want to reason about, well, how can I prove that?
the money that came from there, that I got it right. Like, how can I make sure that the intent was to send it to me? How can I make sure that the amount was correct, right?
Because like we talked about earlier with TransferWise, for example, there's a lot of web scraping, there's a lot of this sort of... craziness going around less so now more so then right and you want to make sure that you have either done it a few times that you're quite confident that you've run it on big data sets like you need to be certain that you can pass these things correctly. And then you want to sort of grab that information and
Then we get into the whole banging of how do you then move them? There's like Swift and that's pretty taken care of. The only problem here was that it was slow, right? So what we would do is that we would basically assume that if we could read the intended money here, we could effectively pay it out here immediately and then just do the in-between transfer later.
So we would basically have big stacks of money around the world and we could immediately pay out locally or take money in locally. And then we would transfer later. so in that case it was more a question of can you verify the amount here and can you verify that it's the sort of the correct amount and the correct intended amount and can you then make that payout over here zu kommen mit So then can you do the payout here? Is there sufficient funds?
And then after that, you sort of say, okay, now we've effectively solved the user journey, right? We've taken money in, we've paid money out. Where effectively the user is happy or both users are happy, right? Nobody knows that we didn't actually move them And then we can take it slow, just do it properly, make sure that we interact with the Swift network. That's a bit slower. So you need to have a buffer in both ends to do it, right? and the most important part here is
the ability to scrape, right? So one of the critical points here was that you have to be able to always scrape these web in some capacity and you have to actually write your your code back then it was you know scraping you have to write it quite robustly and sort of Reason about what happens if the layout changes, because banks change their layout surprisingly more than you would.
Because they don't care about your ability to scrape it. That's not their business, right? Which is fair. It just means that you have to think about the robustness of a scraper like that. Because what happens if the... If the item is not classified the same, if the DOM element doesn't have the same class or the same ID as it used to have, can we find it by reasoning with words? Like, can we figure out, well, if it...
Like if it says some, can we maybe then figure out what's over here next to it? Because that might be the number I'm looking for, right? So there's sort of like reasoning like that, that goes on. Yeah, and then there's... Accounting, like a lot of accounting, a lot of keeping tracks just for legal and compliance reasons on where is the money going, who is it going to, KYC, all of these. There's a bunch of tooling.
after that, right? Which is sort of where I was introduced to this whole compliance thing. Like how do you make sure that the people you're operating with are in fact the people that they say they are and are in fact the right people that you're supposed to be operating with, right? So that was also our journey. How do you even know that? How do you make tooling for them to identify themselves?
at the latest possible time so it's the least amount of burden or the least amount of friction and make it as easy as possible. There's a bunch of providers out there that can help you do that. You can also do it yourself. it's not super complicated, but you are now dealing with people's private information. Because now you maybe have a copy of their passport. How do you deal with a copy of a passport? You have to sort of be like, okay, where do I store this?
How do I make sure that it's encrypted? How do I make sure that none of the developers can access this, right? Because you don't want developers to just SSH onto the box and just be like, oh, let me just download all the... So you need to make reasons, you need to think about how do you then protect this data from yourself in some way, right?
Yeah, so that it goes deep and it goes far, right? You can talk about this for a long time and about all the reasonings you have to make, but at some point you also have to protect your systems from other developers. from other... Also, high-level people in the organization needs to be kept out. Because the higher up you are, the more spearfishing or whaling, you are effectively the subject of, and the greater the risk, right? So you want people to not...
Maybe you even want the CTO to not have access.
Like there's a bunch of... considerations like this in in the banking one uh i was a senior developer i had access then we had One of the compliance officers, she had access to the Swift network, but the CTO and the head of the... COO type person didn't raise because that would pose too much of a So there's also like these human factors you have to think about when you reason about the security of a system that actually impacts and it has something to do for real world people.
That was a very long spiel. Sorry. It's fascinating when there's so much on the line, like most of the apps I've worked on, like if it went down, it's like, yeah, fine. Okay. It's not ideal, but it's not the end of the world. I kind of want to pick up on something you mentioned quite a while ago, actually, about type checking and survey and stuff like that. I'm very much a fan of the dynamic typing. I do have a background.
in static programming. Because I started as a mobile developer, so I spent a lot of time with Objective-C, with Java, and with Swift. and so i do have that background but ever since i kind of switched to ruby um i just love that dynamic style and i just find it easier to work with but I also get that when it's critical infrastructure, the type checking is just almost essential.
So I'm curious about why you went with Ruby and Rails when the infrastructure is so critical and type checking is scary. something that's important to you and it's kind of bolted onto the language. So what motivated you to go with Ruby and Rails instead of something that has it natively? Yeah. It is It is to a high degree about the speed and the very well built.
testing frameworks that is around it, right? So even if it's not statically typed, you can test a lot of things and you can test it in a lot of ways. Typing or types is just one way of ensuring that the data is correct. that doesn't necessarily guarantee that the actual data you are trying to read in is of the correct format or whatever. It just means that once you read it in, it will fail because it's not the right.
so it depends a little on what you're trying to do you can also because it's dynamic you can then sort of do things like okay you gave you told me it was a csv And it's obviously not a CSV. And it obviously doesn't have a string where you told me a string was supposed to be. But I can see how this can look like a string. Maybe I can just make it into a string for you and move.
You have these options of making things go a little bit faster and a little bit smoother because you can just fiddle a little around. What tends to happen in these systems as they grow older and as they mature is that part of the core of the system gets replaced with something like Java or Rust or something like that. So that once you have a really solid part, let's say the transfer engine, right, that transfers from one account to the other, that hooks onto the Swiftnet.
Maybe you want to move that into Java or something that's more static, more slow, but more hyper-robust and more maybe even performance optimized for IO or whatever it is that you need, right? but then what tends to happen is that you still want the speed and ease of using Ruby on Rails and having Only a few developers do a lot of features so you can figure out what's good or bad or what works or doesn't or do a lot of development around.
trying out this new standard to verify with the engineers that it will in fact test correctly with this airplane. You know, sort of having that capability to do feature spikes and to do tests or like mini projects. It seems to me that that is... to something really lost when you use a more There's some other language, right?
But you are right, like once you have it down, once you have it locked in, once you don't have any changes there, what tends to happen is it gets replaced with something that's faster or better suited for the purpose. Yeah, I've seen that happen. So the reasoning is that you can go really fast with quite few people and you can actually get something that's meaningful.
that will affect people's lives, that will better somebody's life. You can get it done pretty quickly, right? Like you can get a... at the science system for to basically do We're not talking like 3D sketches of airplanes. That's not necessarily how you design all of it. That's just the building of it. But you also need to reason about the logic of who does what. What are the processes? And you can get something like that built in seven, eight months, right?
As far as I know, and what I could see in the market, right, and what happened was that we basically went a lot faster than everybody else, right? with the same amount of errors. It seemed that there was no real... drawback to using Ruby in that case right and so It was a pure benefit to use the language and the framework. Nice. There's really nothing that can compare to the speed of development of Rails. I think we kind of take it for advantage, for granted rather, as Rails developers.
Go and look at the outside world is when you realize, bloody hell, this is a good deal. Yeah, it is quite insane. Like I had a colleague who did, you know, sort of. an AI based feature spike, you know, like, oh, can we just time box this to like three days? And in three days, we had a tool that seems to be working, did integrate with AI, had UI, had test. were behind a feature flag and was effectively shippable.
it was not you know it was not the best or the most robust or whatever but like in three days a single person made a full feature spike that could technically be deployed and could work right that's pretty amazing I'm curious real quick, like... I'm... In my experience working with critical systems like this, once it's built, it's hard to change. And mostly because, too, the requirements probably don't change very frequently either.
How do you approach in those cases that change? Is it like build something new and cut one off and switch to the other? What kind of systematic approach do you take to that? I think so for me it's certainly about you have to reason about the protocol and the process somewhere other than the application. So you have to diagram it or put it into dance or whatever you like to do, but we need to have a different place to talk about it than in code.
Because we need to reason about when somebody has a change, we need to be able to reason about it before we spend time in code. And we need to figure out if the... the signature of that particular change. Let's say you want to change one hashing algorithm with another.
them right you need to be able to reason if that will break the system just in principle before you start to even try and implement it right so my I find that It really benefits to have it in a big diagram or have it written out in long form in a document or like somewhere that you can reason with people who aren't developers.
on whether or not it will work or not before you even try to implement. Because if you can reason that it will in fact work, it will plug in, it will replace If you can reason about that before you actually see the code, when you come to the developers, it's a much smoother experience to say, okay, I need this piece replaced with that piece. And yes, the hashing algorithm is not the same, but it is whatever based on the same something, something. So it will plug in, right?
And then we can either have both. If they plug well, we can replace one with the other. Then you need to migrate the data, right? But now that you're migrating the data, it's really nice to also have that conversation with beforehand and with others who are not developer. Well, we have existing data, and you want me to plug this thing into this place in this big diagram? What happens to all the old data that went through that point beforehand? What do we feel we need to do about that?
revision it and say that version less than something can never be opened again or can never be edited again but can only be viewed like what's our reasoning on that right So having somewhere else to talk about it than in code really helps to figure out if you should replace plug in or make it as a different service that you call sometimes, whatever it is. I find that having something other than code really helps. So it really needs somewhere else to have that conversation, right?
Yeah, you know, it makes me think about the old days of Yahoo Pipes, like where you could visualize and like in real time update the data flow. aggregate thing you were trying to collect. Are there any tools like that that you use to like maybe automate this process or is it just a matter of getting it done? I mean, I've tried a bunch of them. I haven't found one that I really like. I haven't found one that really does it for me.
I tend to try and make them diagrams as code because then they can at least get checked into Git and we can have version control on top of it. But I haven't really found anything that really does it for me. I've done. I tend to either do whimsical or draw IO or something like that. And then it is just a lot of work to keep it maintained. And that sucks a bit, but, you know. It's hard to do without somewhere else to reason. And especially when you need to talk to...
Because it's not only a technical problem, right? It is also a business problem and it is a regulatory problem. If I'm going to change this piece of the testing bed. for an airplane or this piece of code for the design software for an airplane. I need to be able to have a conversation with somebody who is not a software engineer about the consequences of that, right? Because I need to talk to the civil engineer. I need to talk to the test manager.
And I need to talk to a lot of people about the consequence of doing it and the way to do it. And are there any consequences outside of my system that we need to have a conversation about? It is just sort of like trying to get as many people joining that conversation as possible. And for that... seems to work better if it
not quite code, because then more people can participate in that conversation, right? Yeah, I think that makes a lot of sense. I'm still personally trying to find the silver bullet. The closest I've gotten is Mermaid. Yeah. And that also works quite well, actually. It's just... It works pretty. it works pretty reasonable and it is definitely diagram as text right so I've also used that a ton and it's decent enough I find that
It has some problems when you get up to really large size stuff. It's hard to get enough in there and still have it be... viewable and understandable as a human. But I mean, a lot of like building these systems and a lot of participating in building them is surprisingly not about code.
It's about talking to a bunch of people who have nothing to do with software development and figuring out if you're going to break their lives, right? Like, are you going to break something completely insane that you had no idea that you ever were ever? Because, you know, in the case of this, well, I'm going to change this thing in my design software. And then suddenly somebody in... the factory somewhere.
is totally confused about how to build an airplane because I did something that, you know, so it's really not that much of a software challenge. I mean, it is, but it is to a high degree a communication challenge and sort of like, how do we even communicate what this software does to people so that they can use it and so we can reason about the safety of it together, right? Yeah, it makes sense. I mean, I've worked on things at this level of complexity that you have to get right.
I worked at Morgan Stanley. I mean, we weren't doing like banking level stuff, but we were doing like investment level and business information stuff, right? And so, yeah, the world didn't end if we got it wrong, but we kind of needed it to get right. A lot of it was just a matter of, like you said, making sure that all the people who understand all of the concerns and all the things that it touched are able to weigh in.
and be part of the conversation, and that way, yeah, you don't do things without understanding the implications of how it's going to ripple. I'm working a contract now, and... You know, I'm still trying to figure out the ins and outs of the system. And they're managing. funds and stuff like that right and so yeah i'm definitely feeling it here where i don't want to make any changes that ripple through the system
Right. And so this is where you, when you're talking about like testing and documentation and things like that, where the documentation really isn't around. And there are a lot of other things I wind up having to ask a lot of questions and I'm still not sure. So, yeah, a lot of this really helps in the sense of, oh, okay, if we're going to nail down robots. reliable things. then yeah, you know, all of these things are important.
And I think part of my issue with some of it is just that I like to just get in and solve problems, right? My mentality, I was talking to some other friends the other day, and my mentality is go, go, go, go, go, go, go, go. And so, yeah, having to slow down and, you know. I joke and say I dot all my T's and cross all my I's. It just, it's, you know, it's not my natural state.
I think a lot of developers, yeah, they just want to write great code that solves problems. And so the meetings are tedious. The conversations are boring. And why can't I just go in and just make it work? And the reality is... you know, yeah, what we're talking about here. And so I think it's not just we have these specific concerns whether they're regulatory or something else. I think it's also that there's enough complexity to it. to where you really do have to get in and understand the new...
Yeah, and I think also the breadth of what you're doing, right? Like, I know this for myself, right? When I was younger and started developing code, right? It was like, well, there's only one truth, right? one plus one is two or it's wrong, right? But that's just not the case when you have these really big connected systems, right? Where like, well...
What happens if I change this protocol? Well, now it's right because it's a different protocol, but now it's wrong because it's a different protocol. So it can be both at the same time, right? because now all the other people who used it for one protocol can't use the other. It just gets really messy.
I think at some point I realized that I liked doing documentation and that really helped to sort of make me be confident that this could be some way you can do a career, like you can do a career in something that's not obviously developer-ish territory, right? because I actually enjoyed doing like architecture diagrams and reasoning outside of the code and all of these things I found was pretty interesting and that makes it easier for to be me, right?
It's more fun to have that conversation with everybody to figure out, well, if we do this thing over there and wherever we are operating in Australia. That's fine, but now we've done this change. So what happens to our operations in Austria? Is that the same tax system? Can we do both? It can become really complicated when you're operating under multiple jurisdictions and multiple governances. then that can become complicated as well.
You know, you're doing currency conversions or you're doing unit conversions or you're doing all of these. Like it's so hard that I know that even the space agencies are getting it wrong and sometimes their stuff just won't. work when you get it into space right because somebody miss converted inches to centimeters right and so it is just and that should like to if you go to a
fresh out of college developer and says to him that I need a unit conversion system or unit conversion method that can go from inches to centimeters. That's pretty straightforward. That's pretty straightforward.
ask, right? And you can do that method pretty quickly. It's not really super hard. What is really super hard is getting people to use it right so you need to build a unit conversion method that is easy to use fun to use at hand when you need to use it so that you will in fact use it because otherwise you're going to do it in your head and then it doesn't really matter that you build a piece of software right so like
There's a lot of other components to this than just the software. Like there's a lot of things like, you know, there's a lot of backup, a lot of everything, but it is... Also about making sure that the people who use it, use it correctly. And the way to do that, like we talked about before, like with the TSB and everything.
it's not to blame the person for not using the thing it's to blame the thing for not being available when it was needed right so you need to be like if if i have a unit conversion thing and it's hidden behind you know a multiple three dot options that you need to figure out five levels in that's the wrong place to have it then right you don't blame the person for not using it you blame the system for not putting it in his face when he needed to convert
measurement. Yeah. I like that. Just from the standpoint of... I mean... I guess where my brain goes is everybody has a responsibility to understand. Right. And so everybody has the responsibility then to, you know, to be aware that this method exists. But at the same time, if I create it, then it's also my responsibility to make sure everybody knows it's there and knows how to use it.
I also agree, though, that if you make it easy or natural to do the right thing, then people will generally do the right thing. And so, yeah, I think I agree with you, but I don't want to absolve anybody of the responsibility of learning to do the right thing. No, no. I mean, that's not what it is. I mean, you can absolutely end up being blamed for crashing up. Like, it's not.
entirely out of the realms of like the pilot can be blamed right but it is also about looking at did we actually provide the correct tooling at the correct time with the correct intensity and all of this right And it's the same here, like, well, the system didn't work. Did we provide enough? Did we scale enough? Did we have enough compute resources? Did we have enough database storage? Was it just so slow that people couldn't use it and it crashed and somebody couldn't get their vote in?
Because if somebody couldn't get their vote in, then, you know, we get into the territory of, oh, somebody prevented me from voting situation, right? it like yeah they could have done it again like they could have just waited five minutes till the peak was down and voted again but that's like it's also not really their fault right uh so it's there's a lot of There's a lot of sort of like ancillary non-developer stuff to talk about when designing. I'm pretty sure there is on all systems, right?
But the fun thing here is just that this happens to include other engineers, which is very fun. And it happens to include things like cryptographers. people who actually like politicians or whatever it is. It's just I like the types of people that I get to talk and interact with in this rather than... I get to talk less with marketing and social media and more with people who build spaceships and airplanes. And I find that's basically what drew me here, right?
It's this idea that I get to talk to other people who also have this engineering. spirit and this idea to build something in the same mindset that I do. Awesome. All right. Well, I think we're going to go to picks because we've been talking for an hour and 15 minutes. This is cool, though, if people want to connect with you or if they have questions about what you're doing. Maybe you want to pick your brain about some of this stuff and how they find you on the internet.
I think go to LinkedIn. I've more or less tried to absolve myself of all social media. So LinkedIn is probably your best bet. It's my name. It's right there on the screen. There it is. And then also go on Discord. You can ping me there, PM me, like you did. I'm both on the Kamal, the Rails, and the Ruby servers, so you can just go find me there. It's E-K-A-M-P-P, so just my initial letter and my last name. Nice. I didn't know that there were Ruby and Rails discords. I'll have to go find those.
I can send you invite links. That would be perfect. Cool. All right. There's one for like Rails development. That's the Rails one. And then there's one for Ruby, which is more like application support and like, oh, I don't understand what this method does or I need to change this. Whereas the Rails one is more for like... I need approvals or reviews of PRs for the actual call. So that's more about...
supporting of that. So it's less interesting if you're trying to solve a business problem, but it is kind of cool if you want to just peek at what the others are doing. Nice. Okay. Well, let's go ahead and do some picks. Valentino, do you want to start us with picks? Sure. Uh... I've got nothing but AI picks today. There's been a lot of activity in the Ruby world. Andrew Cain has released yet another AI gem.
Bless his heart. It's called Informers. And it basically lets you run inference in Ruby on Onyx models. So lots of hugging faces. models are ONIX supported and you can now just run inference with those. And he also has Transformers RB if you are interested in running inference on non-ONIX models. So now we have like a complete picture. Hopefully those, there are some...
Some missing pieces still for larger models, but it has a ton of pipelines already built in. So I would suggest checking that out. The next pick, somebody built this awesome chat app. where they've aggregated all of the Rails and Ruby articles and blogs and guides and everything like that. So you can basically rag the entire Rails community ecosystem.
in a chat GPT style interface. And it's remarkable. And it does like citations and everything like that. So you can see the original article that referenced whatever it was you're interested in. It's really awesome. It's called Ship on Rails. Bye. And it's built with this really cool tool called Nausea, which provides basically like a Rails app ready to go so that you can ingest your own documents to rag again.
and do a very similar thing. It's really cool. So I'd recommend checking all those out. Night. How about you on your... I'm not sure what to take for picks this week. I haven't really thought of anything. It's a cheap one, but I'm going to go with BlueScore because I'm quite enjoying BlueScore in the last week as I finally signed up for it. And a lot of Ruby folks have signed up recently.
It's quite a lot of activity on there. I'm quite enjoying it as a social network. Hopefully it can stay the way it is. I'm active on both Blue Sky and Mastodon for now. Yeah, we'll see what happens because Blue Sky is still a VC-funded, privately-held company. So we'll have to see how long things remain good. But for now, I'm quite happy with it. What else can I pick? I haven't really watched anything or anything new recently because I think...
I had picked the TV show called Shrinking, which I did the last time, so I don't want to pick that again. I'll go with the music pick, because I always go with the music pick when I'm struggling. New Tears for Fears album. Check this one out. It's really good. It's a live album with a few new songs from 80s pop band Tears for Fears. So, yeah, nothing else. I'll go with those. Awesome. I've had a number of people tell me I need to join Blue Sky, so... Um...
Anyway, let's see. I'm going to throw in some picks. So I played... I played a new game. I actually played a couple of new games this week. because the game convention that I usually teach games at is going to be on Saturday, and so I need to be able to... these games. So I'm going to pick one of those. It's called M-L-E-M. And essentially what it is, is it's, how do you describe it? It's cat astronauts. And the way that it works, it's a relatively simple game. It was almost too...
But if you're kind of a casual... board gamer, right? It's definitely approachable. So let me pull it up on board game geek. But yeah, so the way that it works is you have astronaut cats that are... going, going into space and they're basically occupying planet. And so you pick one of your cats and put it on the rocket. And somebody's the captain and they roll the dice. and if the dice match what's on the space on the board that you're on.
then you can choose which dice to use in order to move up the board, right? And so if you have boosters, boosters move you up for free. The other ones, you have to give up the dice in order to move. And so if you ever get... to the point where you roll the dice and you can't use any of the numbers, then you can push the ship. At any point, people can get off the ships, onto moons or planets.
You get points for having the most cats on planets. You get points for every moon you occupy. If you go all the way to deep space, then you get bonuses for that. And then the different cats have different abilities, right? One of the cats lets you start further up the board. One of the cats lets you get off the ship after it crashes. So you can go to the planet it's next to or whatever when it crashes. One of them gives you a bonus.
doubles your points for the planet it's on. One of them doubles the points for the moon it's on. Anyway, you get the idea. One of them doubles the points if you're in deep space. But deep space is hard. I played it once and we never made it to deep space. We got close. But then I was talking to some other people who had played it a couple of other times. And one of the times they played, they made it deep space like three or four times.
So, anyway, so if you're the captain, you're rolling, and then you just rotate who the captain is. If the captain gets off the ship, then the next person in line just finishes out that. Anyway, that's basically the game. BoardGameGeek has it at a 1.68 weight. It says ages 8+.
And yeah, I have an eight-year-old. She could definitely play it. It says 30 to 60 minutes. That's probably about accurate. Plays two to five players. I played it with four players, and that felt pretty good. So it's M-L-E-M. And, yeah, anyway, check it out. And then I'll just put a link in the comments here. And then let's see.
I'm not sure if I really have any other picks. I know I've kind of been... doing other things and getting involved in other things, but... I guess the big thing that I went through this lately is I went up to a retreat up in Sundance, which is actually like 45 minutes. You've heard of the Sundance Film Festival. They actually do that in Park City, not in Sundance. But... From here, it's about two-thirds of the way to park.
And, um, anyway, it's, it's up Provo Canyon and these guys were running essentially a mastermind for three days for, um, Christian. And so we did a lot of faith-based stuff. We did a lot of business-based stuff. And it was amazing because they just kind of helped me get clarity on what I want to do and who I want to be and what kind of difference. So... just keep an eye out because I have stuff in the works.
But it was terrific. And I really want to just put forward, I guess, that just connecting with other people who are doing what you're doing or who are learning what you're learning or who are maybe a little ahead of you on the path you want to go. It's so helpful. So, so helpful. You know, kind of like we were talking about with, like, the Discord servers, right? So much helpful stuff there. So, anyway, those are my picks. Emil, what are your picks?
And so recently, I'm going to pick a book. I recently read a book called How Big Things Get Done. And it's by a Danish guy called Ben Flubier. So what he did was that he went, I don't know if you guys know it, but he went through statistical data on overruns on big projects. How far behind is an IT project in general? How much of an overrun on cost do you get?
how much of an overrun on time do you get and he goes through sort of like these different industries and different factors that play into to why why big things fail and why some big things get What are the ways to make sure that big things get done? And what are the pitfalls and everything? And he goes through everything from the Sydney Opera House to big government IT projects.
It's just super cool because it's very sort of data driven and he has spent his career collecting data on why big projects fail, why all kinds of projects fail. So it's a really interesting set of... window into do this if you really want to mess it up So by inversion, if you don't want to mess it up, don't do all of this. Do the other thing, right? That's kind of an interesting statistics to have. So yeah, how big things get done. I found that was really cool.
Awesome. All right. Well, thanks for coming, Emil. We're going to go ahead and wrap it here. Thank you for having me. It was a pleasure. And until next time, Max out.