Hey Ben how are you doing my friend?
Hi Matt I am good.
I think this time both of us have got our microphones on the right setting pointing the right way.
Ah I'll believe it when I hear it man.
We've not had great success. So far we started out strong and then we we both upgraded microphones and things and then the number of commbinations and permutations of things that can go wrong exponentially increased.
Right. This is like when you have some difficult thing to do and you're like procrastinating and you're like I know I'll just like work on the build system or whatever what we're doing instead we were gonna record a podcast but instead we're just gonna mess around with microphone setups and audio engineering and we've turned that.
Ah, exactly yeah.
We've turned our podcast into an excuse to do that.
It seems about a right? Yes I was meant to be writing peer reviews all day today and so I've been doing literally everything but that why why is it so difficult. Why is it so hard to write human you know things for humans when code is straightforward. You know like I'd rather sit down and write some lock free queue than to write things nice or otherwise things about my my peers. Um, it's hard. That's why humans human stuff's difficult. That's what it is.
Right? Well because there's no type checking.
Maybe LLMs will do the type checking. Maybe that's what I should do is I should you know pace pace. No I I should not do that. This is from me to my peers to tell them about what what? how I think they can improve not for some LLM to turn it into word soup for 300 anyway this is not what we were talking about I've just got.
Totally sidetracked at the beginning. But um, so um, I was really interested actually in something you and I have spoke about outside of this before which is your sort of of philosophy of developing a new product or project or or machine or whatever. I guess project is the main thing and you know how how you approach it right? Yeah and obviously testing is a big part of that. But there's a there's ah 1 aspect which I've taken on board I think it's really important and ah, you should talk about it I'm talking about deploy first. Yes.
Aha. Now. Ah, yeah, yeah. Yeah, you're talking about Deploy First? Yeah yeah I ah I mean it's generally everything that you want to do in any sort of project is think about. Um. The the risks the things that you ah know that just ain't so ah and one of the easy things to know that just ain't so is oh yeah, we'll just deploy this out to whatever thing right? and we'll put it on a computer somewhere.
Right? right? I have I have you know `project.py` or I I just need to copy it somewhere and then just run it right? How hard can it be things. Yeah.
Right? Yeah, how hard could it be right? exactly so that is a place where um, you can I think there's a general pattern of these things which is. Um, it usually makes sense to confirm easy things are actually easy and one of those things should be deployment deployment should be easy because hopefully you're going to be doing it a lot so starting with ah deployment.
Right.
When it couldn't possibly be easier when you have like 1 line of code that like writes to your logging system. This is like hello world. Yeah, even if your logging system is just standard out for now right? It's it's like well what could possibly be easier than that. Well it's like oh if it's so easy then why don't you start with that.
Yep, right.
And then and then there's this little demon that shows up and and and says well what if it's not easy. What if it's a huge pain in the butt. Why don't We just do it later when I don't have to worry about it after I have written the fun lock free queue and all of the other cool stuff and I don't want to have to deal with deployment.
Um, what if it is. Um, ow that that came back to haunt me pretty quick. Ah no right? exactly. It's messy and it's It's a bit like writing reviews for humans right? It can go wrong in so many different ways. In a beautiful platonic ideal of my editor and my TDD-based sort of workflow. It's like just some bits get tarballed up; and it has to be working or I have to copy it to a machine and I have to get the environment set up and all of the things.
Um, yes, yes, and the and the more stuff you have to deploy the harder. It's the classic thing of like it's really hard to build big big complicated things you build big complicated things by building tiny little things and combining them all together.
Yeah.
You know, no one likes to review the 10,000 line pr no one wants to be on rotation when the huge change that someone has been working on for three months and then went on vacation rolls out um you want to do things 1 tiny piece at a time and the best way to do that when it comes to deployment is take the dumbest.
Possible thing that could even remotely smell like the thing that you want to build and deploy it right. Is it a web service. Okay, cool, get some basic web server in place and you know make your helloworld html and you know deploy it and make sure that you can pull it up on a browser and then it works you know is it some running service. Okay, cool is is. Is does service actually run if you if you kill it what happens right? does your alerting work right? If you if you go under the machine that it's running on and do like a peak hillll. Do you get the alert on your phone saying that it shut down because you know you are assuming you know that that that infrastructure is in place.
Ah, yeah. Yeah.
And if it's not or it doesn't work the way that you think or it's broken or hasn't been configured yet. There's going to be no easier time to troubleshoot that than when you have 1 line of code. So I'm a big fan of starting out with this because it sort of fits.
Um, this model of well if this is so easy then let's just do it right now and if it's not easy then we'll know about it sooner rather than later and we can account for it. Um, and the other thing it lets you do is you sort of as you're as you're making changes to the code.
Yeah. Right.
It's like oh we're going to need, you know access to a large disks. We need to cache some things on disk when we're processing. Okay, cool. You're rolling out that changes 1 kind of atomic change rather than getting to the end of you know some big development cycle and being like okay we're going need a big disk or we need 2 network cards and we're going to need a backup server over here that's in hot deployment mode. And then now you've got to configure all of that all at once for this one deployment and everything's got to work perfectly the first time and that sucks. Yes.
Right? You're sort of what you're hedging for is incrementality from like literally from hello world exit 0 through to massive system that does all of these things but you can do it 1 thing at a time and be sure each thing works individually as opposed to trying to as you say debug...
..12 different things that happen at once I also get there. There's a lovely aspect to to doing deploy for us which is like I can point at usually some kind of dashboard that has like a green light that says my service is up and running. Obviously we're talking very specifically here about the kind of services in a. Corporate environment or or similar here on me obviously like if you're if you're building a game may actually know maybe if you're building a game. You make sure that you can take your your dumb apk or whatever it is and then install it on your phone and say look it just says Matt's game and you can click the button and it quits and that's all you can do. But.
Ha ha.
Now I can commit that in the Repo I can make my Ci step do that every time build the apk all those kinds of things. Yeah, that's an interesting interesting point but I was thinking more like like literally our world where there usually is a service that you use you know some kind of docker base thing or use some kind of Nomad or other system homegrown system to say I have a new thing that's running please make that run as well and do all the normal bits and pieces you know, know where its logs go make sure they get archived all that kind of stuff. So yeah, deploying something and that I can point at even if it's not doing anything yet and say well it's running all we have to do now is add the code. But.
Right? right? right. Right.
Do something I was literally just speaking to somebody about doing this this very thing It's like we internally we wanted to make a server that publishes um some specific form of derived data that my team works on and um, there's a team that's going to be consuming it and I so I said to to the the head of that team.
And.
I think I might just deploy something which publishes the value 0 continuously for you because I can point at it and say it's up and running you can get. We can get all that of boring is the TCP connectivity right?
Um, yeah.
Is the multicast groups so they configured right? all that that nonsense can get out the way and then when you're receiving a bunch of Zeroes later on I can replace it with you know some high quality predictions for the market. It's an exercise for the reader at that point right? You know everything else because we all know that ah the where the bodies are mostly buried is in the wiring.
The right right? right? man.
I Do like that. So Let me let me ask you?? Ah, ah, something about that which is how do you feel about like continuous deployment because it seems to sort of lead into it once if this incrementality aspect of it If you're if you if you start from deployment then do you keep it up. I suppose.
Yeah, absolutely and I mean I think ah I mean me personally it I It would have to be a very unique circumstance for me to not take the time and energy that it takes to continuously deploy a system these days.
Right.
It would have to be like there's only 1 piece of hardware that this can run on or with and it costs more than your house. So we only have one and it's in production. Ah, and so I'm like okay well if that's the case then maybe we can't continuously deploy it. But maybe we can and we'll figure out a way.
I'm feeling a bit seen here. We have 2 computers that are a little bit like that. But 1 of them is a backup staging instances that we deploy to every day now we don't do it continuously in our world. Um, a lot of what we do on our side is sort of based around the market hours of a market and starting up.
Right? Yeah yeah. Yeah, yeah.
Intraday is an important test but it's not the same as what we normally do which is run continuously all day and so we want to make sure that we we can in fact, run continuously all day but it's still continuous ish. It's what it's intermittent regular intermittent deployment regular deployment I Suppose is what you could say.
You know we we have ah a staging environment where every morning it starts up with the latest version of the code then it's the expectation if you've committed it and it passed all the tests it goes to staging like as a matter of course and then fairly soon. It'll end up in production. Um, but no the the deploy first thing is.
Continual-ish deployment? Right? right.
Something that you said fairly early on in our me knowing you and Mika and it was definitely struck a chord of being like no, that's a cool thing to do that's an awesome thing. So definitely worth talking about.
Ah. Yeah, yeah, and I really like your idea of like I'm just going to set up the service that publishes all Zeros and then you can integrate with that because you know as you were saying like you know you want to get all of these sort of basic things out of the way again, making sure the easy things are easy. Um.
Right.
Because if you deploy that and and the service that the whoever it is is building is like Wow I'm getting all ones. Um you know that there's something very wrong and it's now still you you know it has to be like a small set of things. It can't also be bugs in your code or bugs in their code or whatever. It might be.
Right.
It's like well we just did this one so stupid thing and that didn't even work So we're clearly missing something right.
Yeah, it's much easier to to to debug something when it's when it's simple.
So And then you know you're always like building on top of all of this kind of infrastructure. Um your deployment environment your logging Environment. You know any tools that you build for Observability. It's It's really easy to just. Kind of assume all that stuff works and so you better make sure that it does one of the things that I do pretty much in every system that I build these days that um has any kind of alerting based into it which is most of them is I build ways to intentionally trigger faults right? So like in a web app. For example.
Um, right.
I'll have a route that you can hit that raises an exception right? And um, you know the kind of exception that it raises might be variable. You might even be able to pass in different parameters to get different types of errors so they can be handled in different ways. But I want to be able to deploy my product my system to production So like you know going a continuous deployment model I'm just going to you know. Make a change or whatever it is or just you know have the system be running in production and I want to hit that route that creates an exception in the production service and then I want to see my phone light up and say like there's an error in production right? and I want to be able to do that at any time if I have even the slightest whiff.
Um, yes.
Of a hint that maybe the alerting is broken in some way and there's something terrible going on that I don't know about I can immediately dismiss those fears or most of them by being like well let's cause an error on purpose and make sure that we get an alert right and make sure that our logs work and everything else.
Right? That's a super and of course yeah those those things are the kind of things you have to be have a good working relationship with your operations folks so that they know when you're about to do this, you know, maybe you you so you put your you know equivalent of your pager duty in maintenance mode. But.
Of of course.
You still make sure that it appears in the UI and then you trust that pager duty will in fact, ring your phone but that's a really important thing I mean it sort of comes back down to you know? um, who watches the watcher. You know where where do you draw the line about that that observability aspect of your environment?
Right.
And actually now I think about it is a complete non-sequitur. Um, but that kind of worrying about like how do I test the thing that is like the last line of defense like when if an exception if my my web server has an exception I mean.
Man.
You mentioned killing the process. That's another thing that you might reasonably do is log into the box and go kill minus nine that pid and watch it die and then also watch your phone light up and again you don't want to be doing that every single day but it's nice to be able to do that as part of your like ah checks from time to time. Um.
Right.
But yeah, yeah, the how how do you test that your tests Sorry how do you test that your monitoring is working I mean I guess and again I guess I guess again that things like prometheus and Grafana You can just go and look at them. They're there. Yeah.
Right? Yeah, but you know there are lots of situations where you have an absence of evidence problem. It's like okay this you know, um, locked thread count is always 0 right. Have we ever seen. It not be 0 do we know that this metric works. Do we know that it's actually measuring the number of locked threads and not like some other random variable that is unassigned or unused or whatever.
It the that's a very good example as well. Because yeah, if you say things like that. It's like it's one of those things is almost always going to be 0 whenever you take a look at it just because of the I mean assuming you're meaning like some kind of like a lock that you take out before you...
Right.
..do some work. Yeah for the first approximation. It lives at 0 But it's very important when it's non-zero and maybe if it's stuck at non-zero. You know that there's a problem. How do you test that? Yeah I mean and how do you test that right? do you just put a ah you know i've.
Right? right? right? or any kind of monitoring up.
I've been I mean again, this is perhaps a different thing again. Ah you know you can put a big um honking great big global variable I know this is a C trick where I'll do `extern bool HACK = false;` and then I'll poke it in the debugger and then make one of the things actually like `if (HACK)`.
Then you know `if (HACK)` effectively just doing us in our infinite loop and then I can test it ah exogenously. But that's not very reproducible. But then do you really want to put code like that in production you know I mean I look that specific trick.
Yeah. Um, I I mean I have to say yeah I don't know like I'm pretty bold about putting stuff into my systems that give me more observability right? um.
For sure for sure. But there's observability and then there's like deliberately putting ah something which is known to be broken in your code so that you can test that the thing that detects that it's going to be broken works live in your production system.
Yeah, yeah.
And I don't know you know I now I've said it outlined as explicit as that it sounds terrible especially in the context of say trading systems right? There are famously cases of companies who are trading companies that have ah made mistakes of this flavor that have subsequently then folded. So.
Ah, right.
It's not something you want you want to do lightly but on the other hand I don't also know of a better way of of being sure you know" Portland Sure" 1 might say which is a whole other conversation. That's a whole other episode....
"Portland Sure" right? that probably that's a whole episode is "Portland Sure-ity" um yeah I mean one of the questions that I frequently ask? Um, when it comes to. Because you know I mean it would be easy to caricature me as a person who just like yeah I wrote all the unit tests and the L pass and I'm going to deploy it to production and not think about it anymore right.
Not just easy but done every day at work and online and like online/on air.
Ah, right? right? if only we had comments for this Podcast. We don't I'm sure that that would be in the comments right?? Um, and all of the jokes that you've seen maybe about the like you know, like the um, the paper towel dispenser. Over the trash can that like once you trigger it like the paper towels just continuously stream out. Yes and the and the and the sub title is you know all the unit test pass that kind of thing right.
Because it sees it as a yeah hand underneath or what was it. There's another one with there's ah um, like ah ah, a pipe and it's got a hole in the pipe. But there's a second hole in the ah in the pipe and the water is leaking out of 1 and then going into like the l-shaped pipe and the but is squirting out. It's like again, everything's fine here like the test pass water gets from one end to the other Ah yeah yeah.
Oh yeah, ah, ah, yeah, right? All the tests pass right? Yeah, but. 1 of the things that that I do and I I ask of the people who who work on my teams and I will gladly ask of anyone that asks me for advice is if you build something if you created some new capability functionality within your system have you ever actually seen it work...
Right.
...Like in a live running system. It doesn't necessarily have to be the production system. It could even sometimes depending on exactly what it is just be your workstation but have you ever actually seen it work because if you haven't you have no reason to believe however, many unit tests you've written. That it does and so whatever you need to do and you're talking about you know, poking variables into the system or all these other things. Whatever you need to do to create the behavior create the effect that you just spent days weeks months trying to build do it? Um, and.
That's a really interesting point there? Yeah yeah.
Design the system so that you can do it and design the deployment system so that you can see it happen right? like all of these things need to be done if you have this model of like okay I'm going to just you know, write a bunch of unit tests And. Ah, you know do a continuous deployment of my system into production and it's all just going to work great and I never have to check anything and I never have to do anything that even smells like manual testing I'm going I've never been able to develop software like that and I love tests right.
Right? That's that's really no that is genuinely very interesting to hear and heartening as as a mortal who doesn't like it's not quiet as not quite as test because but. But you know like I've always felt slightly dirty doing that I feel like I'm admitting something here but like you know the the fact that I've even tell you these tricks that I have right for like well this is a specific deploy I'm going to do it just to see because it feels ephemeral and it feels like it could break again.
Mortal! Haha.
Because it's not automated and so I know that that's sort of wrong on the 1 hand. But if I'm now going to take the other side of the and play Devil's advocate to my own like feeling then it's like seeing it work once is still infinitely better than deploying it and never actually having seen it work at all.
Ah, yeah, yeah. Right.
Even if it breaks the next day you know what you've done is you've shown that the onramp works and then after that you kind of assume maybe too much that it the the transitively you test all the bits around it and then no one should break the onramp or whatever the the thing that actually causes it to happen and obviously if you can. Contrive it to happen in a controlled way and you can have tests that do it. But that's 1 thing but there yeah, but yeah as I say I've always felt dirty like I'm doing it wrong when you you have the oh um, you know what? I'm going to do is I'm going to put a massive sleep in this thread and comment it in and out...
Right.
..just so that I can show that my slow thread detector thing fires up because you know I can't otherwise you know I can write all the tests in the world and all I'm really doing is showing that my mock when it returns timed out I appropriately log it timed out and it doesn't feel very satisfying. Ah.
Yeah, yeah. Right? right? No Absolutely And and I mean the only thing that I would The only thing I would be concerned about with that is if you now feel like you have to do that sort of what I would call exploratory testing every time you make a change right.
Right? right?
Like I don't feel comfortable deploying this out to production until I go do the sleep thing again that is telling you okay you you probably need to write some tests here or you maybe need to design this in a way where it's easier to test or you need to design in a way where it's observable where it's like maybe you're not testing it with automated tests.
Yeah.
But you have some test environment that can simulate it or reproduce it and and and get that out of there because you know if if you're if you're stuck in this world because I mean the purpose of this kind of like have you seen it work once goes right back to what we were talking about at the very start of this conversation which is it's what you don't know.
That ain't so or what you know that ain't so that gets you in trouble and and and seeing it work is an opportunity to prove that you're wrong that your assumptions about how things work aren't so and if you pass on that you're just sort of waiting to be wrong in a fantastically horrible way right.
Um, right? Ah yes yeah. Yeah.
But what you don't want to do is turn that into a crutch like once you've done it one time; once you've had that opportunity to to disprove yourself and you've failed to disprove yourself if you feel like you have to do that over and over again, you're missing tests so you're missing some other form of feedback that you. You need to create because it's not scalable to do that for every piece of functionality in your system every time you change anything. Ah yeah.
Yeah, yeah.
Exactly Ah, here's the document that says this is how you're meant to break it and poke this? Yeah um I mean I could probably make a case in it very extreme circumstances where you might need to do that like where if your code is say extremely Performant. You can't have loads of if statements in the middle of it for all these different things then maybe but and but but then typically you've already take paid a massive amount of cost to develop that thing and then you put a bloody great big um ah like comment at the beginning. It says nobody touches this code under any circumstance without running these tests and it's.
Tucked away in the corner of your code base again. The lock free stuff of which I was alluding to earlier exactly exactly this you know you spendt ages getting it working and it's almost impossible to test that it is completely right? under all circumstances because you know it's that kind of multithreaded um thing. But but yeah has that.
Yeah, yeah, yeah.
That that flavor of like well once you do get it working at great cost. You don't touch it and then it's fine and then if you do you maybe do have some extra manual steps around it. Um, but yes, yeah, sure.
Right? right? Yeah and I and I have those things too like I have things in my codebase right now where it's like you know little functions that have been written little you know main entry points that you know run a whole bunch of multithreaded code in a tight loop hoping that if there's any concurrency issues in there. We will find them knowing that. That is not going to save us. But at least if we detect it that way we found 1 right um.
Right? Yeah, exactly is. That's exactly the kind of thing you know you do want your unit test to sit there for 6 minutes while it just runs every possible comp. No nobody wants that but you do it's nice to have. Maybe even if you ran them daily. Maybe if you just written them when you make the changes. Yeah.
In the end now...I Just run them when I change it you know? yeah because again, it's ah it's a very small amount of code that is under test there right? It's a very like specific thing and you design the system so that it is a very specific thing. You don't want to scatter that across the whole codebase right.
Yeah, you have to know that they exist. Yeah, it is yeah you can isolate. You find the abstraction that means that like the horrible code lives in 1 place and then then you test the crap out of it using non-traditional means.
Yeah.
And then you kind of say all right? We're done here with dust your hands off and go I Hope we never have to touch that again. Ah.
Um, right right? It's actually reminds me so like the the cliche of putting all your eggs in 1 basket right? There's the original version of that hadmore in that cliche.
Interesting.
It's put all your eggs in 1 basket and then watch that basket. So it's not necessarily a dumb thing to do is putting all your eggs in 1 basket. It's you can you just watch the basket right? yes.
Yeah, you consolidate your risk into 1 place where you know where to look as opposed to scatter it throughout your codebase in this particular instance. Yeah, which I mean we've seen the number of times you know, like anytime you have some complicated thing. Um that you have to do exactly right.
Exactly exactly.
And then if you have to scatter it through your codebase then you've probably designed your apis wrong. It's much better to put it in 1 place and have an api that meet means that the awkward to do thing is in 1 place so that when you inevitably get it wrong. You only have to fix it in 1 place as well. Yeah, no, that's that's cool I mean that's an interesting.
Yeah.
Um, what I thought of when you were talking about ah like the manual testing and stuff and if you've never seen it fail then or succeed. Sorry if you've never seen it succeed that's kind of inverted from literally every test I ever write which is I start up my id I open new file in the in the relevant place and I do.
Whatever boilerplate I need to do to get an empty test and then I set the first test I do is `def should_fail(): assert false` and then I hit the button that says run my tests and I sure as heck it should fail on the line that says assert because the number of times that I've misspelled the word test in the file name like "_tset""or something like that.
Nothing. Yeah.
Some other thing that means that it doesn't treat it as a test and it either runs it as just a regular python file and there is nothing to do because it's just a bunch of files or it's the C++ and whatever you know any number of reasons why it doesn't actually execute it as a test. Can give you the most horrible sense of false security. You're like hey I'm writing these tests one after another not a single one has failed I am great and then ah I am this is oh my gosh I'm gonna have to this ah this day will go down in history as being the day that I got everything right?
Yeah, right I am a superstar.
And then you realize you've started by misspelling the file name of the program of the test and now you feel very very silly. So yeah Assert force Yes, to be honest, Yeah, absolutely yeah.
I think all of the deploy first stuff sort of comes from from a base philosophy right? and it and it is the same philosophy that tells you you know, make sure your tests fail as you expect them to fail right? when you don't have any behavior you run your tests they fail and they fail the way that you expect if you.
If you build something. How do you know that it actually works have you ever seen it work. Well, you need to figure this out and it's it's like ah it's like a scientific mindset right? It's like how am I going to prove that I am wrong and just sort of thinking about those things in a very systematic and continuous way.
Yeah. Ah.
Like every little step you take how am I going to prove that I'm wrong here right? Um, and it the.
Right? I think yeah, it's very easy to fall in the trap of asserting the behavior that you know to be right because you're still I mean but that finds a remarkable I mean still finds a remarkable amount of Idiocy in my own code is number of times are like oh yeah, this is how could there possibly be a bug in this thing. It's so easy to write test for it. It's almost rude that.
Oh yeah.
Rude not to and then you find the bug Anyway, oh gosh. But then yeah to then turn it on its head and say like okay well I've written this ah list of processor or this this lock free ah queue how could I show that it was that it isn't working or how could I show that it is working even right? you know that's.
Yeah.
You know and well um, does you say spawn up your 12 threads and then you get them dumping stuff in then then you make sure that you can read it out the other end or whatever and then yeah, um, ands yeah, that's cool, but but this was meant to be deploy and then we turned it into tests which is the way we do these things but they go hand in hand I think it's about.
It's about discovering the things that are broken as soon as possible howsoever they are and then developing confidence that it is actually working and then yeah if you're deploying your software first then you could always point like I can point to my my Ceo who when he's been telling me when is this thing going to be up and I can point him at the publishing 0 version go like. We're only a configuration file away from this being something important to publish instead of a bunch of zeroes but like the plumbing's all done and that's really satisfying.
Now I mean that's that's how I sound it is right? Well and. Yeah, yeah, yeah, yeah, well and and relating some of what we were just talking about back to deployment one of the things that you may discover if you start asking the question of have you seen this work is you need an environment to watch it run.
Ah, yeah, yeah.
And sometimes that can be your production environment. But um, it's better if it doesn't have to be right? And so um, you know I think we've talked on a couple of episodes before about you know branch-based environments and things like that.
I I think so if we haven't we should because it's a very good topic. Let's assume we have but you've got a very good setup on your current project where you effectively every branch in Git is its own environment and so everything...
...gets deployed to an environment with that name and it's like a unique DNS name and blah blah blah but it means that you can basically just for the hopefully lowish cost that whatever provider you're using. Um as the backend for each branch. Um, you get a copy for every pull request effectively.
Um, maybe we have yeah. Right? Exactly exactly and so that that really gives people on my team at least no excuse to say have you seen this work because they're like well it's too hard to see it work I'm like well then this design of the software is wrong because everything else is set up to make this super easy. Um, but.
Yeah, if you if you don't have a way to easily run your system in a realistic environment that gives you confidence that it does actually behave the way that you think it behaves it's going to be very difficult for you to answer this question. You know of like you know, have you.
Yeah. Have you seen this work. Yeah, well then that's something we need to solve. Yeah, if you can't see how to do it then that's a structural problem with either the way that the team is set up I mean and I'm now thinking that like this is exactly the problem with my team right now is that there are a number of things that are very, but.
Have you seen it work right? It's like well I'm not sure how I would do that I think a lot of software.
Byzantine and bespoke that are difficult. We kind of lean on the crutch of having a onesize- fits or staging environment where we do discover things but because it's the tragedy of the commons of like everyone in there if 2 people have made us screw up because they couldn't test it any other way then we've got two problems in the same environment and's you know you say to?
Right.
And they well sorry they can quite reasonably say how could we have known this beforehand and I'm like well ah sorry you can't so I'm I'll tell I'm taking a note here "be less rubbish". Oh dear, this is all my fault. Sorry my team.
Ah, yeah, well yeah, and I mean and you know it's it's like time in the testing environment is like time on the mainframe you know you go schedule it I get from two o'clock to four o'clock on Thursday is day afternoon right? Yeah yeah, yeah, well.
Ah, no, it happens I mean for us there's a physical hardware component which is unfortunate. Um, you know, not everything quite a lot of things can be run either on our local development machines or in small like cloudish environments but we have got some in our particular case.
And that's can make it difficult.
Some very obscure networking stuff. But I have got a po o out to buy more hardware so you know it will happen. But yeah, still less than ideal and some of this stuff is like so let me all Rightre where we we should probably wrap up.
Yeah.
We're about, the right amount of timing but 1 of the things that I find hardest and this is where I lean back on and I was actually only quoting you the other day about this when I was telling people. It was okay that these kinds of mistakes were happening which and I blamed you; is "fail fast"
Yeah.
Right? These were things that we couldn't otherwise test and when I say that we know that we have configuration files that like um have the essentially command line parameters for a whole bunch of interconnected programs that run in an environment right? You could imagine this. There's things like the names of Kafka topics. There's the name of brokers that are the brokers for this environment versus some other environment. There's every other like command lines which you might imagine that you might have and so we have one that's like this is the staging environment command line. This is the production command line. This is the development one and this is the all and for like n things right? and there's always a bit of wiring somewhere right? I mean.
Yeah, yeah, yeah.
There are ways and means of like making them automated or whatever. But for whatever reason it's like the place where we do go. Oh I'm gonna turn on this flag for our staging environment that makes it on purpose different from production because we're testing something that we want to run for a long time and then see if it compares good. Yeah, all that good stuff right? But it's also the number 1 place to typo.
And. Yeah.
Ah, Command line flag name and you could try ahead of time running on your dev machine but you will be publishing to a topic that is used in the staging environment or God forbid in production. So you better make sure you don't type in that bit of the commander and so people don't quite reasonably. You're like okay I run it.
Right.
Maybe and I Maybe if I'm feeling really brave I carefully comment out all the things that I know to be production affecting and then I make sure and obviously there's some network partitioning as well to prevent the worst of these things from happening but it's still ah a risk. So ultimately, really, we all look at it. It goes through code review.
Ah.
2 or 3 of us stare at it and then we commit it and then and only then do we de define discover oh that it's you know dash dash blah underscore thing rather than dash dash blah dash thing and you're like of course it is but none of us could have seen that stupid thing. So the the only sort of comeback I have is it fails immediately at like seven in the morning when it deploys.
Right? right.
And we've got plenty of time to go and make a patch release to to fix it before like we care about it. But um I'm now just trying to think how that works in in say your branch-based environment. How do you deal with the fact that there are some.
Yeah.
Configuration-y things that maybe are different in production right? You're like okay, this really has the dash dash. No, you are allowed to here are the credentials for the thing that you can do.
Ah, so the way that we do. It is the configuration is literally just code. We don't have configuration files we have classes and everything and there aren't very many of these but everything that is oh this happens in production or this not. Is not is keyed off the branch name right? Um, and it's probably less than half a dozen things that I can think of yeah and and there's and not There's not surprisingly unit tests for all the configurations. So it's sort of like oh when you have this...
Okay, and so that's I mean yeah, that makes sense but but so this...
..setting set then it creates these objects instead of those objects and there are unit tests for all of those things that confirm ah like there's a suite of tests for the main configuration that's got the special bits in it. There's a ah test a unit test for like the deployed branch configuration which includes the main configuration but also all the B R ones.
Yep.
There's one for ah, local testing and there's one for sort of like our operational scripts and things that run in the same environment and all of those things are unit tested.
Like that. Got it that makes some that makes sense I think I don't know if that could work for us now. Um, one of the things that I like about the command line flag based version is that we often use that locally a lot as well. So I want to run something that looks like the stage of environment. But I'm gonna do tons of changes to how it looks and and then I can also.
I.
Into into subjectively paste that into like slack and say hey this is a reproducer for that issue. You can run this locally and it go whereas if it was like a local config that I then actually had to edit I'd have to check it in somewhere and say you need to pull this version. But yeah, that's an interesting way of solving this this problem. Um.
Um, yeah. I'm right. Yeah, it's interesting because on this project and this is not something that I have done before is we I think we talked about this a little bit into the in the transition from linux to Mac episode where we sort of readed all of our operational things in Java because we were forced to because we were.
Ah, Mac right? yeah. Right? and rather than running you know said and a and gawk or whatever you like well I'll just write the 3 lines of Java that does it and then it works but on both operating systems the same. Yeah.
Changing operating systems right? Yeah, yeah, yeah, yeah, and 1 side effect of that is that we have very much sort of turned into this for better or worse I'm not saying this is a good idea as I'm just saying this is what we did. Um, we've turned into this thing where you know to the the way that we do things intersubjectively on the team is someone writes a little Java main function and then they check it into a branch and they're like here's this thing I'm trying out. And sometimes we actually wind up copying and pasting that code and is it's like I I want to try this over here I'm just going to copy and paste it and run it. Um, which is a little more complicated than copying and paste in the command line arcs. But it's maybe I don't know it's um.
Right? Like a github gist kind of thing?
...the sort of like environmental shift that we had to to be able to develop on max is sort of forced us into this mode which I've never done this before but it works okay like I don't have any major complaints about it. Um, but 1 of the things that it does do is it allows us to lever all of the. Regular environmental tools and libraries and and checks and everything else that we have where it's like if you make a configuration for one of these scripts. It's real clear what it has access to and what it doesn't right like that's pretty bulletproof.
Mother. Yeah, yeah, yeah, no, you just triggered a memory actually like at at Google there were definitely some tests that would like take the string and run it through the command line parsor and then make sure that the output was what you expected kind of level things for testing things like this and it kind of.
Me.
Flavors similar flavor but like some of these things have complicated independent interdependencies that you would be like well you know, even if you got it right? The the you might be using the right channel but you're using the wrong broker and it's hard to right test it I Yeah I don't know I maybe I'm making excuse I am making excuses for myself. But but um.
A.
But you've definitely given me a lot to think about. There's some definite improvements we can make I mean there always is with these types of things and yeah so I think given the time we should probably leave it at that. Um, this has been incredibly useful and I think you know I can definitely claim this back this time back because I'm gonna.
Yeah, there's pretty much an infinite number of improvements that you can make to these things.
You know as in like company time. This is a perfectly good use of of company time to to teach me some ideas about how to improve my setup and then hopefully to tell our listener the virtues the many virtues of deploying the hello world app before you've even written the rest of your code.
Um, yes, and then make it fail. Yeah yeah, cool.
And then then make it fail ah cool all right? Well um I guess we'll leave it there until next time.
Till next time.