Hey, Ben.
Hey, Matt.
How on earth are you?
I'm doing okay. I'm still, I'm still a little sick.
I was gonna say, you sound a bit nasally still.
Yeah, I got, I got a little bit of not really congestion, but just sort of like, I don't know about you, but I feel like at some point in the winter, I get a cold. And this was obviously not true in the last two years of the pandemic
Because you didn't see other humans.
Yeah, right. But at some point in the winter, I get a cold and that cold lasts until spring. That's been my experience too. Right? Yeah. It's like I feel better, like I'm not really sick, but like that little lingering cough or that little sort of like, you know, rough sound in your throat. It just, the only thing that really cures it is spring. And I hate that. I don't know, really. And
It's always, you always sound the worst after you've stopped feeling bad. That's what I might discover. So like, you feel dreadful for a couple of days and then you get no sympathy off anybody. Yeah. And then you sound all stuffy and crap the next day. And it was like, oh, are you all right? I'm like, I feel fine now, actually, but thanks for asking.
Yeah, right. Exactly. So yeah, probably a more long-winded ans answer than you wanted for
That was. Yeah. This is, look, this is the SYN / SYN|ACK of the podcast here. Not, not the, I I'm not actually genuine interested in how you are. I'm afraid.
Oh, man. Yeah. So what are we talking about today?
That's an excellent question. Well, we were chatting yesterday at breakfast, such as it was, we, we got a coffee together as we'd kind of got on the same train. And then we got talking as we do, and then we, the, the conversation was so interesting that instead of going to our desks once we actually got to work, we sat down still wearing our coats and carried on the conversation about integration testing of all the things. I mean,
Yeah. It
Seems unlike us to, to be so excited about something that many people might consider very dull, but maybe not our listener who is that now going "testing? Amazing Tell me more."
Right, right. Yeah, it's kind of shocking to me. I was looking back over some of the old episodes and it's kind of shocking to me that we haven't talked about this the best I can tell. I mean, we've definitely talked about shades of it, but,
You know, we talked about acceptance testing, which to me Yeah. I sort of broadly think of as an integration test of, of a sort. I mean, it it is, yeah. Pretty much an end-to-end test type thing. And that's obviously has its place, but not sort of like the integration test that we, that we know and love or I've written before now where
Yeah. Right,
Right. Often come to regret latterly. Mm-Hmm.
. So yeah, so I mean, basically, you know, as we were kind of talking about yesterday as you, as you said integration tests are I think a, a bit of a double-edged sword. And one of the things that has stuck with me for a very long time is a talk that JB Rainsberger gave mm-hmm.
It's a very sort of clickbait, you know, of course in this world, you know title. But it's, it, it, it makes you think, and I think the central point is, is sort of bang on, which is integration tests, which we're gonna define as tests that exercise large chunks of the system, perhaps spin up external dependencies like a database or, you know, some other sort of like network service. You know, if you had like a if your system was built out of microservices, you might spin up like a whole bunch of containers to run all your different microservices and have them all interacting with each other. You know, anything that, that talks to like a third party service you know, that that's kind of what we're defining as an integration test here. And, and usually in those kinds of tests, you're testing lots of different pieces of behavior all at once, you know.
Right. An example of this would be like, all right, I'm gonna, I'm gonna log in as a user and I want to test to make sure that we can log in. It seems like a reasonable tester, right. Make sure users can log into our system. Right. Certainly very important. Yeah. So I'm gonna start up the web server. I'm gonna start up a database. I'm gonna populate that database with some user records. I'm gonna, you know, go to the website. I'm gonna enter in a value for the username. I'm gonna enter in a value for the password. I'm gonna hit the login button, and then I'm going to get redirected to a dashboard, a user dashboard, and I'm gonna confirm that all of that works for some definition of confirm and some definition of works, right.
Those are the kind of tests that I think JB was talking about in his talk. And his central point was those tests are a scam. They seem like a good idea. They seem like they give you like a little bit of a safety net and some confidence that things work and they do. But the costs of those kinds of tests are very subtle. Sometimes they can be very hidden, and they are often paid in the future with a ton of interest. And so if you build your system with the idea that you're gonna test everything through integration tests, what you often find is that over time, your tests, your test suite gets very slow and it gets very brittle and it becomes extremely difficult to make changes to your system, which was kind of the whole point of writing tests in the first place. Is that right? You want to be able to make changes to your system quickly and have confidence that you haven't broken anything. Yep. And it's like, well, you still maybe, maybe have the confidence that you, you have haven't broken anything, but you can no longer make changes to your system quickly. Right? Yeah.
Yeah.
And so that is why integration tests are a scam, scam, that sort of Yeah. And that idea has kind of stuck with me for a long time. And so a natural question that one would ask if you say, okay, Ben, let's take that as a premise, is what should I do instead?
. Yeah. What should I just, should I just accept that occasionally I'm gonna make an an innocuous change and then everyone can't log in and, and that's okay. I mean,
Exactly.
Maybe, maybe it's okay if you're Twitter Too soon, except not too soon, because by the time this goes out that no one will remember why, why? That's funny.
Sick burn. Yeah. That's, that's true. That's a good point. But yeah, so, and that, and that is sort of what we were talking about yesterday is sort of like, okay, well if you're not gonna do that,
What
Are you gonna to do?
Then? How can I just develop confidence that I can make changes and I haven't like painted myself into a corner where to, in order to make those changes, I have to like update the integration test because I've moved the login box up a little bit and that was totally not the, that's the point of my change.
Yeah,
Yeah. That kind of stuff. So what do I do? Tell me. Yeah.
Well I was gonna ask you some of the things that you've done. Obviously I've done a lot of things on this, but if, if you want to throw
No, I won't. I was playing the fool. Go on. You asked me. All right. That's, I don't, so like yeah, the, the, the, we, we, the, again, we were talking about this yesterday, which is not a, a good thing to keep bringing up on the podcast cause nobody was there yesterday, which is why we're recording this today. Yeah. But we talked a lot about transitivity of tests mm-hmm.
.
So we have tests that say, does our interaction with a database work, can I select rows out of a database and read them back in again? Okay. That could be a test. Now that does sound integration-ey and maybe it is integration-ey but like I, I'm, I developed some amount of, of confidence that I can talk to a database that could be a first step.
And then I go, well, now I understand how databases work. I'm gonna just get rid of the database and replace it with some kind of fake or a stub or a mock or any insert your favor, not a real thing in technology here. And then I can write the majority, the line share of my, can I access, quote the database correctly tests, using tests written against that fake stub mark, blah, whatever mm-hmm. I can log into the database directly now. Mm-Hmm.
Yeah. But the, the marginal cost of dealing with the, the, the Machiavellian case compared to like the integration test cost, which is giant and as you say, brittle and often fragile and, you know, flaky. I'm, I'm definitely like, it's definitely not as certain because for each step, I'm, I'm only 99.9% sure. Let's say mm-hmm.
Yeah. Yeah.
The, the chances of a real bug falling in that 3% crack seem diminishingly small to me, and the cost for a big integration test, or at least developing all your tests as integration tests now. Right. Maybe there's a call that says like, it is so fundamentally and critically important that users can log in, that you do have a once a day CI thing that stands up the system and does the thing or pre-deployment or even you have a human do it. Right. I, at that point, I prefer not to do that. But like, you know, if you need it then this is, there's a business case for it. But if you build a system where that's the primary thing that you, the primary way you make your tests, then you are definitely making it hard for yourself. Right. You can't iterate as quickly and it's, yes. So that would that I, I think I went all the way to the far end of the
That's really good. I think, I think that's a really good baseline cuz I think that describes the sort of basic answer, the, the short answer to, okay, if I'm not gonna do integration tests, what I'm gonna do, well, I'm gonna break the problem down into lots of little bits and I'm gonna use the transit property to as make sure that all of these little bits can talk to each other. And each of those tests for each of those little bits can run super fast and be very reliable. Cause they're not, you know, necessarily talking to external systems or anything like that. And when they fail, they will give you a very focused answer of this bit is broken. Right? It's not like, well, something in the login process is broken. Broken, and I don't know
What it is. Let's say somebody changes the "crypt" method. Mm-Hmm.
Right
Now, maybe you've got a much more localized understanding of where the problem will lie.
Yes, yes, yes. And I think that this sort of, that sort of difference has a number of very interesting properties to it. One is it's sort of the difference between if you're doing any kind of validation, data validation, like checking individual fields versus checking like a check sum or a cryptographic SHA.
Right? Right, right.
Like if you just take a whole bunch of data and you're trying to be like, what are the differences between these two pieces of data? Well, if you, if you run it through diff if it's text data, you can see the differences, right? Right. Like, oh, this line is here and it's not there. If you take both of them and you do an MD five sum on them and you compare the MD five sum, it's like, well these sums are different. And it's like, well why, why did it, what happened? Now it's much easier to compare the MD five sums. It requires no understanding of the data itself. You don't have to look at it, you don't have to understand what it is, what each of the individual lines are. It's very simple, but all you really get outta that is if everything works properly, you knew that everything worked properly.
Right. Anything breaks, you're left in this world of like, well I know it doesn't work, but I have no idea why. And so I think that is why, again, creating the bulk of your tests using this, this sort of integration style will lead to a world in which you will operate very slowly. You will be able to make changes very slowly cuz you will be constantly in this world of, hey, these two MD five sums are different and I don't understand why. Yeah. and so there, there's that, there's that property to it. The other thing that I think that it does that it's, that's kind of interesting is it does put a, a bit of a bur the, the alternative that we're proposing here where you, you break things up into small pieces and test them does put more of a burden on software engineers who are changing the system.
Right? Right. And the burden that it puts on them is that they have to make an effort to understand when they're making a change the parts of the system that interact with that part. Right. Which is not strictly true if you say, well I'm just gonna test everything through integration test cuz I can lean on the integration test to tell me that. Right. Right, right. If I go and I make a change to the system, I can just ignore everything that it talks to and everything that talks to it and I can "trust". That the integration test will catch that. Now in practice, I will tell you that any integration tests suite that is actually comprehensive enough to test every part of a system using that approach will finish after the heat death of the universe
Just too immense
It's like like grains of sand in the universe.
An interesting observation that I'd never really considered is that you are definitely reducing the dimensionality of the problem by testing unit by unit. Like I'm, I'm, I'm testing my password hashing thing. Now, of course there is an infinite, almost infinite number of inputs to my
Right, right. Yeah, absolutely. And I think one of the cool thing is, is that if you sort of do what we're talking about here, you will arrive at that sort of naturally and you will be naturally incentivized to sort of make that that contract with the outside world as small and as simple as you possibly can. So going back to our sort of database example, right. With the login, right? Like, okay, if I'm gonna break these things apart and I'm gonna have, you know, some system that interacts with the database to say, Hey, I need the, the, you know, I need to get the the salted password for this user, for example, right? Like the fundamental operation of get me a sallted password or get me any field that's related to a user has nothing to do with SQL or databases or, you know, servers or network connections at all. It's a very simple operation, right? And so there are various points in this sort of chain where you can create abstractions that are significantly simpler than the actual underlying system that's backing them,
Right? Right. You could reasonably have said, you could have exposed the sequel to everybody, for example. Like, and then in the first case, right? Because it's convenient, Hey, select star from users where name equals quote, whatever I have in my street. That's, that's one thing. But this forces you down the, the route that says, no, no, I want to pr provide the API of bull. Does user exist? Mm-Hmm.
Right. Right. Right. And that has a lot of like really good design benefits that are very practical. They're practical in the sense of like, it's helping you. Right. Better, faster, more reliable tests. But it also sort of, in my experience, it actually sort of creates a, a nice sort of framework for discussion among like the people that you're working with. Right? Right. Right. Because like people have these sort of like esoteric, almost like philosophical debates about software design all the time mm-hmm.
I've never thought about that, but yeah.
As opposed to I want an object hierarchy here. It's, no, we should use functions as, no, we should use a list. No, we should use a set, whatever. And it's like, if you can tie that back to something concrete that everyone agrees is valuable, of course, then you can actually have like much faster, more coherent, more aligned discussions around software design because they're based on fundamental principles that you all share, right? Yeah. Yeah. And you don't get the opportunity to do that if you don't create the constraint by saying, well, we can use whatever design we want cause we're just gonna create this with an integration test anyway. Yeah. Right. Or we're just gonna test this with an integration test anyway. So it doesn't matter if you create a nice layer of abstraction over the database, you can just have the templated string with the sequel in it and, you know, inject whatever values you directly got from the UI into that script.
Terrifying
Because, because that's not gonna, that's actually the easier thing for you to do, as opposed to the easier thing being, creating a nice abstraction that you can then swap out. Right. and this actually leads me to another thing that we were, we were kind of talking about, which is a technique that I do all the time, which is using the sort of test fakes that you create as part of doing this process as sort of the repository for institutional knowledge about strange errors.
Right? Right.
So like you build your interface to your data store, right? Right. And,
And there were air quotes for the listeners there around the data store. Yes.
And it's, and it's a database under the covers. The, the normal implementation is a database, right. Some thin wrapper around some, some SQL driver or something like that that you have. And maybe you even think about it, it is like a SQL based thing, right? So it's like, I'm giving this thing sequel statements and it's giving me results. I don't really understand what's going on behind that abstraction, but that's what's happening. Yeah. Like that's the interface that I have. Right. And you have your sort of real, okay, we're using Postgres, so we use the Postgres one, and then maybe there's like some, you know some really stupid stub that you create maybe where you just have like hardcoded query strings with hardcoded responses, and you can use that in some of your unit tests when you're just testing things like, yeah. Right. I just, that's not the point of this test. It's not the test if the database works or if sequels correct or anything like that. I just wanna make sure when I run this grade and I get this result, do I process it
Correctly? Right? Yeah. Which is a very common thing to wanna do. Right. You know? Yeah,
Exactly. Exactly. But another form of implementation that you might have there, in addition to the real one, in addition to the super dumb fake one, is a realistic one that lets you simulate error or failure modes. Right? Right. And this is something that like a lot of times you will have some strange esoteric error, like oh yeah, the cluster was rebalancing and in the middle of the rebalancing we executed this query and there was some sort of weird, you know, data partitioning condition that the database detected and it threw this era that we'd never seen before. Right. And we have no way to reproduce it because it just depends on like the timing of the database, like actually just doing its rebalancing and, but
We wanna lock in the good behavior of if we see that again, right? Yes. Because we we fell over in prod
It created, yeah. So, so like where does that knowledge go? Right? Well, in a lot of organizations it just goes into a ticket or a document, and that's, it's sort of is that's where it is. Right. But there's an alternative here, which is you can take that hard one, hard fought knowledge where somebody had to wake up at two in the morning to deal with this thing and take it and encode it into a sort of a test double, like a fake database whose purpose is to kind of mostly behave like a real database, except it lets you puppeteer all of the different failure modes that you've seen. Yeah. So when you go and you write that code to say like, Hey, we have this one in a million error condition that only ever happened one time, and that happened 12 months ago, and I'm changing the error handling code for that right now because we're making another change. How do I know if this error handling code actually still works in
That weird case? Yeah.
In that weird, that one weird case, right? Well, if you encode that information into your test doubles, you can have a lot of confidence that works, right? Because you can say like, yeah, well, well we, we reproduced the error that we got in the client code by rera, that same weird error that we saw out of the database, the client code failed in the same way. So we knew that we sort of reproduced it and then we fixed the client code so that it handled that error so that it wasn't an issue. And we have a test, you know, that makes sure that like, when that client code runs in this error occurs, it, it works. So I can go back later and I can say, okay, well if I change how that's handled or I handle it in a different component, I can reuse that same fake database in a completely different component. It's like, oh yeah. When you're interacting with the, the storage system,
You have to know....
Watch out for that, sometimes it'll return that error.
Yeah. Yeah, yeah, yeah.
Because it can happen sometimes and you can have a reasonable degree of confidence that if it happens again, you will handle it. Right. So there, there is another sort of opportunity that you get by, by, by using this approach to sort of encode that information in a way that is much more immediate than tucked away in a ticket somewhere that no one will ever absolutely look at unless something goes
Terrible. At the very least. I mean, I, the fake stuff, I, I can see that it could work, but I I, in my experience, when those things have happened, definitely get setting out and like making a mock that walks the system through that exact system, exact set of circumstances, and then giving a test the name that explains it and maybe has the reference to the ticket where it was filed, or the page of duty alert or whatever. Mm-Hmm.
mm-hmm.
And in some cases they're almost impossible to puppeteer the system into a set situation where Exactly. Exactly. You can make it happen even re remotely or, or at all, you know, like some strange admin commands and you know, nobody wants to have any chance at all that, like these unusual things where you're like maybe even killing processes to try and kind of make it do the error. No, that's not code you wanna write in case it ever Yes. Somehow makes it remotely near production
To that. Right. Right. Yeah, exactly. And then you wind up doing it, it's like, it's like very, very expensive to write those tests and run them and make sure that they work. And it's for this error that has only ever happened once. Yeah. And you start asking yourself like, yeah, we added like two minutes to our build to like, you know handle this, bring up this, this database and tease it into a state where it produced this error and then crash and then do the thing. Yeah.
That's not, and it's flakey,
This certainly ever happen one time. Yeah. It doesn't and it doesn't always work. Yeah.
You can easily argue for like, well, let's just cross our fingers and hope it doesn't happen again. And which isn't a satisfying answer compared to here's a 15 line test that sets up the, the fake in the right way and calls the login method and says, yep. Look, even if we get a split during the login, we just retry.
Yep. Done. Yep. Exactly. Exactly. So, so I, I think that there are a lot of benefits in addition to the, I think sort of more immediate benefits of the tests run faster and they're more reliable. There's a lot of like design benefits to saying like, okay, we're gonna decompose our system in this way. We're gonna be able to test it in this way. We're gonna think about it in this way. There are trade offs, like I said before. It means that you need to think a little bit more about the interactions with components when you're changing them. You have to have a sort of a wider view on the system. Right. But I think that those tradeoffs are, are very, are very
Worth it. Yeah, I agree. So there's another sort of aspect to quote integration tests that is something that certainly I've done and I know other people do as well. And that is like, sometimes you don't know how a third party api, a third party server and how it works. Right. And so you want to kind of write Yeah. Yeah. Test in the same way that I write test to test code that I might be interacting with kind of exploratory, even if it's my own code, you know, t d D style. Like, Hey, I wonder what happens if we pass in this string to my code. Mm-Hmm.
Yeah. Absolutely. There's, there's a real power in writing tests that you intend not to keep. Right. I, I sometimes refer to this, and you've heard this before as fake make it till you fake it instead of fake it till you make it. Yep. Right. So you, you can write tests against a real system. And this, like, I've definitely done a thing where it's like, all right, I'm gonna get some read only credentials to this database. It might even be a production database cause it's got some data and it is interesting to me, but I've got some read only credentials. I'm confident that they're read only credentials,
What does the error message at you look like? You know, we've got
Yeah, exactly. Exactly. And so and I will sit down and I will write tests to sort of explore that knowing that I'm going to either dramatically change these tests before I actually go and check it in, or I'm just gonna delete them. Right. and that can be really useful, especially if you have a setup where your tests run automatically. It almost becomes kind of like a repel where you can just be typing it away in a test and be like, oh, what does this thing do? I'm gonna assert that the return value of this, you know, get user count is null. It should always be an integer. Right? Right. And then I say, oh yeah, it's 105 in the production database. Cool. What if I point it at a table that doesn't exist? Oh. It's null. It should never be null.
Yeah. But in this case it is actually, actually no. Right. Yeah. So that kind of like exploration could be a great way when you're in that stage where we're trying to figure out how to break this thing apart, right? Because again, one of the things that, that you get from this, one of the trade-offs from this this approach is you do actually have to think about the design of this and how do I create these, like easily testable seams that's not always obvious when you're first, you know, using an API or thinking about a problem, right? Like, you have to sort of come up with that. That's your job, right? So having sort of like everything just sort of laid out in front of you in a messy way that doesn't try to do this so that you can see it all and you can interact with it all.
You can understand like how the system works and then say like, okay, I think that if I built a little abstraction around this part, I could basically treat it like let's say for example, a a sequence, right? So I just have a sequence of lines that is coming out of this thing. I don't really know what it is, but I can create an abstraction on top of this API that will, that will basically treat it like that and then I can build my real implementation under that. And all it does is produce a sequence of lines mm-hmm.
The API that I now know how it works, I can model as a sequence of lines. And so the wraparound that is extraordinarily simple. And then I can do some additional things where I, you know, kind of like we were saying before, maybe I write an integration test for that real implementation. Maybe what I do is I just get really heavy into mocks and I actually just like mock out the interaction with the API. That's another approach, you know, you can sort of do either way. Maybe what I do is sos something I do a lot is rather than having an integration test, I test for integration. So when my system starts up, I say, I'm gonna take all of my objects that I'm instantiating that represent real world systems that I didn't want to write unit tests for, because they would be slow and unreliable, and I'm going to make sure that they work at the point when the system starts up so that it will fail fast if they don't.
So in the case of the database, I'm gonna make sure that I can connect to the database. I'm gonna make sure that my credentials are good. I'm gonna make sure that the tables exist, I'm gonna make sure that the tables have the scheme as I expect. Yep. And if all of that stuff is not true, when the system starts up, it doesn't really matter what's gonna happen after that. Yeah. Yeah. There's nothing else is gonna work. Nothing will work. So it's just gonna shut down and be like, Hey, I expected there to be this table and it doesn't exist, so sorry, I can't do anything. And I think that that is a much more sort of practical way to get confidence that those external dependencies really work the way that you do. Because you tend to, you tend to build them up in this sort of iterative form where, like at the moment when you're writing the code, you're pretty confident that it works.
You know, you've done your, your make it till you fake it, or you've written a little, you know, command line test driver program to make sure that it works. Or you've, you know, stepped through with the debugger and you've made sure that it works. It's like, all right, I'm pretty confident this works. But the thing that you have no control over is the outside world. Yes. Right. Did the configuration of the database change? Did the schema change? Did other things change? And so you can try to test for those things in an integration test that runs in your ci but that test is gonna be unreliable. And it might be an unreliable in a very unfortunate way, which is if you introduce a bug that causes a problem with the database, and it's critical, you have to fix this right Now, you may wind up in a situation where it's like, oh yeah, we up the database and I need to go change this code in order to fix it, but I can't deploy that code because it depends on a continuous deployment system that actually checks the real database. Right. Which is not working, which
Just not working. Right. And then you're
Like, so you wind up having to be like, yeah, so you wind up being, having to do this thing like in a panic, like turning tests off Yeah. Which is like, okay, everything's on fire. You're panicking, you're turning off all the safeties.
That doesn't seem like a good idea.
Right. That's a situation when you, that's the very moment when you need those tests to give you the confidence that you can just very quickly get a production fix out there. And not have broken anything.
That's an interesting observation. Yeah. I hadn't thought about that.
Yeah. So, so that, that kind of style. Yeah. I think where you're testing your external dependencies at the sort of the last responsible moment, at the moment when you know, okay, this has to work now, it's kind of, okay, if it wasn't working before, but it has to work now. You know, that, that kind of thing can be, I think, I think a better, a better way. And then you can write those tests to be very focused, right? It's not the sort of general like, oh, do databases work at all
These are preconditions under which I've performed all my other tests. If I can prove those at runtime, now I can transitively believe that the rest of the code should work.
Exactly. Nice. Exactly. Exactly. So, so I think that's another alternative to sort of the like, okay, I wanna write integration tests just to prove that the database is configured correctly. And it's like, okay, yeah, I can see why you'd want to do that, but rather than writing like, you know, a unit test that does that, maybe try a different approach. Yeah.
Well, this has pretty much covered everything that and more that I had in my list below here. So, I mean, I think we should exhort folks to seriously reconsider any integration testing that they're doing with with these kinds of alternatives.
Yeah. Might be a scam.
Might be a scam. You heard it. Here. Might be a scam. I was gonna say you heard it here first, but No, this is what, how old is that talk? You heard it here first
At least second
Oh dear. All right, well that's, that's how much we got our finger on the pulse here.
Right? Yeah.
Cool. All right, my friend. All right, I'll yeah. See you the next time.
Good As always.