Quick note, this episode isn't sponsored. I'm building a new kind of IDE called Rex because existing ones make it hard to work across multiple projects in parallel. I'm sharing it to get feedback from listeners. I'd really love to hear your thoughts. The link is in the description. And now let's move on with today's super interesting episode. Welcome back to the Deep Dive. Today we are tackling a subject that I think keeps a lot of engineers, you know up at night.
And I want to start with a bit of a provocation. A counterintuitive premise if you will. I love a good provocation. Let's hear it. OK, in modern cloud architectures, perfect code. And I'm talking structurally sound, bug free, beautiful logic, the kind of stuff you'd like. Frame on a wall can still create catastrophic failure. Oh, absolutely. That is not hyperbole. I mean that is a Tuesday in distributed systems. Right. And that's terrifying.
We all grow up, as, you know, junior engineers thinking if I write the function correctly, the software works. It's a very binary sort of true, false view of the world. It is. But you're telling me that in the systems we're discussing today, production outages rarely happen because of a syntax error or a logic bug inside a single function? Rarely. I mean sure, bugs happen, people make typos they have off by 1 errors they, you know, forget to close a bracket.
But the really nasty ones. The ones that paid you at 3:00 AM. Exactly the ones that take down an entire platform for hours and make the CTO sweat through their shirt. Those usually happen in the empty space between the services. The space between. It's not about what the code is doing, it's about what the code assumes the rest of the world is doing. And that is exactly our mission.
For this deep dive. We are taking a whole stack of research architecture guides from AWS, some academic papers on things like metastable failures and Bayesian frameworks, and some really deep analysis on microservice pitfalls. Specifically looking at a paper titled The Hidden Danger of Assumption Drift in Distributed Systems and we are going to use it to well to upgrade your
brain. We're moving from junior thinking, which is, you know, is my function correct, to senior thinking, which is what assumptions is my system making and when will they inevitably be wrong? Exactly. And honestly, if you are preparing for a high level system design interview or a coding interview for a senior role, this is the secret sauce. This is the stuff that separates the pass from the strong hire.
I want you to walk out of this deep dive ready to stand at a whiteboard and just blow an interviewer's mind. It's the difference between knowing how to code and knowing how to engineer, and there's a massive, massive gap between those two things. So to guide us through this, we're going to anchor everything on that concept I mentioned a second ago, Assumption drift. Assumption drift.
It's a great term. It basically describes what happens when different parts of your distributed system invent their own version of reality. I love that phrasing. Inventing their own reality. It sounds like a sci-fi plot, but it's really just bad architecture. It sounds philosophical, but it's completely mechanical. Service A assumes a request takes 2 seconds, Service B assumes it can take 30. Their clocks are different, they drift apart. And in that gap?
In that gap, failure happens. And not just failure, silent failure. The kind that's almost impossible to debug because every individual part thinks it's doing the right thing. So here is the road map for today. We're going to build this up gradually, piece by piece. We'll start with the foundation layer, Why distributed systems are just fundamentally weird compared to the code running on your laptop.
Then we'll dive deep into this assumption drift, how it creates these solid killers in your architecture. And finally, we'll arm you with the defensive toolkit we need to talk about patterns like jitter, circuit Breakers, and deadline propagation. Not just what they are, but how to explain them in an interview to show you actually know how the sausage is made. Let's do it. OK, let's start with the weirdness.
You brought a classic analogy to the table from some of the AWS source material, Pac-Man. Yes, the Pac-Man analogy. It's perfect for explaining the leap from local to distributed execution. It really crystallizes the problem. Walk us through it. Why is Pac-Man relevant to, you know, modern cloud architecture? OK, so imagine you're writing the code for the original Pac-Man game. It's the 1980s. It runs on a single arcade machine.
You have a line of code, something like board dot find Pac-Man. Simple enough. Find the yellow guy on the grid. Right, and what happens? The CPU looks at the memory, finds the coordinates and returns them. It's nearly instantaneous nanoseconds, but more importantly, the code shares fate with the machine. Fate sharing That sounds dramatic. It is dramatic. It's a critical concept. It means if the power cord gets
pulled, the whole thing dies. The board, the ghosts, Pac-Man, the code looking for Pac-Man, it all goes dark together. So there's no situation where just one part fails. Exactly. There's no scenario where the fine function keeps running, but the board itself has, you know, vaporized. They live together, they die together. It is a single, cohesive universe. One brain, one body. OK, that makes perfect sense. It's all one machine. Nicely. Now let's teleport to today.
You're building Cloud Pac-Man. The board is on a server in a data center in Virginia. The user is on a phone in London. The game logic is running on a server in Tokyo. Now you run that same line of code board dot find Pac-Man. Syntactically it looks exactly the same in my editor. It looks the same in your IDE, but under the herd it has just exploded in complexity. That single function call isn't a simple memory look up anymore,
it's a network request. And network requests don't just work or fail, they go on a journey. A perilous journey, it sounds like. Very. In the AWS Architecture guide they break this down into what I like to call the 8 failure modes of the apocalypse. And for an interview you need to know these. Not just memorize them, but feel them in your bones. OK, let's run through these because I think people, myself included, really underestimate how many physical steps are actually involved in just
sending a message. OK, so let's trace the packet. Step one POST request. You, the client, try to put the message onto the network. And that can fail right out of the gate. Immediately your network card, your Nic could be fried. The local router could be down, the operating system kernel could be out of memory for network buffers. You haven't even left your own machine yet, and you've already failed. OK, so failure mode one is we trip over our own shoelaces
before leaving the. House, you got it. Step 2 deliver request. So the message is on the wire. It's traveling under the ocean in a fiber optic cable, but it has to actually get to the server. And things can happen on the way. A lot of things. A backhoe could cut that fiber optic cable, which happens surprisingly often. Yeah, a cosmic ray could flip a bit in transit.
Or, and this is a subtle 1, the server could crash literally milliseconds after receiving the packet but before the operating system hands it to your application. So the message arrives, but the person who is supposed to sign for it is dead on arrival. Perfect analogy. OK, Step 3. Validate request. The server application wakes up, it gets the message and it looks at it and says. I have no idea what this is. A version mismatch or something? Exactly.
Maybe you upgraded the client to send Jason, but the server is still expecting XML. That's a valid message, just not one it understands rejected. OK, so we're three failure modes deep and we haven't even started to do any actual work yet. Not a thing. Yeah, now Step 4, update server state. This is the first time the server tries to actually do the thing you asked. Move Pac-Man. But maybe the database is locked, Maybe the disk is full. The application logic itself
fails. OK, but let's say it works. Pac-Man moves. We're good, right? No, you're only halfway there. Now the server has to tell you that it worked. This whole thing has to happen in reverse. Oh man, OK. Step 5. Post reply The server's network card might die just as it's about to send the success message you're. Kidding me? That's just cruel. It happens. Step 6. Deliver reply. The network goes down on the return trip. A shark bites the undersea cable on the way back.
Your reply is lost at sea. This is brutal. It's a miracle anything works at all. It really is step 7. Validate reply. The client gets the success message back, but it's garbled. Or maybe the client app crashed and rebooted and doesn't even remember asking the question in the first place. So it gets an answer to a question it forgot it asked. Right. And finally step 8, update client state. The client gets the success message but then fails to actually process it and update
the screen. The logic is done, but the user never sees it. So that one innocent looking line of code board dot find Pac-Man is actually a gauntlet of eight discrete death. Traps. Precisely, And here is the crucial insight for the interview. This is the thing that trips up almost everyone who hasn't really lived this pain. Lay it on us. The unknown state. The unknown state. This is different from true or false. Fundamentally different in local code. A function returns true, false
or it throws an error 3 states. In distributed systems there is a fourth, and it is the most important one, unknown. Explain that in practical terms. Give me an example. Let's say you're building a banking app. You make a request to transfer $100 from your checking to your savings. You send the request, you wait, and nothing. The connection just times out. OK, so it failed. I'll try again. Did it. Well, I didn't get a confirmation, I didn't get a success message, so I assume it
failed. And that assumption is one of the most dangerous assumptions in software engineering. Look at our list of failure modes. Maybe the request got there, the bank moved the money, but the reply got lost. That was failure mode 6. So the money is gone from my checking account. Right. Or maybe the request never got there at all. That was failure mode too. In that case, the money is still there. You as a client are sitting there holding a time out exception. You do not know.
You cannot know if the money was transferred or not. That is incredibly uncomfortable. Humans hate it. We are wired for binary outcomes. We want to know, did it happen? And the system is just shrugging its shoulders and saying maybe. So if I'm in an interview and the interviewer asks what happens if this request to the payment service times out, the wrong answer is we assume it failed and retry. That is a fatal error in the interview.
If you assume it failed and you retry, you might transfer the money twice. If you assume it succeeded and you don't retry, you might never transfer the money. You have to design for the unknown. And this all comes back to that concept you mentioned earlier, independent failure, yes. Unlike the local Pac-Man machine where the CPU and the memory share fate, in a distributed system, the network can be perfectly fine while the server dies. The server can be perfectly fine
while the network dies. They fail independently of each other. So we've established that the environment is hostile, the network is unreliable, and we can't trust timeouts to tell us the actual truth of what happened. Correct, that's the foundation. Now let's talk about how we as engineers make this whole situation so much worse for ourselves. Let's talk about assumption drift.
We have this source, The Hidden Danger of assumption drift, and it introduces the idea of the almost correct system. The almost correct system. It's actually far more dangerous than a blatantly broken system. How so? That seems backwards. Well, if a system is broken, like it crashes on startup, you fix it. It fails loud. Failing loud is a feature. It's obvious there's a problem. Right, you get an alert, you find the bug, you push a fix. An almost correct system is
different. It passes all the unit tests. It works perfectly in staging, it works when you demo it to your boss. But it has these hidden latent assumptions that only drift apart and cause a catastrophe under a very specific kind of load. And the classic example of this which the source walks through is a time out chain disaster. This is a distributed systems Horror Story, and it's a true story that happens all the time.
Let's imagine a simple chain. You have a user, we'll call them the client, a load balancer in the middle and a back end service doing the work. OK, a standard 3 tier setup. Now let's look at their time out configurations. These might have been set by different teams at different times and they all look reasonable individually. The client, say a mobile app, wants a snappy experience, so it's developer sets a time out of two seconds. It makes sense. I don't want to wait forever for
a page to load. If it takes longer than two seconds, I'm probably going to refresh or just leave the app. Exactly. The load balancer, which is a piece of infrastructure, needs to be a bit more patient to manage traffic, so it has a timeout of five seconds. OK, still seems fine. And the back end service which does the heavy lifting. Maybe it's a complex database query has a time out of 30 seconds to make sure it always finishes its jobs, even the big ones.
So we have two seconds on the client, 5 on the load balancer and 30 on the back end. Individually these numbers seem perfectly fine. The back end needs time to crunch numbers. The client needs to be fast. But watch what happens when you put them together. The client sends a request, it goes through the load balancer, hits the back end, and the back end starts processing. But let's say the back end is having a slow day. Maybe there's a called cache, Maybe the database is under
load. It's going to take 10 seconds to do the work. OK. So it's going to be slow. Now at the two second mark, what does the client do? The client times out, it gives up. In its world, the operation has failed. Right, the client says this didn't work. And typically, what does a well behaved client application do when something fails? It retries. It retries. It sends a second request for the exact same thing. But here is the critical moment of drift.
Does the back end know the client has given up and left? No, of course not. The back end is still working on the first request. It has a 32nd timeout. It thinks it has 28 seconds left to finish the job. Exactly. So the back end is chugging along processing request one. Suddenly request 2 arise from the same user. Now the back end is processing both requests. But the client is only listening for the answer to the second one.
The first one is an orphan. It's an orphan, it's a ghost, it's doing work for nobody, and if the back end is slow because it's overloaded, we just doubled the load on it. And if that second request also takes more than two seconds. The client retries again. Request 3 arrives. Now the back end is doing triple the work. The system isn't just failing, it's actively tearing itself apart in a feedback loop. And the crazy part is that the code in the back end is technically correct.
It's dutifully processing every request it received. It is. It's just processing a request that nobody wants anymore. This is assumption drift in its purest form. The client assumes if I timed out, the request is dead and gone. The back end assumes if I'm processing a request, someone is patiently waiting for the answer. Both are totally wrong. Their realities have drifted apart. The client lives in a 2 second reality. The back end lives in a 32nd
reality. They're not in the same universe anymore. And in that void between universes, your system dies. So how do we fix this? What's the senior engineer move here at the whiteboard? The big shift is moving from these implicit contracts, I guess it will finish quickly to explicit ones. The first step is what you could call the alignment rule, which is generally you want your timeouts to increase as you go down the stack. The client should be the most
patient. If the client is willing to wait 30 seconds, then the load balancer should be willing to wait say 29 seconds and the back end 28. The system gives up from the back, not from the. Front so you shed load at the source, not at the client. Exactly. Yeah, but the gold standard, the thing that really shows you know what you're talking about, is deadline propagation. Deadline propagation? That sounds fancy. It's incredibly elegant.
Instead of every service having its own static timeout number like 30 seconds, the first service in the chain, the client or the front end said I have a total budget of 2 seconds for this entire operation. Then passes that deadline down the chain, usually in a request header. So if the load balancer takes 5 seconds to do its part, it looks at the budget and tells the back end you have 15 seconds left go.
Recisely and if the back end gets a request and sees that it only has .1 seconds left in the budget, but it knows the job takes at least a second, it doesn't even start, it fails fast. It immediately returns an error saying I can't possibly do this in time. That saves so much wasted work. The ghost work. It saves the system, it aligns all the assumptions. Everyone is looking at the exact same clock. It stops the ghost work, the work being done for clients who
have already left the building. OK, that covers the timing assumption, but you mentioned that the client retries and it seems like retries are a huge double edged sword. Oh, retries are selfish. That's the phrasing from the Amazon Builders Library, and I absolutely love it. Selfish. How is code selfish? Think about it from the server's perspective. When you retry, you're demanding more resources from a server that is likely already struggling.
You're essentially saying I know you're busy or slow or failing, but stop what you're doing and process my request again right now. It's like shouting at a waiter in a slammed restaurant who is already dropping plates. It doesn't help. It's the exact same dynamic, and this leads to these catastrophic events called retry storms. If you have a deep architecture service, A calls B which calls C which calls the database, and they all have a default retry 3 times policy. You can do the math.
Oh boy, I think I see where this is going. If the database at the bottom slows down, service C retries its call to the database three times. OK. But from service B's perspective, it's call to service C is just slow, so it retries its call to service C3 times, and each of those calls will trigger three database retries. That's nine database queries now. And service A sees that service B is slow, so it retries its call to service B three times. So you have 3 * 3 * 3.
That's 27 queries to the database for what was originally a single user action. If you have 5 layers of services all retrying 3 times, the load on the database increases by a factor of 243. You turn a small hiccup into a self-inflicted distributed denial of service attack. A complete meltdown. You've amplified a tiny problem into a catastrophic failure. So do we just not retry? Is that the answer? That seems wrong too. No, we have to retry. Networks are flaky.
We all know that from the 8 failure modes. We need to handle transient failures. A dropped packet shouldn't kill the whole operation, but we need to do it defensively. We need to do it with back off and jitter. OK, let's unpack these. Back off is pretty standard. I think most people have heard of that. Exponential back off. It's the most common type. Instead of retrying immediately, you wait one second. If that fails, you wait 2
seconds. If that fails, you wait 4 seconds, then 8, and so on. You give the server breathing room. You're being polite instead of selfish. But the AWS source makes a really big deal about jitter. Why isn't exponential back off enough on its own? Because of synchronization, this is a subtle but critical failure mode. Imagine a popular mobile app and the server it talks to reboots. So you have thousands of clients all at the exact same time getting a connection failure at 12.000000.
OK, they all fail simultaneously. They all have the same back off logic so they all back off for one second. What happens at 12.000001? They all hit the server again at the exact same time. Whim. They all fail again. They all back off for 2 seconds. What happens at 12.00 point 033? You get these perfectly synchronized spikes of traffic. It looks like a metronome. Tick slam, Tick slam.
And the server can never recover because it just gets hammered in these synchronized waves it can't get its feet under. It never jitter is the solution. Jitter adds randomness. Instead of waiting exactly one second, you wait one second, plus or minus a few 100 milliseconds. Or a better strategy is to pick a random number between zero and your current exponential back off cap. So instead of a metronome. You want rain.
You want the request to patter against the server randomly, spreading the load out over time instead of concentrating into these sharp spikes. I love that visual metronome versus rain. That's a great way to explain it in an interview. It is, and if you are designing a system in an interview and you mentioned retries without also mentioning jitter, you are setting yourself up for failure.
The interviewer will look at you and ask what about the thundering herd problem and you'll have to backtrack. It shows a gap in your knowledge. Noted. OK, there's another huge piece to this puzzle. If we are retrying requests, we run the risk of doing things twice, like that bank transfer we talked about at the beginning. This is where we have to talk about idem potency. It's a fun word to say idem potency.
And it basically just means doing something more than once has the same effect as doing it just once. Correct. The mathematical definition is FS, FX, inquest FX. In a distributed system. If you have retries, which we've established you absolutely must, then your operations must be idempotent. There's no way around it. But not all operations are naturally idempotent, right? Like adding an item to a shopping cart. Is. No, it's not, So we have to categorize them.
First, there's natural idempotency. This is the easy stuff. An operation like set user status to active. You can call that 10 times in a row, the status is still active. That's safe to retry. Then there's what you might call business idempotency. This relies on a natural key in your business domain. The classic example is create customer e-mail. The e-mail address is the unique key.
If I try to create a customer for bob@example.com twice, the second one should either fail with a user already exists error or just return the existing record. The system state doesn't change incorrectly. OK, that makes sense. You rely on the uniqueness of the data itself. But the hard one, and the one that always comes up in system design interviews, is operations that are inherently not idemitant. The classic one is charge this credit card $50.
Right. The credit card company is more than happy to let me charge a card $50 twice. That's a perfectly valid sequence of operations. Exactly. So you can't rely on the data itself for these cases. You need to introduce an artificial key, a unique ID or an idem potency key. How does that work? When the client decides to make the payment, it first generates a unique ID, usually UID. Then it sends the request saying charge $50 and use this
transaction ID 123ABC456. And the server has to remember that it's seen 123 ABC 456 before. Yes, the server stores that idem potency key, maybe in a dedicated table or right alongside the transaction record for some period of time. If it receives a request with that same ID again, it doesn't try to charge the card again. It just looks up the result of the first attempt and says oh I already did this, here is the receipt from the first time ah.
So that solves the unknown failure mode perfectly. If I get a time out on my payment request, I just retry with the exact same ID. Exactly. If the first one failed before anything happened, the second one goes through. If the first one actually succeeded but the reply was lost, the second one doesn't double charge, it just returns the safe success message. It handles the ambiguity perfectly.
So the So what for the interview is if you design a payment system or an ordering system or anything transactional and you don't explicitly mention idem potency keys alongside your retry strategy with back off and jitter, you're essentially designing a system that will either lose or steal money. You fail the interview, plain and simple. You cannot build reliable distributed systems without these fundamental patterns. You're building a casino, not a bank. OK, we're getting into the
really deep water now. We've covered the basics of defensive engineering. Now I want to talk about something even scarier. Metastability. Metastability. This is one of my favorite topics because it's so non intuitive. It comes from a fantastic paper called Metastable Failures in Distributed Systems and it describes a specific type of nightmare scenario. The nightmare that doesn't end when you wake up. Pretty much, yeah. We often think of systems as being in one of two states, up
or down. Metastability introduces a terrifying third state that's, well, kind of both in neither. And the definitions from the paper here are stable, vulnerable and metastable. Correct. Stable is the happy state. Normal operation, traffic is flowing, latency is low, everything is fine. OK. Business as usual. Vulnerable is the interesting one. The system looks fine from the outside. The dashboards are green. It's serving traffic, but it has
lost its safety margin. It's like an airplane that has lost one of its engines but is still flying. It's walking on the edge of a Cliff, but it hasn't fallen yet. Then comes the trigger. It's often something small, a momentary network blip, a minor traffic spike, a single server rebooting, and that little push sends the system over the edge into them in a stable state. And this is where it gets really
weird. In a normal failure, you remove the trigger, you fix the bug, or the traffic spike ends and the system recovers and goes back to normal. But in amid a stable failure, removing the trigger does not fix the system. The system stays down. It gets stuck. Why? What's holding it down? It's held down by the sustaining effect. There's some kind of feedback loop in the system's own recovery or error handling logic that keeps it pinned in the failed state.
The analogy I liked from the source was getting stuck in a hole. Yes, that's the perfect way to think about it. Tripping on a small rock is the trigger. It causes you to fall into a deep hole, but once you're at the bottom of the hole, the rock doesn't matter anymore. You can remove the rock, but you're still in the hole. The depth of the hole is the sustaining effect. Let's make this concrete. The look aside cache failure is the absolute classic example of
this. This has happened to basically every large tech company at some point. You have a standard setup, a web app, a cache like Redis and a database. Very standard. In the stable state, the cache is warm and has a 90% hit rate. The app is handling say 3000 requests per second or QPS with a 90% ten hit rate. 2700 of those requests are served instantly by the fast cache. Only 300 QPS actually hit the slow database. And let's say the database can handle 500 QPS. It's happy.
It's not even breaking a sweat. OK, the system is stable, but it's hitting capacity limit is 500 QPS at the database layer. It's vulnerable. Exactly it's vulnerable. Now the trigger, the cache server cluster has to be rebooted, maybe for a security patch or an upgrade. For a few moments the cache is completely empty, so. The cache hit rate drops from 90%. Then what happens to the 3000 QPS from the app? They all go straight to the database.
All 3000 QPS hit the database. The database was built to handle 500. It's now getting 6 times its capacity. It immediately overloads. CPU goes to 100%, connections are refused, queries start timing out. The database melts. Now here is the metastable trap. The cache servers are back online. Now they're up and running, ready to be filled. How do you refill a Lookaside cache? You you have to read the data from the database and then you write it into the cache.
But you can't read from the database. It's overloaded, it's on fire. Every request to it is timing out. Oh no. So the cache stays empty. The cache stays empty. Because the cache is empty, all 3000 QPS keep hitting the database. Because the traffic keeps hitting the database, the database stays overloaded and down. Because the database is down, you can't refill the cache. It's a perfect vicious. Loop. It's a death spiral. Yeah, even though the original trigger, the cache reboot is
long gone, the system is stuck. The sustaining effect is the application's own logic for refilling the cache, which now acts as a weapon that keeps the database down. And the system was vulnerable the whole time because it's secretly relied on that 90 cash hit rate to survive. It had hidden capacity. You thought your system could handle 3000 Q, but in reality your database, the hard limit, could only handle 500. The cache was just masking that
vulnerability. That is genuinely terrifying. How do you even get out of that state? It's very painful. You usually have to manually intervene and stop all traffic at the edge at the load balancer. Like turn off the website for everyone and let the database recover. Then very slowly run scripts to warm up the cache by hand, then cautiously let a small percentage of users back in.
It's a major, major outage. Another aspect of this sustaining effect that the paper mentions is work amplification. Yes, this is another one of those super counterintuitive things. We often assume that handling an error is cheap. You know you have a tried out catch block a try to catch return error. Right, it seems like the catch block does less work, it's just creating an error message. How hard can that be? But in many complex systems, the error path is actually more
expensive than the success path. How is that possible? Think about everything that might happen when you catch an important exception. You might capture a full stack trace. That's CPU intensive. You might write a huge detailed log entry to a file on disk. That's IO. You might do Adns look up to log the client's host name for debugging. That's a blocking network call. You might send an alert to a
monitoring system. Oh wow, so if the system is overloaded and it starts throwing a lot of errors. The very act of handling those errors consumes more CPU, more IO, and more network resources than the success path would have, which makes the overload worse, which causes more errors. It's another one of these feedback loops. Another loop, to be a senior engineer you have to look at your air handling code and ask is this cheap, is this lean?
Or am I accidentally building a mechanism that will DDoS myself with stack traces when things go wrong? This is incredibly heavy stuff. It really really shifts the perspective from does my code work to how does my system fail. How does it fail and can it recover on its own? That is the only question that matters. OK, let's talk about the advanced patterns. We've identified these horrible problems, retry storms, metastable loops. What are the heavy duty tools we
use to fight back? The first one in the toolkit, and it's a big one, is the circuit breaker. Wikipedia defines this pretty clearly. It's a direct analogy to the electrical fuse or circuit breaker in your house. It's the exact same principle. If you plug too many things into an outlet, the wiring gets too hot and the breaker trips to cut the power and save the house from burning down. In software.
If a downstream service is failing too often, we trip the circuit breaker to protect it. What does that look like in code? What are the mechanics? It's a simple state machine. There are three states. The normal state is closed. This means the circuit is complete electricity or in our case network traffic is flowing. OK, that's the happy path. Then if the breaker detects too many failures, say more than 50% of requests are failing in the last 10 seconds, it transitions to the open state.
This means the wire is cut when a new request comes in. The circuit breaker doesn't even try to call the downstream service. It fails it immediately with an error. And this directly prevents the retry storm we talked about. It gives the struggling downstream service a break some breathing room to recover. Exactly. It stops the bleeding, but you can't stay open forever. You need a way to know when it's safe to close the circuit again. That's the third state half open. The scout.
The scout. After a configured timeout, say 30 seconds, the breaker let's one single request through a probe. If that single request succeeds, the breaker assumes the downstream service has recovered and it moves back to the closed state. Everything is back to normal. And if that scout request fails. It goes right back to the open state and starts the time out timer again. It's a really elegant little state machine, but it directly solves that sustaining effect of
the database overload. It does, but here is the nuance, the senior interview detail that separates the candidates. It's the mini circuit breaker problem, or the problem of granularity. What's that? Imagine you have a service that calls a database that is sharded into say, 100 shards. Shards A through Y are perfectly healthy, but Shard Z is on a faulty machine and is melting down. OK, a partial failure.
If you have one big global circuit breaker for the entire database dependency and Shard Z starts failing, what happens? You trip the main breaker. Now nobody can talk to the healthy shards A through Y either. You've taken down the whole system because of one small partial failure. So you need to have a circuit breaker per Shard. Or partition. Or per instance you're talking to. The granularity of your circuit breaker matters immensely.
If you make it too broad, you create unnecessary outages. If you make it too narrow, say a breaker for every single customer ID, you might use too much memory tracking the state of all those Breakers. It's always a trade off. Always, yeah. Discussing that trade off is what gets you the job. Now let's go even deeper down the rabbit hole. We have a paper here from Google Research on D2, TCP and deadline awareness. This feels like the absolute cutting edge.
This is for when you're designing extremely high performance, low latency systems. Think Google search, high frequency trading or ad bidding platforms. And the specific problem they describe in the paper is the old DI problem, which stands for Online Data Intensive Applications. Right, and the classic example is Google search. You type a query. A query hits a root server. That root server then fans out your query to maybe the 1000
leaf servers. Each of those servers is responsible for searching its own little chunk of the Internet. And they all find their results and reply. And they all reply at roughly the same time. Which creates a traffic jam. The paper calls this in cast congestion, a fan in burst. Exactly. You have 1000 responses hitting the same network switch all at once. The switch's memory buffers fill up instantly, packets start
getting dropped. And standard TCP, the protocol that the entire Internet has run on for decades. How does it handle this packet loss? Poorly for this use case. TCP is designed for one thing above all else, fair share. It tries to be fair to everyone using the pipe. When it detects packet loss, its algorithm tells everyone to slow down their sending rate, usually by cutting it in half. But in this scenario, not everyone is equal. Some responses might be more
important than others. It's not about importance, it's about time. In an ODI app you have a hard deadline. We must return search results to the user in 200 milliseconds. If a packet containing a piece of a search result is going to arrive at 250 meters, it is useless. The root server has already given up and set back the results it has.
That late packet is garbage. So standard TCP might slow down a packet that only has 10 milliseconds left on its clock, causing it to miss the deadline while giving that bandwidth to a packet that has plenty of time to spare. Yes. TCP has no concept of time. It only knows about bytes and fairness. It will happily make everyone late in the name of being fair. So what does D2 TCP do differently?
How does it fix this? It adds deadline awareness directly into the congestion control algorithm at the kernel level. It uses a mathematical trick they call gamma correction. Gamma correction sounds like something from video editing. It's a math term, but the intuition is actually pretty simple. It modifies TCP's back off behavior based on how close a flow is to its deadline. How does it decide? How does it know the deadline? The application tells it.
The application says this socket connection is for an operation that must complete by this time stamp. DGTCP then knows the time budget for every packet. And what does it do with that knowledge? If a flow is very near its deadline, it's in danger of being late. DDTCPS Gamma correction tells it do not back off. Be aggressive, push through. You need to finish now. And if a flow has a far off deadline, plenty of time. It tells that flow to back off
very aggressively. You have time, Get out of the way and let the urgent traffic pass. It's prioritizing traffic based on time remaining. Exactly. It moves the entire network, stacks philosophy from fairness to urgency. That is a profound shift in thinking, and for a system design interview, mentioning that standard TCP itself might be the bottleneck because it ignores deadlines, that's a serious differentiator. It shows you understand the stack all the way down to the transport layer.
Most engineers stop thinking at the load balancer. This shows you think deeper. OK, we're rounding the final corner here. We've talked about how to design these complex systems with circuit Breakers and deadline propagation. The last part is about verification. How do we know it all actually works? This brings us to a fascinating paper called the Zebra Conf. The big headline here is that most complex failures aren't caused by code bugs anymore, they're caused by configuration errors.
Heterogeneous configurations. Right, In a massive distributed system, not every server is running the exact same configuration. You might be in the middle of rolling out a change. You might have different hardware types that need different tuning parameters. And the paper points out this insidious problem where parameter A masks a fault in parameter B. Yes, it's a combinatorial nightmare.
Maybe you have a retry limit of three on one group of servers and five on another, and that specific interaction accidentally hides a bug in your timeout logic. System works until the day you decide to make all the retry limits 5 and suddenly the whole system falls over because the bug in the timeout logic is now exposed. So for the interview, the point is not just to design the code, but to design the configuration management process itself.
But don't just say we put configs in a YAML file. That's not an answer. How do you test config changes? How do you verify that a change doesn't violate your system score assumptions? How do you prevent config drift from killing you silently over months? And they mentioned a technique called Bayesian risk refinement. That's a very fancy mouthful, but the concept behind it is we can't possibly test every combination of configurations.
The search space is infinite, so we use probability Bayesian logic to guide our testing. We build a model of our system and use it to find the riskiest configuration changes. We focus our limited testing resources on the areas where we have the least confidence or where an error would be most catastrophic. It's intelligent risk based testing rather than just brute force exhaustive testing. Exactly, and it's how modern large scale systems are actually
managed. All right, we have covered a massive amount of ground today from Pac-Man to gamma corrected TCP. Let's bring it all together. Let's create the Senior Engineers interview checklist. OK, if you were walking into that interview room or logging onto that Zoom call for a system design round, here is your mental CHEAT SHEET. Let's hear it, Item 1. Assume failure. Don't design a happy path.
First start with the failures. Ask yourself what happens if the network dies after the request is processed but before the reply is received. If you can answer that, you're on the right track. Always handle the unknown state. Item 2. Check assumptions. Explicitly state your assumptions about time load and data. Are your timeouts aligned? Remember, upstream timeouts should generally be longer than downstream. Or better yet, use deadline propagation. Don't let your system's reality
drift apart. Item 3. Control flow. Don't just let request fly blindly into the system. You are the engineer, you control the flow. Use exponential, back off. Use jitter to turn a metronome into rain. Use circuit Breakers to give services a chance to recover. Control the storm before it starts. Item 4. Consistency. If you retry, and you will, your operations must be idem potent.
State how you will achieve this. Use natural keys, business keys, or for the hardest problems, explicitly pass a unique idem potency key. Do not design a system that can charge a credit card twice. And finally item 5. Meta stability. This is the advanced one. Ask the big question. Does my recovery mechanism act as a denial of service attack on myself? Do my retries create a retry storm? Does my cash warming logic keep my database down? Look for those deadly feedback loops.
That is a ridiculously powerful list. It's the difference between saying I hope this works and I know how this will break and I've designed it to recover gracefully. I want to leave our listeners with one final thought. We started this whole deep dive with the idea that perfect code can fail. And I think we've seen exactly why that's true. There's a quote I love that I think sums this all up. Great software isn't built by eliminating bugs, it's built by
eliminating surprises. I love that. That's perfect. The bugs will always happen. The cosmic rays will flip the bits. The backhoe will cut the fiber optic cable. Those are just facts of life in this field. They are not surprises. But the surprise The system collapsing into a heat because of a simple retry storm or getting stuck in a metastoble loop for hours. That is what we as engineers can and must eliminate through good
design. The code inside the brackets, the if statements, the for loops. In many ways, that's the easy part. The void outside the brackets, the empty space where the messages travel and the assumptions live. That is where the senior engineer lives. And that is where you need to live to ace that interview. Good luck, you've got this. Thanks for listening to the deep dive. We'll see you next time.