Working at Amazon as a software engineer – with Dave Anderson

have not heard any other companies where the largest outages are taken so seriously by everyone, the whole leadership. Operational excellence and dives deep and all these kind of things.

tie in together and there's this culture which you know sometimes people complain about like new external hires sometimes amazon has had these pockets of a bunch of vps hired from a different company where they don't join the on call don't join this f1 and it's like oh my god these bad external people who don't understand how we do things but i would say

long-time amazonians like we would sometimes have for example in aws you'd have the engineers call where they're all discussing fixes and then sort of separately like there's also the vp svp call of like give us the updates what's the current status and some of this is a scale thing right like netflix is down in like half the country because there's some global networking outage and we're like talking to the backbone providers.

trying to get them to fix things faster. We're telling them exactly where the backbone broke because our alerts are much better than theirs. I was shocked sometimes. The connections that you have and the data you have sometimes are so mind-bogglingly big. What is it like to work at Amazon as a software engineer? Dave Anderson was an engineering manager and director of engineering at Amazon for more than 12 years. He's now retired and so can speak with more candor than usual.

In our conversation, we cover engineer and manager career levels at Amazon from L4 to L10, the interview process for engineers and Amazon's unique bar raiser interview. Amazon's infamous pip slash underperformer target. and how performance management actually plays out.

on-call and incident management inside Amazon, and many more topics. If you're interested in understanding how Amazon works differently than most big tech companies, and why startups often have success in hiring engineers and engineering leaders from amazon this episode is for you if you enjoy this show please subscribe to the podcast on any podcast platform and on youtube dave welcome to the podcast

Thanks. Glad to be here. So you've worked at Amazon as an engineering manager. How is Amazon called? SDM? SDM. Yeah. Yeah. I started off as an SDM and then went up that engineering management chain. You're a software development manager. Before we get into what it's like to work as a dev manager there. You manage and work with a lot of software engineers. What is it like to be a software developer? And what levels does Amazon have for devs and also for managers? And how do they overlap?

Yeah, the numbers overlap except for, so when you get hired out of college, like the entry level that lasts like a year and a half to... sometimes more years, is level four. That's for whatever reason, we start at level four for engineers. Then there's level five, which is like the beginnings of middle level engineer, which equates to... The first level of SDM, Amazon technically has SDM at level five and six.

Level five is more rare and it's more for, you know, you almost want to call it junior, except no one would want to call it junior SDM because it'd be a little insulting. It's actually where I started. So there's level five. Then level six is what you might call the senior software engineer or software development manager, which is like the software development manager. Level six is the most common management position where.

You manage a two-pizza team, you own your space, and you usually don't have managers underneath you. And that's senior software engineer, which is where I would say a lot of software engineering careers end, as in that's a senior position. You're the... You could almost equate it to like a level six manager has a level six engineer in a perfectly designed team, which is a level six engineer, which sort of be the head engineer for a team.

And then you hit level seven at Amazon, which is principal software engineer. And they would theoretically be over a group of teams, the lead, you know, sort of directing, you know, and as you could imagine, as you get to like larger groups, you get to. more broad ownership. A principal engineer wouldn't be necessarily focused on one system because you're theoretically leading across multiple teams. So you get more and more architecture, bigger scale designs, things like that.

And that equates to level seven manager, which is senior software development manager. which would be a manager of managers, the first level where you're really managing multiple teams. Then you get to level eight, which is a senior principal engineer, which is Just more senior, bigger orgs, bigger scope, etc., which equates to director for development management.

And then you get up to level 10 because there's no nine for whatever reason. I don't know if I ever got a straight answer on why there's no nine. It was like this idea of a senior director position that never got created. But anyway, you jump up to 10 and you get to distinguish engineer or VP, which is sort of the pinnacle of where any of us normal humans would get to. This episode was brought to you by WorkOS.

If you're building a SaaS app, at some point your customers will start asking for enterprise features like SAML authentication, SKIM provisioning, and fine-grained authorization. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand, and you can ship quickly and get back to building other features. WorkOS also provides a free user management solution called AuthKit for up to 1 million monthly active users.

It's a drop-in replacement for Alt-Zero and comes standard with useful features like domain verification, role-based access control, bot protection, and MFA. It's powered by Radix components, which means zero compromises in design. You get limited customizations as well as modular templates designed for quick integration.

Today, hundreds of fast-growing startups are powered by WorkOS, including ones you probably know, like Cursor, Vercel, and Perplexity. Check it out at workos.com to learn more. That is workos.com. This episode is brought to you by Modal, the cloud platform that makes AI development simple. Need GPUs without the headache?

with modal just add one line of code to any python function and boom it's running in the cloud on your choice of cpu or gpu and the best part you only pay for what you use With sub-second container start and instant scaling to thousands of GPUs, it's no wonder companies like Suno, RAMP and Substack already trust modal for their AI application. Getting an H100 is just a pip install away. Go to modal.com slash pragmatic to get $30 in free credits every month. That is modal.com slash pragmatic.

Why do you think it starts with L4? This is so strange because at Google, Uber... I think even potentially Stripe. It all starts at L3. It's also for a historic reason, but L4 at most companies is like the mid-level engineer. And here at Amazon, it's the entry-level one. And I don't even know why. It's not like we don't have different pay scales. So I'm pretty sure if you get to the FCs that I don't remember the levels for the FC workers that there's like one, two and three.

their leveling starts down for uh fulfillment center sorry so when you get to the fulfillment centers of which i've had very little to do in my career so like you know once in a while i'd touch on something so i'm pretty sure that their levels start lower um And if you get to a few weird positions like...

QAT was a position which I don't know exists anymore. It existed and then didn't and then existed again and didn't. So I don't know what the current status is. It was a quality assurance tester, which is like a non, not like QAE, which is quality assurance engineer.

they would be level three. So there's like, there's weird quirks where you'd touch level three once in a while. But for the most part, yeah, Amazon just starts off at level four. No idea why they decided their numbering schemes needs to start there. And how easy or difficult are promotions? I think you kind of alluded a little bit to that when you said that a lot of people get stuck at L6, which is the senior level. Yeah, I would say the like four to five.

for engineers isn't too bad at all. Or for just about any position. I mean, you know, there's level fours for some other roles. But I would say that's not too hard. If you're competent, you complete your projects on time. You hit like two years in or so on most teams and they're just going to end up promoting you like, hey, look, because the idea of like a level four is.

Sort of like you need assistance. Imagine hiring someone straight out of college. There's very few college hires who are competent enough to be handed a project and actually work on it. Just think about how you were in college. You finished. Are you really ready to work on large-scale work now? Someone's sitting over your shoulder helping you, explaining what you're doing wrong. Don't forget to test before you push.

So level five is sort of like you can be trusted to work on a project, not a huge project, but a decent sized project. at the start of level five, at least on your own. So hopefully after a year and a half or two years of work, you've repeatedly demonstrated that you can work on your own. So you're promoted to level five. Past that, you get to a lot harder. So there's this very detailed document of the criteria.

for what's the difference between level four to five five to six six to seven for both for every position at amazon actually like there's a big archive in sharepoint last i saw um which was uh just the detailed of like Level five you and it describes it in fairly good detail. I mean, hand wavy, as you could imagine, in terms of. a project with a scope of a team, a project with the scope of multiple teams. Yeah. So those are actually what's mostly used when managers go to write promotion.

So Amazon's, in comparison to a lot of companies, Amazon has a very document culture. Everything, including projects, the famous six pagers, right? Yeah. Yeah. And so you write a famous six pager for promotions as well. And so even for promotions for even for promotions. And it's actually. I would say one of the aspects that's the hardest for managers and one of the big differentiators between having a good manager and not is one of the reasons you want a manager who can write.

is that your promotion document is the thing that gets you promoted or not. I'll just contrast that quickly with I worked at Facebook for a bit. I don't know. And I remember being in OLR there and a manager said like, oh, my engineer is so good. They really need to be promoted to level six. And a couple of people said, yeah, yeah, are you sure? Like, yeah, I remember I heard they were good. Yeah, they're good. Okay.

Sounds good. Moving on. And then flabbergasted. I was just like, my mind was blown. I'm like, are you kidding me? Because at Amazon, what you have is this document, which has like a lot of required fields where you have to explain.

write this big narrative you first of all there's some like normal feels like how long is their time and position uh which again is like a criteria to look at and say like oh only 13 months that's not very long are you really sure you can perform them that fast like this it's a very uh um It's a hard process to get through, let's say. But then you have to write this narrative, multi-page narrative, explaining how they meet the criteria of the next level.

So, you know, everyone has the doc of here's what a level six engineer does. You have to be able to read through that, understand what the criteria is. And then you have a narrative where you explain how they've done that thing at the next level repeatedly. And then you need quite a few people at the next level or higher saying that they believe you have met the criteria for the next level. And they usually have to give some anecdotes of why they're pretty convinced that you should be promoted.

Level four to five is pretty darn easy. Again, like you say, they completed this project, this project, and this project. And then a few level five engineers will write down saying, yeah, yeah, totally cool. Thumbs up. By the time you get to like level seven to level eight, like it is. serious amounts of documentation, years worth of project completion. You have to have a lot of VPs and directors on your doc writing out narrative blocks of text, explaining all the things they've done with you.

And so I would say that each level you go up, it gets... I would probably say exponentially harder to get promoted, which is why four to five is not a big deal. Five to six is a pretty big promotion. Six to seven is... Like it's already hitting like levels of. Slightly less bad for management, which I can explain in a second why, but principals have additional hurdles they have to get past, sort of unfairly in my mind.

technical assessments of exactly what they've done by multiple principals who seem to have incentive to say no, it's really tough. And so I would say... In ways that makes it have the wrong incentives for managers, one of the big things for managers is how big is their team? Because of your level.

if you're a level six manager and over time your team scope has grown and you eventually got some more managers reporting to you and then they get more engineers and then you start getting more managers more engineers and someone looks at your team and says this dude has 55 people reporting to him and he's level six. Like that makes no sense. We really need to promote them. So managers have this sort of artificial pressure.

And incentive to get their teams larger, which is until recently was one of the big things almost everyone did was like. A huge part of your job as a manager to grow your career was just grow your scope. Figure out more projects, more teams, more everything else. You can almost guarantee a promotion if you just grow your team enough. Incentives are an interesting thing. Taking a step back.

We talked about what it's like to be inside Amazon and get hired in there. How do you get hired as an engineer? And what is the interview process? Is it the pretty standard? You know, the ones we know about Google and Meta does, you know, algorithmic coding interview, et cetera. Or are there some unique things for Amazon? For the most part. I would say it was very very similar.

In fact, I would say at least when I went to Facebook, I was there for about a week and I started getting added to interview loops because they heard I was a bar raiser at Amazon, which I'll explain bar raiser in a second for anyone who doesn't know.

But I got added to the loops and I was like, oh my God, this is the exact same thing. Everyone's doing the same thing. It's extremely similar. At least the groups I was in, I was just added to the loops. It was exactly the same. Nothing was different. For the most part, with some exceptions, interview loops are five people.

You do a phone screen. They just sort of check whatever they feel like checking to make sure that you're like a vaguely competent candidate. And then you have an interview loop of usually five people. of which four is usually on the team or closely related to the team. And then one person is a bar raiser who is a essentially a third party person who can veto.

a higher decision and they also are running the loop as in, um, You're putting someone who's very experienced interviewer on the loop to make sure that everything goes well, everyone interviews well, that the loop is set up correctly, and they have the ability to say no. because like their special power and so but they cannot say yes right so they're not the hiring who says yes is it the hiring manager so a yes requires a yes from the hiring manager in the bar rate

So it's like a combo decision. So either one of them can say no and it doesn't happen. Now, in reality... Once you're there long enough, there's lots of ways to get around that as both. Not as a hiring manager. You can't really get around the bar raiser. But as a bar raiser, a few times I've had hiring managers saying no. I'm like, that's cool. I'll just pass them off to my friend here.

It was a hiring manager. Like, dude, quick, quick, before anyone else gets him. Because sometimes hiring managers have weird opinions. But yeah, for the most part, it has to be yes from both the hiring manager and the bar raiser. I think we can all imagine the interview going through. As a candidate, correct me if I'm wrong, but you apply to Amazon, you have the first... a screening, you're through, you're put on to, maybe you have a technical pre-screening.

But then you have the on-site or it might be virtual, but you'll probably have coding, architecture, hiring manager, maybe multiples of some of these. You have all these interviews. So we talked with all these people. You probably think you did great on some, you did on terrible on some. What happens behind the scenes? How does a debrief happen? So the loop is set up and there's...

At no point. Amazon is very bad at consistency across all of Amazon. So just about everything I say has a grain of salt. The only thing is not just like, yes, bar racers are always included. But other than that, I don't know how many times I've said. this is the way you do things. And someone else is like, not at all. You're totally wrong, which is funny because we're both wrong. I worked in a good number of groups, so I have fair confidence that I've been in.

Like five, six, six groups at Amazon. So like very different orgs. So I have a pretty good idea of what's consistent. You set up a loop for engineers, for example, you would have usually like two coding interviews. A design interview, if it's a senior engineer, you'd add an architecture interview and then at least one leadership principle style interview, which is like checking your leadership skills.

um level four interviews are sometimes a little different because they have no leadership because they're college hires and like It's a little bit of a waste of time to spend much time asking them about times they've had conflicts with coworkers. It's like, well, they've had no coworkers. They lived in a dorm. I've heard some really weird stories when I try to ask those kind of questions.

Yeah. There's one time my roommate was staying up really late at night and I had a test in the morning. I'm like, okay, maybe we just move on. So in normal interviews with experienced people, you have all these things. Everyone takes detailed notes on both the questions they ask.

and the answers they give and your interpretation. Like there's three parts of a good interview notes and everyone takes notes. You interview independently. You shouldn't be talking in between the interviews. Like each interviewer does their thing. And you take these notes. Then you get into what we have as a debrief, a discussion about the candidate. So the riser leads that meeting. You start off the meeting. For the most part, you started off by saying, okay, everyone read.

So you all start reading and you read. This is pretty typical across Amazon, right? A lot of meetings. Yeah, Amazon's document culture here. It's like you start off, you say everyone read and you're going to spend like if you have a... depending on how long the debrief is half an hour hour like you sit there and you read you read the questions you read the candidates answers you read the interpretation of those answers um

And then how the rest of it proceeds is very much up to the bar raiser and like their org and like how they tend to do things. I usually would say like some version of now that you've read all the feedback. Are you changing? Oh, sorry. Part of the interview notes is what is your vote for higher or not higher?

And so I would sort of carefully go around and say like, what are your current thoughts on higher or no higher? I would usually not say, are you going to change your mind? Because people are reluctant to change their mind. I like to say just like, now what are your current thoughts? You've read all the feedback. And you sort of get an idea of like, where is the room landing in?

I would probably say like 90% of the cases, it's not all yes or all no. I would say almost always there's at least one no and one yes, even on the worst or best candidates. And so... You preferably then go through and try to pressure test both directions of like, what happens if we don't hire this person? Are we missing something really good? Or what happens if we hire this person? What's the big risk that we're overlooking?

And, um, the bar raisers main job is to make sure you have a really good discussion that you don't overly obsess with one little bit of feedback. You know, sometimes people are like, Oh my God, I hated that answer. Yeah. We need to say no. Okay. That was. One sentence given by the candidate for one of the interviewers. Let's try not to obsess over that too much.

In fact, like as what you said was, sometimes you're almost always you'll have a good interview and a bad interview like that's extremely common. And so. Hopefully what the bar is doing is making sure you have a good discussion over like. everything's seen. And one of the reasons, in fact, what we do two coding interviews almost always is...

because it gives you more chances to have good answers. I don't know how many times I've heard, oh yeah, they couldn't go their way out of a paper bag. And the other person says, they did just fine for me. And now you have a good discussion.

why do we get two very different answers? What exactly did they put as their answer for the other person? Let's compare them. Let's take a look. What kind of hints did you give? What kind of hints did you not give? Then you just try to arrive at the idea of like, should we have them or not? So is it fair to say that the bar raiser tries to... you know, make the debrief.

a bit more fair you know even out things compensate for less experienced interviewers and maybe some of their biases given the borrower is It's both an experienced interviewer, but is there like a special like training or like how do you become a bar raiser? Yeah, for the most part. So you. Again, org-wide, there's a lot of differences across Amazon. For the most part, it's like once you've had 100...

plus interviews. Sometimes 50, you can start training, but like usually 100-ish interviews is when you get to train as a bar raiser. And for the most part, the training is... You watch a bar raiser do their thing for a while, and then you start to do the bar raiser thing while they watch and then criticize you behind your, you know, later on. Criticize. And that can last anywhere from like...

10 interviews to, I've, I've literally, I've seen 50 or plus more of like, they keep watching them and saying, they're still not ready to do this on their own. Um, depending on how picky the bar is or is and, or how bad the person in training is. But for the most part, I would say in a lot of my interviews...

I had more time interviewing than everyone else combined. And so a lot of the bar raisers do have 400, 500, 600 interviews under their belt. And a lot of the interviewers... have done 10, 15, 20. So so you are not just like a bit more trusted, you are wildly more experienced than other people on the loop. And so very frequently, you're both a sort of a louder voice in the room in terms of people can just trust your opinion on these things, but also.

I will look at their question. I don't know how many times I've seen, like the question the candidates answer, and then their interpretation, I think, is wrong.

I'm like, that's actually a pretty good answer. I can see where they're going with this. And you're like, absolutely wrong. And I'm thinking that's actually not wrong. It's not it's interesting. You know, it wasn't the approach you liked, but I actually see where they're going with that. And I think it was fine. So. I would say that that's sort of where the bar raiser thing comes in is like trying to, like you said, sort of fair, like a fair evaluation of a candidate.

Because we're not trying to hire people who have the exact answer that they want to hear. What you want to hire is... people who can come up with good answers as they work here for many years and so um you know one of the things you're looking for for example is more of like growth uh you know the possibility of growth like is this person willing to hear things you know sometimes it'll be

In the first interview, they said this. By the last interview, they said this other thing. I'm like, that's interesting. They were already pivoting their answer, sort of recognizing what we want to hear. That's totally fine with me. Like if they recognize ownership is sort of a key leadership principle and they sort of switch their answers to demonstrate more ownership. Great. Like that's sort of what you want is people who can change their mind over time.

Once you get hired and you start inside of Amazon, Back in your day, what was the tooling stack? I know it's changing these days because I'm hearing more and more of it. It's pretty much AWS, which is super unique across all of the companies, by the way. Google doesn't use GCP internally. microsoft is starting to use azure a little bit but not nearly as much and you know meta has their own thing Yeah, I would say it was changing over time.

Actually, one of the interesting things, like I was mentioning before, people would say, you're totally wrong. No one used that. There were homegrown web hosting stuff that we used in... marketplace back in the time, which was like originally was Grupa and then was multiple versions of new web hosting platforms that teams were building on.

And you get over to AWS and they're like, well, obviously everyone's using all AWS systems now to build things. And I thought it was sort of funny because it's like, obviously we're all using this and a lot of teams weren't yet.

and then you move over to uh idos and devices and like the mobile stacks and that was pretty much using straight up like the the development tools that need to be used you know by apple and by um google in terms of like the android development system and stuff and so it was

highly dependent on what stack you're building on is it like a homegrown web hosting platform or are we building you know aws tools or are we building devices and um And that's one of the reasons, like when we talk about a job I didn't mention of the bar racers is also... For example, I've seen people complain that a candidate only coded in Python for their answers. And definitely a bar raiser answer for that is we don't care what language they code in.

Because every time you change jobs, you're frequently going to use different tools. There's just such a wide selection of tools. And again, there's some homegrown, some AWS, and you want people to pivot to different tools. You want them to be able to say, sure, I'll flip to Kotlin or something for coding because why not? This team does it.

Or, you know, this software stack we've decided to switch. And you want them, for the most part, we want engineers to be fungible to be able to switch over and, you know, you move over to... We had some like not C++, but C code inside of one of the AWS teams I was on, for example, which as a side note was sort of funny because

I think I knew more C than a lot of the engineers because I'm old enough to like, that's what I was learning in college. Oh, yeah. And so I remember them saying something about like stupid memory management. And I'm like, wait, C? Like, cool. Let me. I haven't looked at the code in a while because that's not an SDM thing at Amazon. But I'm going to look at this like I'm sort of curious because you're suddenly flipping back into my wheelhouse now that you're looking at really old code.

One thing I've kind of heard a few horror stories about Amazon is on-call. and how brutal it can be. What is on-call like? How is it organized? How important is it? And I think we all know, or most of us will know about Amazon's customer obsession. How does that translate onto on-call? Yeah, I think it's a super interesting, and also I would say one of the more interesting differences between Facebook and Amazon.

So, at least in the teams I was on at Meta, there's no emergency support. There's a completely separate team that was dealing with emergencies. I think that's true of Google as well. At Amazon, it was...

It's a core aspect of most... teams way that they operate is if you wrote the code you're also supporting it in production and not just like when we talk about supporting we're talking about literally you're the one deciding which fleets host your code in which data centers and how distributed you need to be and

How are you sharding your database and all this kind of stuff? It's very, you know, the ownership stack is the ownership leadership principle comes into play on a lot of this stuff. But that also means that your team. in most cases, sets up a rotation where there's someone responding to emergencies in real time when emergencies happen. The philosophy behind it is if there's problems with your software, you want the problems to land on the person, people who know it best.

And the pain of bad software lands on the people who have the responsibility to fix it. When done well, and there's a couple of caveats there, when done well, I really appreciate it. As a manager of a team, I've been paged in way too many times at 1 or 2 a.m. because there was a problem, but also... You go into work two days later after a couple big emergency nights and we say, okay, we're stopping all feature development and we're fixing these problems. That's the done well part.

We are not patching this. We are not setting it to reboot every night to fix it. We're not letting people get paged at 2 a.m. because that's ridiculous. So we're going to fix it right now. We're going to rewrite this code, whatever. I would say like contrast.

In the negative way, there were some systems at Facebook. I remember people talking about like, oh, they've been having problems for months. Like they're just restarting this thing constantly. Some point in time, we'll get someone to fix it. And I was like, that's interesting because. at least the teams I ran, we would have fixed that thing right away because it would like, you would lose your mind if your system was always having errors like this.

On the other side, like the poorly run teams where you hear like it's been nine months now and everyone is paged two or three times a night. Anytime they're on rotation, they never sleep because they're getting paged every single night. That's just a nightmare. That's just not okay. Things are broken there. That's a management problem, I think, more than anything.

So, yeah. So the rotation is usually on most teams. Let's say you have a team of like seven people. Seven's a good, maybe eight. Eight people. You usually have someone go on rotation for a week. And for that week, you're responsible for responding to emergencies. On most of my teams, that would be, let's just say like two times in that week, you would be paged sometime off hours.

A lot of times it's 9 p.m. Frequently, it's when code gets pushed. So if you are pushing your code at 10 p.m., which is a common enough time of like, hey, we have a downtime in traffic. Let's push the code and make sure it works. The pager goes off. It's not a big deal. I think when you have a team having issues and you're getting paid two or three times.

Every night, that's just a nightmare. But again, most of the time that was fixed pretty quickly. Like the engineers aren't okay with it. Hopefully the manager is not okay with it. And then you get a result. Yeah. Well, I think there's always a, there's always a downside of being too close to the problems, right? Cause you do have to worry. But if it's set up in a way that you're actually paged for things that truly matter and matter to customers, you're now going to fix it faster.

you know we've all used software where like there were some annoying bugs and you can tell when it's there like months later that that is not paging anyone inside of it they might not even know about it or the people and i think it's like Can you tie responsibility for your schedule in terms of what are you working on next? Tie that to the responsibility of responding to problems and tie that to the responsibility of having a good customer obsession.

And that's where it's done well. I sometimes have to remind people over and over again. I'd hear from an engineer like, Oh, yeah, that was such a hard on call last week. And I wasn't paying attention as a manager or something. Or maybe I'm the senior manager.

And I'd be like, what do you mean like last week? And I have to go like quickly run a report because usually manager is not paged into everything unless it escalates. Like if they can't fix it within 10 minutes, then I get paged or something. And they say, oh, yeah, I got paged eight times last week. I'm like, well, what did you have to fix? They say, oh, we haven't had a chance to fix it yet. And you're like, oh, my God, like, dude.

Why are we working on anything else if you're getting paid seven times in a week? Like, that's not okay. So I usually would say, like, you should only be okay with being on an on-call rotation if you also have the capability of saying that this is the next most important thing we're all going to work on. I think it's not fair when you put responsibility on people if you don't give them the authority to... to also redirect your resources. Those things have to come together. And so...

I would say most teams at Amazon had that. And certainly there were teams where they felt like they were helpless and just had a terrible time of it. But well-run teams, you would say. They're not going to be okay with, again, a bad operational situation.

And then when outages happen, what is the categorization? I think there's different levels between the severity of the outage, right? So like a page comes in, someone decides, the on-call decides that if it's an outage and then they have to sign.

a level to it. How does that work? And it was different across teams a little bit of like what the exact severity meant again, because Amazon hates consistency in all ways. And for the most part, I love the lack of consistency because you can just invent things and just tell. One of my favorite things, as a side note, was... Someone would say that they're doing something and I don't like it, right? And I would say, I'll quickly go edit our wiki.

And write in a rule against whatever they just did. And then say, as you can see here in this document, that's not allowed. And they would look at the wiki and say, well, that's true. Okay. I was never called on to the fact that I just edited the rule set or the instructions or the step-by-step, whatever, that I just edited and then showed them the instructions. It was great how people would just believe these things.

That was like a bar raiser thing, too. It's like, that's not allowed. And I am the bar raiser, so I would know these things. Like, oh, OK, cool. Smart hack. Oh, man. Yeah. With confidence, you can get past almost everything if you need to. But yeah, so most of the time in most teams, there is severity one through five. And five means we're never going to fix anything because it doesn't matter at all. Most things are like level three, which is... It's a problem. You should work on it someday.

Severity two and one are the ones that would like page people, make people pay attention, make their phones freak out. Severity two and one are just emergencies. And for the most part, severity two is what we use for most things. Severity one is like company wide emergency is sort of. you know, Sev One, it's a, it's, was usually like the phrasing for it. In certain groups, like, let's say AWS in,

SEV2 is there's a system problem and we need to fix it. And you'll probably let your manager know. SEV1 is usually like, this is where I, so for a while I was on the encore rotation of manager, you get paid instantly for SEV1s. It's like. You would have a big list. You could set up a distribution list where a whole bunch of people would get alerted for someone, including maybe your VP, maybe your SVP.

um for for aws style set ones that was frequently like i was writing to jassy right away with a list of other svps and vps on the list saying we have an aws outage like blah blah so i actually have a really interesting story and one of the reasons i i asked that this We had a senior director join from Amazon. He was a senior director there and he became the senior director of my group. Amazon doesn't have senior directors. That's that level nine thing that doesn't exist.

Okay, well, maybe people like to invent their own titles. Or maybe it was a director who became a senior director, but he was a senior director, which meant that he had about like 300. engineers in this group and we we had an outage so my team was payments and the way uber did it so amazon sev one was l5 like first l1 was the smallest and l3 is internal l4 is like you know It's just the opposite. L5 is when it's actually company-wide, so it impacts the whole product.

We're actually losing revenue or something like that. And then there's like levels, there's like low, hot, medium or high. And my team pretty frequently had an L5. low outage, which meant that payments were down in one part of the world. Typically, we had a third party issue. Let's say one of the payment methods in India just stopped working because the thing had an outage there. And it was pretty routine for us. This is something that we couldn't do anything about beyond alert the third party.

And so we were just having one of these things and we always have a call. It's kind of routine. Like everyone's just sitting there bored. So, okay, have you guys paged the third party? Yeah, we paged. Suddenly, this new director. is in the call saying, all right, what's the situation? Can I help anyone here? And I'm like, It's like, Jeff, what are you doing here? He's like, this is L5, right? It's the highest one, right? I was like, yeah, it is. He's like, okay, I'm here. Use me.

How can you put out the fire? So then I learned that at Amazon, VPs jump onto step one and we were just having really casual, like for us, like enough.

5 wasn't really like this we just knew this was just our regular alphi because we had to follow we couldn't do anything about it but it just made me realize how amazon's taking it way more seriously and this this senior like jumped out of some important meeting to go there and for me it was just like whoa like this is this is new for me yeah so it just shows

serious amazon scenes and i have not seen any other companies or have not heard any other companies where The largest outages are taken so seriously by everyone, the whole leadership. So I think that's, you know, it's a bit of a story there. Operational excellence and dives deep and all these kind of things tie in together. So and there's this culture.

which you know sometimes people complain about like new external hires sometimes amazon has had these pockets of like uh you know a bunch of vps hired from a different company where they they don't join the on-call you know don't join this f1 and it's like this like oh my god like these these bad external people who don't understand how we do things. But I would say, yeah, the longtime Amazonians, like we would sometimes have, for example, in AWS, you'd have the...

engineers call where they're all discussing fixes. And then sort of separately, like there's also the VP SVP call of like, it was the updates. What's the current status? Because. And some of this is a scale thing, right? Like Netflix is down, like we're sitting there and like Netflix is down in like half the country because there's some global networking outage and we're like trying to...

Also, Amazon scale. We're talking to the backbone providers, trying to get them to fix things faster. We're telling them exactly where the backbone broke because our alerts are much better than theirs. I was shocked sometimes. The connections that you have and the data you have sometimes are so mind-bogglingly big. It's like you see the news articles coming out and it's like...

Like we know about the backbone outage in whatever place because we can see exactly where it is in the network based on all of our alerts and alarms and all that kind of stuff. But yeah, it was very expected that. Depending on the scale of your outage or issue going on that your VP would be joining in the call. If not, not usually to run it unless it's really important, but definitely like. What you want and what you usually have is we need X done and you need it done right now.

That's the kind of thing a VP or director, whoever the really senior person is around for is like, oh, okay, we need a... the kind of thing that would come up sometimes is like, Oh my God, we need a hundred extra servers right now. You know, these big beefy things, we need them provisioned immediately in Dublin.

And they say, okay, I'm calling the head of EC2 right now on the phone and we're going to get some engineer to allocate those to us in the next about two minutes. And so boom, boom, boom, they get this thing done and a bunch of servers are turning on because... The person can do the direct call and just say, I need this right now. And they say, got it.

I think that that's useful. I mean, it sounds to me, this might sound a bit cheeky, but if you get it to Amazon, and usually being on an outage, you don't really want to be on an outage as an engineer. If you're in one of these outages and it already happened, it's kind of an awesome, awesome opportunity. Where else would you be able to even experience this?

I've heard from, I believe, like if you're talking about like building skills in terms of like running, running software, not like just writing software, but like running software. If you want to go to a startup and run software. If you get an Amazon engineer who's been in the thick of things, sometimes it's like, you're not just good at writing software, but... Twilio calls. See, I got on a phone with the CTO of Twilio, a company. It's like...

Yeah, we're having this issue. It's in this specific region. And we're like, we have engineers in the room. You know, we're talking to them on the phone in a separate phone call from the engineers talking about whatever tests they're running on whatever trace routes and figuring things out.

Like the skills you build in terms of product and what can we ask the customer to do? Can we ask them to test something? Can we get access to their hosts so we can try something out? It's such a cool experience. I think it sucks that it's happening at 3 a.m. sometimes.

I would say most of the time it's not because it usually problems happen when you push code. It's just the way things work. Funny thing of like the amount of pages that happen when Amazon historically did lockdown sometime around Thanksgiving time, you would just stop pushing code changes out. in this exponential drop-off in pages because...

changes or what breaks things in the world. But yeah, I think the experience, I think you talk two years later about like, oh, we had such a good time on the team together, didn't we? You always talk about the outages and stuff because that was the exciting times at work. One interesting thing about Amazon that's also just famous for and people get surprised for is the frugality. Can you tell me what engineers experienced or what did engineers complain to you who joined?

from like companies that were a bit more generous. You know, like I worked at Uber. I visited friends at Google Meta. You have like these like, you know, just kind of standard perks like lunch. I think at Uber, we had like these vending machines. You could get like whatever, like mice. Free stuff from the vending machines. Yeah, yeah. Our vending machines had profit. They got a profit from our vending machines. It wasn't even like a cost. Oh, my God. Yeah, no, no. I think Amazon...

And there is some interesting points that... Sometimes we had these arguments of like, oh my God, like the amount of money that you pay engineers, why can't you be a little bit more generous? And some of it comes down to frupidity, like the stupid frugal thing. It's just like just this ridiculous, stupid level of frugality. But sometimes they do. There is interesting explanations because Amazon has fulfillment centers and we have the majority of Amazon's workers are.

very, very low margin per person. You can't afford it. And then they talk about For I think legal and other reasons that they can't offer different benefits and different things, perks to corporate workers versus fulfillment workers, whether or not that's legal, whether or not that's just like. Now we're at risk for unions forming across the world because they're giving unfair benefits to corporate workers.

You look at something like a 401k match, for example, was something that came up. It was like, couldn't we just do a double the match, do a hundred percent of whatever? And they're like, yeah, we could for engineers. But we definitely can't afford that for FC workers because our margin there is like 0.5%. If we give them any more money, we're losing money on every item Amazon sells. We have to be very, very careful. It's super expensive to have a million workers.

I mean, for good reason, like it's just expensive. So when you try to give, you know, try to give comparable benefits to corporate workers and you have these issues, but then you get to just. how Amazon operates is unless there's a really damn good reason to spend the money, we don't do it. And that's the same thing for resources. I think if you had to choose.

I guess as a business owner, a person who wants a company to be successful, you would definitely want your company to be frugal. Like the amount of money wasted when I was at Facebook, it just blew my mind. Some things are great. It's very relaxing to get free food. It's no big deal in comparison to the amount you're paid. You calculate it out and it's like, okay, what?

You know, a few thousand dollars a year spent per person on food compared to their salaries. Nothing. So why not do it? But then you get to. Just like the stupid levels of waste of like building products that no one needs or something like that. And at Amazon, I kept being like, oh my God, we would never. I remember saying once like this project looks like a duplicate of this other project. Shouldn't we shut it down when I was at Facebook?

shouldn't we shut it down and they looked at me like i was crazy and like why would we shut it down like well because we're wasting money and they're like so we have billions like what like it's just kind of like confusion over like why would we not build something pointlessly just for fun and Very deep in Amazon's culture is this like.

We do things efficiently. Now, sometimes you're doing things efficiently to launch something faster. So you'll rewrite someone's code because it's faster than integrating with theirs or something like, you know, you're always balancing. speed of building versus efficiency of resources and all that kind of stuff but i think um If your answer is always, unless there's a strong reason to do it, let's not do it.

leads to like at least the first half of my tenure at Amazon, engineers would get like one monitor of only 16 inches or something ridiculous. And like literally, and it was, I mean, it was both hilarious, but also frustrating. It was like an intern would leave.

And before the IT group came up to take their equipment, it was like the sharks would swim in. All the engineers would run over and take their equipment, take their extra keyboard, take their monitor and stuff because they wanted two monitors. And the only way to get it is to steal someone else's.

Yeah, that was a fluidity. Oh my God, it was so stupid. I think it is context, like just understanding where the company is coming from and that it is a unique setup in the sense that there are so many fulfillment center workers, which most companies do not have. Yeah. I wasn't aware of that part. I mean, data driven. I've been on teams where they wrote six pager explaining why they needed.

you know the top of the line mac pros instead of the middle grade mac pros and they explained the you know productivity benefits and someone said okay not now you've explained it like finally someone came up with an actual math reason not just like we would like it and boom everyone got their stuff and we moved on but um

I would say like if you had to choose between frugality and not frugality, you'd want frugality. It just makes things operate better when you spend money on the things that need to be spent on and not waste time elsewhere. But it definitely.

can be frustrating trust isn't just earned it's demanded Whether you're a startup founder navigating your first audit or seizing security professional skill in your governance risk and compliance program, proving your commitment to security has never been more critical or more complex. That's where Vanta comes in. Vanta can help you start or scale your security program by connecting with auditors and experts to conduct your audit and set up your security program quickly.

Plus, with automation and AI throughout the platform, Vanta gives your time back so you can focus on building your company. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SOC 2 and ISO 27001. With Vanta, they centralize security workflows, complete questionnaires up to five times faster, and proactively manage vendor risk. Join over 9,000 global companies to manage risk and prove security in real time.

For a limited time, my listeners get $1,000 off Banta at vanta.com slash pragmatic. That is vanta.com slash pragmatic for $1,000 off. Now, one thing when I did my Amazon deep dive on Amazon's engineering culture, one of the most negative things that came up Amazon's so-called URA target, unregulated attrition target. If rumors have it that every year about 6% of staff is expected to leave as unregarded attrition, and what this results in is...

basically focus plans, which is what comes before PIP and there's PIPs. And the people, again, I have friends at Amazon and many of them mentioned that either they were on a PIP or they had someone else be on a PIP. There is a bit of a scare, uncertainty, especially for people entering Amazon. You were on the other side of this. You were a manager, plus clearly you were also maybe subject to this on your end. How scary is this and how does it play out in reality?

I would say for most people, it wasn't scary and it like... So, you know, at various times it was between six to 10% of people had to be, and exactly what's measured was a little different over time of like whether or not you were like six to 10% had to be rated poorly or six to 10% had to be.

put into like coaching or whatever or six to ten percent have to be fired um sometimes different levels of goals in terms of like the cascading level of like as you're managing people out there's goals to hit um It wasn't scary for the vast majority of people because you're looking ahead. You're thinking about how do I get promoted? How do I do these projects well so I can get ahead in my career? I don't think you're...

I don't think most people are thinking I'm probably in the bottom 10% of all my peers, right? I think most people don't think that. You're not worried about being the bottom 10%. And I would say until it pops up, right? You move to a new team, your manager doesn't like you. And you think about like, huh.

my manager doesn't like me and seems to like all my peers, suddenly it becomes very scary. Because I've actually heard this from people I know of the same story that I, they've been with Amazon for some time. Great. Either a new manager came or they moved to a new team.

And that's when things became, you know, I actually had a friend who was put on a pip. He thought it was really unfair. He was there for four years. He actually made it out of it, but he kind of became bitter, especially towards his manager. Almost never happens. Yeah. He actually turned it around, but yeah, still. Yeah.

I think there's a few edge cases where people can make it out of pips. And mostly that I would say the vast majority of people who get out of those is something like a college hire. And, you know, this is just a common thing. I make fun of college hires because of like how much they move from kids to adult over time, right? Like you're in college and you're a kid and then you become an adult.

How many times they get hired in like three months into their job and I see them posting on Instagram at four in the morning and then the next day they don't show up to work and they're like, I was sick. I'm like, yeah, I bet you were sick. Alcohol poisoning is definitely an illness. Like that's kids, right? It's like, dude, you haven't done any work in two weeks and you're out late and you show up to work at 1 p.m.

Then you can get a pit that you can get out of because they decide to buckle down and actually do their work. You let them back out because you think they're competent. They just need to grow up. But if you think about the incentives, you're running a team. I have 50 people working for me.

Because usually the criteria of that like 6% to 10% is only if you're in a big enough org. So let's say I have 50 people. So now the criteria hits me because usually it's 50 or so when you start to have these expectations. If I need to rate 10% of my team poorly, let's say. And I have 50 people. It means I need to find five people. By the time the review period rolls around, I need to be like considering who's not doing great.

Um, review periods every six months, every 12 months, 12 months, 12 months. Yeah. Yeah. And it, again, like at various times we had checkpoints, six month checkpoints. Sometimes the HR would care that you're flagging people. Sometimes it's like a, um,

For the most part, especially when you're talking about like firing people, it'd be like by the end of the year, you need to have fired six, five people. And sometimes it's like, well, we already did four along the year because of various things that happened. So that fifth one is the only one we're looking for. And so you're going through the review period with your 50 people and you need to find one person. It's not a big deal, right? You talk to...

That would probably be like seven or eight managers at least. One of them has some employee who's not doing great or one of your managers is annoying as hell. And so you like, no, you're okay.

But I would say it becomes an issue when... your team especially if your team is more stable like if you're constantly hiring people like there's always going to be new blood someone wasn't interviewed well someone's just like not at all suited to be here and that makes it easy when your team's more stable and you've been

for the most part with the same people for two years and you're running into yet another review cycle and you're like oh my god like we already got rid of everyone who is not doing great everyone on my team is doing well now you end up in frustrating situations um And most of the time, I would say it was not a big deal and wasn't that stressful because...

Again, those expectations don't come into play until you have a bigger organization. So like you're a dev manager and you'd say, no, everyone on my team is good. And as long as you have backbone, you're okay with that. And then you get to be a senior manager and you have 47 people. The criteria doesn't apply to you still now.

You know, boy, you have 47 people. You have no one underperforming. Are you sure? Like 47 people. Like now you start to get to the point where your manager sits down with you and says, let's go through and tell me the bottom five people of your org and why they don't deserve to be rated as least effective.

And then you get to bigger orgs and that's where you get this weird, like the organizational pressure of like, now you hit your director. You know, I have 150 people reporting to me. I need to come up with the. i don't know 11 people who are in coaching by the end of the year or something and like now i sit down with my managers and say okay guys like we have eight i need 11.

Sorry, we have to talk through all the people who are like on that borderline and explain why they're not whatever at this level.

because for the most part they were very very hard asses on making sure you hit these um the the goals and then from a software engineering perspective you know you're kind of like blissfully unaware you're kind of working away let's imagine that you know like one day your manager probably had this discussion with their director or their manager and they identified you as someone.

Usually when this happens, in your group you circle down and say, this is this person, these are the people who are underperforming. Is there also a part where... There is like reasoning given of why so that the manager can go back and say like, all right, here's, here's the difference. Here's why you're not meeting expectations. Or is it just more like, well, you know, you're, you're, you're bottom of the pile. Like, sorry, here's a pip.

So I guess the first thing to say is like there's often a... a definition issue and even even with the managers sometimes like boy this is where managers screw up the most often and you almost need like a bar raiser for hr processes um The description of that lowest rating and the description of letting people go is all about not... not meeting expectations. It is least effective.

And so the argument is, we explain to people is, if we have 100 people, we want, let's just pick an easy number, the bottom 5%, five people out of that 100, we want the least effective of them. It doesn't mean not effective. It doesn't mean that they're not doing their job. But if you pick the bottom five people and say, would you rather keep them? interview and find someone who might be just this amazing, awesome hire. And most of the time you would say, well, I mean...

the bottom five out of 100 people aren't great. Like they might be okay at their job. Comparatively, right? Comparatively, like they're a fine, let's say engineer, right? They're a fine engineer. Most people would get that work done in... a week and it takes them two weeks because they're slower. Like it's just like they're not great.

If their manager is doing well, their manager has been telling them this all along is like, hey, do you notice that your work is getting done slower than everyone else on the team? Right. Because again, this is the bottom five of 100. Like they're slower than everyone else there.

making more mistakes, like whatever it is that makes them at the bottom. Hopefully they've been getting feedback along the way. And then in this discussion that inevitably happened, right? You have all the managers in a room with their senior manager, maybe with the director, and you're having detailed discussions. And we're saying, okay, their productivity is lower. They make a lot of mistakes. Their peers complain a lot about them because almost always there's like a lot of peer like...

I really don't want to work with Dave on this one because, boy, Dave always is slower and you have to do most of the work for Dave, right? So... You have this collaborative discussion with whatever feedback you have available to say, okay, definitely Dave has to be in that bottom five.

The manager of Dave has been giving him feedback throughout the year. And maybe this happens not at the very end of the year, like preferably just like all of a sudden, wham, everyone gets surprised. But if it has happened, then yes, the manager should be able to sit down and say, You are being rated as least effective or you're being put into coaching or whatever, you know, whatever it is, the step is because here's the, you know, the situation, you know, you in the last four sprints.

Specifically, in the last four sprints, you have missed your... projects haven't been completed like we thought would be completed in the sprint, every single sprint for the last four sprints. We've already talked about this. The issue usually comes up is if the manager has been saying, you're awesome, you're awesome, you're awesome. And then suddenly, like, terribly sorry, you're in coaching now because you're not awesome. I've actually heard quite a bit of that.

The problem that some of the people that I talked with who were in the situation, their manager was new or did not understand. and some of those things. But this does make sense. In some ways, it feels to me...

Correct me if I'm wrong, but this feels somewhat similar to Netflix in the sense of, you know, Netflix is also very open about having the keeper tests where they regularly re-evaluate people and they're not saying you're not meeting expectations. You're just like, well, if someone is not...

they actually do the other way around they're saying if someone is not worth fighting for keeping yep we don't necessarily want to retain them in fact we might let them go and they're they communicate subfront whereas with amazon if i understand they're saying you're the least effective which Again, people got into Amazon. It's hard to get into. They're typically good engineers. They might be awesome at a lot of places, but Amazon says, okay, we want to...

This is their performance philosophy. Yeah, yeah. And I think a good number of the times, if you talk to someone and say like, You know, someone who's getting this kind of feedback and they're saying, but I did complete that eventually or something like that. But do you agree?

that every person on the team would have gotten that done faster than you. Like, you know, it's, well, yes. It's like, okay, so at least you would agree out of your 11 peers on the team that you are the slowest engineer. Like, you know, it's like...

I think people understand one of the hard things is just like the difference between least effective, which is just like, hey, at the bottom of the pile, you're going to be at risk. If you think, if you look around the room and you're thinking, yep, I'm the worst one here, that's not a great situation to ever be in. It's just never safe. And at Amazon.

It's definitely not safe. Some other companies where they just might do layoffs once every four years, you might be safe for quite a while. But Amazon has this sort of regular cycle. Well, also, I feel some... companies, and I wrote about this, I said some companies are starting to become a bit more like Amazon. So Facebook has done 5% performance based layoff for this time. I think Stripe was doing something similar.

Previously, I think we saw there was these kind of golden years, if we will. You know, when the job market was really good, Amazon kept doing this. And Amazon was really the kind of the bad company, if you will, the only one who were doing it besides Netflix.

It seems to me Amazon hasn't changed anything, but some of the other companies are doing. So maybe as software engineers, we should accept that some of these top companies in terms of brand compensation, they're going to be tougher places to be at. pushing people to do better and they'll keep comparing people with one another.

In terms of like as an individual, if you're joining as an engineer or manager, you know, anyone, I think one of the things to keep in mind in terms of like ego and like how does it feel to be told all of a sudden like you're not doing well. I guess partially is like awareness of, I would actually say as a manager, even like 50% of that performance frequently is your relationship with your manager and like your team. How will you fit in with the team, with your peers, with your manager?

So many times I've had someone who was either doing amazing on one team, they moved to the next team and they're like actually not doing well at all. Or someone who was not doing well escapes to another team before they get fired. And they do well. I've had people who I was planning on firing who managed to transfer off my team in time.

And they ended up getting promoted someday. And the same thing, I've had people who said like, I'm about to be fired. I'm pretty sure that it's coming towards me. Can I get off the yoga? This is like my my sneaky recommendation for anyone is like if you start to hear performance feedback whatsoever from your management chain, if you have any opportunity at all, get off your team fast as possible. Like that's that's the.

As we frequently say, it's like unfair thing that managers know is if your manager says. you know, that wasn't so great what you did recently. It's like, okay, well, switching teams like fast as I can because most managers won't, not if you trust your manager, they might be actually just giving you honest feedback, which you'd like to be able to receive.

But for the most part, if you've been working for someone for three years and suddenly they start giving you performance feedback, that's a really bad sign. If you run for the hills fast enough, it's possible you'll get away before they flag you in the system as non-transferable. Yeah, and I guess...

It's just worth keeping in mind that for people who have not worked at these are companies, internal transfers are possible and you do want to take advantage of it. When it makes sense, it's still easier to do an internal transfer. to start a new job search process to start, you know, like your network will, if you change companies, you'll have to start to rebuild your network and so on.

Yeah. And I think, again, it frequently is not that you're a bad manager, a bad software engineer, a bad project manager or something. It's just, you know, sometimes it doesn't work out with you and your manager. And I think... The same thing. If you're like, boy, I don't get along well with my manager, but I like this team. That's not the safest thing to put up with. I've heard people saying that like my manager doesn't like me, but.

i really like what we're working on i'm like dude get off the team like if you want to keep your job get off the team i feel amazon mike might make some of these kind of you know universal truths a bit like more Because I remember, for example, I worked at Microsoft at a time where Microsoft just didn't let people go. And there were people who were doing really bad. We knew. They were doing bad work.

But for some reason, management didn't want to let anyone go. So there were dead weight. And by the way, when this policy... I wasn't there when it changed, but I heard when it changed, they were one of the first ones to let go and no one was surprised. But the point is like, This is usually how companies work. Either normal times or when times are a bit tougher. Right now it is a bit tougher and it's just good to prepare for that. Speaking for myself, I was always paranoid.

In most of my companies, especially when I joined the first like six to 12 months of, I assumed I probably was not doing that good. So I tried to do my best. you know hope that was enough over time you could become more comfortable but as you said i was as a manager as well if you're not getting what your manager what you said like like you need to switch managers like try to fix it if you can but if you can't sometimes you're just Not much you can do. Yeah, I think that...

There's the mistake that people will sometimes make is like my manager doesn't influence my job that much because I can work independently or, you know, I don't need to figure this out with my manager because. I can work with my peers or I have this great engineer on my team I can work with.

But especially at a place like Amazon, like your manager is responsible for your rating, which is like tied directly into comp. Like they're definitely responsible for your promotion. Like you're never ever going to get promoted if your manager doesn't like you. And every year, at least, your manager needs to pick out a number of people who are not doing great.

And so if you're not getting along with your manager, you need to fix it or you need to move on to another team. Like that's just a solid thing at Amazon. And there are bad managers. I think like half the time someone gets fired, it's probably because the manager is like, you know.

it's a relationship issue. And I guess it's fair to say, this is specific to Amazon and maybe it's some other companies you can get by, but I think it's fair to say the other thing as well, right? If you have a great manager, someone where... They have your back. You've got your back. You work well together. It might be even worth following, depending on the situation, but oftentimes I hear people following their managers to other companies.

you know, they joined a year later, they reach out, do you want to have a coffee? Guess what? They're going to be their right hand at that company as well. I actually know people who have risen with, and you know, these good managers usually, not always, but oftentimes they're good at. you know get getting ahead especially because they do bring some of their team so if you know or if you have a good manager or if you had in the past just hit them up again

It's rare. It's not that easy to have that spark. If you've had no managers that you have a good enough relationship where you want to go move with them, then... You've either had a really unfortunate choice of managers, or maybe you're having the issue with the relationship. Sometimes it's you, not them.

But yeah, and I've heard before for managers, especially as you get higher levels, like if they don't pull in a number of people into their org as soon as they transfer over and like there's not a list of people ready to transfer over. Then maybe there's an issue with that person because you want them to say, every time I move groups, I would frequently start off with how many open roles am I going to have? Because I have a number of people who I want.

you know, who will be happy to come over and join and take all those open positions. And that's what you expect. That's what you should have. And especially within Amazon, that's a totally expected thing. The person comes over and then suddenly all the spots fill up with the people who have followed them for the last three org moves. And it's a good way to shuffle. I think it's healthy.

Especially, you know, companies like Amazon, where you want fresh blood and new ideas, new thoughts, like it's a good way of. you know, orgs sort of shuffle around, you know, your VP leaves and suddenly half your management team leaves. Like, that's a good thing. That means the VP was liked by some people. They move off to somewhere else to bring some new ideas in and then you get a whole fresh swath of new hires.

So I'm getting a sense that Amazon, and obviously I got that sense when I was refreshing Amazon, but talking to you even more so that Amazon is just not a typical, what you would expect a large company to be. And one thing I've noticed that I think underscores this. is when startups hire from large set companies.

When I'm talking with founders or people who have hired in the past or engineers who work with them, there's a lot of eye-rolling. Like, oh, we hired that person from Google and it didn't really work out. A small company or Meta or some other company. One thing that I rarely hear is that this Amazon hire didn't work out. In fact, I see so many Amazon people go to either large tech companies and then some that even have maybe a shinier brand like Google, Meta.

open AI, etc. And they do pretty well there. But they also go to startups and they do pretty well there as employees, not just as founders. Why do you think this is? I mean, I think we've already touched on a few things. Why? It could be because... Amazon feels a bit like a scrappy startup, even though it's a big company. I mean, corporate-wide is not a scrappy startup. It's a big old behemoth that can't turn to save its life.

Not that I'm biased or anything, but sometimes like HR policies and stuff like, you know, with a few Amazon wide things, it's like good luck freaking changing anything. It's like bang my head against the wall so many times trying to get things changed. So corporate-wide, they're not. But when you get down to a dev team, I loved the fact of almost everything is controllable at the lowest possible level.

And so, you know, everything from stupid changes people can make or awesome changes they can make that you have a lot of control at every individual dev team can pick there.

process can pick their coding language can pick how they're deploying their code like what tools they're going to use and sometimes it becomes stupid like it's just like what there was a team that built their whole um very important middleware like service out of lisp and it's like what are they thinking like it's why i mean it's sure but

knows the damn language. Like no one else has written anything in Lisp and like no one knows what you're doing. And I think it was like two engineers on the team had this great idea and they were at the whole damn service and like, great, they built the service and they all transferred off the team.

And then they had to rewrite the code because no one knew how to support it. And it was this nightmare. But they could. Right. And thankfully, no one came up with an Amazon wide policy saying you're not allowed to write anything except if it's in Java or something.

So teams would regularly build stuff in whatever language they want, in whatever tool set they want, at whatever pace they want. They can do Agile, they can do Waterfall, they can do Scrum, or they can... not do sprints they can i liked the fact that unless there's a strong strong strong reason for something to be dictated by amazon or vp or director whatever else for the most part the culture was

You can't tell me what to do in a good way. As in like, you know, the director would say something like, why don't once in a while you hear someone say something like, why don't we line up all the sprints? And everyone just starts stomping their feet saying, hell no. you can't they want to do a three-week sprint but we like four-week sprints or whatever it is and it's like um

I use that as an example because that did happen a couple of times where like someone thought like, you know, what we should do is line up all the sprints across our organization and all the engineers are like, oh, it does sound like a nightmare as a dove. Yeah.

And I appreciate it. I really like that culture of like someone says we should all flip to Macs and like, no, no, I like my Windows laptop. Why? Because I do. Or like someone else has a Linux laptop. It's like, well, I'm just going to do that because I feel like it. Unless there's a reason to dictate it, like a really, really strong reason, the default answer is just everything goes to the lowest level. And so...

Again, like you're on call because it's your software and it feels in most cases like you're in a seven person startup or a 10 person startup because like you picked your language, you picked your your. how you deploy it. You picked how you QA it. You picked, you know, how you monitor it. You set your own monitors and alarms. It's not like there's some global Amazon click the alarm button. Like, oh God, no. Like you have to make all this stuff up and people make it up wrong.

But you learn, right? I think there's a lot of skills. you know i can't name the exact gaps but sometimes you'll you know i would talk to the facebook engineers like they've never dealt with an emergency of like how fast you know it's like where are alarms you know where are That was an example. I said something about like...

where are your metrics and stuff like that for the system? And they're like, I don't know. I'll have to go look for them. And I'm like, blew my mind. Because if you're an Amazon engineer, you have a freaking hot link on your... browser right because you have to click the button really fast when there's an emergency and see all your alarms and they better be on like one big dashboard with the alarms that you know about and the metrics you know about because

You support this system and you know exactly how everything works and you know that your memory usage is higher than usual. So what the heck is going on? And so... i think it just builds a lot of skills at that like really low level of like we own the whole stack we own the product we you know frequently like you're making product decisions because there's not enough product managers for your space or things like that so

On the frugality side of things, for example. Well, so I'm sensing that you're, like, as an Amazon engineer, you're actually really close to, you know, the metal, the work. You're in a small team. You have a lot of ownership. You're a whole team. You have to wear multiple hats. You cannot afford to buy all sorts of used budget because Amazon doesn't want you to.

Plus, well, that's not my problem is something that, you know, that's one of the quotes in the leadership principle is like you can't say like. That's a product decision. go la la la i don't want to make a decision like no dude like sorry you're an engineer i know but you have to make this product decision now and then you also need to listen to customers and stay close to them yeah

Which is already, I guess, like a startup founder would say, these are my ideal folks. Plus, on top of it, when you leave any big company, Google, Meta, maybe even Microsoft, you're used to internal tools that do not exist outside of the world. But with Amazon, especially these days, If you're using AWS, guess which one is the most popular cloud service? I would say both lucky, but also I definitely heard teams at Amazon switching to...

publicly available tools in part because they say, I wanted to learn this. Like this is a popular tool these days, which is again, is like one of those big advantages of it. a system like amazon where you can just choose to use whatever you want is someone saying the hot new way of doing android development is x so we're going to do that for our for this next project because we you know

us collectively on this team want to learn that tool. So we're going to use it like that's pretty awesome. Like that's such a good way of keeping on top of tech is being able to say we're just going to use whatever we want. And we've read a lot about this thing. It looks like it's popular. So we're going to try it out. This is awesome. I mean, my view changed a little bit about Amazon because it just seems like

Unlike some big tech jobs that you either go to big tech or become a founder, Amazon does open multiple paths. You can keep going on in larger tech companies. You can found a startup because you've seen how it's done. You can join a startup. You can join a scale-up. Pretty cool. Yeah, and I think the downside which I would say absolutely exists in Amazon is... related to the like independence thing of like everyone can do whatever they want is

Imagine the manager who says, we're going to be working on these projects. No, we can't go do that operational excellence. We're just going to put two people on call and we'll get paid 17 times a week. And like, there's not. the uber amazon policy holder who says like you manager are doing it wrong you need to be fired right

So you do end up with these pockets of just absolute horribleness because people can do whatever they want. And so once in a while, you talk to someone and you hear about what it's like on their team. And I'm like, that sounds horrific. Like, I can't believe. your team sucks so much. And so when someone says, You know, you'll see on Reddit or something about like, you know, oh my God, don't go to Amazon. Everyone leaves after six months. They don't.

the actual stats are not in some pockets they do but in some pockets they do absolutely i have met teams that had no no one like 13 engineers on one team no one had a tenure more than 12 months because they all

They were all leaving. People would join the team, realize how terrible it was, and they transfer off. And it just like just pop on, pop off, pop on, pop off because the team was terrible. The manager was bad. Their senior manager wasn't paying attention. Like there are these pockets of terribleness. And so.

Again, if you're in a bad spot, move somewhere else. Internal transfers are great. And I assure you, I was in six orgs. I had a bad manager. I guess sort of like two bad managers. They were both fired. Everywhere else was great. And those places became great after I had new managers. And so I had great managers. We had great orgs, great peers. And you would just see pockets of terribleness here or there.

Almost like the inevitable downside of having thousands of little startups around the way is like once in a while you end up with a really bad situation. I know you had a short cent of Facebook as well, but in total, how long did you spend at Amazon? I think it was a little over 12 years at Amazon. A little over 12 years at Amazon, a bit of Facebook. Afterwards, you had a little stint. I took a leave of absence towards the end. I took a leave.

And then you worked at Bezos Academy for a little bit as well. But then you made the decision to retire. And you're still young. You don't look 65 to me. So how did you decide on... retiring and what did it take to build up a kind of financial independence to be able to do this?

What do you do right now? Yeah, yeah. So many years ago, I have no idea how I got so lucky that I stumbled on the idea of financial independence and read a bunch about it and learned math, you know, like financial math in terms of like... withdrawal rates and investment things and index funds and stuff. And so many, many years ago, in my 30s or whatever,

Yeah, I guess that's a while ago. Early 30s, let's say. I realized how much money I would need and keep an eye on our expenses every year and stuff like that. Amazon pays well. Stocks go up over time. They used to go up. They did. They used to back in the day, back in my day. Hopefully they will. Yeah, hopefully they will again someday.

And I would say almost like as for tech workers, it's like keeping an eye on your expenses is probably more important than keeping an eye on your income because you're probably making enough to retire if you don't spend a lot. And so thankfully, my wife and I don't. And it comes down to that kind of thing.

A good number of years ago, I built a lot. I mean, I don't know, eight years ago, I built a spreadsheet and I could tell about when we would have enough money where I wouldn't need to work anymore. And that was my position when I was getting promoted to director at Amazon. I could see like essentially like my next vestings make us way past my safety margin. And so I don't need to work anymore.

So there's just like that fundamental, like, what am I going to do with my life when I don't need to work? I can keep working. I enjoy working, like some aspects of it. I don't like waking up in the morning and commuting sucks. I really appreciate sometimes like hanging out with smart people and talking to them or doing creative things.

So Bezos Academy poached me out of Amazon at a convenient time when I didn't need to work anymore. And I was having a little bit of that existential crisis of what do I do next with my life? So they poached me. I worked there for a bit. Ended up deciding that that's not like the right kind of job for me. And so I thought, well, like my dream since I was young was to write a book, you know, actually like fiction book. I want to write sci-fi or fantasy about dragons and stuff.

How do you force yourself to write? Because I've been wanting to write for years and I have like literally 20. stories of like 50 pages each um and so i thought well how do you force yourself to write regularly newsletter is an obvious answer so i'd written years before while I was at Amazon, I'd written a few very popular articles on LinkedIn.

That just sort of blew up. One of them was one, the main one, in fact, was like the Amazon Leadership Principles article that a lot of people know about where I'd written an article. Or sorry, I written an email that I would forward to people when friends or family or relatives or whatever else said, hey, I want to interview at Amazon. I'd forward them this email that essentially said, here's what you need to know about interviewing at Amazon.

So I turned that into an article on LinkedIn and it became wildly popular. And so I thought, well, why don't I do that more? Like, that's sort of fun. I'll do a newsletter for a bit. I start a newsletter. And like by the second or third article, I had like thousands of subscribers and I was like, holy smokes. OK, well, I need to write a paid newsletter is what I need to do just for fun. And this was literally like I just see the tools.

where i can click paid and say like you could pay and like how much should they pay me i don't know five dollars um so i just pushed some buttons said i i thought i'll just write two articles a week then so things have changed over time but in general it just blew up and For a while it was, I'm going to write a new article. And this is Scarlet Ink, right? This is Scarlet Ink, yeah. So the short thing for Scarlet Ink is I got that...

domain a long time ago when I thought like, I want to do some kind of writing a blog, a newsletter, a book or something. I wanted writing. Because I always liked the red pens at Amazon when I mark up documents, you know, document culture, right? You sit there with a piece of paper in front of you. And I'd always use the red pens because I could see it better against the black printout.

And so I thought red ink, but red ink was taken and it was really expensive to buy the domain. Scarlet. It's even more literary. Yeah. Scarlet ink. It sounds really fancy. So, um, so I got Scarlet ink. I wrote the Scarlet ink newsletter. It was really fun.

And I was writing for a bit and I kept saying and thinking like I'm just sort of on a sabbatical where I'm writing and then I'll figure out what I want to do for a job. Like, do I want to take another nonprofit job? Do I want to go to a startup? Do I want to go to, I don't know, Google or Microsoft or something?

And I did a couple of small interviews. I talked to some people at Google. I talked to some people at other companies and it was like, meh. But the newsletter kept growing. And then you start to get to the point of like. But why would I go to a company again when I'm making... So like... I don't know, a couple years, a year and a half ago, two years ago, I sort of crossed where we were making in my newsletter more than we were spending. And so like...

I haven't actually withdrawn any of my retirement money, withdrawn any from the, I mean, it's only grown because I don't need to touch it because I have this newsletter going. And then it gets to this weird point of like, why would I go get a job? When I'm writing a newsletter that already pays me a salary and I'm not spending all that much time on it. I'm not commuting. I'm sitting in my little office here writing one article. I end up writing like one article a week.

I enjoy writing. I sit down, I write for a few hours. Sometimes I'll write over a period of a few days. Like I'll just, you know, sit down for a couple hours, write something, brainstorm. I'll be on a walk and I'll text myself an idea of something I could write about. And so you sort of like the idea of getting a job fades away when you realize you have so many other things you can do in life. Sitting in the office and being forced to do it for 40 plus hours a week doesn't sound quite appealing.

So it sounds like you built up really good savings as a mix of just working in back tech. being at a good time where there were stocks awarded. Yeah, back when stocks went up. It was appreciation. But you also read upon financial independence, the planning. You actually started to make a plan.

And then there was this point where you crossed the boundary where like, okay, you are financially stable and you could afford not to work. You kind of explored some things, different things, you know, like the nonprofit job at the basis academy. whether you should work at other companies and then you just did something that kind of like Just worked out. Yeah. Sounds like it's because you didn't.

Is it fair to say that maybe it worked out because you just didn't really have the pressure? I mean, if this once a week news that didn't work out, you probably would try something else, right? Because you have no pressure right now. Yeah, if you imagine, yeah. I don't know if it's responsible for the newsletter working out, the fact that I didn't need it. I mean, if I really wanted to make it work, it probably would have worked also. I think a little bit of what might have happened was...

If you're not trying to impress people, your work product for creative things is probably a bit better because I wasn't... And I don't try to appeal to like, what do people really want to hear? Like, what's the hot topic? I've only written about AI like once or twice. Like, I think only once, like real an article about AI, which is mostly saying like, no, it's all overblown. which I still believe in. But...

For the most part, you're writing about the things that you think are interesting or what someone else has talked to you about. And you're like, oh, that was such a cool thing. I did actually some one-on-one coaching for a while too. That was one of those side things once I started writing. I think when we first met, you were still doing a little bit of that.

Yeah. Yeah. So I was doing hourly coaching and my wife and I argued about it because it was like I was making a lot of money doing hourly coaching. I kept raising my rates and she's like, stop it. You're not going to get any more work. I'm like, well, I know because I don't want to be doing this anymore. So I kept raising the rates trying to send people away. And then I started feeling guilty because I was making such a huge amount of money per hour.

And people are still signing up for it. But then you start to have this like performance pressure of like, am I worth that many, you know, that much money? Like, oh my gosh, like, holy crap. So I lowered the rates a little bit and then people were still buying and I ended up just turning it off. I just told everyone like, sorry, I'm.

I'm not getting joy out of it anymore, and I don't know why I'm doing it. It's a weird existential problem to run into when you realize that incremental money won't change anything other than a nicer car or something, which is like... like fundamentally something that give you, if you feel that way, like this is how you reach financial independence is not to keep.

growing your spending. Because a lot of people will be like, I got a raise, so I spend more. If you stop that from happening at the right time, then as your career improves and you make more and more money, the only thing it's doing is increasing your savings rate. And that's how if you have a good tech job over time, like, you know, you can afford things and save 15 percent. And the next time, you know, your next raise, you can save 20 percent of your salary and then 25 and then 40 and then.

you know by the end we were my wife was working at amazon which is partially how this happened i mean she she retired a couple years before me but We had two tech workers. We were saving like 85%, I think, of our income. You can retire pretty damn fast if you're saving 85% of what you make.

And then where did you learn about financial independence or for people who are thinking about like, okay, I'm actually inspired. I'd like to learn more. Where did you start? And do you have some pointers of books, resources? Yeah, the internet. I would say Mr. Money Mustache, and he doesn't write as much anymore, but you could definitely, he has some pretty fantastic archives. Mr. Money Mustache is a blog newsletter.

Mad Scientist was another one who also... He's been retired for years. It's sort of funny. These... Financial independence newsletters in particular are just sort of hilarious because... they'll be like, I have $10,000 and I want to retire in five years. And so they write like mad for five years and they're like, I've made it. I think I've made it.

and then you notice like the newsletters just sort of peters out and stops like the right once every two months because they're busy with their life and like yeah they're the the driving force that's what they're going to do it was their thing so uh almost no active newsletter exists that was from the time of which i originally read about it um

But yeah, Mr. Money Mustache, Mad Scientist are the biggest ones that I read about at the time. But there's a good selection of... Actually, hold on one second. There's a... I can look it up right now. There's a book that I recommend on my website, in fact. You can set it over and then we'll put it in the show notes afterwards. This was really fun. Yeah. Both about software engineering and a little bit on management. We didn't go into too much detail at this time, but perhaps on a follow-up one.

Well, thanks very much for being here. This was fun. Thanks for having me. It's been fun. I enjoy chatting on these topics. I hope you enjoyed this deep dive into how engineers at Amazon work, as well as how working at a large tech company like Amazon could help getting to early retirement like Dave has done so. Thanks today for a great conversation.

You can read more from him on his newsletter at scarleting.com and you can find him on social media as linked in the show notes below. We previously did a more detailed deep dive on the engineering culture at Amazon. Check it out linked in the show notes. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. This helps more people discover the podcast and a special thank you if you leave a rating. Thanks and see you in the next one.