Successful projects, there's a narrow path to success and there's a million paths to failure. A narrow path to success. The default state of a project is to fail. Projects only succeed because people drag them kicking and screaming to the finish line. If you leave a project alone and...
nobody's worrying about whether it's going to ship. Nine times out of ten, that project will not ship. The ability to kind of come up with a technical demo, kind of moving mountains. One demo of like, hey, look at this thing happening, can cut through weeks of conversation about whether it's possible. And if you just go off and...
produce that you can really blow people out of the water in the context of like a large company being technically strong if you work on software is really a superpower because it makes you connected to reality it makes you connected to the real world to the actual product that you're building it's sort of like a lot of the work
these other people are doing around strategy and marketing and figuring out what needs to be built, that's a huge multiplier. How do you ship projects at big tech companies? Sean Godeke has worked as a staff engineer at two companies that employ thousands of software engineers. He spent five years at Zendesk and now works at GitHub.
Sean wrote a blog post about this topic titled, How I Ship Projects at Big Tech Companies. The post went viral online and sparked a lot of heated discussion. In this episode with Sean, we dived into... What does shipping projects actually mean when you're working inside a larger company? What is more important in order to be seen as someone who ships projects? Your technical skills as an engineer or your soft skills to navigate the organization?
The story of why Sean was considering looking for a new job when his manager told him to not worry too much about test failing in the staging environment. How do Gen.AI tools change the ability for us engineers to ship projects? Sean is a great person to talk about this as he's worked on the GitHub Copilot team and now works on GitHub models. Advice Sean has to succeed in a full remote team when working across time zones.
and a lot more if you enjoyed the show please subscribe to the podcast on any podcast platform and on youtube sean welcome to the podcast Thanks. It's great to be here. I'm a big fan of all your stuff going back many years. So really exciting to get to talk to you. So to start off, what... is shipping. In your post, you wrote, and I'm going to quote you, I know it sounds extreme, but I think many engineers do not understand what shipping is, even inside a tech company. So what is it?
My post kind of came from this experience of seeing a lot of engineers around me and myself in the past getting into this frustrating position where they were delivering code. They were feeling happy about their output, particularly the volume of their output, but they weren't getting the kind of... recognition they felt they deserved. And the sort of management chain seemed happy or seemed unhappy or even worse, just seemed totally indifferent to what they were doing.
um so that's that's kind of the context in which i i talk about shipping and the context in which i wrote my blog post so in that context uh i say that shipping is uh kind of a socially constructed fact which sounds like this postmodern way of talking about things, but it actually caches out really concretely into the proposition that a project is shipped when your management chain believes that it's shipped and like talks about it being shipped. So shipping...
Kind of means doing things that your management chain is happy with and I say management chain because like for small projects this can be your manager and just your manager. But for larger projects, it's usually a layer or two above your manager, particularly for like staff roles. And sometimes it goes all the way up to the CEO. And in those cases, the metric of whether a project ships is the CEO happy with what you've done.
That can be a kind of frustrating way to think about things. I think for a lot of engineers, certainly the commenters on Hacker News found it that way. I think partially because it relinquishes a lot of control. You know, it's not necessarily about the quality of your work. It's kind of about how it's received.
But I do think you have a bit of a happier experience going into these projects with open eyes. And I think you tend to be more successful if you're actually driving for... this kind of like organizational definition of shipping rather than just saying I closed all the Jira tickets so the projects shipped I deployed the code to production so the projects shipped
That definitely does not mean that it's shipped. You can deploy something to production and it can be a complete disaster. This episode was brought to you by DX, a platform that helps organizations measure and improve developer productivity. The DX team includes notable researchers behind the frameworks like DevEx and Space. So they're constantly asked by leaders, what metrics should we use to measure developer productivity? To help simplify the landscape.
DX just recently published the DX Core 4, a unified framework for measuring developer productivity that encapsulates Dora, Space, and DevX. The DX Core 4 includes four dimensions, including speed, effectiveness, and quality. that provide a focused set of metrics that help track productivity at all levels of the organization. You can learn more about the DX Core 4 and how it's being used in companies like Dropbox, Etsy, Vercel, and Pfizer at getdx.com slash core4.
Again, that's getdx.com slash core4. Something that feels wrong about this definition is... It kind of sounds like, you know, we've shared successfully if the management chain all the way up to CEO is happy and they're like, all right, great job. But it kind of goes really against like, I think what a lot of us feel is shipping, which is...
Does it work for customers and users that people care about it? This goes a little bit back to, you know, like agile, like the original agile manifesto, which talks about customers doing stuff for customers. How do you think that? is connected because clearly i can see on one and you know if the customers are happy the ceo is happy but you might have things where like i don't know the customers are not happy and the ceo is not happy
You've been doing this for quite a while, but surely you needed to reconcile this and also think about this fact. I want to say a couple of things on this. First, you touched on this in the question, but I do want to stress that in any remotely healthy company, Your management chain wants customers to be happy.
They want customers to be happy because they want customers to be paying money. They want more people to be using the service. You should be aligned with your management chain's goals here. If your management chain want customers to be unhappy and want the company to make no money, then yeah, you might.
ship and make them happy by deploying code uh well by doing whatever but you know you probably shouldn't be at that company you're definitely not going to have a fulfilling experience so i think like This idea that you're making your bosses happy instead of making customers happy kind of presupposes that you're working at a company where those things are in tension.
Certainly in my experience, I've been lucky enough that at Zendesk and now at GitHub, people tend to want customers to be happy. The other thing is sometimes your management chain has good reason. to make decisions that are going to make customers unhappy. And sometimes you don't have the context for that. So there's lots of reasons.
that are very important to the company. We could spend the entire rest of the podcast listing them off, and that might be a really interesting podcast. But just to give you an example, sometimes a company is in trouble and needs to position itself as being in a particular space. Recently, AI is... a good example of this so you need to ship some kind of box ticking feature in order to like
have a really positive impact to the success of the company. Users might not like it. That's a shame. It would be better if they did, but sometimes you need to ship it anyway. I want to be clear that GitHub is not an example of this. Our AI features are actually top-notch. That's part of the reason I'm here. Another really common reason why companies might want to justifiably ship something that users don't like is security and regulatory compliance.
If you've ever been at a company that's gone for, say, FedRAMP, you'll have to ship features that kind of satisfy these. What is FedRAMP? Oh, I'm sorry. It's a US government kind of regulatory set of... requirements, I guess, that make it so, as I understand it, the entities within the US government can use your software. Yep.
Yeah, it's one of the, I remember as well, it's a series of regulations if you want to do business. And there's, I guess, examples, not just this, but a lot of other regulations. Yep, exactly. And some of these things, particularly in the realm of security controls, mean that you're compromising the kind of customer experience a little bit and doing things that customers won't like.
For instance, you have to be more careful about the way you manage API keys. You might have to make them expire over time. Users really like having their API keys live forever. It's very convenient, but you can't always do that. Those are two reasons. There's a million others, but just to tie it back to the original question, like the fact that you're writing code that doesn't make customers happy might make you feel bad as an engineer, but it's not necessarily counter...
to the interests of your company. And there's lots of cases where I think it's quite literally your job, particularly as a very senior engineer, to kind of suck it up and to prioritize the company's goals over like your own goals. and how you might be building the project if it was just your personal project and it didn't have the kind of stakes that working at a large company has.
You touched on how as a senior engineer, very senior engineer, staff level engineer, it is your job to help the business to understand the business as well. What was the point where you... recognize this you learned this because this is typically a learning that i mean ideally it comes as as you grow but what was the point where you realize okay that is my job now that i'm at this staff or above level uh yeah i
I learned this one the hard way. Sort of the story of mistakes. So I was a very enthusiastic and very motivated engineer early on. I was just happy to be there. I was really excited to not be in academia anymore. So I was absolutely... absorbing things. I cared a lot about what I was doing, which was good, but it kind of pushed me into a couple of pathways that I think weren't super productive. So one of them was back at Zendesk, you know.
At this point, like six years ago, we had this suite of staging tests for the staging environment that were pretty unreliable for a whole bunch of reasons, not worth going into. But it was my like... holy mission for six months to like keep them green.
and keep them like up to date and i shipped fixes i shipped improvements i shipped reliability improvements to the to the tests i spent a lot of time and effort kind of like catching up to other engineers work where they shipped some feature that changed the ui which changed the test selector you know this this was something i cared a lot about um and i felt like i wasn't getting a lot of like rewards for it but you know people seemed kind of happy about it and then after about six months um i
said something in a meeting where i complained about you know some feature we did that that kind of broke the test again uh and my manager said you know something like oh well you know you just don't have to fix them and i was so mad I was so angry I left the meeting. I think I actually left the meeting midway, which I've never done before or since. I just got up and left and I went into another room. I went, this was in the office. I went into another office and I updated my resume.
that was what i did was what was this feeling of like like engineering craft is not like i don't know the number one or or what was it it just went against like your core beliefs at that time i think i had just put in so much work that honestly a lot of it was like a sunk cost thing. I felt that it had to be important because I'd spent so much time on it in hindsight. I did think like it sort of offended my sensibilities to have tests that were constantly failing.
That felt bad as an engineer. Yeah, I was so mad. And then... I think a couple of days later, like when I recovered, when I kind of felt good again, it sort of like instilled in me this sense that you can care a lot about something and you can have good engineering reasons for caring about it.
the company might not care at all. And if you're in that position, that's your fault. That's not the company's fault. You either have to deal with that and like accept that you're not going to get rewards for doing this work that you're doing and not expect them or you need to like reprioritize so you're more in line with what the company wants.
that's sort of what i took from that experience um and it took a couple more instances of me being invested in things and kind of you know suffering this misalignment between what i wanted and what the company wanted but that was definitely like the first kind of And I think I was a mid-level engineer at that point. I don't think I was even senior, but that started like the pathway for me to be thinking about these things in the way I do today.
But how did you reconcile this, right? Because I can feel your frustration on why, you know, if you have a test suite, like why I have a test suite which is not green all the time, etc. But in any end, how did you come to understand why?
either was not important or not as important as you thought for these staging tests specifically? Or what was important? What did the company care about that maybe you missed at the time? I think... I made the mistake that a lot of more junior engineers make of seeing something written down as like a procedure or a rule and assuming that it was true.
So I had thought that like this social fact that you couldn't deploy with red tests. So we had to get the test green every time we deployed, which was a lot of work. I thought that was like an immutable fact. And then... So sometime after I got knocked back from this and I stopped maintaining the tests and they were red more often of the time, we had this critical deployment that had to go out and the tests were red. The tests weren't green.
And I remember we were just like, oh, whatever. We'll manually test the red cases. We don't actually need to get the suite passing. We'll just deploy the code. And I was like, oh, yeah, when the company needs code to go out. the company will find a way to get that code out and these like rules that are in place to like keep code quality up and stuff those aren't hard rules those are like soft rules those are rules that will bend uh
if the company has like a clear and present need to get something done around them. I think that was kind of like, that was the end of me worrying too much about those tests, I think. Yeah, and I'm just going to assume that these tests were more flaky.
Less reliable, closer to end-to-end tests, which end-to-end testing suites are always a huge headache for every single company that does them. I always see companies investing a lot and then scaling back and all those things. So there's also that part, right? We're not talking about like unit tests or tests.
That signal, if they break, there's a 100% chance of regression or like 99% chance of regression. No, no. End-to-end tests operating on a small set of accounts such that one test could change the state of the account and fail a subsequent test. Ruby tests, Capybara, and Selenium for anyone listening who's in a similar painful situation. So jumping back to shipping projects and production, in your experience, what are things that get into the way of shipping?
to production emphasis being both on shipping and and production yeah i mean this is another another list of of things we could go through for the rest of the podcast because uh Successful projects, there's a narrow path to success and there's a million paths to failure in my experience. A narrow path to success. Yeah.
Projects want to fail. The default state of a project is to fail. Projects only succeed because people drag them kicking and screaming to the finish line. If you leave a project alone and nobody's worrying about whether it's going to ship, nine times out of ten that project will not ship um so yeah many many things uh to get concrete um one like common category of things is that it's some detail nobody thought of so just to
do a rapid fire of like examples uh from my experience hey we got to like a week before ship and we realized we've forgotten to do the security review or we've gotten to like before ship and we thought Team X was going to build this thing, but actually they thought we were going to build this thing and this thing doesn't exist. Or we were relying on this service, but this service is actually in beta. It's not actually deployed yet.
You know, it's slated to go out Q2 next year. Sorry. Or, hey, this service is in production, but you want to run a thousand requests a second through it and it can only support 10. It's just not designed for your stuff. So it's just... endless endless endless like you're coordinating with some other team and there's a misunderstanding about who's delivering what and in big companies like projects need to go out with
So on the order of like five to twenty five teams coordinating sometimes like There's a lot of cooks that go into like shipping anything in a large company and it only takes like two of them to to be really badly misaligned to sync the whole project. So one thing I've observed at companies like this where it is really complex to ship, as you said, five to 25 teams.
collaborating together, which is a lot. But again, if you worked at these larger companies, the ones that you have are a similar size, it does happen. But there's two ways I've seen people respond to this who are tech leads specifically. One type of tech lead goes like, you know, puts up their hand like, look, this is like too much for one person to do. Our TPM was not on top of it. Technical program manager who typically coordinated like.
not my fault right like you know what can i do and then there there's also the the second least who like you know who don't have this response and and they actually one way or the other go through it but what is your take on this because when
You do have such complexity. It can feel like a full-time job of just coordinating. And, you know, there is some pushback that I hear from engineers, warranted or not, saying like, look, this is not software engineering. This is project management. And my main job is software. engineering, I would need someone else to do it. How, when you were in this situation, what was your kind of response? And also, especially keeping the mind that you do want to ship in the end of the day.
I think there's two coordination tasks that you're sort of talking about there, and luckily they can be done in parallel, or I don't think anything would ship at big companies, no matter what. There's the task of making sure... Every team is actively working on the project and coordinating in communication and agreeing that they're going to ship by a certain date. That's what I see as more the...
the TPM or program manager role of coordination. But there's also in parallel, I think, a purely technical sense in which these teams have to coordinate. And maybe you've worked with different TPMs than I have, but...
In my experience, you need an actual engineer on the team who is a a full-time engineer usually a senior or staff engineer to have the entire technical structure of the project in their head in order for first something to ship and this is something i said in the blog post and this is something that actually was not controversial everyone seemed to agree which is
that projects ship because one single person has the technical context in their head. They don't ship because teams agree and manage to coordinate. teams agree and coordinate because one person among those teams has the whole flow in their head and is able to kind of anticipate problems and answer questions because of it.
And I think a lot of companies kind of formalize that into having a DRI engineer as well as a TPM and as well as all these other things. There's a single engineer that's responsible for like... the technical success of a project. And, and that's, that's the person I'm talking to when I write my blog post, uh, because that's the, that's the position I've been in a lot of times and sort of, you learn, you learn hard lessons.
And DRI being directly responsible individual? Yeah, that's right. Sorry. Directly responsible individual. That's what it stands for. So I like how you're phrasing this, but is it... Am I understanding it correctly that you're saying that if you're the tech lead on a project or you're an engineer on your project aiming to be this operating at the staff plus level?
regardless of your title like you're like okay i want to know what's going on then it is your job to understand the whole project understand your team's work how it ties with everything else and understand all the parts how it can go wrong and you know make sure that These are addressed at least. And then if that's done, you're part of the project should be fine. You know, there's a lot of coordination, communication that you cannot control, but you can control making sure.
do I have this whole thing in my head, understanding it and having clarity and feeling good about it? Is that roughly? Yeah, that's right. That, I think, is what I'm saying. The question of whether it's your job is an interesting one.
I don't actually know like if people typically do this work because it's their job. I think they typically do this work because they care about the project succeeding and they recognize that it's not going to succeed unless somebody steps up and kind of thinks through.
the whole thing end to end. But whether it's your job or not, if you are in this position of shipping these projects, it can feel like thankless work. But in my experience, it is really not thankless work. People do notice and people do talk about it.
And your name will come up eventually as somebody who can be relied on, which will lead you to getting more impactful work, which will lead you to promo packets and so on. At least that's what I've been lucky enough that that's happened to me. So I think whether it's your job or not, you know.
it can sort of be like perform for the job you want right not the job you have maybe like i'm gonna plus one that having been a manager on my on my team i always remembered people who help either they bring up they bring up risks that are on projects but they also bring solutions saying oh here's a risk here's what i think i should do and when they do it enough times i'm like okay this person they're actually
really good at whatever they're doing whatever their level is and the lower the level right the more ammunition that is for the next promotion and also on on uh calibration meetings for promotions or or performance reviews It was a very powerful argument when an engine manager brought up, well, this person, they are de-risking the project. The project shipped on time because of them, because they stepped in, even though it wasn't their job. So it does take, I think, a while to catch up.
managers are usually they want to boast about their direct reports at all these occasions, which is invisible to engineers. I agree with you. It's hard to go wrong with this. So talking about kind of. technology skills or hard skills versus soft skills or politics. A lot of what we talked about shipping to production seemed like it's a bit of a soft skill, understand the business, know the priorities, you know.
make your management chain happy and these things. But how do you think the technology skills fit into shipping? You mentioned they're important, but can you give examples on... how important they are or you know what what comes first right because there's i'm trying to figure out if if in your opinion or your experience is one of them more important than the other
I wrote a lot about political skill because I think engineers typically understand that it's important to be technically strong, and they typically don't understand that it's important to kind of be aligned with the politics. That's why my focus was there. But yeah, if I had to pick one, I would say the technical skills are more important because shipping projects only succeeds. You can only ship a project when you understand it end-to-end.
And it takes a lot of technical skill to be able to understand a project end to end and to be able to pivot and to be able to answer questions that come in from various angles. And one thing that I didn't write about a lot, but I think is really underrated is speed in shipping projects.
not just speed of implementation but speed of reaction, speed of communication. When something comes in out of left field you need to have contingency plans like right away you need to be communicating it right away when there's a blocker you need to be addressing it right away because all these things add up like because there's so many things that can kill a project like the one like you know grand theme that can kill a project is being too slow
Because if you take an extra week or an extra two weeks, that's an extra two weeks that something can happen. You really need to seize your opportunities and try and ship as quickly as possible before anything can go too badly wrong. And you just don't have the firepower to do that. without enough technical skill to be able to see-throw it end to end, which is a lot. Reflecting on this, I'm just thinking back of the best tech leads I remember, especially when I was a manager on my team.
And, you know, they weren't the ones who so whenever I think you solve with the problem, oh, here's a problem. This team says they need another like two weeks to build this stuff. The tech lead that was kind of OK was like, OK. Let me talk to them, see if we can get it down to a week and a half or a week, if we can reprioritize. The awesome tech lead was like, why? What is keeping them up? Let me...
I just looked at their code base, and I don't really see a reason. I actually talked with their engineer, and they said they just need this small ticket done. I already wrote the diff. It's ready to land for them. We'll get unblocked ourselves for tomorrow. And like, it's this.
As you said, the person with the technical skill who can actually go there and understand and challenge, and also sometimes, especially if you're working at a company which has an internal open source policy, meaning you can make changes to their code base, like, yeah, just... just make the change and then the team it's it's it's incredible how technical strong technical skill with the ability of getting things done can really just move things
Have you observed similar things? Yeah, no, I've seen that too. I've seen that so often. One thing I've seen is like the ability to kind of come up with a technical demo, kind of moving mountains, like one demo of like, hey, look at this thing happening could cut through.
weeks of conversation about whether it's possible. And if you just go off and produce that, you can really kind of blow people out of the water. One thing I say a lot is that in the context of like a large company, which of necessity has many
non-technical managers and non-technical stakeholders being technically strong if you work on software is really a superpower because it makes you connected to reality it makes you connected to kind of the real world to the actual product that you're building uh and it's sort of like A lot of the work these other people are doing around strategy and marketing and figuring out what needs to be built, that's a huge multiplier.
like a tech company but it only kind of starts multiplying when it locks into somebody with the technical skill to actually make it happen so if you're that person just by like having a lot of technical surface area you can like unlock the the sort of multiplicative factor of like all these other people in your organization just by being willing to answer technical questions, to make technical demos, and to implement things.
quickly that are bothering them that they think are important. You can really move an organization just by being technically strong if you're willing to put that technical strength in the service of people who have like...
thought hard about what needs doing but don't have the technical skill to do it i see a lot of people who are so technically strong but are pouring that technical strength into like making the code a little bit cleaner or writing tests that are a little better and that can be really useful but Can we zone in on to this specific thing? Because I hear what you're saying is if you are technically strong and...
You can surface in a way that is a lot more helpful to the business, visible to others. But what I'm seeing is a lot of most software engineers are technically strong. And as you say, they might be pouring it into less visible things. What might be some ways that I can... You know, like surface this a little bit more. I think the most impactful piece of advice I can give is just to go out and talk to people. And that's a little spicy sometimes because going out and talking to people.
sort of circumvents some of the process that these large tech companies have in place by which like communication goes through from project managers to designers to engineers. But if you're an engineer and you think you have a lot of horsepower,
and you're looking for a place to apply it and you feel like the backlog on your team is not the most impactful place, sometimes the best thing you can do is to go and talk to your skip level, go and talk to your project manager, your product manager, your designer, and just say,
What do you think needs doing? What do you think I can work on? One thing that I... I do see engineers do this, and this is maybe not the most politically savvy way to use this superpower, but one way I have seen engineers use this superpower is to go to support, to their support counterparts who raise tickets, and to invest that horsepower into making things better.
better for support, fixing bugs that waste time for support. This is not the kind of work that I think is going to be super impactful on a Promo packet, but it is worth doing. A lot of engineers who already do that for support could be pretty well served by taking that and also doing it for design and product instead of waiting for these design and product plans to percolate through to
to epics, to issues, to scoping, because a lot of the time there are quick wins that you can just knock out quickly. I like this. It might make your manager unhappy and it might not be a sensible move depending on the climate of your...
of your team to go around the process like that. Yeah, I like this, but I do think that if you're unable to go to customer support and fix a persistent issue, you know like on the side at all like over a year or two years then like it is a good question of like how effective can you be like i've actually noticed that some of the The most effective tech leads did do, not on a regular basis, but every now and then, or they were able to do this.
So, yeah, I guess if you're trapped and looking for inspiration, this could be one way to try and also just figure out like what the response is, right? If your leadership team is like, oh, you should have not done that. Like, well, do you really want to be working there when you're like fixing, doing something delightful on top? of the job that you have is taken like that. Probably not. Again, it depends on the external climate and all that, but it can be.
I like the suggestion. I think it's worth trying to understand why if your leadership team comes back and says you should not be doing that. Because yeah, maybe they're wrong, but maybe they're right. Maybe there's some existential problem facing the company that you ought to be working on instead. A lot of the time, engineers can spin their wheels. It doesn't matter.
A problem affecting the company can be as serious as it gets, and there can still be engineers spinning their wheels because their team structure doesn't empower them to act. I've seen that happen. Yeah, it's a good question on who is it down to. Obviously, as an engineer, especially when you're senior above, you should understand that.
If you're not as experienced in the management team or your manager is not efficient in telling you like, this is really important. The company is on the line or our product is on the line. Well, yeah, that's a little bit on them as well. That's true. Speaking of trust with your leadership team, as a tech lead, it's really important that you do have trust with your leadership team. And this sounds pretty vague, but it is important. What are good ways you've seen of doing this?
Yeah. So by far the best way to like maintain trust with your leadership team is to ship, is to deliver and is to deliver consistently. So if you're that engineer we talked about earlier, who's like... saving projects and de-risking projects and making sure things get across the line, that builds trust. At minimum, it builds trust in leadership that you're technically strong.
And that goes a long way. That gets you put on higher profile projects. That gets you kind of called in when something's important and kind of starts this snowball. In order to do that, in order to ship on things that your management team cares about, I think you do have to maintain the capacity to do that. So what I mean by that is if...
If you're at 120% on your team's Jira tickets and you're just burning down and burning down and burning down, you're not going to have the capacity you need when something comes in that is more visible and is more kind of higher profile. I do think if you want to like maintain this trust with your leadership team, one big part of that is to kind of keep your regular working cadence at like closer to 80% or 60% and then kick it up to 120, 130.
when something does come across. And that way you can kind of save your powder for when you really need it. I think that's really important. The other thing is to try and communicate like them. So I think a lot of engineers kind of get the sense that talking like senior management is this kind of slimy thing, is this non-engineering value. You're not a hacker if you use words like circle back.
or synergize um and i think yeah maybe that's true but if you want to build trust with your management like you need to kind of play the game a little bit and you need to like learn how they communicate um and then sort of communicate back to them a little bit like that Just to make that really concrete, if you're shipping a project the most important thing, almost like more important than technically delivering, is understanding what the project is for.
So a project can be for box ticking for an audit. It can be for attracting new users. It can be for extracting more money from existing users. It can be for satisfying some VP or CEO's dream of what a future product could be.
many reasons and all of those mean you have to work differently. Products for new users have to be really polished. Products for large enterprise users don't have to be polished at all but the requirements are usually really inflexible. Stuff like that. But you have to know what the
what the project is for. But management won't tell you that unless you can communicate in a way that they know how to communicate back to. So if you go to your manager and you say, hey, can I build a really crappy UI for this?
that is terrible to use, every single manager in the world, particularly in writing, is going to say, no, of course not. You have to build something really nice because they're afraid that you're going to go and build something awful and then point at them and say, hey, they told me to do it.
You know, so you really do have to like, to pick up this like crucial information, you have to be able to communicate this like, you know, is it okay to make trade-offs with user experience instead of saying, is it okay to build a crappy UI? Like you have to be able to talk. to talk like that. It's a hard lesson for some engineers to learn, I think, but it's really important. Yeah, this is very interesting because it triggers memories for me where I was the tech lead on a project.
which it didn't make sense for us to build it. And I talked with my product manager and she confirmed like, no, it doesn't make sense in fact. We're only doing this because there's an executive who struck some sort of deal with this company. We needed to integrate something. And she said, look, it makes no business sense. We tried to talk them out of it. We got details, but it came down to something.
political honestly that they were saying oh we have a deal with this company we cannot not do it etc so it became a little bit of ego and she said like look here's what's going to happen we need to build it there's no way out of it we need to do it as fast as we can
We're going to retire it in a year. We're going to have very, very few users of this thing, relatively. And the PM wasn't telling me. She was like, these are the constraints. So as a tech lead, I kind of turned to the team and told them, look, here's the deal. We need to build this. We can do it quickly. We can skip some of the tests. It kind of needs to borderline work, but we don't need to pour much complexity. And this was a complex project. So in the end, actually, it helped us.
deprioritize a lot of the edge cases it was like don't care just add an error message don't do this and in the end actually what happened is we shipped it we had like a ridiculously low number of users as we predicted and and we retired it in 12 months later the code was deleted And on one end, it was this box-ticking exercise, which didn't make sense, but...
We had to do it. And the shorter we did it, the better it was. It was the first time I realized that you could do stuff that doesn't really make sense. It was a necessity and we could have argued a lot more or dragged our feet about it. But in the end, we're like, you know, let's do it quickly. Ship it. Thumbs up.
I'm really impressed by the trust relationship that you had with your team and with your product manager that you could just be upfront about it and say, hey, we're doing this to check a box. We can trade off user experience, no problem. That's, I think it sounds like it made things easier.
It made that this was a rarely like it was a really like close relationship with the product manager. We were like really kind of one unit. It also helped that we were in a different office location from HQ. I'm not sure if at HQ it would have been said this.
clearly just because then people talk and you don't really you know i think it kind of undermines you know like leadership morale when you say like basically the leadership like we don't agree with leadership and we're right to do so and actually the the data showed that we were right to do so but either way So one thing that is changing software engineering a lot is Gen AI.
It's now in a day-to-day tool chain. Clearly, you're working on some tools that I know you cannot talk about at GitHub, but a lot of software engineers are using tools like GitHub Copilot or some competitors to make their work easier. But I'm wondering, because you're working on both this tooling, you're using it, you're also seeing what developers use, what parts of the software engineering job do you think are becoming easier with Gen.AI tools?
specifically with with getting things done right yeah um well One thing I said earlier was that the sort of cardinal virtue of shipping projects is speed, is getting things done quickly because so many things can go wrong when you delay. And one thing Gen.ai is really good at is speed. It's getting you like 80% of the way and it's getting you like volume kind of for free.
i'm a huge believer in like gen ai for programming i i use it every day i use it all the time um i use github copilot i i chat to copilot in the editor like when i'm on personal projects i i use chat gpt and it's just The level with which you can like go from zero to something is so good. I do think like, it's the kind of thing where I think engineers that tend to work on one code base and tend to not work across.
a sort of broad spectrum can sort of underrate how powerful it is because it's it's it's not going to be better than you at writing like ruby and ruby on rails in the code base that you've worked on for five years you're going to be better at it than that for sure But it will be better at you probably at working on like Golang in your company's load balancer repo that you haven't touched and are unfamiliar with.
It'll be better at you in all these domains where you don't have experience. And because so much of shipping projects requires understanding code and writing usually small bits of code in repos that you're not familiar with, I think Gen.ai can kind of... be this real difference maker there where it can you know you need to build like a page in your company's internal admin tool in order to ship a feature
That probably doesn't have to be super polished. That probably doesn't have to be like, you know, 100% there. Jenny and I can probably do most of that for you. You know, that's the kind of thing where like you can save a day or two and that kind of compounds over the course of a... of a project so uh we had simon wilson on on the podcast who is software engineer he's using a lot of gen ai and one interesting thing he said that gen ai makes him a lot more ambitious so
Is it fair to say that with Gen.ai, if you have it at your workplace, which most companies will have at this point, as a software engineer, you can just be more ambitious and do, you know, either sign up or start doing these projects or just contribute to Project Witch?
before these tooling, you might have been like, nah, too much time. I'm not going to be able to get my head around it. It's a code base that I've never touched. I've never programmed this. Let me just leave it alone. I personally haven't noticed. this huge increase in ambition at work, just because typically the hardest parts of complicated software that has to be correct, the hardest part is that extra 20% on top of what AI can give you.
I would be reluctant, I think, to kind of swan into, let's say, the load balancer, like GitHub's load balancer and say, hey, me and Copilot built this feature that I think is really cool. I wouldn't expect a super positive response. Where I have found like the chat GPT or touch GPT, like Gen AI lets me be more ambitious is personal projects for sure. When I'm working on like.
stuff by myself or learning stuff by myself i almost feel like i can learn anything i almost feel like i can i can get into any area uh like so much quicker than before just because the ability to ask questions of um of copilot of chat gpt the ability to like just ask endless questions of someone that never gets tired of the questions that always gives you like considered answers that's that's sort of the dream right when you're when you're learning something new so i
i love it for that kind of work and i love it for like prototyping and spiking things out and you know i sat down the other weekend and i was like what would it look like if you had five models playing texas hold'em against each other who would win um and i you know Under a day I had that I had that working and I Turns out GPT for I like
it's just uh it's just not something i would have done i wouldn't you know building a texas hold'em poker like simulator to like back this back this app it's just you know i don't think i would have bothered doing it because it would have taken you know multiple days it would have taken a lot of effort but
With LLMs, I just kind of had it up in half an hour. It was great. And this is a little different question, but related to this, do you see parts of software engineering that are getting actually harder as a result of more?
gen ai being used more people using gen ai or or just because of these tools themselves uh yeah i mean i'm i'm in a great spot to talk about this as somebody who's worked on copilot and is working on github models one part of software engineering that's getting harder is building these gen ai tools which is increasingly becoming a large part of software engineering
really hard to make into into working software uh the text text modality is hard to work with uh hallucinations is hard to work with the fact that it's all moving so fast it's a really fun area but man isn't more difficult than the work I was doing before.
on so many different levels. Like even just the fact that many people you work with are kind of struggling to keep up as well. You just don't have this kind of baseline level of understanding that I did say at Zendesk when I was working on apps. We'd had apps for years and everyone had been working on app stores for years. Everyone knew what an app store was. Now I'm at GitHub working on LLMs.
people are kind of learning as we go like what are evals what are what are logits what does it mean to put like a a grammar enforcer in front of the sampler when you're when when you're like you know enforcing this json schema how does that work like so many so many things the learning curve is just it's just so steep so that's definitely getting harder if you actually have to like build these build these tools man yeah that's harder than than standard kind of engineering crud work
Well, and I guess more and more of this over time, I would think it would leak over into software engineering in the sense that these days, if you're a backend engineer. you're kind of expected to understand how like the front end works at a conceptual level you know if need be okay like you're probably not your job but like every now and then you should be able to touch a little CSS or HTML Same way, I would expect the term full stack engine is exploding for good reason.
And I would assume that in a few years, software engineers understanding things like what is a rack pipeline or what does it mean to train a model versus to fine tune it. Again, you probably don't need to do it, but... Just knowing some of these and increasing more concepts will just be part of the day-to-day job, especially as more companies incorporate LLMs into their different projects. Yeah, for sure. Assuming we still have.
rag pipelines assuming we still have fine tuning like i think some of those things are definitely still up in the air i mean even two days ago uh open ai released a new a new way of fine tuning models that isn't low rank adaptation so who knows where it's going to go Exactly. Well, it's an exciting space. Yeah, sure is. Another area that is very challenging and interesting is working across time zones.
You're doing this. You're based in Australia. So I'm assuming you work with teams who are in vastly different time zones. What is your experience working? With teams who are in different time zones, what works well and what doesn't? Yeah. Well, one thing I do want to say just on the topic of me being based in Australia is...
A lot of people got angry about my post about shipping and said that they're going to plant their feet and be really disagreeable and tell their manager to screw off and do whatever. I got to say that the calculus on that looks a little different when you're an Australian engineer.
And you don't live in San Francisco and you don't have, you know, 20,000 jobs that you can walk to from your, from your front door. So I think one thing about being in a different time zone is, is, is I think you sort of don't have this, there are some areas in which.
your experience is not like your US peers, which definitely affects how you work and maybe makes you a bit more prone to kind of do what the company needs as opposed to planning your feet. But in terms of my direct experience working across time zones, At Zendesk, I sort of worked with like teams in different time zones, but my team was kind of co-located, which was nice. And then I had a bit of a rude awakening at GitHub. I was hired as the first Australian engineer.
on my team. Yeah, just me with the goal to kind of grow this APAC group. But I think two weeks after I was hired, there was a hiring freeze. um as there were many hiring phrases around that time and so for for nine months uh it was like five u.s engineers and a u.s manager and me in APAC doing my APAC thing. And that was a very interesting experience. It was a very lonely experience in some ways, but there were...
There were real upsides that I think are kind of hard to anticipate if you haven't been in that position. One of them is that like my mornings were really busy because everybody was like closing up their day and I had to catch up on everything that had happened. but my afternoons were almost all clear. So I was in this position where like every afternoon I could do deep work. And I think that really set me up for success.
I do think like I'm a pretty high agency engineer and GitHub was lucky that it got somebody like me who is happy maybe with a little bit less support. i do think like being the solo engineer for nine months in a different time zone like not for everybody yeah so now having seen you know, like both co-located teams working with different times as also you working by yourself. When it comes to remote work, like...
So talking to engineering managers or people who are organizing this, like what is a strategy that you've actually seen work? the best and and most efficient am i incorrect to assume that it's going to be the co-located teams where there's multiple people yeah definitely uh definitely don't just hire one person
in a time zone if you can. Right now we have like a sort of squad of APAC time zone engineers and a squad of US engineers and I think that works well. One thing I think that's really important for managers kind of Trying to do this and I recommend all managers do this
I think every US company should open many hiring racks in Australia, please. That would make me very happy. But one thing that I think is a really good benefit of this approach and something we've done at GitHub the whole time I've been there is to sort of double down on the ability to have people working.
uh follow the sun 24 24 7. so many many many times when we've had critical projects or like critical bugs we've had the US squad do a bunch of work and then hand off to the APAC squad and then the APAC squad hands off to the US squad and the result is that like it sounds like a silver bullet but you can basically 2x the speed when you're delivering some things where the handoff cost is pretty minimal, like bug investigations, that kind of thing. And we've really been able to deliver
I think some projects that we just could not have delivered if we'd been restricted to working in the US time zone only. Users as well. When a user reports a bug in the afternoon and they go to sleep and they wake up and it's fixed, that feels really good. And that's harder to do with NA engineers. They kind of have to work really late for that to happen. But we sort of got that for free. Like anytime an important bug came in.
just got given to the APAC engineers. And if we were able to solve it by the end of the day, it was just this really good user experience of like, wow, they literally fixed it overnight. Like I wake up and it works. That's amazing. I do think one way in which GitHub is really well set up for this kind of work is that it's very async first, and that's super important for managing these.
these two kind of time zones because you can't all be in meetings. You need to be producing kind of written artifacts and demos, these sort of things that can like speak for you and do work while you're asleep. So definitely like. you need to kind of hire people who are comfortable both writing and reading
documents and demos and that kind of thing. I think that reading part is important. I think it's comparatively easy to hire people who are happy to write memos and write RFCs. I think it's slightly harder to find people who are happy to read RFCs. But that's almost more important on these distributed teams. People need to be communicating async for it to work. And you're now in a, I think of...
An increasingly enviable position in the sense that you are working full remote at a team and a company that does support full remote. We're seeing data that the percentage of these job... is unfortunately just going down a bit. Who knows if it'll stabilize or go up in the future, but right now it is going down. And the willingness of engineers wanting to do that is not going down. So I'd like to ask...
In your experience, what are the practices and the traits that help you excel in a full remote environment? And what do you think you needed to... uh to actually do this because as you said you know you work for nine months by yourself and you mentioned that it's not for for everyone yeah i mean To sort of sound like I see some people talk on Twitter these days, I think part of it is that I'm built different. And I mean that in like a maybe, you know.
not necessarily entirely positive way, but like some people I think need human interaction, need to be like talking with people they work with, whether it's on zoom or not. I think I can go a long time communicating with people. I think that's just kind of.
how i'm put together which makes it a really good fit for me uh sort of first and foremost that's that's sort of how i've like because i i know a lot of good engineers who i think could excel in this in this role kind of flame out because they can't stand being by themselves and that's just not something that's that's that's bothered me so i've kind of had that obstacle like removed which is which is which is really nice um of course then you still have to excel uh and i think you know
what i focused on what i've done well you know we've we've talked about it you can go and read the blog post like but the the one sentence is you know if you're in this enviable position and you want to keep it You kind of have to align with what the company needs from you and you have to take what your job is really seriously. I see a lot of engineers who like have their own goals.
for how they want to work you know they want to have all their code perfectly clean or they want to always have unit tests or they want to work on like front-end stuff only and they sort of push those goals up against the company's goals and that's fine you can you can do that and you can make those decisions
But if you're on a spot that you really want to keep, like working fully remote for a company like GitHub that I love, you know, I have a lot of reason to try and kind of absorb the company's goals and to try and make them my own. i think that's broadly been successful and has kind of put me in this position where like i'm able to keep doing this and hopefully i can keep doing this for some time to come i i like this reflection especially on the part of
you can go a long time without communicating async. I guess a lot of this will just be learning, but is it safe to say that, did you know what it would be like to work proper, full? remote before you did it or or was it like just you know something that you needed to go through to see because as as i understand at zendesk it was more in in office and and together with the squad and this was the first time where it was a proper formal job for you well
Recall the timing, right? Like I moved to GitHub in 2021 and Australia, Melbourne had one of the longest, one of the most extreme lockdowns in... the world so for a decent amount of time before i moved to zendesk i'd been working like co-located in a time zone with people but i've been working from my house i hadn't been working from the office i hadn't seen these people so i think
That was pretty unpleasant in a lot of ways, but in hindsight, it did kind of give me the confidence that like I could do this remote role at a company like GitHub and that I could kind of, I sort of knew that I could work without going into the office. Before, I wasn't so sure how I would cope with that. It turns out I cope really well, but I don't know if you can anticipate that ahead of time. You kind of just have to try it and see. On the flip side, a lot of us who have worked...
throughout 2020, 2021 have likely had that experience. I think people are going to reflect on how it feels. Could you do this for longer? Did you actually thrive? Some people thrive, some people hated it. Yeah, for sure. I do think it was very different being like in the same time zone as people from being alone in my time zone. That was a big shift. That was something that I think I got lucky that I ended up being a good fit for that because had I known that was going to be the case,
I might have been a bit more skittish about taking the role. I probably still would have done it. Oh, yeah. Well, some things you just got to wing it, and it sounds like that's what you did. Let's close up with some rapid questions. So I'll just ask a question and just like bird out whatever comes first to mind. What's an interesting framework or library you've recently used and what did you like about it?
I mean, this is a silly answer, but Ruby on Rails, man. I love Ruby on Rails. It's not new. It's not sexy. But I am a huge Ruby on Rails believer. And it's just... It's the most programmer-friendly way to build large web apps. And its flaws are honestly endearing to me. Is GitHub still using Ruby on Rails? I heard that.
it was started on Ruby on Rails or it migrates Ruby on Rails? So internally it's being used or this is just using on the side? Oh, no. GitHub is still like in very large part still a Ruby on Rails shop. We run a lot of different things for a lot of different services. Sometimes Ruby on Rails doesn't make sense, but if you go to GitHub and you go through github.com and you interact on GitHub, almost every single thing you do is going through Ruby on Rails still.
That's a good, fun fact reminder. What is a book that you would recommend and why? Well, one tech book is actually underneath my mic right now. It's the little book of deep learning. by Francois Fleuret which was sort of something I picked up when I was getting into LLMs and I wanted to have a bit of a better understanding of like the
the sort of maths that was backing them. And it was just a really nice short primer on how it works. It didn't go into too much detail. It was excellent. But to recommend a book that I like. I read every year. I got a shout out, The Name of the Rose by Umberto Eco. I read it every year. It's an incredible book about an Italian monastery in the 1300s where somebody commits a murder and then they have to do an investigation.
I think it's a very funny book, but it's a book about people doing their best in complicated systems, which kind of speaks to me and maybe some of the things we've talked about. So yeah, Name of the Rose by Umberto Eco. Go read it. Well, thank you very much for sharing all these details and being on the podcast. No problem, it was a lot of fun.
Thanks to Sean for the deep dive into the topic of shipping projects as a tech lead with the focus on how to do this at midsize and above companies. If you'd like to read more about Sean's thoughts on software engineering, check out his website or connect with him on LinkedIn, both linked in the show notes below.
If you've enjoyed this podcast, please subscribe on your favorite podcast platform and on YouTube. And if you're interested in more tips on how to do the project as a software engineer, check out the deep dives from the pragmatic engineer linked in the show notes below. Thanks and see you at the next one.