S is a satisfaction metric. Just ask people, right? Are you satisfied with the tools you use? Are you satisfied with something? Because people can usually tell you pretty quickly. Like I've seen data across an entire tool chain that was pretty fast. Like it was good for the company. But you ask the developers and they're like, this, this is nearly.
impossible. Performance, outcome metrics, we definitely want. A is activity, all accounts, anything that's account. We usually have a lot of activity metrics. C is communication and collaboration.
This could be looking at meeting schedules. It can also be looking at how well APIs are working or if they're breaking. And then ease, efficiency, and flow. Those types of things, those types of data points and metrics give you additional insight. Nicole Forsgren is probably the best-known expert on developer productivity. She is a co-creator at the widely used frameworks Dora and Space, lead author of the book Accelerate.
Nicole started her career as a software engineer at IBM, where she experienced firsthand the challenges of being given unrealistic deadlines by management who understood nothing about developer productivity. Seeing how developers burned out around her, she entered academia, researching DevOps and developer productivity. She then joined Chef Software and then co-founded a bootstrap startup called Dora. Dora was acquired by Google and Nicole worked on developer productivity here.
She later joined GitHub as VP of Research and Strategy, and she is currently leading research initiatives at Microsoft. She's also working on a book related to developer productivity, one that we'll touch on in the podcast. In our conversation today, we cover... Dora and Space, examples of using these frameworks and advice on how to make the most of them. Measuring developer productivity, whether measuring PRs makes sense and why this area remains challenging to quantify.
Traits of high-performing engineering teams and why time to onboarding can be very telling about the efficiency of a team and many more interesting topics. If you're looking for advice on making your engineering team more efficient or are interested in ways to measure developer productivity, this episode is for you.
If you enjoy the show, please subscribe to the podcast on any podcast platform and on YouTube. Thank you. This greatly helps the show to spread to even more listeners and viewers. With this, I'm very excited to have Nicole on the podcast. I wanted to put a pointed question at you. Yeah. You've been studying developer productivity both at the theory level but also very practical level. What is your take on the importance of PRs or diffs?
And the background is here that there's now a renewed conversation about this. There's been this viral threat going around on the concept of ghost engineers who might not be producing as much. kind of like code on a monthly basis, then, you know, that's something that can be measured. And also companies like Uber and Meta, they do have tools for managers and directors to track the number of diffs per team. They don't say that they're going to break it down per engineer. It is possible.
And also there's a new DXCore4 framework, which is something that is going around. And they also suggest that the number of divs at the team level should be measured. I'm curious, how important is the diff frequency and how good of a predictor is it for productivity or unproductivity at a team at an individual level? I love this question. Thank you so much. I will say...
PRs and diffs both are good signals and are terrible signals, which is probably not helpful. But let me dig into this a little bit more. When we think about using the word productivity in particular, many folks especially in leadership, are thinking about the traditional economic definition of productivity, which is the rate of output, right? What do you produce? And then sometimes that'll be a ratio, right? The rate of output per...
inputs. And so as we know, that doesn't apply very well to software or it's really hard to disentangle, right? Is it a feature? Is it a shipment? Is it a full product? Is it... what, right? It's different than in like farming. I grew up, my grandfather was a farmer, right? Like how much do you put in? And then how many potatoes do you get out? Like that's fairly straightforward. And when we think about GDP, that's often what it's looking at. But in technology,
Things are just different because many times you'll get many, many, many more things when we think about cameras and phones or photos, right? We could take a handful of photos. Now we can take millions of photos.
And it's basically free. And so many things in technology are just flipped. So with that context, I will say PRs are important in a few ways. One way is that... it does give you a relatively reasonable view into what work is getting done and what work can get done through the current system and processes that you have.
With a couple important caveats and disclaimers. Many senior engineers don't have very high PR output or code commits. Now, for those of us who have been in software for a while, this isn't surprising. Right. Because as you become more and more senior, you're doing things like unblocking folks. You're doing things like architecture and design work. You're doing things like supporting reviews and mentoring and maybe like pair programming with juniors. And so your commits.
your PRs, yours, finger quote, right, aren't showing up. This episode is brought to you by DX, the engineering intelligence platform designed by leading researchers. Measuring developer productivity is a decades-old challenge, and many technology organizations still struggle to get it right. DX takes a scientific approach that offers complete clarity into productivity and its underlying factors.
It combines data from development tools with self-reported data collected from developers so you can see both the what and the why. The insights from their platform cater to leaders at every level of the organization. from CTOs to platform teams to frontline managers. Learn why some of the world's most iconic companies like Dropbox, Played, Twilio, Vercel, and Gusto rely on DX. Just visit getdx.com slash pragmatic engineer. That is G-E-T-D-X dot com slash pragmatic engineer.
And then you're also pulled into recruitment, your leadership meeting, performance calibration. Oh, there's this team on fire. We need you to help out. The higher you are staff and principal level, it happens all the time. Exactly. Exactly. And so I think it's really important to think about not just what we're measuring, but in which context and who we're measuring because that's not, I wouldn't call that output. I would call that one aspect of throughput.
right? Because it can be really interesting and important to see if there are blockers, why is it? If there's friction, why is it? Is it an overly complicated code base? Is it a really, really difficult to use? tech stack and developer ecosystem? Is it that our PR assignment process is just like bad? Either it only goes to one person or it doesn't get shared out very well. And so one person or just the assignment is a blocker. So there are a handful of things to look at there.
bad as a just straight solo metric, but I think everything as bad as a solo metric. What we always want to do is take kind of a constellation of metrics that help us think. more holistically about the context in which developers are working. What are the trade-offs that they are making? So I've been working with some folks to kind of kick up. We started the Eng Thrive effort at Microsoft.
And it's a way to understand and start conversations at several levels in the business about how can we help our engineers thrive. And we use a handful of metrics across many dimensions. And it was inspired in part by space. the space framework, because then we can look at qualitative feedback, right? Microsoft.
You know, famously has NSAT, which is kind of like a satisfaction qualitative survey. That's like every six months, right? I think I remember when I was there. It was still going on. Yeah, and some of them are coming out quarterly now. 10, 15 years ago. Yeah, some of them are quarterly now so that we can make sure we stay on top of any trends, any changes. But then we also want to capture some objective data from our systems because that can provide additional insight.
And you're not asking people for it all the time. You're not interrupting their work. And so there you're looking at examples of quality and outcome metrics. You're looking at examples of... any delays that may come up. Now, I'm just giving like broad examples. This isn't necessarily what N5 uses, but you know, like build time or deployment pieces or security outcomes or quality and stability outcomes.
PRs and the data that are around PRs, right? So it can be number of PRs, it can be time for PR, it can be the time that awaits from review. can be helpful in this constellation, right? Because then you're also seeing, and you can think about a story of why people are making decisions to do the work that they're doing.
And what are the impacts and implications? And I love that because it can help inspire larger questions around when we're making investments. Now, say investments, whether it's money or capital or time, where should those go? How do we want to balance feature delivery and customer value delivery with internal investments? Because those are so tightly related.
Right. I like to joke. This is our hot AI moment. Right. And everyone's like, we'll build all the AI things. Just everything's AI. You have to go faster. It's real hard to build the AI things when like your pipeline is trash. Like it's super hard if your developer experience just isn't there. I agree. I also feel that the reason I've been thinking, why is this trending right now? Like people seem to be really touchy feely about the fact that you can measure diffs.
This survey was specifically about remote engineers, you know, like potentially doing less. And I feel that what is happening, there's a lot of things that what you're talking about and what people talk about in developer productivity, it's like you have a team that's...
Doing well. Everyone is, you know, doing their work and we're trying to figure out how to do better. What's happening? Where are people stuck? Where are the bottlenecks? But there's another question, which is performance management. which is I have a team and I'm a non-technical manager, or I might be removed from the layers. I'm the director or CEO, and I just want some stats that tell me who's performing well and not. And there's an assumption for some reason for people who are not.
uh working here that i can just look at some numbers may this be output or some scores and i'll see people scored and my question to you how important or is it possible to you know, run a good engineering team without a manager or a lead who is hands-on and kind of understands this context and, you know, like knows that what's going on is kind of involved because I'm kind of seeing that there's a real desire.
from people, investors, founders who are removed from technical details to get some of these stats, some of these developer frameworks so that they can tell without understanding anything, oh, this engineer is good and productive, that engineer is not. A, is this realistic? And B, is this something new? Or have the business always been asking for something like this? I'll start with that last one. Is this new? I'll say it's not new.
Right. Like we've, we've been, we, the Royal way. That's good. Business and researchers have been watching people work and trying to measure how people work and trying to assess how people work with summaries and numbers. for a really long time, right? There's something in research called the Hawthorne effect, which is when- The what effect? I love it. So this gentleman was working with, I want to say it was like some type of production for a process.
And he would come into the office, he would flip the lights on, watch people work, count things, and he would leave. And they would count things and the numbers would be different. Well, so he was like, well, it must be because the lights are on. because i guess there's if i'm remembering correctly right like he turned the lights on so they start turning the lights on but it's not like consistent it turns out like people are doing more like when you're there
When someone is there. When they know they're being watched, right? Athletes tend to perform better when there's an audience, right? As developers, we'll definitely do it. As soon as we know what's measured, we're like, all right, we're optimized for that. Absolutely. And I think this is where we have some real limitations around things. You touched on it a little bit when you said we can measure PRs. So many times people will default to the measures that are...
Easy and quick to operationalize. Number of commits, number of PRs, number of lines of code, number of what can we grab quickly? And I think this is one thing that prompted space. The space framework is counts can be useful. Yes, if you have access to data that you can quickly operationalize and grab, sure, right? Inserting caveats around, like, what's the accuracy, the reliability of the data? How are you gathering it? Are you doing cohort analysis? But also...
that will only show you a few things. And the few things it will show you are the parts of the business that are easy to operationalize. And it will ignore everything else. And so we often want to think about...
You know, S is the satisfaction metric. Just ask people, right? Are you satisfied with the tools you use? Are you satisfied with something? Because people can usually tell you pretty quickly. Like I've seen data across an entire tool chain that was pretty fast. Like it was good for the company.
But you ask the developers and they're like, this is nearly impossible. We are going to break soon because so much of it was in systems. So you'd think it was automated, but every single handoff point had to be manual because they were not integrated. And that's one of the biggest weaknesses for operationalized data is missing that handoff. The boundaries between systems and the boundaries between processes. So people can tell you pretty quickly.
So performance, outcome metrics, we definitely want. A is activity, all accounts, anything that's account, right? We usually have a lot of activity metrics. C is communication and collaboration. This can be looking at meeting schedules. It can also be looking at... how well APIs are working or if they're breaking and then ease, efficiency and flow. And so that's where we're looking at time handoffs. You know, those types of things, those types of.
data points and metrics give you additional insight. And if you can grab at least three different categories, you're going to be in a much better position. Now, you also asked, and I'm kind of tying into the earlier question, which was, Can you evaluate and understand how well a team's doing? Can an edge leader, an edge manager do it if they're not like there? I think at that level, right, when you've got a tech lead, when you've got an edge leader, an edge manager being.
absent and unaware of context, does no one any favors, right? Yeah. At some level though, I think there can be benefit to having a holistic set of metrics to help leaders. guide decisions around which type is an example, what type of blockers or barriers or friction points are affecting most of the company? Would that be a good use of a central resource? What types of barriers?
show up that are probably out of the realm or scope of control for a development team to fix, right? Sometimes their documentation can be something like a central build system. No individual team is going to be able to fix that. Or it will be heroic effort. But the larger the company, the more impossible it is. Exactly. And so this is where I think there can be some value.
Especially when we're doing things like cohort analysis and kind of holistic measures and having it be a discussion where people can get curious and can talk through what they think they're seeing and then go, confirm because it's just not realistic for an executive to meet and be fully embedded with every single engineering team that exists and every single product team that exists and every single marketing team that exists to try to figure out, get a good feel for the business.
This makes a lot of sense. And I really like about space how, to me, that was the first framework that kind of made it really clear that this thing is complex and we can measure a lot of things and we should measure multiple things. And sometimes it will... tell us different stories, but we need to look at the whole picture. But before we go too much into space and what happened before, before space, there was Dora.
You mentioned your first startup. It had a massive impact on the industry. The book Accelerate, I think, had a huge impact as well. I see it on the shelves of almost every engineering leader. And if you've not read it, I think it's a great... read to start with, at least. And in the Dora framework, there are four key metrics. The deployment frequency for D, lead time for changes, change failure rate, time to restore service.
Still today, a lot of engineering leaders who I meet, they read the book. They kind of hold it up and say, like, I've read the book. I've cracked it. What for us, what we need to do to be a highly performing team. Here are these four things. We're going to measure it. And if we have high scores on it. will be there now clearly things have voltzons but i'm interested
Where do you see the usefulness of Dora? So teams who are optimizing for this and getting to this thing, how far can they go? And what is the point where it's time to look at other things? For example, how space and other... uh things evolved from here dora is a lot of things uh dora is a how i kind of refer to the entire research program but a lot of folks just when they hear dora they think about the four metrics and so i think
understanding that there are both of those are important. And here's why. The four metrics end up being, through a lot of statistical tests over 10 years now, like even when I'm not doing the analysis, it ends up being a very good... indicator, a really good signal for how well your development pipeline is working, right? Is it relatively fast? How many things can you deploy?
are the quality outcomes at the end of that. Right. And so for folks who are just starting out, that can be a pretty good place to start. Now, I will say sometimes folks just hate me because they're like, well, this doesn't work here. You know, I'm in air gap systems. I'm in wherever. Find ways to...
accommodate for your environment. I do work with folks that work in air gap systems that use Dora. And so instead of the final... And by air gap systems, can you just tell us for the ones of us who are not familiar with that term? Yeah. So air gap systems are where you have some...
some gap in your deployment process, usually right before final deployment, where there is no... communication between the two sides right there's there's no input you can't just push and it deploys all the way through um so some folks that i work with that have air gap systems is the u.s navy right
It kind of makes sense why they would have that. Right. Many times it's for security purposes, but sometimes it's because you literally have to take something out to a ship or a submarine and install it there, right? That's a good way to ship on a ship. Yeah, right? And so there, you know, we say, okay, well then what is, what do we want to consider our deployment pipeline? When we want to think about optimizing and improving this, what should that be? Now, at some point you do want to consider.
flying out to a ship or a submarine and installing it. But for the teams on the ground, you can go until that pre-prod environment because that's within the scope of control. And that for all intents and purposes, the way they're thinking about it. That is what they're thinking about. Now, the same is true for iPhone and Android apps. You can't push to the App Store every single day. That's not real, right? But you can go to pre-prod.
And that can give you great insights. So don't, don't get super caught up in like precise definitions because we can, we can always accommodate right now. Your other question is how far should you go and what else is there? Now, sometimes folks also tell me that they hate the DORA4 because it just tells them how bad they're doing. But that's where the rest of the research program comes in, right? Because there's a BFD, a big...
friendly diagram that kind of points to the many things that can improve these four metrics. Things like improving automated testing, things like improving continuous integration, things like improving... you know, your cloud deployments and your cloud usage. So it can help. It's a signal, but it's not the action, right? How well are we doing? And it can be good because as you generally start to improve.
you start to improve. If there's a huge drop-off, you should probably take a look at your system, right? So from a signal... perspective, it's pretty good. It's not everything though. And one thing I always try to point out is that Dora only goes from commit to production. Now, back when we were starting this, this was like 2013.
end-to-end software systems had not been studied very much and measured because it's so, so difficult to measure them. And so we focused on that part of the engineering system because we thought we could have some really good impact, deliver some good insights. Also because that's the part of the process that can be highly engineered, right? When we think about writing code, we want things to work better and be faster, but also sometimes we just need to take...
30 minutes to go walk around outside to solve a problem. Sometimes we're sketching on a notebook to solve a problem. Sometimes we're having discussions around prioritization. Once you commit though, you should have opportunities to optimize.
all of that system. It should be high predictability and low variability. And so that is also a nice way to think about that because if your tool chain is really out of whack, if it requires a ton of manual intervention, if all of your tests and your builds are failing, if... It's highly, highly variable. Sometimes it's super fast and sometimes it absolutely breaks and you can't figure out why. It's a great place to start because you can engineer the heck out of it.
But it's not everything. Developer experience ends up being incredibly important. Now it's related because if you can't get feedback on something for two weeks, and then you find out that the thing you committed, you thought you were done with, you were no longer done with.
I have to drop all of my work. I have to reset my environment. I have to get my head back in the right headspace. I have to figure out everything that's happening. So fast feedback is important, but the rest of the developer experience, the holistic developer experience is also important. Can I set up an environment quickly? Can I provision the resources that I need quickly? Can I write code and do local build and test quickly with good insights? And so that's where...
identifying the opportunities to really understand and get to, I like to look at this as a constraints problem, theory of constraints. Where are my biggest blockers? Because I can't fix everything and I sure can't fix everything in the next month. Do I understand correctly that Dora started with developer productivity, if you will, as you said, until we ship, just to get some data from there?
And a lot of the research that has been happening since thinking about space, thinking of the DevX framework, thinking about some new things coming are... are starting to look at not just this but also the other part or get a piece of the puzzle i'm just putting it here as you mentioned developer productivity and how all of this
is affecting one or the other is this a fair way to to say where the space is progressing or how do you see things evolving since dora uh together with space and some of the newer research that you're embarking on yeah um so A lot of people have studied various ways to improve the act of writing software, right? I don't want to say development because development is a lot of things. But when you're coding in an IDE, what are ways that we can improve that?
But I think there has been kind of this continued evolution, both with the tools that we have, with the processes we have, with the things that we've learned prior to think about this from an end-to-end type of a view. So not just commit and not just writing code. Although many people focus on that and they're the experts and they can tell you everything down to the millisecond of how long things take and at what point you lose focus and you break concentration. So I think all of that...
All of these individual pieces are very, very important. And for some folks, myself included, my team, we like to look at the full experience to kind of test what that looks like. And I think it's fair to say that that is part of the evolution of space is, you know, we learned so much about the, you know, commit to prod.
And how else do we want to think about that? And I had been chatting with so many companies for so long that kept saying, well, how do I define a metric then? How do I pick the right thing? Or once you're in DORA and let's say DORA, you know, the analysis, some kind of constraint analysis tells you that.
build is the biggest problem or PR is the biggest problem. Then the question is, well, how do I measure it? And so space was useful here because it can measure the entire developer end-to-end tool chain. Or you can just come up with space metrics specific to PRs.
How satisfied are people with the PR assignment process? What's the outcome of PRs, right? Like what percentage of PRs get through within, you know, a day, a certain timeframe or number of viewers, activity metrics, how many PRs do people... How many PRs are they assigned? Communication, right? Like what does the communication piece look like? And then efficiency, how long does it take? Where are the delays? So you can apply space to a specific component.
as well. And it turns out when I went back and looked, Dora is an instance. It's an instantiation of space. Oh, wow. That's kind of reassuring. Yeah. So deploy frequency is a count, right? That's an activity metric. Lead time to deploy is efficiency and flow. And then change, fail, and time to restore are both performance metrics.
Getting into developer experience, as you mentioned, this is becoming increasingly important, especially at mid-sized companies and larger companies. Maybe startups don't care as much just yet. What does this term mean to you? What is a place that has good developer experience? I think developer experience is what a developer...
goes through, like, what's their lived experience? I'm very, very reciprocal and using the same word. But in research, we like to talk about, like, what's a lived experience, right? What does it feel like? And what is it like to do the job day to day? Now, many times people kind of conflate like a good developer experience with just using the word developer experience. But I do think it's important to think about them at least a little bit separately because the experience is...
just what the work is, right? Could be good, it could be bad, that's what it is. And then if we want to improve it, we can look for ways or we can identify ways that take away from or inject negative... like not wanted things in that developer experience, which often looks like friction, which often looks like blockers and barriers, which often looks like confusion, right? If we can't find information, if we can't find the docs, if we can't find... any of the things then that gives us you know a
an improving developer experience. Or sometimes there's a degradation in developer experience when we roll out. I'm sure people will think about this, right? When a company rolls out a new HR system or a new expense system or a new budgeting system. it's going to slow me down. And that might not be what people typically think of as my role as a developer. But if I need to ever interact with these systems with a type of regularity, you've got an initial learning curve, which is understandable.
If that delay or if that friction remains for an extended period of time, that is going to affect the way I do my work. It will affect the time that I have available. Just as you say, an example that came to my mind when I worked at Uber, when I started. things were pretty kind of free. You could access a lot of systems and there was some logging of accessing sensitive data. When you went to profile, you had to just write quickly while you're doing it and then boom.
And then as Uber grew, obviously they needed more compliance. So what happened is when you wanted to access that systems or even data as I was debugging customer issue, I needed to fill it out and my manager or my director had to approve it.
Now, suddenly stuff started happening. We're like, OK, I need to just fix this bug. And I couldn't because my director or my manager was traveling. And it was like two days because then the skip level. And suddenly I lost context. I didn't get back. And I mean, this.
policy change that had nothing to do with developer. It was just really like... hammering down i never thought of it as the developer experience was there i was like oh there's just you know things but the result was like things were tasks were getting done slower especially these debugging tasks
That's an interesting way to think of it. I don't think we thought of it like that. And I love that you bring up not just the time delay, but also the cognitive implications, right? How does this affect your cognitive load in terms of having to come back to it later and regain context, but also...
Maybe this is just me, but I'll joke that I just end up having to use computers in anger too often, right? Because when it keeps breaking, I can handle so much frustration and still be at my best in terms of creativity. And sometimes it inspires me. But at some point I'm like... Not today. We're done, right? Like I've got to jump over to some administrivia that needs done anyway. And it just doesn't require the type of problem solving and creativity and innovation that great work does.
And I just I can't do great work when I'm just surrounded by friction and interruptions and broken systems and unnecessary process. Some of it's good, but also how can we find ways? to minimize that process, right? Security and compliance is super important. Are there ways where any environment that gets spun up automatically has all of the, or most of the preconditions in the settings done versus
A developer, every single developer having to do it manually every single day and every single time, like multiple times a day. Now, in your experience, because you talk with a lot of different companies, large and small, you also work at some very well-known ones. Again, pretty pointed question. Do you think developer experience in general is better at some of the largest companies, you know, the Facebooks, the Microsofts, the Metas, etc., or at small startups?
Which, you know, like these big companies that I mentioned, they do invest a lot in developer productivity and sometimes smaller companies look at them as envy. They have platform teams, they build custom tools, they have teams that measure all these things and try to improve it.
Startups, you know, small, like two, three, five person startup has none of that. They just kind of do whatever they can. So do you see a difference between them? Because sometimes startups move pretty darn fast. Yeah, I will say yes and no.
a joke I used to make all the time when people would come to me with Dora and they would say, well, this can't apply to me because I'm at a startup or there's no way this could apply to me because I'm in a large company. There has to be something different. I'm like, well, startups tend to be faster. They have less bureaucracy. They have less levels.
Large companies have more funding. They have more resources. They have more ways to help improve what that developer experience looks like. So TLDR, no excuses. But what do I see? I see a mix, right? I've been at a large company that had, I'll Google, right? Probably the best developer experience I've ever seen. It was incredible. I was committed to coding the first day. It was great.
They have really taken time to invest in that and prioritize that and research that, right? I worked with some incredible researchers there that did great works here. Jasper, Emerson. Murphy Hill, he's at Microsoft now. They care deeply about that. And it's evident through almost every system that they have. They do not tolerate friction.
I've been at other large companies and advised, especially I've seen, you know, as I kind of talk to large companies, some of them are very good and some of them are very bad. Like it's... I don't want to say bad. It's not a value judgment, but it's incredibly difficult to get work done there. Why? Because many times they're working in large legacy systems, legacy code bases. They're working with layers of bureaucracy that...
slowly built up over time. And now it just makes it incredibly difficult to do work. Startups have often a slightly different set of constraints, right? So they can move fast, they can pick whatever tool they want, they can just go. That also means they have less infrastructure in place, right? They have to build that as they go. They have to kind of acquire that. If they have to go through security and compliance, there's not a team.
dedicated to security and compliance. At Dora, we were a tiny little three-person startup that pretended to be a 10-person startup. We did SOC 2 compliance because some of our customers required that. Getting a SOC 2... And basically a two, two and a half, three person company. That's a lot, right? Congrats. We did that at Uber, but we had a massive team dedicated people. It stopped work. Now, shout out to Jez because he did the lion's share of that work.
there was no development happening through those those that week or two right it just doesn't happen unless it was like emergency bug fixes or something yeah so there's there there are always trade-offs wherever you go One thing that keeps coming, I just keep asking, because I've also been thinking and covering and being involved in developer productivity here and there for many, many years now. Why is it so darn hard?
And you've been doing this for a lot longer than I have. It just feels, you know, like we're software engineers. We write code. We have an output. Okay, we're people as well, but we're kind of pretty organized people. It feels...
eventually we should we should converge on something that's kind of you know like easy enough to understand and and the space should go down a little bit you know we should understand more and more and there's less ambiguity it doesn't seem to be happening how are you seeing it why is it hard
I think it's hard for a lot of reasons, right? One is that most of the work that we do is, let's say, invisible, right? Like you can't walk down to a production line and see where something breaks. That's part of it. The systems that we use and we work with are increasing in complexity, right? When we think about cloud systems, that changed the way we had to think about...
distributed systems and orchestrating them and architecting them. Now we have AI. AI also changes the way we need to think about the infrastructure and the pipelines and the tooling and the model management, right? How we think about that and how we account for that, just its existence and its emergence with continued growth changes the complexity of the work that we need to do and the supporting systems that we need.
Another thing that can be challenging is many times leaders and execs aren't feeling the pain, right? How many CEOs have been developers? and are currently, so some of them are developers, but also how many of them are currently using our systems to do work and using them for at least a week or two, right? They're not managing their calendars.
Which they shouldn't be. But if they had to feel that pain, I think the motivation would change, right? This is so interesting because when I talk with David Singleton, who used to be the CET of Stripe for seven years, he told me that... Every few months, he takes a few days where he does development work, which I found very interesting because not many CTOs at his level do it. He had thousands of people underneath him. At the same time, Stripe is one that invests very heavily.
into platform teams, into developer tools that they had one of their AI systems early on to help developers. I was thinking, and he told me like, look, if you're not doing this, you're not, he said what you're saying, but very few people do it. So I think that's just a really interesting point. I think anyone listening who's in a leadership position, especially in a C-level position.
If you're not doing it, there's companies that are doing it a lot better and you might be missing some things. Right. And in that case, because maybe for whatever reason, an executive team can't be carving out several days. To do development work, Courtney Kistler is at Starbucks now. I met her back when she was at Nordstrom. She had a phrase that has always really kind of stuck with me and resonated, and it's honor their reality.
If you ask a handful of developers and engineering managers what it's like to work in systems, you don't even need a survey, right? Like if you want to send someone, if you want to chat with a handful of them, you need to honor that reality. You might not agree with it, but that doesn't mean that isn't.
their lived experience of doing work. And sometimes that's going to be the best proxy you can get. But getting it is super important and listening to it and being curious about it is super important. This episode was brought to you by Sentry. Buggy lines of code and long API calls are impossible to debug, and random app crashes are things no software engineer is a fan of. This is why over 4 million developers use Sentry to fix errors and crashes and solve hidden or tricky performance issues.
Centrica's debugging time in half, no more soul-crushing lock sifting or vague user reports like it broke, fix it. Get the context you need to know, what happened, when it happened, and the impact, down to the device, browser, and even a replay of what the user did before the error. Sentry will alert the right dev on your team with the exact broken line of code so they can push a fix fast.
Or let Autofix handle the repetitive fixes so your team can focus on the real problems. Central helped Monday.com reduce their errors by 60% and sped up time to resolution for next door by 45 minutes per dev per issue. Get your whole team on Sentry in seconds by heading to sentry.io slash pragmatic. That is S-E-N-T-R-Y dot I-O slash pragmatic.
or use the code pragmatic on signup for three months on the team plan and 50,000 errors per month for free. Now, most people listening who are software engineers or engineering managers who are pretty hands-on or stay close to the... It seems they will agree with both of us that this is important. Developer productivity is important. It's hard.
If you're above a certain size, at mid-size or above, you kind of need some investment. It's nice to have dedicated folks who help out. May that be provisioning infrastructure, standardizing things, or building systems. The number one question, or not the number one, but a very frequent question I get from people, especially engineering managers, senior managers, directors, is...
I see this as a problem. I think we should invest more into developer productivity or internal tools that help us with developer productivity platform teams. I need to get funding for this, and I'd like to make a case. What is your suggestion of doing so? And I'm asking you as someone who's advised a lot of companies and seen them, what are ways you've seen engineering making the case that...
There should be an investment in platform teams that directly or indirectly address some of these topics. Because, you know, the business, the usual pushback will be engineering expensive enough. We gave you enough resources. You should just start it out as you are. Yep. In general, I would say some of the best ways to communicate include both data and stories, right? If we can have some numbers, that can be incredibly helpful. It can drive it home. If you only have numbers, it...
it becomes abstract. It's just another spreadsheet or a number on a document. Now, if we only have stories, it can be easily dismissed as a one-off. That's just that one thing. Someone is just like a disgruntled engineer, right? Someone, some team just doesn't like, it was just that week. And many times resourcing will come down to like, what are the trade-offs?
You want to take a realistic look. Are your engineers having to rewrite, like re-roll scripts every single time they deploy? How much time does that take? So go around to do just a rough... Like at Chef, we did this early on and we just had a spreadsheet. This does not have to be.
you know, super intense or rigorous. We had a spreadsheet where at the end of the week, we asked engineering managers to like kind of chime in on like about how long certain things took, because then you can do some kind of like rough back of the napkin math to say, this is how much time we're losing in aggregate.
Now, if you want us to repurpose engineers, we can, but this is the impact on the product. Because we can come with a recommendation, but it's going to be someone else's decision. And the more information we can give them about the trade-offs that exist in the system...
the more valuable it will be and then pair it with a story. Now it can be an actual story. It can be a quote of a comment, or it can be a very true amalgamation, like almost a persona of what a day in the life of development looks like. Now. and then what it could look like later. And then it comes back to being a decision around like, where are we going to invest the resources? We're always in a constrained environment. No one has unlimited resourcing, maybe a couple of AI companies, right?
We always have constraints. So what's the best way to do this? I will say one additional thing to keep in mind for folks kind of kicking this off or making decisions about whether they should be investing in this is to acknowledge. both to yourself and to leadership, that there is a tipping point in this trade-off where further investments, explicit, dedicated investments, won't deliver that much more.
progress. Which I think is important because there's, you know, I've talked to execs before who were like, I'm not going to take this meeting. It's just a black box. Everything's broken all the time. And so being upfront about the fact and saying, I know.
that there are so many problems and we could spend all of the company's resources fixing all of it. I know that's not what we want to do. I am trying to optimize our workflow so that we can deliver value to our customers. And I think this is going to be a very important leverage point.
And once we get to a point we'll continually reassess where it's good enough, we will retain these investments, right? We'll keep a platform team, but we won't need additional investment until it starts negatively impacting the business. And that kind of opens up a...
a door to have a real conversation around, like, what are the constraints that we have, right? Don't just ask for headcount. Yeah, I love the both that you're saying, making the case, well, first of all, making about trade-offs, because I think that's a lot I used to work with. The fact that...
mentioning that you can later repurpose teams. And I think just the reality right now in the market, it's all about efficiency. There's now a push with a lot of companies where they're hesitant to hire more software engineers. I do think folks will need to get a lot more creative and also just accept the reality that we've had a really good run of the last 10 years where so many places got almost unlimited investment in these areas as well.
which meant that engineers didn't have to feel some of the pain. We might need to just decide that some of this pain, it is here. Let's solve it. Let's try to work around it. But let's solve the most important pain points. So I feel there might be a bit of a... turning point that a lot of companies, not all of them, were like what you've been used to, you know, getting resources to just building teams that helped it.
either help yourself or or sort it or look at tools that can make it change up your workflow so is this also a little bit what you might be seeing yep yeah absolutely so talking about highly productive companies in now in 2025, some of the most kind of productive companies that us air teams, engineering teams, I would say that just get stuff done.
What are some of the common traits that you see across demand? Are there some common practices that they use? May they be a startup or a team within a larger company? Many times what I get instead from a company is how do I make my company high performing, right? But it really is very contextual to teams, right? Because there are team technologies, there are team onboarding, there are team cultures, there are team norms. So it can be.
a group of teams, right? But I often see very different, let's say like performance profiles, right? Across a company. Now, what makes them great? And you can be great anywhere. I've seen a great team working on mainframes. I've seen great teams working on SaaS software. I've seen bad teams working on SaaS software. I've seen bad performing teams working on... We've all seen it, no? We've all seen this, right? We might have even been that team.
Maybe. Yes. That's a yes. The thing that I see in these teams is a few key characteristics and values. Psychological safety is always in there, right? I'm sure several of us have heard about that. You feel safe taking risks. You feel safe asking stupid questions. You feel safe doing so many things. Another part of psychological safety is...
depending on your team and also knowing that your work has purpose. And that can come back to asking stupid questions, right? Sometimes stupid. They're very important questions as like, how does the project that we're working on right now? relate to the group or the business goals. And does it? I was at a company once where we rolled some of these things out and a few engineers quietly back channel came to me and they're like, I don't think this aligns to any of the key.
You know, the exec OKRs, what am I doing? And so we were able to have a discussion around it does. It's just worded differently. And you're not only impacting your line of business, you're impacting three others or. I don't see how it aligns at all. You should chat with your manager because this might be a good opportunity for prioritization. So psychological safety is one. A very related one is getting curious.
being very open to asking questions and learning more. And I especially see this in large companies, being open to the possibility and frankly, the high likelihood. that the way you do things now is not the best way to do things. Just because something was difficult or impossible to solve five years ago, doesn't mean it hasn't been solved. And so when you open yourself up to the possibility that there could be better ways to do things.
There may be other teams doing better than you. Learning, then you can learn what that is. It offers an opportunity for improvement, right? Like I've worked with teams kind of in larger, older orgs where...
I came in, this was a problem. And they're like, oh, well, but that's not one we're going to look at because that just can't be solved. Because the last time they really put their heads down to try to solve it, it was seven or eight years ago. And it's actually been solved now. I can point you to conference talks, right?
And so being willing to consider and ask several questions ends up being super important. And then tactically, like onboarding ends up being a great indicator, right? How long does it take a new hire to commit code? And how can we improve onboarding practices and environments? Because it often puts a spotlight on things that are really, really difficult. I was working with one really large org once where they're onboarding time for a brand new...
college hire engineer, was the same as a senior developer who just switched divisions in the company. Because the tooling changes, the documentation changes, the context they needed to get. Wow. Was basically non-existent. Those two things shouldn't be the same. Just to be clear, it was a long time, right? We weren't talking a day. It was probably like... We were talking several months. Wow. Okay. That's...
Okay, that's a real unexpected. I mean, that's just terrible. Let's say it how it is. Some of the best teams were at 60 days. Wow. Which is not great, right? No, it's bad. It's bad. It's bad. And... When we worked with them on some general interventions, right, what are some of the concrete things they could do to improve it? There were basic things like improving documentation, improving how easy it is to find documentation.
creating a dummy exercise for a dummy pull request very, very early that has to be completed within the first week or two. What we found is that even if you know, so some research on this has been done at Microsoft, my colleague Brian Hough did a bunch of this, which is really incredible. And shout out to the teams that partnered with us on this. Even something like a dummy pull request, right? Not dummy, but like...
fix a title, right? Ends up being hugely impactful. It increases productivity. Some measures of easily automated measures of productivity through the whole rest of the year, right? There was like a 30 to 50% increase. Because you've gone through the system. Now, people push back. They're like, well, what if you're gaming the system? Great. This is the best way to game the system because then someone has gone through.
every single step of the process, they are now more familiar with it. They've done it early on something that was fine, but like kind of didn't matter. Like they didn't have to have full context of the entire code base. to get familiar with the tool chain. This is just so interesting. It was great. And they could surface obvious problems that other people...
When I joined Microsoft, I joked because some people just knew where every body was buried. They knew where every document was. They knew where everything was. But it's because they'd been there for 14 years and they had a list of bookmarks. Bookmarks are not transferable, right? Now, I mean, the same was true at IBM. The same ends up being true at many large companies and even small companies, because if there's just a handful of you, you can ping somebody and ask.
But it slows things down. When I used to work, I had a dock of the things to do. And I think most people had after a while because there's so many systems.
But what you said is just so fascinating, especially about the onboarding. Because I didn't think, you know, as we're talking developer productivity, I thought about a lot of things. And I think a lot of people are like this. They don't think about how onboarding can predict. But as you said, it's not just the onboarding, but the internal transfer.
And one super interesting thing that might tie back to this is a famous thing, a famous law that a lot of engineering managers will quote is Brooks Law, which says for a late project, if you add more people, it's going to be even more late.
This is true at most companies where onboarding takes a long time. You know, like the project is like estimated two months. If you add like three more people, they'll take a month to onboard and suck time away, etc. But if you're working at a company that's known for super fast onboarding,
externally at least, is meta. New hires will often push something code on day one, day two. They have monorepo. If you move teams, you will probably work on the same code base. If you have a... organization where changing teams means that the person who changed teams in a day, they will push code to a different part of the stack.
Brooks Law might not entirely apply, assuming you can pick up parallel tasks, which a lot of people don't realize. Because I once had this experience at a company where, this was actually at Uber, where people said like, If you get more people on your team, can you get it done faster? And at first I was at Brooks Law, but my director said, no, no, no. Those people, they're onboarded. They know it's the monorepo, the mobile monorepo. They can do...
Productive on day one. And we added more people and we got it done faster. And later I was thinking, how did we, because this is 50 years ago, it was coined 50 years ago. But as you said, I think this is the part where you need to be open. to things potentially changing around you for the better. And that's where the question kind of becomes when someone's onboarding to a team, are they onboarding to the entire system and the entire environment in which they work?
Or are they onboarding to a slightly new team with maybe a slightly different culture and learning a different part of the code base, but everything else remains unchanged? If that's the case, maybe not... similar productivity on day one, but a handful of days to really dig into the new part of the code base, new functionality, but everything else is just done. Whereas sometimes...
Even if you're within the same company, you're onboarding to an entirely new system. You have maybe the same tool chain, but it's configured entirely differently. So it's basically a different tool chain. You have different request processes. You have different documentation. If everything's changed... That's onboarding. That's new hire onboarding. They just already have their login. Yeah. Related to this, cultural turnarounds.
A lot of companies that will turn to experts like yourself and seek out different methodologies know that we've got a problem. We're not very productive. We look at our competitors. Other companies, they seem to be getting stuff a lot faster. We're going to turn around the culture. We're going to go from a... you know, like do whatever it needs to be done. Have you seen successful turnarounds? And if so, how did they happen and over what...
timeframe. And, you know, let's talk about like companies that are like, you know, like not, not, not, not a tiny startup, but like company that like a hundred plus people or even bigger. And I'm more interested in the kind of the generic learnings because what I see is. You know, people at the top, they think like, oh, yeah, we'll just hire some consultants. You know, we'll become a.
tech first company in no time. I've actually worked at a company like this a long time ago, a financial company that kind of advertised itself. And on the ground, developers are just rolling their eyes like, okay, yeah, we're getting, you know, another agile training or whatever that is.
And you feel not much is changing and there's usually a disconnect between the two. And there are some companies that over time, by the way, like they do manage to become a lot more tech first companies, like old school companies who you'd be surprised, you know, like. A lot of banks. are here. I've worked with a lot of banks that have undergone that transformation successfully. Parts of the government. For the ones that succeed.
What do you usually see in terms of success, the time it takes, and what do engineers feel on the ground? You're right. I end up talking to a lot of folks about it. I think it's important to realize that if you're trying to change the tech culture. That's not going to be possible. And by that, I mean, you need to think about changing the culture of the company or the culture of the organization. You can do things to have it and help it influence.
the technology culture but it's really difficult to try to just carve out the technology culture and change the way people think about software without changing the broader culture and i will say one you know I think notable cultural transformation is at Microsoft, right? Satya Nadella has really changed what Microsoft is like compared to the Balmer years. I used to work at the Balmer time. It was very different. Oh my gosh. It's very, very different.
Now, there are a handful of things you could do in general and then specific to tech. I'll probably focus on the specific to tech ones because I'm sure other people are familiar with communication and commitment and those types of things. When we think about technology, I think we have additional opportunities because there we can look at some previous research that's been done. John Shook did some great work. We used to think that to change a culture, you would change.
or to change the way people do work. You would change the culture. You would change the way you talk about it. And then that would just trickle down into the way people build tools and kind of execute their work. What he found is that when you change the way people do their work, that bubbles up to cultural impacts. And I think that's where having an environment where these changes are allowed and supported is super important, right? From that broader environment.
But, you know, as an example, people talk about Numi all the time, but I, the cart factory, but I love talking about it, kind of situating it in companies. If you're in... a company or in an organization or a team or a situation where your tool chain is just incredibly slow. It's very high friction. There's a lot of process to go through. You can talk about doing agile.
And it's going to help to a point, right? People can wrap their minds around it. They can think about what it means. But actually changing the core culture isn't going to happen if you're working in systems that are incredibly slow. No. If you introduce new tools and new ways of working that are actually much faster, that actually make it easier to communicate with others, that make it easier to surface and fix problems early, that make any prioritization decisions that you're talking about.
Fairly easy to get done. Now you've changed people's lived experience. You've changed what it feels like to do the work. And that's paired with larger communications, larger education, larger. you know, commitments to what is happening. Now, commitments is also important because are you committing to improving the difficult pieces of the technical stack and the workflow to get to the desired outcome? Are you investing in
the communication that's required, right? You have to say things at least three times for it to kind of land. And it needs to come from several areas and several levels of the business. And so I think those two together can be incredibly powerful, but it also needs to be explicit. And just being a little bit more specific, say I'm either a...
a staff, senior, plus engineer, senior, staff engineer, principal engineer who joined a company, or an engineering manager, a senior engineering manager. Not a C-level, right? So I can't set the things. In this sense, what is it that I could do? Like I've kind of onboarded, I took it in, I didn't rush to, you know, like copy paste whatever I saw the previous place. You know, this is a frequent criticism of big tech employees joining startup, bringing process, but we didn't do that mistake.
I took it all in and I realized there are things that are just really inefficient. I can see, you know, if I was a CTO, I could do stuff, but obviously I cannot do that. What are ways that I can improve? the engineering culture, the efficiency? And is it going to be what you just said, which is around the lines of starting to change my team's lived experience, for example, and hope it bubbles up?
Yeah, I think it's going to be a mix, right? Just on a smaller scale, right? So what is within the scope of your control? And it'll start by, I love that you said, you know, it's when you've joined, you know, you've done your first 90 days, you've done a listening tour.
You've been able to kind of identify what some of the challenges are. It can be super important to then write up a summary, run it past one or two people to make sure it kind of makes sense. Yeah, sanity check. Meet with the teams. this is what I'm observing. This is what I'm hearing from you. Does this resonate? Does this sound right? Is there anything I'm missing? And then end that without a big call to action, but say,
I would like for us to look for opportunities to improve this. And I would love for you to be involved if that's something that you're excited about. Give people a little bit of time to kind of like marinate in that kind of digest. And then go back and say, what are some ways that we can improve this? These one or two things really stood out to me. What does that mean for you? And then...
So depending on – this is where some resourcing comes in, right? Because we want to change the way that people work. And so that could be creating new education if it's just about changing a process. It can be changing. It can be improving the tech.
And so we can have discussions about either kind of carving people off or maybe setting up a hack day. Sometimes a hack day can be great to just deal with paper cuts. And that already, the thing that's great about that is you get this quick win. It's visible. It's a quick win. it impacts people so they see it and if it doesn't impact me personally maybe it impacts my co-workers so i'm at least seeing progress and then we have continued communication right again what you said which is
talk with people and basically make allies. Because when you start a new company, you have no real social capital beyond maybe a fancy title or if you have some impressive credentials, but not there. And I think what people forget a lot of times, because I've seen so many times where someone on paper, you know, comes in at a high level, tries to make changes.
there's a pushback because, you know, like we just want to do our work, right? Like what is this person trying to do? And then eventually they leave, eventually disappointed that this company doesn't want to change. But what they miss doing is first... of all not realizing it takes a longer time you do want to you know you do want to build a team who who supports you you understand them they understand you and now a team and then later another team joins in and do this so i
I feel like you've kind of conveyed this really important thing. And the fact that it's also not a sprint. In a sprint, I mean, you can do something, but it'll probably be washed away with like a sandcastle on a beach. Right. Yep. And I will say, I have found having aligned incentives ends up being important as well, right? If you say something's important, acknowledge it.
And follow up with that actually being important. So if you're just on the team, sometimes that can be a little tough if you're just an engineer, like an IC you just started out of school, right? But you can thank people. You can highlight what they've done.
You can send a thank you email and CC their manager. And as a manager, you can say, this is something that I find very important and we decided together is important. Any work that's done here, even if it's not feature work, is going to be valued. And then reflect that in any performance reports or one-on-ones because it can be really discouraging to go along with like some transformation effort or fix effort or whatever. And then...
It just kind of to your point, right? It just kind of washes away. It just goes away. So far, everything we talked about might have might as well been two years before because. Before ChatGPT came out, before any of the AI tools went mainstream. But these days, software engineers are using AI tools, coding, ID assistance, and many others. It is changing software engineering as a whole.
How has it changed your thinking about developer productivity and how we should think of developer productivity and developer experience? With AI, I'm getting so many questions now about how can we improve the way developers work? How can we make sure that we don't detract from their work? Does this impact space or DoraMetrics? So I'll quickly say DoraMetrics...
in general won't change, right? Their utility, I expect to remain about the same because we're looking at how well that kind of outer loop is running. In terms of space, I think it's still incredibly applicable. right? Because we want to know how satisfied people are with the development experience. We want to know about performance outcomes like quality and security. We want to know about how many things they're able to get done, the way they communicate.
I think efficiency and flow is also super important because it's changing the way we do work. When we're writing code, we're writing code, but we're also now reviewing a lot more code. It's almost a review exercise instead of just purely writing. there may be opportunities, I think, to expand space, right? Because trust ends up being a much larger factor.
to everyone. So I studied trust with sysadmins and people doing high consequence, highly complex work, and trusting a system and trusting the outcome of a command and a new tool was incredibly important. Developers didn't see that as much because we go through school, we use an IDE, we use a compiler, right? Does the compiler work? People are not asking, does the compiler work? And I understand the underlying math is very different, but suddenly you've introduced this question.
to an everyday, all-day workflow, is this the right output? Can I trust what I'm seeing? So I think it's good to kind of capture that and watch. Now, what does that mean kind of more broadly? I think there, and we're seeing a lot of opportunities to kind of reinvent development in software engineering, right? We're seeing it in the IDE right now. We're seeing it in tests. We're asking questions around code quality. There was a great study done with GitHub.
Office of the Chief Economist here that found that four developers that had been seasoned developers, right? They have at least five years of experience. They were finding that, you know, they're saying that the code is easier to read. They have good approval ratings. They're seeing over a 50% increase in unit tests getting passed. So it can have some really good... And this is just from devs using tools like Copilot or other coding assistants. Yep.
Yep. It's just from that. Interesting. And these were the reviewers saying that, you know, the PRs that are coming in just seem like nicer to work with. And, you know, you can have a PR review cycle that. is done you know we're used to linters why not have an ai take a look and offer some suggestions you know that doesn't mean it's inevitable approved yeah i think there's already like startups are building it but i cannot see this not being the case
Oh, absolutely. It's just feedback loops, right? Right. Some folks, it's not quite on their radar, though, because we are so many times when we think about a developer, we think about someone like sitting there typing, right? Or like it's hackers or something, right? Yeah. It's very IDE focused. But there's so much more to the development tool chain.
Right. One thing that my team and I are focused on that I'm like real kind of bullish about is how do we think about the rest of the software process? How do we think about roles like release engineering and deployment? Because historically. That's been very difficult and something that a person makes a decision on. And it's basically a risk-based decision, and they have to take data from the builds, from the documentation, from the downstream system, from...
so many different places and make a decision, this is an opportunity for LLMs to really jump in, right? Because the person is still making the decision, but instead of spending days... collating all of this data and trying to summarize it. You can instead get a quick summary with links out to what you want to look at so you can confirm or see if there's any like nudge or edge case or something. And then you can make a more informed decision.
So I'm personally excited about several ways that AI can kind of help bolster and support and, you know.
reinvent and disrupt right the way we do the full engineering lifecycle earlier you you told me that some of the most productive engineers uh that that you or teams but but also engineers you've seen are curious but also they're they don't hold on strongly to beliefs they're open to things changing and you gave the example of like you know like a team tried an approach eight years ago and it failed but now it's actually it was possible
Do I sense correctly that it's probably a smart approach for a lot of us engineers to just kind of put away a lot of our reservations because LLMs are just fundamentally changing a lot of things. They'll turn out to do some things wonderfully. Some things will just crash and burn. But unless we do this, we'll probably be that team that kind of close itself down to how things are. Is this the kind of advice you give to?
What would you tell people? Because there are some experienced engineers, they've been around the block, and there's some people who are skeptical. Like, okay, they're just a fancy autocomplete. We're giving them too much credit here. It's not going to change. what we do at the fundamental level or there are concerns that it will introduce errors or security bugs or other types of things that like i would never do right um yeah i i would say
If you're in an organization that is deploying or allowing tools like Copilot and Tab9 and Cursor, really take it for a spin. See what works, see what resonates, see what lands. If you don't like it, don't use it, right? If you only like it for specific types of use cases and specific types of tasks, leverage it for the things that it is good at.
And don't use it for the things that don't fit into your workflow, that don't fit into your mental model. But give yourself a little bit of time to open up what that mental model is. Because it can shift and it can shift in ways that are very, very valuable, right? It can open up time.
You know, now you don't have to do like all of the setup in your code. You can just focus on really solving the problem, really looking externally, really trying to figure out where the blocker is or what the longer term architectural. components should be now when i talk with developers and also when i'm building with with uh ai i see two types of usages one is you have
Just specifically talking about the coding. So, you know, I know what the problem is. I know what I want to do and I now need to do the work. One of them is you have the inline AI assistance, which, you know. show automatic code completion may this be co-pilot a cursor or some other ones and there's a
And, you know, it kind of helps you move faster as you go. It continuously gives you that additional context, but it also kind of takes away your attention a little bit. As soon as you type it, it shows up things. An interesting thing that I talk with a developer who's actually building a game. He uses AI very differently. He says he actually goes to one of these tools, ChatGPT or Claude or one of the others. And he goes there with like a skeleton code.
paste it in with a to-do and it generates and i asked like hey why aren't you aren't you using it in your id and he said like i actually i find that i can focus way more when i I don't have this thing interrupting me all the time, but I'm like, no, here's, I need help. And I want to get your feedback and I'm iterating with you versus going, I'm in my flow state. Are you aware of either some research or even just personal observations on?
how developers can preserve flow? Is it important or can you actually have a flow state either way? Is it something that everyone needs to figure out for themselves? I just want to say that story you started with, I think is incredible because...
someone played around with AI and they figured out the workflow that makes sense for them, that works for them, right? It doesn't have to be, you know, line completions inside the IDE. It could be writing up a chunk of code and then asking it for review. I will say, anecdotally, I've seen folks sometimes adopt that.
type of workflow, or they'll turn off autocomplete or auto suggestions, you know, as they're typing. I haven't seen any hard data on that, although I wouldn't be surprised if there's some coming out. In terms of flow, There are a few things that we've discovered and noticed. So there's a paper that some colleagues of mine did here in MSR around, they use the CUPS model. So it's how does...
how does coding change, right? You write something, you wait for feedback, you have to reassess it, you have to decide what's coming next. And some of that work and early work we did with the GitHub team even influenced how...
the timing of suggestions happens, right? We ended up slowing some down because as you pointed out, if you're getting like suggestion after suggestion, after suggestion, it ends up being an interruption. When I think about... your question around how do we think about flow, I think everyone will...
kind of need to play around with this and see what works for them and what that means, right? Because when you're coding in an IDE, I know sometimes I'll turn it off because I just want to code, right? I did that this weekend because I wanted to play around with some code. And I wanted to be kind of driving and doing all of the writing. But that also is, I think, one helpful way to think about it is imagine a world, we're here, right, where there's just a different way to do the work.
And maybe that means you call it something else in your head, right? Maybe instead of coding, maybe I'm driving and reviewing or I'm guiding, right? Because if we can think about it differently, then we're not comparing it to something that's not the same thing. And we can find what that new workflow is. I like it. I also like your initial suggestion of driving versus guiding. It is just very different. But I do think that a lot of us engineers just need to accept that it's different.
It will change. We'll look back at this 10 years later. People will look back. People are starting their career now. They'll be like, oh, yeah, a little bit low. Anyone who's a coder, it's like, oh, Stack Overflow, it always existed. I mean, interesting enough, now we've kind of stopped using it. But there was a time where it was just a given, and there were people before and people after. So I think we're going to have that.
It's a good and timely reminder. And also, we're just in the beginning of this, right? Like this whole thing, it's just wild how much it's gone in such a short time. I don't think... There's been any technology in the history of software engineering that has spread across the whole industry so quickly and is changing how people work, changing their productivity, what they're doing.
It's still a work in progress. Yeah, absolutely. I totally agree, right? When I think about big, impactful technology shifts in tech, cloud is one, for sure. But it didn't hit... almost the entire industry within a year, right? Like we're seeing AI. And specifically like LLMs, right? To the AI folks out there, AI has been around for years. Anytime you're doing any type of autocomplete in an email or, you know.
IntelliCode, this is all AI. I think LLMs really changed, you know, these foundation models really changed the way we can do work and the types of development we can do. So closing off with some rapid questions. I'll just ask and you fire off whatever comes to mind. The first one is about... coding full-time versus being a researcher you were a software engineer now you're you're a researcher what is the one thing that you miss from coding full-time i miss just like
packing something up and being in the code. The thing I love about research is that it's about identifying and solving problems that are hard to solve where we don't know the answer. Coding is very similar. And I love that. I kind of miss the thought process of figuring out the best way to write the code. What's the best way to structure the code? How do I want to like do some of the work, right? Like I joke, like do the typey, right? I miss those parts of coding.
And luckily, I kind of shifted into something where it still kind of scratches that itch of where are the big problems? What's the strategy I want to adopt, right? There's strategy in writing code, there's strategy in communications and product direction and research. uh research program but plus even if you're not coding you're working with people who do code yes and i like i said i'm i still play around with it but but missing i think a little bit also is like the rush of
writing something for customers that I know is going to be a production system. I had folks at IBM a couple of years ago, contact me and say that one of the systems I wrote there as an intern was still being used. And I was like, awesome. First of all, sorry.
There's just a different kind of excitement around writing code and seeing it deployed and pushed and shipped to customers, right? Which luckily I get a little bit with some of my research. I'm realizing now as I tell you this, like I've found another proxy. I've found a way to kind of duplicate some of my code.
coding just another discipline well and you still code a little bit so yeah what is the next book you're working on if you can tell us yeah uh right now i'm working on a developer experience book And it's kind of geared toward folks who want to think about improving developer experience. They want to learn more about developer experience. It could be...
an IC or an engineering manager. It could be leadership. So we cover things like, I'm writing this with Abinoda. So we're covering things like what type of data should we be thinking about? How should we communicate data? How should we communicate like the ROI, right? A lot of this is communication. How do we want to think about a long-term analysis and data collection kind of?
program. This is not a stats book, but getting people pointed in the right direction, right? Examples of survey questions, examples of types of data, examples of ways to operationalize because... My kind of heuristic is once I've had at least 100 people ask me a question that ends up being in the same flavor of answer, right? It's very, very similar. Something written, something like a book can be really helpful because then...
Everyone can kind of cover basics and then think of the next level questions that they're going to get to eventually. But it's not a good use of time if like I'm a blocker, if we're trying to do something over an hour. What's a practice or a tool that helps you personally be more productive on the day to day basis? Day to day. So day to day at work. I don't want to say a tool. I've got an incredible PM who has absolutely up leveled all of the work that we're doing.
Jason Intimate, shout out. Having a very good PMTPM could just be a game changer. I had an incredible one at Google too. It was just so, so good. But when I think about technology tools, not in my work life, because I kind of maintain this, I for sure maintain like a security boundary. Claude, Claude's incredible. I'm able to use Claude. to do many, many things. And that I would say has been the most notable impact. It is very interesting how...
The competition between LLMs, it's just so great because, you know, OpenAI obviously started all of this and now there's Entropic, Cloud, Sonnet in many ways, you know, like. jumping ahead. But I feel it's just so great for the ecosystem. There's so much competition everywhere and they're all inspiring one another to become just...
So much better. So I feel like as someone using all of these things, we're just in the middle of some of the best time in the sense of like getting all of this, these things for free, because, you know, these teams want to like outclass one another and we're just benefiting from it. And like you said, it's so early that everyone is still learning. And I'll jump back and forth because some models are better for something else. It can change every month, right? Yeah, it can change all the time.
A rubber duck that responds to some of your questions. Sometimes I just want the rubber duck, but sometimes I really want to think through something like, what are my limitations? What am I missing? What are the holes? Okay, no, that's not actually a hole. What else would it be? Ends up being...
really useful. And it's super, super fast. And what are one or two books that you read and would recommend? It's tech related. Inspire by Marty Kagan is great. I'm still making my way through parts of it because it is not a small book. Yeah, it's right over there. It's really kind of solidified and articulated things that I've kind of known, but like in a very fuzzy way. And it's also like introducing some...
new things for me to think about, which I really appreciate. Awesome. Um, another one that I loved, uh, that was just kind of like practical. I just like to dive into like random things. Um, the outlive by Peter Atiyah. It's just, I like to nerd. I want to like nerd out and learn something, which, you know, my partner jokes is like, why aren't you just chilling? I've heard a lot of good things about Peter Atiyah from people working in tech.
And then for a fun book, I always love Ender's Game. It's like an oldie but goodie. So if I just want to escape, I'll just grab a trashy beachy read or I'll grab Ender's Game or Ready Player One or something because it's just... It's just a fun story that I can just race through. They're such fun stories. I feel the movies were fine as well, but nothing beats the book. Yeah. And then for news and articles, Tressie McMillan-Cotton and Anne Helen Peterson.
are probably the most high value rates that I get outside of tech. Awesome. Well, Nicole, thank you so much for being on this podcast. This has been so fascinating, interesting. And I think just like. motivating, but both on how much we've come with developer productivity and also just when we're getting closer to figuring things out, things change again with AI and teams behaving differently. It's an exciting time.
Yeah, it's really fun. This is a great time to be working, especially in tech. Thank you to Nicole for this deep dive into the ever evolving field of developer productivity. You can read more from Nicole on her website or follow her on social media. both linked in the show notes below. In the Pragmatic Engine, we've done several deep dives on developer productivity.
Check these out linked in the show notes below or by searching developer productivity in the Pragmatic Engineering newsletter. If you enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. Thanks and see you in the next one.