Thank you. Welcome to the StaffManage podcast, where we interview software engineers who have progressed beyond the career level into staff levels and beyond. We're interested in the areas of work that set staff plus level engineers apart from other individual contributors.
Things like setting technical direction, mentorship and sponsorship, providing engineering perspective to the org, etc. My name is David Noel-Romas and I'm joined by my co-host Alex Kessinger. We're both staff plus engineers who have been working in software for over a decade. Alex, please tell us a bit about today's guest. Yeah, today's guest is James Cowling. James is currently the CTO and founder of Convex. Previously, he was a senior principal engineer at Dropbox for almost a decade.
We talk a lot about the unique impact that senior ICs can make. Often the biggest impact we make isn't technical, it's our impact on the team. And when that part is done well, our teams make an outsized impact. It was a real treat to hear how James works with teams, and I think you all are going to enjoy it as well. So let's get into it.
Awesome. James, thank you so much for taking the time to chat with us today. If you could start by kind of in your own words, giving us a little bit of a rundown of like who you are and sort of what your story is up till now. Yeah, hey everyone and nice to be on the podcast. So I'm James, I'm Australian, maybe you can hear, I'm kind of losing my accent. I've been in the States about 16 years and I came out here originally for grad school and ended up staying in the tech industry.
I've been a software engineer for my career, and I have a startup now where I'm a co-founder. The majority of the time, I think, you know, where I kind of developed, I guess, skills and technical leadership was at Dropbox. I was there for eight years.
working on a variety of projects. I think the most notable ones were You know, early in my time at Dropbox where I was the technical lead for the product to migrate us off of Amazon S3 to build our own distributed storage system, which ended up being multiple exabytes, you know.
close to a million hard drives spread across the United States. So a very, very large project with a large budget and a lot of risks. I had the benefit of working with some really, really senior engineers who had been very influential. Other companies had gone on to do other things post-Dropbox as well. ended up being ultimately the most senior engineer at Dropbox towards the end of my tenure, where I was not building a lot of...
software directly anymore at that point. And mostly you're doing things like strategy and manager resounding things like that. A lot of mentorship, a lot of direction setting, a lot of working with junior engineers to develop them and develop teams.
So I went through this kind of transition of just building code, building systems, writing code. I've been a manager a few times, but the sweet spot for me is more the technical lead area where I like kind of working with a team and getting stuff done. Awesome. I definitely think we're going to dive into a lot of that history. But first, given your experience, do you have a sense of what should a typical Staff Plus engineer do?
Yeah. So to start with, there's debate at what it means to be a staff engineer. I'll give my definition. I think when someone... transitions from like a senior engineer to say a staff plus engineer there's a shift in not just magnitude of how you work but also the nature of how you work.
So when you're a junior engineer, you're writing code, you're building systems. And as you get more senior, you're doing harder stuff and you're working more and more independently. I think when you start shifting from, say, the senior engineer to staff engineer, a lot of companies have the same titles.
It's when you start getting involved in what I would call strategy. And that's where you start doing actually long-term direction setting and you're acting far more independently. And so I think a lot of engineers make independent decisions on the how to do things. But maybe they're not making decisions on the what to do. And if they are, maybe they're not making decisions on the why. And when you become a particularly senior engineer, a lot of the work you're doing becomes about the why.
What are the problems facing this company? What direction should we go? What have I observed as issues? And then there's an accountability relationship that changes as well, right? So, you know, the thing, you know, I work with a lot of junior engineers who are very talented. I'll tolerate them saying things like, oh, someone should do blah or complaining about things. But when you get to a certain level of seniority, the buck does kind of stop with you.
more senior people than you in the organization. But when you start transitioning to the staff engineer level, it's where you stop looking around for guidance and stop looking around for someone to blame or someone to... tell you what to do and start realizing that it's on you to actually shape the direction of the company. And so that's what I see the engineers who adapt very well to being a staff engineer are those who are very comfortable with uncertainty, with strategy.
with working with people and with direction setting. And I do know and I'm friends with some very, very good engineers who don't get to that level. And that's just fine. There's some engineers who are just super ICs, these super individual contributors who just are incredible at building stuff, but they don't have an interest in...
trying to figure out what the company will look like in two to five years. They don't have an interest in building a team or even dealing with technical leadership sometimes. They just want to build stuff, which is fine. It's obviously a huge contribution to the organization. But a company does need... some group of people to actually decide what gets done and why. And I think that's often the domain of people on this staff plus level.
Awesome. I'm curious. It sounds like a lot of your experience was gained working at Dropbox, but it feels like you have a very... broad understanding of like what a staff engineer plus does. You know, how do you feel like you gain that experience while working, you know, for one company? Like that tenure seems, it's odd in our industry, I think. And it's interesting to see.
Also, I'm a little bit older because I was in academia beforehand. So I did a PhD and I worked in distributed systems and I had worked previously in research and that kind of stuff. I think. I benefit a lot from mentorship and working with engineers at other companies and inside the company. And so I'd have to pick a random number, but, you know, just say...
Half a dozen to a dozen companies, I've kind of gone and done training sessions there where I've helped develop their senior engineers. And, you know, every company has idiosyncrasies. Every company has its own level system and its own like weird corporate dysfunctions.
But there's a lot of similarities that you start to notice. And oftentimes, these come from kind of the discussions where I'll go to speak to a senior engineer and they say, I just can't get blah to happen. Why is that? And when we start chatting about, well... Let's just assume everyone in the room has good intentions. Let's assume that no one's out to get you. Let's assume that we're all smart people and the people that disagree with you are also smart people. Why is it that?
You know, you can't get the thing done that you want to get done. Right. And that could be because there's a lack of empathy. It could be there's a lack of understanding of what their priorities are. It could be that you're not good at communication, which is a huge. issue for senior engineers right i mean it's all well and good for you to have the greatest idea in the world if you can't convince anyone that your idea is a good idea it doesn't matter
There's no point to being the smartest person in the room if no one else agrees with you. At the end of the day, maybe it's not the nicest metaphor, but this is capitalism here. All that matters is the value you're producing. It's not about whether you're right or whether you're wrong. And so when you start evaluating yourself based on, am I getting good stuff to happen?
in a long-term sense. And that's where you can hone these skills. So yes, you know, I was a Dropbox for eight years. I'm founder of a startup now. I happen to work with, you know. almost all the infrastructure teams of the company and so i got a lot of experience with literally hundreds of engineers at dropbox or more but also engines at other companies and and just kind of picking up on the commonalities between the issues that everyone faces
Very cool. There's a lot of questions I want to dig into there. I think actually, one of the things that I'm curious about, I'll kind of work backwards. You mentioned now that you've moved into sort of founding your own startup, which is... Cool. And a pattern that we remark on almost every episode at this point is that a lot of staff engineers start their careers in small startups and sort of that attitude that you were talking about, about sort of like...
not depending on other people to tell you what to do, but finding the things to do is I think something that often gets developed inside smaller organizations. I'm curious how you have found sort of... going in the opposite direction from a fairly established company like Dropbox into starting your own thing. I'm curious what motivated you to do it. And then to whatever extent you're comfortable sharing, I'm curious to also sort of know more about the startup.
Yes, so there's skills and anti-skills you get from both directions. And you can really hinder yourself if you're not careful in both directions. So I think people probably understand that starting at the big company side, the skill is you learn how to do things properly. You know, just say you go work at Dropbox or Google or Facebook, you see.
There's something called a SEV process when things go wrong and you see good engineering practices and you see what code review looks like. And so you understand how to do things, right? The downside is you may never learn. independence and you may never learn ownership and ownership is just incredibly important and the other thing that you may never learn is just say you go and not to throw shade on google here but say you go work at a company like google where their systems are
incredible, right? And say you go work in infrastructure at Google, you can take things for granted. You can just say, oh, things look this way because that's how things always work. And you didn't realize the amount of effort and the amount of... iteration and the number of mistakes that took to get to there, right? So that the zero to one is really, really hard.
And if you're not someone who's built something from the ground up, it's very easy to criticize other people's work. It's very easy to not appreciate the complexity in building something from scratch. So starting a big company, yeah, oftentimes you'll learn, you'll see how things get done well, and you have access to high-quality peers and high-quality mentors. But you may not learn ownership. The other way around, if you started a small company,
And so what's the failure mode? The failure mode is you may just stagnate in your career because you might not, I mean, there's nothing wrong with going and working at a big company and making a lot of money and, you know.
We're very fortunate to work in this industry, right? Absolutely. The flip side, if you start at a startup and try to go the other direction is one, you will learn ownership straight away because you just got to grab a shovel. Now, when I started at Dropbox, infrastructure was i think seven people right so it was you know and there was not really much management going on at the time you know so it was grab a shovel time then too like there was no like someone should do blah is if you said that
That was on you to solve. But go in the other direction. The big problem I observe is maybe we'll call it hubris. Like every now and then I read a blog where someone's opining with like... is deeply held convictions about how you should design a system or you know how scalability works and they've run a system with seven nodes in it right and they've never like really tried something at scale and they've never seen how things fail it's actually it's very easy to
It's hard to get from zero to one, but there's a great subtlety in building systems that can scale to larger use cases, building systems that can be used by thousands of developers and not... And I think it's certainly possible that you can go work at a startup, learn the ownership, but kind of miss the developmental experiences of how to build things well, how to do things properly, and maybe get a little bit too much confidence in.
how hard it is to do things at scale. And so I guess what I would say is, working at a high quality startup is awesome, right? If you go work at a startup, where there's really good people who are like top shelf, who know how to get things done. I think that's the best case scenario, right? Because you're going to have that ownership, but you're going to have people around to learn from. Working at a really scrappy startup.
you know, with like, you know, a couple of friends from undergrad. Hey, it can be kind of fun. You'll learn a lot, but there is benefit. oftentimes to being in an organization with some more experienced people that you can learn from. Not everything has to be discovered from first principles. Sometimes you can just absorb things organically by being around the right people. Yeah, absolutely.
So one of the things I definitely want to get to is talking about the S3 project. I think any listener to this show, their ears are kind of going to perk up. That's catnip, right? That's the crazy kind of cross-organization project that everyone wants to hear about. Before I dig into that, I guess I need to ask you, are there any...
other projects that you sort of wish you could talk about besides that one? Like, what are some of the other things that you've worked on that you think would be interesting to talk about? Yeah, there's plenty. And there's plenty I can talk about that maybe are more controversial and maybe less unequivocally successful.
Right. And so during my tenure at Dropbox, I worked on a lot of systems there. I mean, the big ones, the call out was this project called Magic Pocket, which was the building the storage system there. We rewrote the sync protocol on the server side. You distributed databases and et cetera. And at one point, I was technically the persistent system. So that was anything involving caching databases, multi-homing data centers, that kind of stuff, big data.
Maybe one thing we can talk about is just coming onto teams where the culture maybe isn't quite good or things just aren't working quite right and trying to figure out why that's the case and then trying to turn them around. My life was a little bit painful where like my job was kind of getting thrown into situations where things maybe weren't going so well. And I had to go and join a team and maybe help turn the team around a little bit. Sometimes successfully, sometimes, you know.
I think we always went pretty well in the end. But there's oftentimes a lot more learning experiences from the stuff that doesn't seem so smooth and glamorous in theory. Yeah, I mean, so I'd love to hear about that a little bit more. And I think we'll kind of circle back to all these points throughout because I think we still want to get to the S3 migration. But when you're kind of asked to parachute into a team, it sounds like...
a lot of the times the challenges that you identified were not sort of technical challenges, but they were people challenges, right? And I think that's something that's going to resonate with other folks. So how do you approach that? What's the playbook? Yeah. So I always have this two major...
it's such a huge topic right i'm gonna say there's two major things that i would call out common dysfunctions in a team one is a lack of alignment around a clear strategy and lack of values alignment i'd say And one is just, I would say, improperly scoped incremental deliverables for the team. Oftentimes people think that, hey, we built this giant distributed storage system.
We just sat down one day with a napkin and drew the whole thing out and then just built it, right? Rather than the fact that it was all scoped out into these little stepping stones along the way and limited incremental value. So I'll talk about the former first, the culture and values and alignment. So oftentimes people will voice this frustrations of, oh, that other team doesn't care or that other team doesn't listen to me or, you know, and I find it extremely implausible.
In most scenarios that, you know, engineers at a tech company who are working really hard don't care, right? Or they're not trying hard enough, right? Generally, it's the misalignment. And I'll give you an example here and I'll try to like, you know, sanitize this example. But I went in and joined the metadata team at Dropbox, so basically the distributed database team. And at that point in time, the team was split between Seattle and San Francisco.
And I think probably no one on the team would be bothered if I said that, you know, there was some tension between those two teams. And the teams just didn't seem to really click. And they either thought the other team... didn't care enough or you know didn't listen and there wasn't a lot of productive work getting done across team because there was this kind of gulf in culturally between them and the first step i mean in any
senior engineer, when you try to get into problem solving is start with introspection and the why, like try to figure out what's going on before you try to fix it. And I spent a lot of time chatting with engineers there about, hey, what is the problem?
And when you start asking around, eventually we found out that one team thought the most pressing issue was the usability of the system, that it was not convenient to use, that it didn't have the right feature set, and as a result, it was slowing down product development of the company.
And the other team felt the biggest pressing issue was that whether the system was going to scale and whether it was going to get too expensive and whether we're going to run out of space and all that kind of stuff, you know, existential infrastructure stuff. And when these two teams were not aligned on the why.
the why we should do things, every other issue became a conflict. They never talked about this with each other. They were never like, hey, we think the biggest problem is the usability. We think the biggest problem is scalability. That was just taken as a given.
And then they'll have arguments about what the next thing is that has to get done, right? So the first task was just, okay, let's define for the team what matters right now. And ultimately, what we had to say was, you know, in the short term, we're going to neglect. clients and we're just going to get the scalability stuff sorted out and then we go so we have to have a definitive this is what matters to the team
These values exercises is something that can either go really well or really poorly, depending on how sincerely you take the process. I really, really hate value statements that say things like, we care about... correctness and performance and being nice to each other and just saying a whole bunch of virtues, right? A value statement that has a whole list of virtues is easy to come up with. It's completely meaningless, right? Because it's not actionable.
The best value statements I've seen is stuff like, we care about, like this, when we build the storage system, we had a set of values and they were, we care about correctness and not having data loss ever. So much so that we will push back on feature development and we will not build a feature if we think it might compromise correctness. Or any value where the opposite of the value would also be a legitimate viewpoint.
is a useful value you know we care about the teams might say hey we care about agility and we are going to move fast and we're going to experiment and as a result we are accepting that we're going to make some mistakes that's the value
right you can't just tack a bunch of virtues on so you know getting teams together and talking about what the you know the values have to come from some kind of why statement why does this team exist so you know once that why statement why do we exist well you know we have to do
Whatever, we have to solve this problem. Okay, what do we value? And once everyone's aligned on the values, the conflicts you have tend to turn into what you call task conflicts. I think we should do it this way. You think we should do it that way. We debate it. No big deal.
engineering is all about arguments right but if the value alignment isn't there they become relationship conflicts right you want to do this i'll do that that's because you don't care or you're not listening right so when people are not values aligned
When you haven't got the why and the values sorted out, the what and the how just don't follow. And because engineers were oftentimes so logical and so caught up in the building, you don't take that step back to ask the why. And everyone's debating the how.
without taking that step back. I had the, I guess, I wouldn't say, you know, definitely not privileged, but I had the burden a few times to kill some big projects at Dropbox. And these are projects that, you know, dozens of engineers worked on for years. One of the big ones that we ended up rescuing and turning around was a project that a lot of really talented engineers were working on, but it just didn't seem to be going anywhere.
I was looking at the project and thinking, you know what? I just don't think this is going to succeed. And when I went around to the engineers on the team and I met them individually, and some of these were my favorite people at the company, and I said, hey, why are you building this thing? And they'd say, oh, well, you know. The VP asked us to or something. Oh, that's what the sprint goals say, et cetera. And everyone individually was going and building this thing.
that when you ask them to justify the why, no one could really justify the why. When we went together, no one, apart from saying, oh, we're doing it because the VP said we should, could justify why we were building this system. I went to the VP and chatted about this, and he didn't want them to do this thing that didn't make any sense. Ultimately, we ended up killing that project and replacing it with something that worked. There was a success story in the end. But these are situations where...
It's so easy to get caught up in the status quo. It's so easy just to do what happened yesterday. And I know I'm rambling a bit here, but it's so easy to take your sprint plans and think they're the gospel. And you take whatever was happening last month and make it happen this month and not step back and say, hey, what the hell are we even here? What is the point of this team? Are we doing a useful thing? And anytime I see a manager like...
overly celebrating how many green checkpoints they have in their sprint goals. I think this is not like necessary senior leadership material, right? That's a very harsh thing to say, but you have to celebrate the value you create.
And incremental value too, right? Obviously, like if you have a big product, it takes a long time to deliver the end customer value. But celebrating just getting tasks done is not at all aligned with value. And I'm going to give you one more story here as I'm in storytelling mode.
The original question before I went on a giant tangent here was, you know, what is something to watch out for to get teams aligned around actually, you know, turning a team around getting value? So this is a self-serving story, but, you know, we built this storage system. It was called Magic Pocket.
And that's the silly code name for the storage system that we should have changed, but we didn't. And eventually, once the system was up and running and working, I was like, hey, guys, we have to go and change our team now. We now are called the storage team.
And those decisions are actually really annoying to execute on. We have to go rename all these directories and mailing lists and Slack channel. I think it might have been HipChat back then, you know, all these channels. And everyone's like, oh my God, this is the terrible idea. James is being Mr. Manager again.
it's completely unimportant what the team is called right and my argument to the team at the time was that if there is a point in future where it doesn't make sense for us to run our own storage system right If economies of scale no longer work out, if no longer we have the competency to do so, if Google comes out with this really cheap storage system somehow that for some reason we couldn't compete with, unlikely because our system was really quite efficient.
This team right here aren't the first people in the company to put up their hands and say, hey, everyone, we're moving back to this system. They failed as a team, right? This is a team of the storage experts. And if their priority at the company is the continued success of the system they built,
They've failed in their job as being the storage experts at the company, right? Their job is to decide the best way to store data for Dropbox. And so it's very important that the team mission is aligned around a task.
or a problem that has to be solved for the organization and not a system, not a sports team. And I would see this happen with teams called like the Kafka team. I think at one point we had this chef versus puppet debate and people were like, you know like if your priority is like managing chef it's an existential question for you whether or not we move to puppet or probably it was vice versa
If your priority is configuration management, no problems, right? You just pick the best tool for the job. So there's a lot of this kind of theme here around this kind of, you know, hand-wavy, managerial feeling can feel like bullshit, frankly, sometimes if it's not done right. Why does this team exist? What do we value? What matters to us? But if the senior engineers or the staff engineers or the tech lead or the manager don't sort this thing out, the team will not succeed.
They'll get something done, but it won't be the thing that you need to get done to begin with. What's the process for it? Because, you know, you hear people talk about value statements and mission statements and things like that. And most of the time it is bullshit, right?
And sometimes it's not, but I guess what I'm trying to understand is like, how tactically did you approach that in the teams that you worked with in order to get around from sort of the vague or useless value statements or the ones that, you know, contain a list of virtues? to like ones that are actually actionable. I mean, I've gone through the exercise and written these things down on a piece of paper, right? It never feels...
Great. It always feels like you're doing some silly management exercise where you're writing down these things. Do you do that as a group with people or what's the process? Tactically, I think the first process is spend a lot of time talking in groups at what matters.
Before we get to that, what are the problems we're solving? What's the point of this team? What matters? Let everyone have their say. Everyone's going to have their own individual opinion on what matters. And it's just so critical. It's non-negotiable. that the team is aligned on what matters and what problem is getting solved. Because if people don't agree, nothing else is going to follow. It's absolutely non-negotiable that everyone agrees.
Should we build this system? Why are we building it? How are we going to evaluate the quality of this system? Does this need to get done really fast? Does it need to get done really correct? How are we going to operate? You know, these things have to happen first. And personally, I mean, this is not, I don't have good tips on how to do this remotely. I am, you know, this is controversial right now. I'm more an in-person work person. That's how I have worked remotely many times.
I prefer working in person. I prefer being around people. I'm sure there's equivalence for remote work where you have these breakout sessions or video chat. I just think like these have to be very organic. very casual interactions where you're just chatting like and that's why i don't think you can have a single meeting where you all come in and
write it and come out an hour later. It has to feel real. It has to be like developing a relationship. We're developing an understanding of what matters. Everyone has to say their thing and have to build shared empathy. Once we all agree on what has to happen. I think everything as follows. The other thing I'll say, other little catch is when doing planning, I always find it very helpful to first figure out what the problems are before we figure it.
how to fix them and especially before we figure out who does them, right? And oftentimes people go the other way around where you have a DRI, the directly responsible individual for this feature, right? So if I make you the DRI for whatever, DRI for caching in whatever, and we do sprint planning, you're going to come up with 17 tasks around caching, right? Because that's your job, right? That's your fiefdom, right?
It's very important. Or if you've got too much to do, you might not come up with the tasks that have to get done because you're thinking about your own time management. It's very, very important we start with what has to get done way before we get to who does it.
So I don't have a great tactical framework beyond that. You have to talk to people. You have to be chatting. You have to at lunch talking about, hey, what matters, et cetera, right? And then get down and write, no, I hate the phrase North Star document or mission statement. Don't be formal about it. I mean, you can if you want, right? But just make sure that we've written down somewhere, like, what is the team? What's the point of the team? Everyone needs to know this, right?
And then, you know, what I've done is, you know, you do this kind of all-hand meetings with your team. You know, if you have, you know, teams that end up being, you know, a couple hundred engineers when, you know, they all kind of layer in. And it can be hard in those large groups because you're not in one room.
You have an all-hands meeting and talk to the strategy. It generally doesn't land, right? You'll say the strategy 17 times, and then you've got to go and speak to people individually and seed these ideas. The nice thing is once you've done this work, Culture is viral. And I call the stuff culture, like the team culture. Yes, team culture is about how you interact and stuff, but there's also a team culture in how you develop code. Team culture, how you think about...
manually responding to operations versus building systems to handle it. I'll give an example where I always realized that culture was really powerful when it came back to bite me. I remember once... Early in the development of the storage system, we had a lot of very strong culture around not being cavalier, not pushing code to production after it's gone through this extended release process and being in a staging cluster, et cetera. And I remember...
authorizing at one point this hotfix. We had to push this hotfix out and we ran the test locally. I knew the code was safe. I had looked at it and then manufactured it and we pushed it to production. And our head SRE at the time, Scott McFiggen, who's awesome. All he said to me was, that's not cool, man. He said it just like that. And I felt shattered. He didn't yell. He didn't berate me. He just said, that's not cool, man. And I just knew in that moment that I had violated the team's culture.
I had done something that was antithetical to how we do things on the team. So once these things are in place, they're very powerful and they're self-sustaining because everyone around agrees with them. But yes, the answer tactically is hard.
Take the time to talk, to interact, write it down, and to solidify it. Every time you have a planning meeting, whether you do monthly planning or biweekly or whatever, the first thing you should do in the meeting is like, hey, let's recap. What's the point of this team?
What are the big problems facing us? I wouldn't necessarily go through the values, you know, and let's use this as a basis for what we're going to do rather than let's start with a big, big basket of tasks and see which ones we finished last month and which ones we didn't and we can keep doing them.
Always start with what's the point of this team? Now what should we do? So I feel like there's a lot. We could keep talking about this, like specifically the sort of like team building stuff. And I think. That's an incredibly valuable point of discussion that I think we all love to dig into. But if I can oversimplify, I feel like we've sort of been talking about sort of like how you get team alignment.
you know, maybe a senior or a staff engineer's role is in team alignment. I'm curious to sort of like maybe turn our direction a little bit in another direction. How do we gain alignment as senior engineers with our organization? You know, I think maybe the magic pocket is a really interesting example because there was a huge, as far as I could tell from the outside, that was a huge commitment of capital. He was. I can't even imagine. So I can imagine like some of the stakeholders in there.
But I think for senior staff engineers, what is the role of a staff engineer in a project like that? How do you know that you're working to further your organization's goals? Dropbox had this levels guide internally, which I think was great. I wonder if we could share it. I'll see if we can share it out. But basically, as you grow in seniority, the scope of your influence gets from a team to all. Eventually, it's the company.
And eventually, your responsibility is to the company. I wish that was for every one of the company. But when you're junior, it's too much to think about. But when you're senior, your priority is not to your team anymore. Your priority is to the organization. And if your team is not aligned with the organization, then you're doing the wrong thing. And so the very first step is to understand your priority is to the organization.
If someone else in the organization who is as senior as you or more senior doesn't agree with you, then you should be priority aligned, right? So then maybe there's an information gap here, right? If someone says, hey, this product's too high risk.
And I'm thinking, well, no, it's not, right? Maybe I'm not thinking very carefully about their priorities, right? Now, the reality is I completely understand where they're coming from, right? Yeah, it is high risk. So then you got to start saying, cool, how can I? architect this product in a way that brings confidence to the organization and so what i've seen not work is engineers say well just trust us you know we're talented we have a good track record
we're just going to do some stuff for two years and then come back and just, you know, that's asking a senior lead to like, you know, maybe make an existential company risking bet on a bunch of folks in the organization, right? Then yes, you have a good reputation, but maybe you're going to. fall through. And so I think it's important to understand the CEO or the CTO or the SVP aren't unreasonable people. They just have priorities that maybe
you haven't considered, right? Like to the shareholders, et cetera, right? So the first step is scoping the product in a way that does not risk the organization. And so I have this blog post. It's a cheesy buzzword I made up, but it's called stepping stones, not milestones. And you can just search on Medium for stepping stones, not milestones, or search for my name. And it talks about how to break a product out into these stepping stones.
where each stepping stone, so the distinction I'll make is a milestone is an art, you know, sometimes you go into these projects and we're going to do this giant two-year project and the milestones are called M1, M2, A, B, and C, and M3. And this meaningless thing that, you know, and achieves very little other than like giving us some checkpoints. But how I would characterize a stepping stone is we have a giant, intimidating, open-ended problem to solve.
the world's most efficient distributed storage system and not lose our user's data. That's too hard for anyone to do. And by the way, most of the system was built with a handful of engineers. I mean, literally a handful. And so...
First step, well, let's build something that we can understand. Let's build a very cheap system that does very simple replication. Once we get to that point, we... the scope of future possibilities is now because now we understand the system better right now we have a deliverable that actually has value right so scoping this program to these stepping stones where each stepping stone brings some individual company value right
is understandable has a clear deliverable and also allows us to scope uncertainty going forwards can really help in product execution so i'm not going to go and say hey in two years time we'll come back with the storage system Talk about, okay, here's the things we're going to build and here's how we're going to make a decision whether this is a good idea as we proceed. So we're going to build this system first and it's going to be not cost efficient.
but it's going to be correct, for example. Now, this wasn't the first stepping stone. This was like 75 stepping stones down, right? You know, maybe seven stepping stones down. At that point, the risk the company's taking is much lower, right? Now the risk is a profitability risk, right? It's not an existential risk.
right you know and here's a stepping stone where we're going to build this and here's how they're going to evaluate correctness right and you know one of the biggest kind of stepping stones or milestones in the project magic pocket project was what we call the dark launch right where basically we built this system, we stored, I think, 10% of all user data on it, and we stored that 10% on Magic Pocket and also on S3, and we ran the system for six months.
And the contract we had, and we actually wrote a contract, which is very silly, but we wrote this kind of document. And Drew and Arash, the co-founders kind of signed it. It was mostly a fun thing, but it was also to prevent us moving the goalposts. Our commitment was we have to prove that we can run this system in production mode with no outages, no issues. Now, we knew that if the system failed, we could fall back to S3.
Dropbox would continue as an organization, right? But we would have failed the project. So, you know, we scoped out these kind of deliverables, right, where at any point in time, there was an ability to de-risk the project. Now, you have to take risks.
to achieve great things and there was certainly risk in the project but you know the product scoping was designed such that it's speaking in the language of senior leadership because you should be doing so too it's not like oh what i really want to do is sit down and spend you know two years coding and then
deliver it but i'm going to pretend to have incremental deliverables to keep the ceo happy right no the ceo's language is the correct language so there's the correct language of how do we not destroy this company while we're trying to do this big project right so scoping accordingly i've
always found that I've never run into irrational, unreasonable leadership. Once there's empathy, once there's education, oftentimes you disagree, but that's probably an example that you haven't de-risked the project enough. I don't mean de-risk entirely, right? Because if you de-risk too much, you won't get anything done, right? It's not about being conservative, but it's about being wise. Yeah, interesting. I know that you do a lot of work with, I'm totally changing tracks here.
You do a lot of work with sponsoring and mentoring. And I'm curious, you know, one, what do you think the value of that is for a staff engineer? And I'm curious, do you have sort of an approach that you take to it? Yeah, yeah.
Yeah, I'm going to answer the question in so far. I think it's obvious what the value is to the mentee, right? It's good to be mentored by someone because you learn some stuff from an experienced person. I'll talk about the value to the mentor and also the value to the organization.
As you get more senior, you evaluate yourself based on the impact to the organization. And what you have to wrap your head around at some point that can be hard to do, your impact to the organization is not just your lines of code written. or the things you build it's that you ask yourself the question you know over the past two years if i was here versus if i wasn't here or if i was here versus someone else was here what was the difference to the organization
What value did my physical presence bring to the organization? And when you ask them that question, oftentimes, the best thing you can do is level up the teams that you're working with. The best thing you can do is develop the people around you because probably they're... faster more agile than you anyway in my case they almost always you're better building stuff faster than me i'm getting real slow at the coding but also that's how then they become and they mentor so
I think every senior engineer or most engineers have a responsibility to develop the people around them. That's how you. create a great ecosystem right it's like i don't know it's kind of like you're having kids and raising them well you know like it's if you're a senior person and you mentor junior people then they become senior people and then they mentor junior people so it's just good for the universe to do right
Now, is it also good from a selfish perspective? I would say also yes. So I give some talks every now and then about technical leadership. And a lot of those ideas come from having to like introspect and articulate things to engineers very explicitly in mentorship context. So, you know, actually meeting with someone, listening to their problems.
giving some feedback on how they might solve those problems really forces you to analyze your own perspectives on this stuff. Because the number one skill you can have as a senior engineer is introspection. There's a slight tangent here, but if I have an engineer in my team who has really good taste and really good introspection and they have values alignment with me, I'm pretty confident they're going to do good work because either they're going to do good work.
by themselves, or they're going to screw up, but they're going to realize it because they have introspection and they're going to ask for help. If you have the introspection, you'll be fine. I find mentorship is a great tool to develop introspection. When you have all these big jumble of ideas in your head, you have to articulate to people, you know, hey, maybe you should think about this. And I've had to do...
There's so many areas we've got to give advice to folks. Oftentimes, it's about leverage. Oftentimes, it's about you mentoring a tech lead and they'll be just drowning in work and they'll be saying, hey, the rest of my team. isn't doing the same amount of work as i am and i'm getting stressed and everyone's freaking out and you have to say okay let's stop for a second right why are you doing this work right don't do that work right why is the rest of the team not pulling their weight
Well, it's probably because you don't have a very clear strategy. It's probably because you don't have very clear direction, right? Your job is to do that stuff. If you don't do it, no one will do it. So having these discussions, I felt really helped hone.
my perspective on leadership and being a senior engineer now one thing i'll say is there's good people to mentor and there's bad people to mentor right frankly right and the good people mentor the people who give a shit right anyone who really wants to put the effort in you know
actually prepares for meetings. I love that when people would come in and say, hey, I've been thinking about what we talked about last week. I'm having this problem. Let's talk about how that's gone. Great. People who want to sit there and listen and just get like a free education, that's not mentorship, right? That's just like listening to a talk, right?
But yeah, I mean, generally, you know, I've been involved in formal mentor programs where, you know, people sign up to get mentored. Most of the time, though, you know, it's informal mentorship. It's just finding, looking around an organization.
That was the thing that brought me the most job satisfaction. Looking around the company and seeing the people who just seem amazing, but they're junior, but they just really give a shit. They really stand out and say, I want to work with that person. Go spend some time with them. Help them develop. I guess I can take credit. I mean, two of my favorite people in this category I work with right now. One of my co-founders, I started working with him when he was 20.
And just started at Dropbox. And he became a very, very senior engineer. He became a principal engineer at the company. Incredible engineer. This is Sujay, my co-founder. But yeah, we worked together very closely when he was very junior.
And our first hire, Braden, who's off the charts, talented infrastructure engineer. Similarly, these are folks I've had the real fortune of working with when they were junior. And then seeing them turn to senior engineers, that's the most satisfying part of the job.
So, yeah, look, it's good for the universe because it increases the quality of talent in the industry. It's good for you personally because it helps you hone your skills. It's good for your career progression and it's good for the organization because it gives you leverage.
But it's also, you know, it's good for the soul. You know, it's fun to work with talented people who care. That resonates. I think your approach in general is going to be pretty interesting to a lot of listeners. We're coming up on time and I wish we weren't. We could talk for hours.
But there's two questions that we try to ask everybody. One of them is kind of following the thread of mentorship. If you were to give advice to somebody who is coming up in their career, and who wanted to sort of learn how to be a better engineer, and especially sort of in the areas where staff plus engineering differs from what they may have experienced before that in their careers.
What are the resources or the blog posts or the people or the podcast or whatever that you would point them toward in terms of sort of areas where they can learn more about this stuff? Yeah. The first thing I'll say to folks like that is the first thing you do is hone your craft. The first thing you do is get good at being a software engineer. Get good at building stuff because you don't want to jump too quick into this leadership business before you have the skills to back that up.
And there's plenty of resources for that. Obviously, there's tons of blogs out there and distributed systems and stuff. The best way of getting good at building stuff is working on the best team in the organization. I've had people say, James, I'm picking between two teams. One...
is doing really exciting work, but I'll be junior on the team. So I'm not sure if I'll look so good. And one is really boring, but I'll be the senior person. I think I should do the second one. That is crazy. Go work on the coolest, most exciting thing with the best people, right? And make sure...
When you're doing anything, understand the why, right? That's the best advice I can give to a junior person, right? If you're not yet in the strategy setting phase of your career, understand why you're doing stuff. Why do they do it that way? Why is that system designed that way?
You'll learn so much by doing that more than any blog you read. So the first thing is to hone your craft. And the best way of doing that is on the job and working with best people. Just find the best people you can find and be near them and watch how they work. The second thing I'll say when you start the transition from, say, senior engineer to staff plus engineer is understanding that the nature of the role changes. It's far more about leverage.
It's far more about how you can empower other people to do stuff. It's far more about how you can convince people of things and far more about how you can make high quality decisions by understanding perspectives of others. And this involves skills that we don't necessarily naturally learn as software engineers, right? These are not the math and science skills. These are the touchy-feely human being skills.
around communication and introspection so personally many people are going to disagree with me but i think if you want to become really good at that developing your eq developing your understanding of others right Never saying the phrase, that team's an idiot or that person doesn't care. Spend the time to understand why they're thinking, hey, if you want to do some reading, read about some psychology. Read about how human beings work. This stuff is valuable. Understanding people.
is really, really valuable because we, you know, messy organisms, right? Like we're human with emotions, right? And I've seen orders of magnitude difference in output of teams between teams that are happy and excited and rallying around a cause. and teams that are drifting and lost, right? We're not robots, right? We're human beings. And spending time to understand motivation, understand how people think, that's what I think distinguishes the very impact for people in an organization.
Awesome. That's a great rundown. So we have our last question here, kind of tongue-in-cheek. How much time do you spend coding nowadays? Oh, like a lot. I'm so glad you asked me this right now because if you asked me any time in the past two years, I would have said effectively zero. But now I have a startup, so I spend 80% of my time coding probably.
And I'm warming up again. You know, I got rusty. I got rusty. And so I'm writing a lot of rough code and writing some TypeScript now. And it's fun. And look, one thing I'll add is that like, yes, it's good to get promoted. It's good to get senior, all that stuff. The most important thing to you, everyone who's a software engineer is probably getting paid just fine.
right the most important thing is that you enjoy your job right so find the stuff that you enjoy and do it right if you really enjoy just building stuff then just keep building stuff it's great now figure out ways to build things in more leverage impactful ways right
But I've seen a number of times people go into management and hate it because they've gotten to this career because they're a builder. Turns out I quite like the management stuff too. So I go back and forth and I'm just fine. But I'm enjoying the coding. There's a reason we got into this industry is probably because we liked actually building systems and creating stuff. So yeah, I'm enjoying kind of getting back to the craft as rusty as I am.
Well, that sounds great. I'm glad you're having fun and best of luck with the startup. Thanks so much again for taking the time to chat with us today. It's really, really been fun getting to hear about your experience. No problems, guys. Like I said, I have a startup now where four people, four timers, so it's very small. All of us have built six million query per second systems and three exabyte systems and stuff.
come work with us. We're looking for really good folks and we'll get a link posted up with a podcast where you can get in touch. So if you want to work on something scrappy and fun, but also, you know, based on good principles, come join us. Awesome. Thanks, James.
That's it. Thanks so much for listening to Staff Inch. If you enjoyed today's show, please consider adding a review on iTunes, Spotify, or your podcatcher of choice. It helps others find the show and is a really useful signal to us that folks are finding value in this so that we keep doing it. You can find the notes from today's episode at our website, podcast.staffeng.com. The website also has our contact info. Please don't be shy.