The main reason that I like synchronous collaboration with other engineers is because I can get more feedback more quickly without anything else, without any noise in the signal, as quickly as possible. From Tuple, this is Distributed, where we show how world-class engineers and their teams tackle the challenges of remote work. I'm Jack Hanna, your host. Let's get to the real work. Today, I'm talking with Blake Irvin. Blake, thanks for coming on the show.
Blake, I believe your title is observability engineering lead at SumUp. We'll get into what that means specifically in just a few minutes. But one of the reasons I'm really excited to have you on is we've had the chance to get to know each other over the past year and have had...
a bunch of conversations about the type of thing that we're going to be talking about today. And it's really clear to me that you have a passion for fostering high-performing teams. And so I want to explore how you've approached that both in your current role and previous in a remote.
And so could you start off and just share a little bit more about what your role is specifically over at SumUp? What I'm primarily responsible for is making sure that the... the observability experience and observability tooling that we implement and that we build for our internal customers and our engineering customers at SumUp, which is a fintech, making sure that those tools and processes work well for the internal customers.
If it was more generic, maybe you could say I'm like a product engineer. So I'm doing a lot of product thinking, but also like very hands on keyboard, spend a lot of time like finessing or like improving things with other engineers. work a lot with our SRE and infrastructure teams because they support a lot of our internal observability tooling and services. Probably where the remote piece comes in.
most for my role is that as a global company, even though we have very strong local presences in different cities around the world, my customers are spread all over the world. We've got a bunch of folks in North America, South America.
all over central europe um and we're also in the process of expanding into australasia there's plenty of even if we're not technically remote there's a lot of remoteness sort of baked into how the company works yeah let's talk about that a little bit what is some ups approach to remote hybrid in-person work we're pretty heavily locally focused when teams have been built in such a way that they're all in the same local area they're usually in the office
However, there are many teams that for various reasons are distributed. And there are also teams or there are also folks like me that work a lot with folks that are. not in the same geographical region for those of us that do that we're spending way more time on calls or doing remote work so to speak not remote work in the traditional sense maybe but
We're functionally remote because we're not physically present. We're in different time zones, that kind of stuff. I've seen this organization where teams themselves are no longer co-located. There's just this dynamic where people feel more comfortable with teams being distributed, even if...
as a company we have a central premise which folks are mostly in person it kind of depends on the team i guess for example i'm in the platform engineering group at sum up and there's a lot of the work that those teams do is very focused on solving durability or reliability problems. And so having a team that is somewhat geographically distributed is actually a good thing. It means if there's a long run incident, people get more sleep.
We're getting close to being able to have follow the sun coverage for some of the stuff that we're doing. Yeah. So it's definitely very specific, like per team, but having a healthy remote work culture is very helpful, even if you're not officially a remote company, because if you're big enough.
there's always going to be some remote perspective, like I said before. What does having a healthy remote culture mean to you? If there's some ritual or a meeting where anyone's remote, everyone's remote, right? So... It's much better if you have one person that happens to be two time zones away to all get on the call together, to have that meeting.
Rather than to have five people meet in a room and then that one person be stuck to the wall, the literal thigh on the wall, that's just not, it's not a good experience for that person. Maybe if the team's been working together for like, you know. three plus years they know each other well enough that works in the sense of good communication but i think in most cases that person ends up just feeling third wheel in it or they're like an outsider to some degree which we don't really want
And then I would say like the other thing that really it applies when we talk about healthy remote culture is really strong emphasis on communication. So making sure that all the tools and processors that we have and the communication patterns that we use. but those things are really designed for remote work. So clarity is really important, knowing when to send interrupty chat messages versus an email.
I've trained myself to always consider before I hit send on that Slack message to ask myself the question, is this something that needs to be addressed immediately? And if it's not, I either schedule it for just after lunch on that day.
or the following morning so that all of that sort of non-critical stuff gets batched up together and delivered at like good sort of break points in the day because i don't know if this other person isn't sitting next to me i don't know if they're busy or not right now I don't know if I'm breaking flow for them, and protecting flow is super important. Yeah, that's so interesting because when you're in person with all of the people that you work with, you can just observe whether or not...
They're in that state, right? Like heads down, flow state versus getting them to grab a cup of coffee or go to the bathroom, what have you. It seems kind of like you've taken on this. mental load or responsibility, which is sort of managing other people's flow, ensuring that you're being considerate and being considerate's good. But it also sort of seems like... that creates an overhead or burden on you to think that through when ultimately it's on.
I would argue it's on each individual person to manage the notifications, manage their own ability to digest this information. So I'm curious whether or not this is a pattern across some up or is something that's Blake specific. I'd love to hear a little bit more about how you think about that.
I don't know how broadly adopted that is. I encourage people to think this way, though, because I think I started thinking about this when... the 37 signals folks published a pretty long opinion piece about sort of like toxic chat culture and that really got me thinking about the the nature of interrupted communication right because you can't
One thing that you can't tell when you get that little notification, when you get the little dot turns red on your Slack icon or whatever, your Teams icon, there's no way for you to tell whether that's an urgent message or not.
Right. You don't know that information. Maybe you get like a notification up in the corner of your screen, but still, you don't know if that's urgent or not. You don't know if you should be taking a little bit of your precious attention and pushing it up to the corner of the screen. So I do think there is some cognitive load there, but I think it's worth the price because that time, especially maybe this is especially true as of now that I'm in my forties and like.
concentration and flow are harder and harder states to reach and i have more and more like broader and broader responsibilities so it's difficult to get those those core those like focus focus hours or focus minutes it's really important to protect that resource for your teammates so yeah i don't i'm not sure how broadly that pattern is adopted inside the company but i definitely encourage people to try to always ask themselves the question like is this urgent there's actually three questions
The first question is, is this an urgent chat message, right? And if it's urgent, go ahead and send it. If it's not, consider scheduling it. Ask yourself the question, can I schedule this instead? And then the third question to ask is, should this actually be a call? Should I even be using this sort of like high friction communication mechanism or should I just be making a call? And a lot of times we're doing that too. Yeah. I want to come back to that because I think that.
topic of, hey, what sort of communication should be async versus synchronous is oddly contentious, but I think it's contentious because it's not obvious what the answer is. And I think people have personal preferences. that influence the way they think about this. I forget who wrote this thing piece, but there's a, there was like a think piece that came out maybe 10 years ago about the makers versus managers schedule.
with somebody in the software dev space who has written quite a bit. And it's basically saying, if you're a maker, it's really important for you to have that focus time or to focus with other makers on doing something. because the complexity and mental difficulty of the work you're doing is super high and getting to a state where you can reason about what you're trying to do and reason about whether or not you're making progress on solving a problem is very hard to achieve.
but if you're a manager you are you're going broad not deep right you're trying to field lots of requests you're trying to manage complexity you're gating a lot of decisions so as a manager interrupting stuff is much less of a big deal right you're just someone saying like here's a three sentences worth of context about it's a decision we need to make what should we do and you help them decide but if you're a maker and you're trying to build something and you're
or even let's say even like you're debugging which is even more difficult usually you're debugging you're almost at the point where you're able to figure this thing out and ship a fix and somebody says sends you a message that's hey do you want to get coffee tomorrow morning
that doesn't that's not an interrupt that you need right you can ask that question tomorrow morning or ask it at the end of the day but now yes now suddenly you're out of flow the bug's getting away from you and maybe it takes you an extra like 30 minutes
When I talk to people about these communication patterns, I often say, when you're going to send that message, just keep in mind that it's likely that you will take 15 minutes of that person's day away from them just by sending that message. It's very difficult to have.
a less than 15 10 to 15 minute interaction with some communication thing like that because it might only take you five minutes to answer but it'll take you another five minutes to get back into what you're doing and it's yeah it's just super You know what I'm talking about. If you've ever worked in the maker side of the equation, it's just super expensive to get and have that flow broken up.
Yeah, totally. The time it takes to reload the context, reorient yourself, and then get back in. If you can fully get back into the flow at all. What's really crazy about it, too, is usually the more difficult... the work is that you're trying to preserve that flow for, the difficulty of that work often correlates directly with the value of it, like how critical is the bug or how novel is the feature you're trying to build, right? In those cases, that's even more...
of a shame when you break that right because you're like there's some really bad bug you're trying to get that shipped make your customers happy or you're trying to ship something new that's going to be a delighter for everybody it's like what was the old famous old english poem the one of the nail where she was lost and then there's this sort of this chain of things that goes wrong because this one little thing interrupted what was going on and it just goes out of control.
Totally. And I can definitely relate to that because in my last job before I worked at Tupul, I ran a couple of different teams. And so there was a bunch of folks who were part of my organization, and I was fully in that manager mode and not that maker mode. And coming to Tupul... I lead a very important function, but I am also the only person in that function. So I am squarely a maker and I've.
I like been head on with my own experience of that thinking, man, I probably could have done a better job with some of the patterns you're describing right now in my prior role. Just having those interruptions and interruptions myself. So you. talked, though, Blake, a little bit about this being something that you care a lot about and you encourage other people to think about, which seems like very empathetic and considerate of you. I'm wondering...
Like, what aspect of team communication, team collaboration, what aspect of folks working together do you feel like sum up as an organization is particularly good at? I know you've been at the company for four plus years now. it seems like you've had a pretty big impact on the company itself. And I'm wondering if there's any aspect of this that at the organization level, or maybe even at the tribe level, I think you're organized by tribes that you feel like SumUp is particularly good at.
To be honest, I think this is something that we're still actively working on. I'm not sure if I would say that we're great at this yet, but definitely we've seen a big shift towards... we're all on the same team thinking, despite the fact that we have such a big organization, right? So we have thousands of people in the organization and over a thousand people in product and engineering.
I think when I joined, one of the things that I noticed was that we were struggling to make the transition from startup to scale up, right? You're a startup. Everybody knows everybody else. Everybody's working to ship the same stuff. easy peasy right then you get a bit bigger you need to break up into teams or tribes like you mentioned you start to have whether you intentionally or not you start to have sort of like turf that you own it's easy to lose some of that team spirit or
It can get more difficult to communicate with the other teams because when you do need to collaborate on something, these are now people you don't know. There's an incident occurring or something. Suddenly you're in a situation where you need to collaborate. so i think we've we have been doing a good job working on improving that as we get better as we get bigger just figuring out ways to bridge those gaps quickly it's not possible anymore for me to know everybody
in engineering. It's just impossible. I could spend my entire year having meetings with people and I would come around to the end of the year and I would have new people to meet. A different framing that would be interesting is just like, how does that manifest? Where do you see that mindset, that one team mentality showing up? Well, there's two things that we've...
done a lot of work on since I joined that I've been really involved with. So this is probably not a complete picture, but it's definitely an experience that I've had. There were three tools that we adopted that I think were really helpful here. The first one... was implementing open telemetry flavor distributed tracing with Honeycomb. And since Honeycomb is built to be very collaborative, that was a really good way of having data to show how our different teams' services were interacting.
The tracing practice in home comes specifically to try to encourage people to look collaboratively at big distributed systems problems. The second tool that we adopted like a year later was IncidentIO.
which is also really focused on collaboration around incidents, integrated really well with our Slack workflows, and has a whole bunch of additional features that make it easier for us to collaborate when things go wrong. The third thing... which you probably know what i'm going to say the third tool that we adopted although that whether or not teams use this this one a lot really depends on how distributed the team is but for the teams that are very distributed we use tuple quite a lot
which is of course the company you work for. And that was also done for the same reason. We wanted to remove every possible barrier from... in those situations where there was already some communication challenges to overcome when we wanted to collaborate. So things like you have a new team that's in a different city, they join through a merger or an acquisition, right?
Nobody in team A knows anybody in team B. It doesn't make economic sense to be flying people around all the time. You've got work to do and the technical difficulty of the things you're trying to do. Let's take the mergers and acquisition example. which has been a big, like Tuple's been a big player in the teams that have been handling the one big merger that we did. You've got a big complex technical stack for one product that's in a language you don't want to.
It's like a language in stack you don't want to keep supporting, but you want the functionality to remain. So you've got to merge that into the tech stack that you do want, and you've got to take the team that manages stack B and get them to work together with team A, right? Make that merger happen.
everybody needs to be friends all this stuff needs to work you got to solve all these really hard technical challenges and you got to do it while you're running the business I think for that, like that team, especially, or that business unit, that tribe has been really heavily relying on people to get that done because it just removes.
We found that it removes like a lot of barriers that maybe people are not even aware of, but just giving you that feeling that you're like in the room or you're almost like in the head of the other person when you're communicating. So the marriage of those three tools, right?
collaborative incident response, distributed tracing with a really collaborative tool in Honeycomb, and also Tuple. Those things all playing together have been really good for the teams that have a lot of these distributed problems to solve. At least from my experience. Now, I've been involved in all three rollouts, so I'm probably prejudiced, right? But people do rely on this stuff, and it is part of their daily process.
It's obviously nice to hear the Tubal name mentioned, but it makes sense based on everything that you said. When you have such a challenging situation where it's exciting, right? Like you acquire another company, you want their functionality from a business. perspective that makes a lot of sense. But now the engineering teams are stuck with the technical challenge of taking the functionality, applying it to your content.
and it's not just the functionality that you're getting right it's also like the domain expertise right so sure i had a meeting once a while ago where one of the engineers and the team i was on who had the most experience working with one particular backend service had moved to a different team to help run a different project. And the new engineers on the team that I was in were like, we don't actually know that much about running X.
we need to know some of what you know before you leave. So this person scheduled like a one hour meeting with us, talked to us for one hour about the service and then left. That sounds useful. I mean, it was kind of useful, right? But it was only one hour's worth of knowledge exchange. And as you can probably imagine...
What ended up happening later when things broke or things went wrong that were difficult to reason about is that person ended up coming back. And even if I could go through that process again, I would have basically done something closer to link. mobbing or workshopping or something to go through that transition process until people on the less experienced people would have been able to say, yeah, we're comfortable. Go for it. Get out of here. Go do your new thing because.
We feel totally comfortable that we know the things that we need to know, and then we've gotten our brain down from you. Yeah. And I say that in jest because I'm sure that hour was very useful, but totally insufficient for the task at hand. Exactly. Yeah. It's like going to a really great restaurant and having like one bite of an appetizer and then leaving. It's like you can tell that the food's good, but you're not totally sure what to expect. Exactly.
We're going the right direction, but we're not done. The journey is not appropriate. Yeah. So I wanted to come back to something that we've mentioned before, which is this. dynamic between synchronous and asynchronous communication. Obviously, right now, what we're talking about are instances in which that one hour debrief was quite useful, but perhaps something more like mobbing or workshopping, which...
by nature, obviously, asynchronous would have been more useful. How do you think about this dynamic of when does it make sense? to communicate things asynchronously in writing or whatever tool set that y'all use versus getting together and doing something. What I would call a little bit more high fidelity, but certainly synchronous, like pairing, mobbing, a meeting, et cetera. I think a lot of it has to do with how tight you want the feedback loop to be.
Right. So the main reason that I like synchronous collaboration with other engineers is because I can get more feedback more quickly without anything else, without any noise in the signal. as quickly as possible. I think the asynchronous stuff is great for like, things like status updates, right? Hey, FYI.
this cluster we just scaled up by 20% because we observed this pattern in our telemetry. Just so you know, right? But there's no action required from you. I'm not asking you to make a decision. or in cases where we have all the information that we need to make a decision, sort of like another team that came in to us via a merger from the States, one pattern that they use, which I find really helpful is...
If there's like a decision that needs to be made and it's low urgency and maybe requires a lot of people and we don't exactly know who needs to be involved yet, they will just, in brackets, start a thread in Slack, which is our chat. of choice, right? They'll start a thread in brackets that will summarize what the decision is that needs to be made. Then as the first message in the thread, they will lay out the data that we have so far, and then people can weigh in.
maybe it's just like an emote right like if we get enough thumbs up redo it that kind of thing but in cases where we don't know we don't have all the information yet we're still experimenting or There's a lot of unknowns. I would much rather be synchronous. It's just, I can, of course I can be synchronous. I can open pull requests.
That process is always much slower. And you really have to ask yourself, can you afford that slowness? Especially if you're talking about not getting feedback until something's chipped, right? Because then you're in like escaped defect hell. Or if you've ever worked on an open source project, right? Open source projects, by their nature, because you've got volunteers working on them, the feedback cycles tend to be very slow. And that can be incredibly frustrating.
Nothing against open source, right? None of us would probably be here. The software world would be very different if we didn't have open source projects. Absolutely. If you've ever had that experience, you know exactly what that... low like highly asynchronous workflow looks like and it's really a tiny change can be something where maybe you're a maintainer you get assigned pr review you go look at this thing it's four line change
but you haven't touched that code in like eight months or a year and a half. And so it takes you like two hours just to figure out what is even going on here before you can have like meaningful, like constructive feedback. So that's like an extreme example of slow feedback. But for me, like if I'm trying to feel my way towards solving a problem, you know, building something new, some new feature to solve some problem or some new behavior, like configuration behavior to solve a problem.
Having another practitioner shoulder to shoulder with me that I can bounce those ideas off of. Or in some cases, something I used to do a lot when I was working in the Bay Area was at some point in that process, last couple of companies that worked out there, we were kind of doing like.
two four-hour pairing blocks a day like four hours in the morning four hours at night four hours after lunch and a lot of cases the product person for the that feature would just come over at some point in that session to see how we were doing
And that was way better than waiting for a Jira notification to pop out and then a comment and then saying, did you actually mean this? You can get to the bottom of all of those questions maybe 10 times faster if you're just having a conversation. You don't like going back and forth in Jira comments?
The best thing I can probably say about that is that if you have, let's say, upper management that is struggling to trust you as a team, Jira Commons can be a good way of demonstrating that something's going on. I think if that's how you need to build trust with your managers, you've got deeper management issues that you need to be looking at. And Jira's not the solution in the long run.
I can see the H1 for the new JIRA website now. We helped build trust with management. Yeah, yeah. I mean, it is important to document your work, right? And it is important to broadcast that information.
In my experience, even the process of updating the work tracking tool is better if I'm doing it with someone else that is working on that problem with me, because we can check each other and say, hey, do you think... person a is really going to understand that if we say it like this they don't really have any experience with the thing that you're using as a metaphor and then you can be like oh i actually didn't know that great so let's clean that up and now that
the most snippet of information that you're feeding into the tracker is like much more meaningful yeah totally like needless to say being it At a company that makes remote pair programming software, we pair on a lot of stuff and not just software. We do a lot of things in pairs. Not everything, but we find it very useful for the reasons that you're describing. It broadens your perspective, helps you stay on task.
There's myriad benefits, obviously. People will ask me... One thing I noticed when I started working in tech in the EU was that I ran into a lot less folks that had done a lot of pairing at work. It was much less... uh common and if we started to experiment with it which i did in like maybe three or four or five different companies since i've been here people would a lot of times ask me questions like what kinds of things should we actually be carrying on
right does it really make sense to pair all the time and i was like no not really just pair on the stuff that's really important and then i would like smile at them right And I would say, you understand what I'm saying here, right? If you're not pairing, what are you doing? If you don't need to be pairing, what are you doing? Are you doing some kind of like trivial task that could be automated?
If that's the case, then maybe you shouldn't be doing that. Maybe you should find a way to get rid of that toil so you can focus on the really hard stuff where you do need two brains because it's so novel and so difficult and so important to fix. It's not necessarily helpful to have like dogmatic rules about pairing, but I think asking yourself a question
Am I doing work that's important enough to need multiple brains is a good question to keep asking yourself on a regular basis so you don't ride it all too much into less valuable stuff. Yeah, I like that. That's good. I think we could.
Recorded an entire episode talking about the merits of pairing, when to do it, when not. That is not what we're going to do today, but I love to kind of touch on it. You're tempting me, though. You're really tempting me. Yeah, well, and you're tempting me, which I like. A topic that we haven't gotten to in earnest that I'd love to get your take on is around human connection and particularly how to foster human connection when teams are distributed. Because I find that this can be...
in some cases overlooked, but then other cases addressed in sort of very cringey and cheesy, cheesy ways. And so I want to start by just challenging the base level assumption of this question with... How important you think it actually is to consider human connection when fostering an effective distributed team? I think you cannot overstate how important that is. I've been in a couple of cases where there were like...
teams that were geographically distributed for whatever reason. And there was some sort of like interpersonal friction, right? Sure. And the experiments that we did where we, this is not just at SumUp, but it's, we've also done this at SumUp. The experiments we did where we found a reason to bring those people physically together just for a brief time and not give them too strict of an agenda just let them be together maybe do some work together this is one also one thing that i think is very
important it's like getting people every once in a while getting people physically together even if they need to fly to fly in or something or traveling is super important but The most successful experiments that I've seen like this were ones where we were doing some work together. We were not doing like party games or something, right? We were behaving like adults.
We were working together, but we were making sure that the work was very low stress, like high value, but low stress. So it's sort of like strategic planning kinds of stuff, or maybe like architectural sessions, things like this. So that people have a chance to like have that.
extremely human sort of like interpersonal relationship while they're solving a problem together if you've got distributed teams you can't do this all the time you do need to figure out how to make this work a big part of that i think is figuring out how to foster communication or team dynamics where there's a lot of...
jokes getting made basically having a sense of humor is super important it's it's really it's really not healthy if you only talk about work and i also know some other folks that have highly distributed teams that are big on having agenda-less calls on a regular basis, right? So like everybody logs into me or Zoom or whatever tuple, I guess, if they have tuple, you know, there's no agenda.
it's not a fixed thing maybe you end on some kind of agenda topic but you have a big chunk of time where people can just be together and i think that's super important and don't force it i think if you're always doing scrum.org type party games people know what that is and they i don't think you fully get an open response from people when you do that it's better in some cases to just be patient not plan anything and let people be yeah create the space for it
to happen but don't try to make sure yeah totally you don't manufacture it and you've got to like create that space but also protect that space put walls around that space put walls around that's that block of time that you've got in everybody's calendar
to make sure that they can do those things together. There's other ways to do this too. There's like activities that people will recommend. Like you go out for drinks together if you're physically present or you game together if you're distributed, but you also have to be careful with that stuff because Maybe not everybody does that has the interest or is into that thing. Right. You mentioned, I think earlier.
having like young family and people who are starting a family, they're not really likely to want to get onto a five hour destiny grind or something like this. So you got to be careful about that too. It all depends on the person. I know at least one engineer on our team, he's got two kids, and I can see him doing that. I can certainly see him doing that. I mean, yeah, if it works for them, it works for them, but actually to bring that idea back around to this idea of protecting time.
make sure that time is during working hours, right? Overlasting working hours. Do not ask people to stay late to socialize. Don't do that. If they want to do that, they will do it. If you're helping run the team or you're managing the team and you do this.
What you're basically saying is, I do care about you as people, but not enough to actually give you any of my valuable resources to behave like people, right? So make sure that that stuff is doable in working hours. And this is something to my... current company's horn a little bit. This is something I really value about how we do things at SumUp. We do do stuff after work sometimes, but in some of these experiments I was mentioning recently about getting folks together on a regular basis.
we were intentionally making sure that stuff was happening during working hours we were donating work time to be social and to be human i'd love to hear about a defining moment in your career that influenced the way that you think about work. Because the folks that I've chatted with, and in my own experience, I feel like everyone has a team or a company that has... Yeah, none.
totally so the last company i worked at in san francisco for better or for worse we sort of pioneered influencer marketing although it wasn't called that back then sure the company was called uh wanilo.com like w-a N-E-L-O. Okay. I don't know if the company is still around or not, but in the year that I was there, we did something like a 500,000 to almost 20 million users scale up.
within like less than a year it was super wild also really small team i think when i joined there were eight people on the team and when i left there were 10 so the team was always super small yeah wow we only we were only pairing
We were doing it physically. We were doing that like pivotal lab style, like with like pairing stations, like a machine with two keyboards and two mice. But that was such a good working environment because we were working super hard, moving really fast, but it never felt.
painful because it was so collaborative right there was always somebody there to boost you when you were struggling and vice versa and you're always learning at like this crazy rate because there was always some kind of like puzzle piece
interaction happening where they knew something you didn't know and vice versa and those puzzle pieces were fitting together somehow and something new was happening. That was a really remarkably high-performing team situation where we were just getting a huge amount done per engineer. What were some of the gnarlier or thornier issues that came up from scaling up that quickly? We were doing it on, I forget what version of Rails. And I think Rails itself was...
fine once we figured out our caching strategy. I think the harder thing that we had to manage was like data model stuff for the activity feeds, which we did with a combination of Redis and 2M proxy and postgres and the postgres charting stuff especially was like really wild and i was working on that a lot i guess what we would now call platform engineering there so that was actually another interesting thing about the collaborative approach we
bought a bunch of time from, I think it was PG experts. We were pushing Postgres so hard that we were like, we need somebody that actually contributes to Postgres core. This is too crazy. We can't, we don't know how to solve these problems without super deep expertise. So we bought a bucket of time from them. They were great. So we'd bring them in and have them basically do modern programming with us to try to solve these tuning problems and data design problems.
and what that meant was that unlike a normal consulting arrangement where they leave and you're like hoping that you understood what they did you had actually done it together so it was a really high value session right we were we felt like we were getting our money's worth completely out of the relationship
Things were working. We were rolling with that huge scale change. So I think by the end of that year, we had gone from a single Prosperous database to some sharded thing. So from one giant machine with a quarter terabyte of RAM to 10 of them or something.
It was really just like one single service and one single app, but still it was a pretty wild ride. But yeah, highly collaborative. What were some of the defining characteristics of the team or lessons or patterns that you took from that experience to the other places that you've worked?
i think we were also like pretty early adopters of devops as culture right so we were not treating ops or operations engineering as like a separate discipline so that was one thing also this idea of understanding that the cost of reduced developer productivity was much higher than what might seem like big money costs right i think we updated our pairing hardware like two times or three times that year
every time there was like a machine bump that would come out so we could run builds faster we would just buy new x and turn the old ones in because the price to do that was much cheaper than like hiring more people and trying to spin them up in time to keep up with the growth rate
I think that was a big one. It was just like being very mindful of opportunity costs and also like stack choices, right? Like I think when the company started, we were on some kind of like Java spring framework and pivoted to rails just because we could get
could just build faster on rails so normally people would say rails is much slower than java don't do that but actually with a good caching strategy we could just build much faster with a dynamic language like that which is i think the right choice for the time maybe when If we had gotten big enough, we would eventually switch to a compiled language for backend services, but we were much more about solving those novel problems at the time.
Yeah. It sounds like there's a lot we could unpack from that experience, but I'm looking at the clock and I think we're getting to the top of the hour here and I want to be respectful of your time. Before we take off, what's like one thing that you'd want listeners to take away from this conversation or the perspectives that you've started to share?
here? I think it is a technique. So it's easy to think of this as being a small thing, but it's for me, at least pairing as a practice has been such a transformational thing for teams and companies that I've been part of.
I think it's really important to make sure that if you're going to try it, and I encourage everybody to try if they haven't, but if you're going to try it to do the research to figure out what are the patterns and anti-patterns, to make sure that you have... that the experiment you do is doing it, doing pairing, doing like real-time collaboration in a way that has been proven to work by other practitioners because it's really easy to do those experiments.
have a really bad experience because it's you know like a driver spectator or like pair marriages or something like this make sure that those experiments are well thought through and make sure that you really emotionally or like psychologically support everybody that's participating because it's very intense for people who have done this before to show another person in real time all the things that they don't know but it's really valuable so yeah like i said
I know pairing is like a technique, but it's so transformational, I think, that if you haven't tried it, you really owe it to yourself to try it in a thoughtful way. Yeah, I think thinking of it as a skill in and of itself is so important because...
of what you just said. So many people try it and are like, oh, I've done pairing. It didn't work. One of the first things that really tickled me when I saw the materials that you folks have on your company website was just like this library of information about how to do parent right.
which I would previously struggle to find because I would be like digging into like IRC and like email messages and people's weird blogs and stuff and trying to collect things. Because in a lot of cases I would come to companies, try to get people to start pairing.
read experiment a little bit and then i'd end up having to do like a presentation to explain what good versus or healthy versus unhealthy pairing would look like and having that library of information ready to go was really helpful when i've been explaining to people
Yeah, that's awesome. We'll link to the pair programming guide that we put together in the show notes. It's probably worth a read. Nice. We think so. Are there any other resources that you think we should toss in the show notes for folks along similar lines? I would definitely link to IncidentIO for incident management, Honeycomb for distributed tracing, and obviously Tuple for remote pairing.
I really strongly believe that good tools can build good culture, and I think those three tools have been really great, at least in my experience, for building a better collaborative culture. Yeah, that's awesome. We'll get those in there. And if folks would like to learn more about you or connect with you or some up broadly, where should folks go? For me, I think my LinkedIn.
It's on there somewhere. Now that LinkedIn has suddenly become a very popular social network. I'm B-I-X-U on GitHub. I'm so old that I have a four-letter. GitHub handle. Yeah, that's OG. No, that's serious. Very OG. Yeah, I think those are the best places to get in touch with me. We'll toss those in the show notes as well. Again, Blake, thanks so much for coming on. Great catching up and have a great rest of your day, man.
You too. Thanks, Jack. Thanks for joining us on Distributed. If you want to learn more about what we covered in this episode, or if you want to share your own thoughts, follow at Tupel on X or visit distributed.fm.