#36 - Building High-Performing Teams with Observability and CI/CD - Charity Majors - podcast episode cover

#36 - Building High-Performing Teams with Observability and CI/CD - Charity Majors

Apr 26, 20211 hr 1 minEp. 36
--:--
--:--
Download Metacast podcast app
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more

Episode description

“A high-performing team is one that gets to spend almost all of their time solving interesting problems that move the business forward. Not doing a lot of toil. Not working on things they have to do in order to get to the things they want to do."

Charity Majors is the co-founder and CTO of Honeycomb, the observability tools for engineering teams to debug production systems faster and smarter. In this episode, we discussed in-depth about building high-performing teams by having observability and CI/CD as the critical pillars to support it. We opened up our discussion discussing what observability is and how Honeycomb helps to provide observability for distributed systems compared to the other monitoring tools available. Charity then shared her strong views on how to build high-performing teams by focusing on Continuous Delivery, the sociotechnical aspects of the team, and the 5th key metric as her addition to the widely known DORA metrics. Towards the end, we discussed the engineer/manager pendulum, how one should be conscious about it, and that we should not treat going into management as a promotion or a one-way street.

Listen out for:

  • Career Journey - [00:06:04]
  • Observability and Monitoring - [00:10:52]
  • Observability Mindset - [00:13:08]
  • Implementing Observability - [00:15:09]
  • Observability Pillars - [00:18:35]
  • Honeycomb Overview - [00:20:06]
  • Honeycomb Cool Use Cases - [00:27:02]
  • Writing Custom Database - [00:28:40]
  • Building High-Performing Team - [00:31:20]
  • 15-Min Continuous Delivery - [00:34:45]
  • Testing in Production - [00:36:05]
  • Shipping is Company’s Heartbeat - [00:38:47]
  • Sociotechnical Aspect of Teams - [00:41:01]
  • Good Traits to Look for in Engineers - [00:43:17]
  • The 5th Key Metric - [00:45:51]
  • Engineer/Manager Pendulum - [00:48:45]
  • Effective Manager - [00:54:21]
  • Concerns on Manager/IC Pendulum - [00:55:19]
  • 3 Tech Lead Wisdom - [00:56:49]

_____

Charity Majors’s Bio
Charity Majors is the co-founder and CTO of Honeycomb, provider of tools for engineering teams to debug production systems faster and smarter. Previously, Charity ran infrastructure at Parse and was an engineering manager at Facebook, where she ran next-generation distributed systems at scale. Charity is the co-author of Database Reliability Engineering (O’Reilly), and is devoted to a world where every engineer is on call and nobody thinks on call sucks.

Follow Charity:


Our Sponsor

Are you looking for a new cool swag?
Tech Lead Journal now offers you some swags that you can purchase online.
These swags are printed on-demand based on your preference, and will be delivered safely to you all over the world where shipping is available.
Check out all the cool swags by visiting https://techleadjournal.dev/shop.


Like this episode?
Subscribe on your favorite podcast app and submit your feedback.
Follow @techleadjournal on LinkedIn, Twitter, and Instagram.
Pledge your support by becoming a patron.
For more info about the episode (incl. quotes and transcript), visit techleadjournal.dev/episodes/36.

Transcript

High performing team is one the kids to spend almost all of their time solving interesting problems that move the business forward, not doing a lot of toil not like working on things that they have to do in order to get to the things that they want to do. Not slogging through fixing the bill. Begin not slogging through outages, not waiting for test to pass, not waiting for our deploys to finish fixing something and then realizing it

was a different problem. I think we tend to take for granted that it just it has to be that way. If you get your shit in order you you can have a job where you spend most of your time solving new and interesting problems. Hey everyone. My name is Henry Surya be Robin.

And you're listening to the tekhelet Juno, the show will be bringing you the greatest technical leaders practitioners and thought leaders in the industry to discuss about their Journey ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our Journal. Hello everyone. I'm back here again with another new episode of the technology, you know podcast.

Thank you so much for tuning in and spending your time with me today listening to this episode. If you're new to the podcast know that package, you know, is available for you to subscribe on major podcast apps such as Spotify and apple podcast Google podcast, YouTube, and many others also, check out and

follow technology. No social media channels on LinkedIn, Twitter and Instagram. Everyday, I post quotes and words of wisdom from the recent podcast episode, and I share them on those channels to give us some inspiration and motivation for us to get better each day. And if you are thinking about making some contribution to the show and support the creation of this podcast, please consider joining as a patron by visiting technology.

No dot f /, Patron. I highly appreciate any kind of support and your contribution would help me towards sustainably producing this show. Every week for today's episode. I am extremely excited to share my conversation with Charity Majors charity. Is the co-founder and CTO of honeycomb, the observability tools for engineering teams to debug production systems faster. And smarter charity is a

frequent speaker. And is someone that I admire a lot, not just for her thought leadership on devops observability, distributed systems, Etc, but also for her, Opinions on multiple other topics, such as management, engineering startup culture, diversity, and many more which you can also find and follow on her Twitter and personal blog in this episode charity. And I discussed in depth about

building. High performing teams by having observability and CI CD. As the critical pillars to support it. We open up our discussion discussing about what observability is how it diverse from me. Monitoring and why the traditional monitoring tools and pre Aggregates are not sufficient to help us understand our systems better and to uncover the unknown unknowns charity shares.

How honeycomb provides unique capabilities that can help provide observability for distributed systems and how it works, including an interesting use case that can provide observability for CI CD pipelines. Then we moved on to discuss about building, high-performing engineering teams. Charity, share, it has strong views on how to build such teams by focusing on continuous delivery, carrying a lot about the socio technical aspects of a

team and the fifth key metric. As her addition to the widely known Dora metrics towards the end. We discuss the engineer manager, pendulum based on her popular blog post some time back and charity. Shared what she means by the pendulum and how one should be conscious about it. She gave her word. Of advice and explain that going into management, is not a one way street and that we should also not treat it as a

promotion. This is such an amazing episode that you don't want to miss and I'm certain that you will also enjoy this episode if you like it. Considered helping the show by living it a rating review or comment on your podcast app or social media channels, those reviews and comments are one of the best ways to help me get this podcast to reach more listeners and hopefully they can also benefit from the contents. In this podcast. Let's get this episode started right after our short sponsor

message. Are you looking for a new cool swag tekhelet Journal. Now offers you some swags that you can purchase online. These wax are printed on demand based on your preference and will be delivered safely to you all over the world where shipping is available. Check out all the cool strikes available by visiting technology, you know dot, f / shop and don't forget to break yourself. Once you receive any of those wrecks. Hey everyone, welcome to another

new episode of the technology. You know today. I'm very excited to have someone that I really admire from the devops technology side. Her name is Charity Majors. I'm sure many of you would have recognized her name today. We'll be talking a lot about observability, monitoring CI CD and a few other topics that she is famous for in terms of blog posts. Charity is a CTO and co-founder of honeycomb developer tools, that helps to improve the observability of your application.

She also writes a great Posts at charity WTF and co-host, a popular podcast, called only cats, which I think is one of the best observability of monitoring podcast available out there for charity. Thanks for your time today. Hope to have a good conversation with you. Yeah. Thanks so much for having me. I'm really excited. So charity and maybe in the first place. Let us hear more from you about your career Journey, about your highlights and turning points.

Yeah. Well, I'm kind of an accidental technologist. I went to college for for classical piano performance when I was 15 and I never had a computer when I was growing up. I never even had email until I went to college, but I didn't want to be poor. I didn't want to be broke and I found myself hanging out in the computer lab a lot. I never actually had a degree in

computer science. I had majored in classical studies Latin and Greek and creative writing and philosophy, but I found that I was spending more and more time working in the lab than I was doing any of my schoolwork. I dropped out a couple of X came to San Francisco.

I've been working at startups ever since I've made a niche for myself, being an early or I like to be the first infrastructure higher that joins a team of software Engineers when they think that they've got something that might be real. And I like to help them help it grow up. And then I typically start hiring a team, then I manage the team. Then there comes a point where I'm like, well, I've got everything I do here and I find another startup, but I do it again.

So that's what I was doing when I joined parse, which is a mobile backend for mobile developers, and that was about six years ago. I was there before beta got acquired by Facebook. I stayed at Facebook for two and a half years give you a little bit of context. Parse was we were doing a lot of like microservices before there were microservices. We were doing this really hard code tenancy problems. We had over a million apps running on purse with one database in one pool of app

workers. And we were going down constantly. It was really professionally. Embarrassing to me because every day be like, a new app. Would hit the iTunes. Top ten parts we go down. I tried every monitoring tool. Every logging tool, every AP. Tool tried everything and nothing helped because they were all dealing with metrics and pre Aggregates pre Aggregates are great. When you know what questions

you're going to need to ask. Can you capture that data in that way, but when you have to ask different questions every day and you have no idea what questions are going to need to ask, like, all of the existing tools just fell apart. So, there was one tool that Facebook called scuba that we started shipping some data sets into. And the time it took us to figure out which apple is the root cause of us going down or whatever, just dropped like a rock. It could be hours or days.

It was like opening. And ended like who knew to like predictably it was s not even minutes, but every time we would just like start slicing and dicing we could find the answer, this made a huge impression on me. And when I was leaving Facebook, I was like, oh my God, I can't go back to being without this tooling. I was supposed to be going to be an engineering manager at soccer stripe or something. And I was just like this needs to exist because everyone is starting to have this problem.

Everyone is starting to this problem of high cardinality and high dimensionality, and there's no tool out there that exists. So we started building the tools, Would become honeycomb. We had to start by writing our own database effectively because there is nothing out there that could do this and that was difficult, but it wasn't as difficult as trying to figure out the language because we knew we weren't building a monitoring

tool. We knew that it was something very fundamentally different that it was, you know, for helping you develop your application and it was about six months into having start honeycomb. When I happened to Google the term observability, which no one was using at the time when I read the definition which comes from like mechanical engineering and construction.

Troll Theory and it has to do with how well, can you understand any state that your system can get into just by looking at it for me outside and I went, oh my God, but light bulbs are appeared in my head. This is what we're trying to do. This is what we're trying to build.

We're trying to build a way to understand your system just by asking questions from the outside without having to ship any custom code to understand the problem, which implies that you could have known it was going to be in advance. So it's all about the unknown unknowns and it's all about being able to infinitely slice and dice the Attila. That's coming out. We started talking about it a lot and for the first two years or so. It was like, just nothing. Everybody was like, we've been

told this is impossible. We believe that you're lying, but blah, and about three years into doing honeycomb. One day. It was just like everything changed. I woke up the next morning and the entire world was like, but of course, we need observability. We have always wanted observability. And by the way, we do observability to number was just like, oh my God. No, so there's been a little bit of a bandwagon effect.

Now, the frustrating thing is everybody's like, yeah, we do observe ability to just like, well, no, they're supposed to be very specific technical definition because if you say you want to be able to ask any question of your systems, some things proceed from that some technical prerequisites perceiving that you have to be able to handle high, cardinality and high, dimensionality, and you have to gather the Telemetry in such a way that you have one, arbitrarily wife tortured event

per request per service. And I've written all these blog posts and stuff. The people who follow it closely understand. But like the rest of the markers just like how observability is a new synonym for Telemetry? Got it, and I'm just like, okay. Well, this is better. This is progress. You know, yep is coming in technology industry, any new. Jargon, any new buzzword, everyone taste used. So if you can help us, like maybe share from your point of view.

What is observability? And how does it differ from monitoring? Which, what we are used to? Yeah, I think it's really important. That people like understand the way it's different because monitoring is a very mature. It's an important discipline of its own and there's a lot of best practices and stuff that go along with it. I think that the metrics tooling is great, it fulfills a separate need In the ecosystem from

observability. Monitoring is really about taking from the perspective of the service and saying is this service objectively speaking? Healthy is a service healthy. Do I need to plan for more capacity? What's my consumption percentage? It's usually built on a structure of metrics. There are the generic term for metric, which just Telemetry. But the specific technical term for metric is just it's a number you can append tags to. So like it's really cheap.

You can store a lot of metrics very efficiently, you can age them out, as they get older, you can drop some detail and you can Your system for a very long time. The source of Truth for observability is not metrics because when you capture that one metric that would number. You've discarded all of the context except for a few tags, which you bring back. So for example, say you have a spike of errors in your graph. What's wrong?

Well, what's up durability, you might slice and dice to figure out. Oh, these errors are for all the requests are coming from this version of iOS. From this end point from this language pack from this version of the firmware from this build ID, just like the long string of things that like this is what's different about them.

The well, you can't do that with metrics unless you happen to an advanced capture a metric for this version of iOS this build ID and the ability is always incrementing and always changing. So you have to capture those specific numbers in advance with metrics. Then with observability. You've captured it in this very wide event, which captures all that context and it says all of these things are linked together by this one request from the

users perspective. So you can then ask any combination of question from that on it. So monitor is about from the perspective of the service. Observability is Perspective of the user. What is the users experience? Like which I think is super

important. So the user for monitoring and metrics is typically Ops. The user for observability is typically people who are writing code, people who are shipping code every day, who want to understand the magical conjunction between your code, your infrastructure, this point in time, and that users experience. So, I think to me, when I hear about it, it seems like it requires a lot of mindset change. First of all, like, capturing. All these hide.

Mentionable High cardinality metrics and also like implementing that. So what do you think someone if they want to explore this observability further? What should they do to change their mindset? Well, I think the shortcut is often. Just seeing your own system, your own data in and observability Tool which is why honeycomb has a really generous free tier because we want people to be able to sign up play around with it. Even if they don't become our

users. I still want them to like understand the Mind shift and experience the difference. So I think that's the shortest quickest route to experience. It to understanding it. There's a lot of things will just click if you're used to trying to understand your own system and you see it in different tool. You already know the fields your landscape. And so it's a pretty quick and easy way.

Another way of thinking about this is that this became necessary when the monolith began to decompose because if follows failed with the model, if you could always attach a debugger and step through the code. Well, now you've blown up the monolith and God knows how many services and every time it pops the network. Whoops. You've lost all the context.

You can no longer Trace that code anymore, which is why When you're Gathering the Telemetry for observability, the way we do it. Now, you come is we initialize a honeycomb event as soon as request enters the service and then we pre populate it with everything. We know about the environment, the language, the internals go routines, anything about the context that was passed in parameters. And then, while it's executing in that service, you as a developer can go.

Oh this might be useful stuff it in. We also wrap a lot of common functions database calls. So any time you do a database called, we wrap it and we record the time it took and The Rock query. And the Nice query and results. And every time you do, if you request, we wrap it and we give you the duration, and the

result, and all that stuff. And then when the request is ready to exit or error from that service, it gets packaged up into that one arbitrarily wide track your data blob and shipped off to a honeycomb. So you can think of this as like, packaging packaging along all of that context, that it needs every time a pops, the wire. Well, now you've got that context because we're passing it along with request. That's pretty cool.

So, in the first place who is responsible for implementing this, because in the beginning, you said, Said, it should be developers. First of all, there will be the main users of this. But as we all know, observability monitoring these days are highly associated with Ops, or the devops site. So, who should implement this and how should someone actually think about? Okay? What kind of questions that I would ask about my system? And where does this?

Observability might be useful for my system down the road. Absolutely anyone in the chain can take responsibility for this. I think that most commonly, we have Ops teams who get it. Set up on behalf of Developers and start to teach them which is I think a really beautiful thing. I think that Ops teams are not becoming any less necessary and increasingly their job is not to make the code run, its to help developers understand. This is like the second wave of devops, right?

First wave of devops is like AA. Simple, you must learn to write codes like, okay. Okay, we get the message and now it's like, okay, software Engineers, it's your turn. It's your turn to learn to write operable Services. It's your turn to learn, to support the services, and to build systems that are Not too expensive to maintain but we're here for you. We will help you. So I think that sometimes it's the developers who take it upon themselves to bring observability and often.

It's the Ops teams who bring it in, on behalf of their developers often. It's execs, who are like they've been paying attention and they've got their ear to the ground. They know that systems are becoming impossible to run and scale and maintained without the sort of tooling because forever the way that we've been doing is has been by guessing what we see something wrong. We guess based on past outages and battle scars were like, I

think it's this, right? And then we go and look for evidence that were right, that works really. Well. When you have a lot of people who've been there for a very long time who have a lot of context in their heads and the systems aren't changing that much. But when the system starts breaking in a novel way every single day, it doesn't, do you

much good? So I think that it doesn't matter who takes it upon themselves to do this as long as someone does, because it is very important because people burn out teams. This is an ethical issue. I believe this is a Humane it. Issue, we've all grown up working in the tech industry accepting a certain amount of self-abuse just like masochism, an Ops is just Infamous for this. Well, throw ourselves on the grenade, drag ourselves into the meetings.

Just like in the morning, like, I've been up all night. It's like a badge of honor for us, which it was fun. I enjoyed being the wizard who points at a graph and goes, so that's super fun, but it doesn't scale very well. And our systems increasingly don't reward.

That kind of magical thinking especially these days when people are using lots and More microservices distributed systems even platforms like Cloud where you have so many services first, because whenever you have a platform, you're inviting your users chaos to live at your house and you're responsible for it. So can this observation T concept be useful monitoring business metrics as well.

I think that the separation between business and Technical metrics is pretty arbitrary and detrimental. In fact, I think that you want everyone to be reading off the same answer sheet. You want everyone in the company to have you seen view of reality. Reality and to the extent that you can all be looking at the same numbers. Well, that's just a lot of argument. You don't have to have.

I believe that tools can create silos because the borders of your tool is the borders of your knowledge. I've been in so many meetings, where you spend more time. Arguing about what was true because of what your tool said versus the actual problem that you're trying to solve together. So maybe if you can mention few attributes, I know you have metrics, you have maybe logging and you maybe have traces likes pens and things like that. What are the things that are?

Crucial in any distributor systems these days. I mean metrics logs and traces are just data formats and you can derive all of them from the arbitrarily wide structured data blob that I was just describing. You can derive your metrics from it. You can drive your log lines from it. You can direct your spans from it. You can't go in the other direction. You can't go from a metric to a white truck. Sure blood. You can't go from a log line, to a wide structure blog.

So I really think that we need to stop thinking about this in terms of the archaic tooling that we've inherited and start thinking about At it, in terms of the use, we get out of the system. And I feel like the move to these arbitrarily white shark today to blogs is inevitable, but it also shouldn't be a big deal for people. Right? If you squint, they're just

logs. They're just structured that race is just a wide log line that has a couple of dedicated expand ID fields, and Trace ID Fields. So I think it's kind of a red herring.

It pisses me off that so many Legacy tools are just like you definitely metrics logs and traces, why because they happen to have metrics products, logging products and tracing products that they What to sell you and if they're selling you those three products and you're paying just saved your data, three different places three different ways and they can't

talk to each other. So I see this as a really cynical Ploy on behalf of the entrenched Legacy, this Punk's in the new relics and that data dogs are the world. They count up all the products. They want to sell you in there. Like this is how many you need and it's like, no you don't. What you need is the functionality. Maybe you can share a little bit more about honeycomb for people

who haven't used it before. I must have haven't used it personally, but I heard a lot of rave reviews about it. So maybe we can share a little bit how honeycomb can help and what are some of those tooling? Like what you said, they sell you as a suite of products could not achieve compatibility comb. Well, so, you know, there's a narcissism of small differences. Of course, we're all trying to solve the same problems here. So like for example, we usually start all of us, universally.

We usually start our debugging experience by looking at metrics graph. We see a spike in errors of my God. There's a problem then because you can't slice and dice metrics arbitrarily, you typically jump over to your lung. Tool and you start just trying to visually correlate. Well, that spiked hair is looks like this over here. It's not really science. But you know, you try and find the same thing around the same time, then maybe you want to trace it or something.

So you pick out a tray side you pick out a request ID and then you jump over to your tracing tool and you paste it and you hope that one got captured. And so you're just visually jumping around and you the human or in the middle of the glue. Well, you do exactly the same thing and how do you home only? You don't jump around you just you look at the graph. You see there's a spike or something.

Well, the way we don't honeycomb is he Just draw a little box around, this is what I'm interested in and then we can pre-compute for you for all the dimensions inside the box. He said, you're interested in and outside the Baseline, and then we dip them and then we sort them. So, all the ones that are different rise up to the top, and then you can click on one and just say, she'll be this traced to me, example, trace for this.

So you're doing exactly the same thing but you know that you're dealing with the same data because you're not jumping around. The cool thing. Is it? Because traces and moments were just like two sides of the same coin. Well, it turns out that Tracy was just visualizing by time. So you're dealing with it. The same events, exactly the same data stored, only one time, but you're able to turn it around, inspect it in all the different ways to try and find

the anomalies. I think, the Bubble Up thing that we've done is like, it's a game changer because what it did was it replaced the manual process of what we used to do at parse was scuba. What we used to do is go. Okay. There's a spike. Well, let's see. Is it for all and points? No, it's just for the endpoints that are like in points. Well, is it for all the right and points? No, it's just the ones that are writing to say mongodb. Well is It to all the mongodb and buts.

Well, no, it's only airing to Feast three primaries. Weld, what do they have in common? Or is it to all of them? Well, no, it's just this application idea. You're following the trail of breadcrumbs, which is let me point out. Vastly preferable to guessing and looking for evidence of. You're right. Right here.

You're actually following the trail of bread crumbs to the answer every time, but Bubble Up did is it just replace that whole iterative process because we pre compute them and we show you them all at once yours like Dropbox group. There's the answer, which is pretty cool. This is where we hear a lot these days about AI Ops and like machine learning and all this bullshit.

I have lots of strong feelings on this topic, but I feel like I've never want to take the human out of the driver's seat, but I do want to look for ways for to make machines. What machines are good at machines are good at crunching, lots of numbers to let humans do what humans are good at which is attaching meaning to things. If you see a spike, you don't actually know if that was good or bad or something.

Expected to see something you wanted to see, you don't know that until the human is attached me to that. And then the Computer can crunch a bunch of numbers and back up the meaning assigned to it. So I don't want any developer to give up that centrality of be in the driver's seat. But I do want to look for ways to take out the repetitive.

The loop just like is it this is it that we can collapse a lot of that work so that you just have to point and click and understand the way you described it actually sounds to me as if like, I'm Googling something right? No problem. I start from google.com such one thing and it brings me to more and more links until I get Yeah, so it's about starting up high and methodically working your way to the answer is a process that we're all familiar with and our tools just haven't supported it.

We've just had to make these wildly too fancy. One. Other thing. I will point out is that something philosophically a honeycomb that we really believe it is the power of the team because whenever we're working on these complex security systems. I'm an expert in my corner of it, that service that I'm writing right now. Like the thing that I'm building. I know everything about it. I know the variable names. I know rather Gremlins live, but I don't know your Art nearly as

well yet. When I'm debugging, I have to take the whole system into account. So we talked a lot about how can we bring everyone up to the level of the best debugger and every single corner of the system? Well, we can do that by piggybacking on the way that I interact with my system as an expert and way you interact with your system is an expert. So if I get paged about something, maybe it's looks like

a, my SQL problem. And I don't know anything about my sequel, but I know that the experts on the team are like, Ben and Emily. And I feel like we had an outage like this like last Thanksgiving. So I just want to go look at like Any of them on colic? What did they do? What was meaningful enough to them? That they would run it a bunch of times. Right? What did they tag and put into a post-mortem?

What did they add a comment to, or name and like putting their personal arsenal of useful links? You should wear grooves in the system as you use it and you should be able to draw on the collective wisdom of your team as experts in the system. I trust that so much more than I would ever trust any a. I so if I understand correctly, the way it works, is you instrument your coat, your services to give you this blob of data. Is there any other mechanism that you use?

And how about the platforms that you mentioned? So how do you instrument the platforms? Like the database is the cloud platform that you use? Well, observability is fundamentally about the first person perspective. It's about writing a library for your code. Being the first person actor. Who's this is what I am doing. This is what I am seeing, but data is data and it's better to have second best data that no

data at all. When you've got say a database where you're my instrumenting it and it's not your code. You do want Some ways to get Telemetry out of it. The easy answer is, will accept any structured data. You can throw at us. It's most useful to you as a debugger. If you've gathered into one or trailer wide structure. Data Bob per request for service. But, you know, if you have an elk stack, you can just put a little stanza in your log.

Stash thing that will just duplicate the output streams and send us everything. You're sending to elq. That can be a really nice little bridge for people as they're getting started because they already knows this data. They know how to navigate it. You don't have to do anything change any code that way for things like MySQL. I'm a databases. Right. So this is an old school shit. But what we do is we get three sources of data for my SQL or mongodb or whatever.

We get the slow query log because there's some information that is only sent to the slow query log. We also stream over the wire, we do a tcpdump extreme over the wire and we reconstruct all the transactions and then we ship each transaction in to Honeycomb as an event. Because there's some information, you can only get that way. And thirdly, we also run a Cron job that connects every couple seconds to the command line interface and just dumps out all the internal stats.

Ship those in as a beds to those three sources of data. That's basically the sum total of everything. You can get out of these databases and it's frustrating because they clearly weren't designed, in a way to just, like output all the information that we would like them to have and welcome place, but you can get a pretty good approximation with those three sources. So I'm sure you have plenty of these, right? But any kind of cool use cases that maybe your customers have shared.

The coolest thing. That was a big surprise to me was when one of our customers, your to ago made it. Library called build events for build pipelines and what they did was basically, they made it so that you can instrument, your CI, CD pipeline as a trace. Just like your Trace. So they like overloaded that a little bit if you put the build of its library in your circle or Travis or Jenkins or whatever,

then it will wrap. So it shows each test when it fires off and when it finishes you can see how well the paralyzation is doing, or if it's all sequential or where the time is going. That's so amazing because Bringing Down the amount of time. Takes to build and ship. Your code is so Central to I think every team really should be paying attention to this but there's it hasn't been a good way to visualize it and I thought that was just fucking brilliant.

It's such a good use of visualization by time and that had never occurred to me. So now we try and get people to do it all the time because it's such a great gateway, drug to observability because we're Engineers. We love to figure things out. We love spotting and efficiencies and some things seemed really hard and tough, but that you just see them in the right tools.

Like, oh, duh. I can fix this, but I think almost no one has this experience with the build events are like, oh my God, that's where all my time is going and they go and fix it and it's just you get such a dope, what you did, such a high off of, it's just, it's so fun. I never imagined it that way. But yeah, now that you mention it, I think it's pretty cool. If we can have that data, especially if you can correlate with the real production system,

but what happened? During build time what happened during development? Maybe can't ask mrs. Behave differently during that time. Yeah, so that's pretty cool. So you mentioned in the beginning as well that you have to write your own database and we can share a little bit. Because since you are database person, why the current database tuning is not good enough for

you. It comes down to the fact that the trade-offs that we wanted to make our not trade-offs that anyone else would make for our purposes fast and mostly, right is better than accurate. So, uh, we're fine with just throwing away, some very small tail of data in order to get like sub. S query Layton sees, this is the other theme of honeycomb. It's fuckin fast.

The 95th percentile for queries is like under a second and this is really important because when you're debugging, You want to be in a state of flow, wouldn't be asking this this this this can imagine with Google took five minutes to return to search. You be planning in advance. You be like, oh God. Okay. I'm going to run this query and then I'm going to go to the bathroom. That's not good. So we need to really fast. You're okay. With discarding some data.

We also knew that it was very core that we didn't want there to be any schemas. We wanted people to be able to throw in more Dimensions. Any point in time, a maturely instrumented Services. Honeycomb will have 300 400 500 Dimensions. So the events are that wide. We don't want the friction of, okay. Now I'm going to add another one. We want people to just be able to throw it in, forget about it. Stop sending it, forget about it.

So no scheming changes. No indexes, because indexes again, they violate the theory of durability, which is that, you shouldn't have to predict what's going to be fast. What you're going to need to ask if you'd all be equally fast to query it on. So that meant that we needed to build a columnist or they're not a lot of Open Source freely available columnar scores. We also knew that we needed the data to be very distributed. We didn't want to spend all of

our time managing the database. A store like adding databases for each user, the way it stored is actually it's cool. So when data comes in through the API, the events get dropped into Kafka into a topic just for you, your account. And then the topic is consumed by n numbers of retrievers. All of our sources are named after dogs, because we're a dog

company. So the retriever is just a pair of them for each for, each will consume off the queue and then a thing that we did just unless you're too, we actually made our database, go serverless. So like At first we were storing them all on ssds and am. And then we were like would it

be cool if it was just an S3? We thought that the performance hit would be huge but it wasn't so now like the queries actually they run as Lambda jobs and the data gets aged out to S3 after a few hours on disk and eventually I think we'd like to make it so that you can choose to age it out to your S3, buckets. So that they're really just hosted on, you know, you can

post it forever for free. We have 60 days of storage for free, which I think is the best in the industry and if we're able to do that because we were able to just Outsource a desk. 3r. It's really cheap. Wow. Sounds really cool. Thanks for sharing that, of course. So charity, you mentioned. Also, you're passionate about building High performing team lately. I've seen one of your talk about CI CD and also the blog post. So can you tell us a little bit more about?

What do you think about high-performing team? What is actually a high performing team versus the low performing on a high performing? Team is one the kids to spend almost all of their time solving interesting problems that move the business forward, not doing a lot of toil not Working on things that they have to do in order to get to the things that they want to do. Not slogging through fixing the

bill. Begin, not slogging through outages, not waiting for test to pass, not waiting for our deploys to finish fixing something and then realizing it was a different problem. Just like all of the stuff that makes this job frustrating sometimes I think we tend to take for granted that it just it has to be that way. I've now worked on a few High performing team for it.

Just it doesn't if you get your shit in order, you can have a job where you spend most of your time solving new and Interesting problems. And that is just it's a joy, right? It brings a joy to your work that is just it's Indescribable. And in order to get there. I think that focusing on the door metrics is a really good place to start. How often you deploy. How long does it take? How often is it fail? How is it take to restore? Time to recovery and a lot of

improving. Those metrics, involves leveling up and observability seeing what you're doing, like, putting in your glasses before you go and you drive down the freeway and a lot of teams. They're really Flying Blind. That means that a lot of their work is wasted. I also feel like It's important to note that high performing teams. They need to feel emotional safety. They need to support each other. They need to be kind and the high performing teams.

You don't get them by just hiring the best people or hiring the best Engineers. You don't. In fact, some of the people, I've known, who are the best individual Engineers are pretty toxic to team's overall you, as a leader. It's your responsibility not to coddle the best Engineers. It's a responsibility to build teams that have a very steady strong level of combined output because that's what wins the race. Race, it's not individuals. It's setting your team's up for

success. I just published a slide Deck with the bunch of steps. The other day. I do think that the number one, technical point that people should focus on which, I know that this is something that nobody would disagree with. Right? We should have high performing teams. But another time on moving the business forward. No fuckers gonna disagree with that sentence.

The hard part is how how, and the place to start for almost everybody I've ever talked to is, you need to shrink the amount of time between when someone is written the code. And when the His life. You should get that interval down to 15 minutes or less. And if you get that, you get to spend your time on a lot of problems that are much, much higher up the value chain, but if you get that wrong, you're just going to waste all your time.

Tending, to pathologies. It is really an efficient and effective in my experience. If the number of people it takes to write and build. The system is say x number with a 15 minute or less interval. Well, if that interval stretches two hours, you need twice as many people. If that interval stretches two days, you need twice as many people again. And if it's weeks, you'll need twice as many again, because the cost of context switching and waiting on each other at the diff, get bigger.

And the reviews take longer and now you need a dedicated SRE team to like manage all of the deploys in the failures in the Cheetahs, get patched up. It's just as desk by rho. So that's the place that I think everyone needs to start is getting that fully automated and 15 minutes or less and then everything is going to get so much easier for you or in short. It's continuous delivery, right? So that's what the team should Aspire for continuous discrete for real.

So 15 minutes to me, that sounds really short amount of time. And there's so many things that people wants to do in the pipeline. Like, for example, automated test performance test, security test, whatever test. So is that really important to have 15 minutes as Max limit? Yes. I feel like I'm being generous with that. It is important because you don't want people to have enough time in there to be able to

shift context that moment. When you've written the code, you want this to be a small dip, right? You want this to be like a few lines. How quickly can you deploy? One line of code should be your goal here, right? You want to be short enough that you hook it into people's physiological systems, their muscle memory, you want all of your developers to go and look at their code after they shipped it. So, it needs to be short enough that it hooks into your

physical, like expectation. Just like, you don't feel done, until you've got a new looked at it. And so, that's an hour or more. I'm sorry. That's too much time. Someone's gotten switched over to something else, and that means that they flushed all of this context of their head and they've gotten distracted and they're not going to do it regularly. And if it's an hour or more that I guarantee, you're going to start batching multiple. Changes up together.

When it's really important that you be shipping, one person's change, set at a time because that creates ownership. If you're change going up, you're going to go look at it. If you know, it's your change and 1 to 20 of your co-workers going out at some point in the next day. You're not going to go look at it. You need to construct that feedback loop and keep it crisp. And also notice that you are one of the strongest proponent for

testing and production, right? Is that also one of the reason why? Because you want to achieve this 15 minutes, right? You Don't do a lot of testing and there it all goes together. I think there's been a major sea change over the past five years. In terms of startups that have started, the way people are talking about this stuff. But used to be that we put all this focus on pre-production

stuff. All of our energy went to like every developer gets their own staging environment and oh, this pre-production stuff and everything but we've hit diminishing returns a long time ago there. And I feel like over the last five years. We've started to focus correctly much more on. Okay, but now, what about production, we Tools for production. They're like scalpels that are very sensitive. Very precise. Developer Cycles are the scarcest resource in our

universe. They're not infinite if we're wasting them all in pre-production stuff. Then we're left with nothing for production. I just think that the importance needs to be inverted there. We need to be thinking. First about production. Do we have the tools that we need to ship safely and swiftly and understand what's going on? And then give 80% to production, 20% to pre-production set of 80% of pre-production, and 20% of

production. But the point is not that everyone should be testing production. In the point is that everyone is testing and production whether you admit it or not and you're better off if you just admit it and try to do it. Well don't deny deny deny and then sneakily like SSH into see what's going on. Now. Everybody has to test the production you do. I do? It's not embarrassing. It's a fact of life. Now, how can we do it better? Yeah.

I mean I must admit. I also testing Productions most of the times because you can't really predict what actually happened. Yeah. You have know what the gold standard is to capture. And Play like 24 hours worth of traffic. And I will say the closer you get to laying B on disk the book paranoid. You should be. I have written this Suite of software three times for three, different databases that captures 24 hours of traffic and replays it. Again and again under different

conditions and stuff. You should know when to do that. Major database upgrade. You should absolutely fucking do that. But every day when you're shipping a few lines at a time, no, of course not. But you should invest that energy into observability and if you want to get really precise and really you should be using feature Flags, so, You can flip that on and off so that you can decouple your releases from your deploys. You should invest in observability so that you can

see and canaries. So, if you're really scared about a change, ship it to one percent of the traffic or ship it, to one percent of the users or ship it to one node, build the tools that let you have this very high level of precision around, who you're shipping it to and then the ability to compare the traffic stream so that you can see what the difference is error rates is or what the difference in latency is or anything like that, build those tools to address your

paranoia. Don't say that you're not testing and production. So I'm interested to dig deeper about this 15 minutes. So what actually has to be done in this 15 minutes apart from building software and also packaging it and deploy it run tests. Absolutely run tests. I know that some languages like was girl. I hardly takes any time at all. We can rush it to test but this is achievable even for things like Ruby which is notoriously huge to package, you know, slow to run. All this stuff.

Our friends at intercom who are the ones who came up with a slogan that shipping is your company's heartbeat, which I I really value and respect. They run a ruby monolith and their entire build and test pipeline runs, and deploys, and under 10 minutes. It can be done. Well, this can absolutely be done. It's not hard. It's just engineering work. You just need to commit to doing the ongoing work. Setting yourself a limit set yourself a limit alert.

Would it goes off. Take it, seriously, invest in it over time and you can hold yourself to a higher standard. So I'm mostly interested with this phrase. Shipping is the heartbeat of your company. Maybe you can tell us a little bit more about that. This came from intercom and I adopted it cuz I love it so much. Shipping for any tech company. Giving value to our users is why we exist shipping.

New changes should be new code where users should be absolutely as common as ordinary as uneventful, and is regular, it should be a nothing. Nothing Burger, ask me how many times a day we ship? I don't fucking know the trip all the time like a couple times an hour. I don't know because we're making changes constantly and it is because those changes are small and uneventful that it's

not a big. Deal, I feel like we need to invert the way that we think about this is this social change is a mindset change, but it's not a new thing. The principles of cic have been around for what 15 years accelerate was written with all that data years ago. Like we've known this for a very long time. We just need to look at it into action. And the thing is, this is so good for engineers Engineers,

who have worked with true. See, ICD are unwilling to ever go back because it is like different profession. It is so much more interesting and exciting and up to the moment. See your real impact on users lies like many times a day and that's fundamentally motivating and inspiring. It's real hard to ever. Be willing to go back to a situation where releases are so far away from your lived experience or someone else's

problem. We don't want that and I think this is related to the socio-technical. Mm aspect that you mentioned, right? Like in the first place. You mentioned, you don't need all the high performer people in your team. You just need a team that works well together. Maybe you can also share a little bit. What we mean by By social technical aspect of a team. Yeah. So for example, think of it this

way. Imagine if New York Times is a technical and social organization, imagine that all of the people in the tech side, got flown to the Bahamas for a month and they brought in a new set of people who are everybody's experience talented, good Engineers, Etc. He was brought instead of Replacements to keep everything running and then something breaks. How long do you think it would take them to figure out and fix even a very small problem? First of all, how do they get

into the system? It's a mind experiment to You just how much of the system lives in our heads. These people are not fungible. Right? Engineers are not fungible. People are very important part of the technical system and you can't just separate them. I think it's really important. Especially as we're entering this age with machine learning and AI. This AI that just remind ourselves that no people are

really fucking important here. The part of the system that lives in your head is a part of the system honeycomb. We interview lots of Engineers and what's really important to us as their communication. Ation skill. Now, they need to be able to write code obviously, but the real interview is not us. Looking at their code. It says, talking to them about their code. Just having a conversation about, well, talk to you about your choices here. Why did you choose this?

What else might you have done or like what trade-offs we did you have in mind when you are doing this because there are so many people who are capable of writing the great code but can't communicate about it. And those aren't the people that we want for our team because we believe that high performing teams come from people who are ready willing and able to talk about what they do. And They've done what they've done without getting their ego, too wrapped up in it, but just

being like able to have a conversation about it. Christina and I could have just hired all of our X Google X Facebook, friends. We knew that we didn't want to create a cultural bubble. So we made sure to go outside of our networks, but we were never looking for the best programmers. We were looking for the best communicators, the level at which this team performs. It just, it blows me away. Every single day. There are order of magnitude better than the elite category

and the door of metrics. And it's not because they're the flashiest. Coders is because they're dogged about improving. In their craft at this really inspiring. So what else do you see from a candidate? If you interview them apart from communication skills and some basic programming skills. Of course, they need to have what other things that you look for in an engineer. We look for humility. We look for people who can be gently challenged and don't fall

apart. We look for people who are constantly, I guess growth mindset. Is the term of art for it, which feels a little corporate e to me, but we want people to be able to come curious to a problem. Not to come feel like they're going to be. If I'd if it turns out that they don't know what they're doing. Because it turns out, we don't know what we're doing, either all day long. None of us know what we're doing. Our company values. Are we have values that are like

we hire adults. We don't have your family or your whole life. But we also do, hire people who take a lot of pride in their craft. We don't want to hire people who just want to work. You just do it to pay the bills either because we take a lot of pleasure. And what we do, nobody honestly has more than four hours a day in them of hard, focused, engineering labor and it's so much more important to me, that people know how to manage their time to get them.

Themselves those four hours then for them to like sit at their desk 24 hours a day. We hire adults. Their values are, everything is an experiment and fast and close to ride is better than done and perfect, which trees is all the way back to manifesting in our storage and right? Like that's our philosophy for storing data, turns out. It's our philosophy for people as well.

Feedback is a gift. Is another one because I think that so many pathologies in the modern workplace can be traced back to the fact that people are scared to give each other feedback and so it builds up and it becomes a big thing.

And if we can just like deescalate that make it, not a big thing to be constantly giving and receiving feedback and to be able to give feedback, knowing that we might be wrong about it. Just like your something that I've noticed I thought might help you because we're all on the same team, right? Like we should be able to help each other improve which is so easy to say and so hard to do because we all have our own personal damage. We all have our own prom has and stuff around the stuff.

So we're all constantly having to learn and unlearn bad behavioral patterns, but there was a couple of people have said to me. They started working at honeycomb to like how you guys talk about your feelings and awful lot, and I'm just like embarrassed. It's okay. Yes. Female founded company. Yes, I guess we do talk about our feelings about, but you know what, it's better than the alternative. Thanks for sharing all that.

I think it's a really good Lessons Learned as well for other teams who would like to build High performing team, you know, we spend more time at work with each other than we do at home with our partners and our families. I think that we shouldn't be embarrassed about talking about our feelings and being emotional human beings at work. Because if you're not well, you're just faking it and I don't believe that lets you perform at a high level over the long term. I agree with you on that.

So one thing that you mentioned about Dora metrics, I think in some of the presentations I saw that you actually insert one moment trick not just a yes, you metrics. So what is the fifth? Maybe if we can share I think that every team should be tracking how often someone gets paged out of hours. I strongly believe it's software Engineers need to be on call for their own code, but this comes hand in hand with commitment from management to make Make

sure that does not suck. I'm not just saying software Engineers. You need to go be masochists to just like cops have been all this year. No, the reason to put software Engineers on calls because this is how we make it better. This is how we close that feedback loop, how we make it short, how we get things fixed quickly, so they don't get to infest and like become all infected and gross and big problems be fixing their small. So I think that these two have to go hand in hand.

Yes. Developers need to be on call for their code. And yes manage music. Make sure it doesn't suck and probably compensate them for their time to I It's reasonable to expect any engineer who works on a 24/7 highly available system should expect to get woken up twice a year or so, once or twice a year for their work. I think that's reasonable. I do that's compatible with an adult life. The exception being, if you have a child that is waking up in the

middle of the night. I would never put one of those people on call. If you have a baby, you are out of the uncover rotation until your kid is regularly sleeping through the night. It's not fair to ask anyone to wake up for two things. Right? So with that caveat, these are human systems, right? So there are exceptions. To everything. I'm not actually saying every person must be on call at Facebook. I had a guy on my team who was

totally willing. He wanted to share his part of the burden, but when he took the pager, he got such anxiety and it wasn't because he was getting called or page all the time. He was so afraid. He just had a personality that was just not compatible with it. We tried for a couple months and he gave it a good shot. But I was just like I just felt so bad for him and the whole team did right there just like,

okay. No, Alan doesn't have to be in called what can Alan do instead and so Alan took over. Ownership of the build Pipeline and just like with build cop. So that was like technically more work but it was during business hours and the rest of us were totally fine to get the slack. So it's about pulling your own weight. It's about being willing to support your own code. It's about management.

Making sure you have the time to fix the things that need to be fixed because you can't just put a long call and then tell them to build features all the time. If you're putting them on call it, you have to give them the time to fix the shit. So they're not getting woken up all the time and I just feel like having that metric and holding managers accountable for that metric. You never promote a manager who's paging. Alerts are going off at all hours. That is just a sign of terrible

management. Their team is going to burn out. It should come down on them. The hardest. Wow, thanks for sharing that. I think it's really good key metrics, especially when you have so many distributed systems in production with problems could happen at any point in time. And was, if you have a global system with your users come from anywhere in the world. I think it's also critical to monitor this like how much you get paid tonight, right? Because that's what burns your

team's up faster than anything. So one of your most popular blog post is about engineer and manager pendulum. So because this podcast is also about technical leadership and also Engineering Management. Maybe you can share a little bit. What do you mean by this pendulum between engineering manager? Yeah. Well, when I became a manager I was told it was a one-way trip and I've heard this repeatedly.

People are just like agonizing over whether to become managers because they see it as a one-way trip. They're not sure if they can get hired as an engineer afterwards. They're not sure if they should. You're often told that they shouldn't and I feel like this has said, the best technical leaders that I've ever worked with have been people who have spent time as managers because you learn things about how to help a team overcome obstacles and work as a unit, that you

can't learn any other way. You learn things about how the business works. If you can't learn any other way, but has also, like has been hands on within the past five years because when you take a step away from the day-to-day of building shipping code, like you're on a path that diverges, right? And there comes a point where you are not an effective leader of people who are writing shipping code for the sake of your own career. You need to be cognizant of that and are two different paths.

You can take. If you want to continue managing people. You can make it, your goal to work your way up. The ladder. Now, there are an order of magnitude fewer jobs at each level, as you go up from manager, to senior manager, to director or to, you know, see your director at VP or whatever. There are fewer and fewer jobs. It is more and more competitive. There's more and more bias involved, I think. But for people who love it, it could be very rewarding.

But if you want to stay in a line manager, who manages Engineers who manages the people who are writing shipping code, You're going to need to make your way back to the Well, from time to time. You can't go more than five years without building and shipping code regularly yourself and maintain that level of tension and empathy, and immersion in the problem, set. Your engineers won't respect you as much. You won't be able to mentor and

develop Engineers as well. You won't have opinions that you can trust about the technology. And there are lots of places out there who just take this for granted. They're like manager, shouldn't know anything about technology. They shouldn't have the opinions. We rely on our tech leads for that. And that is a choice. I think it is a choice that is becoming less and less popular or prevalent because I think the outcomes are materially worse. I think that it serves everyone.

When manager has opinions of their own that they can trust about what's going on. Now that said, you do have to train yourself, not just, you speak up all the time as a manager. This can be very humbling thing. Right? Like many of us became managers because we were Top Dog, right? Like we were the most technical of the most senior whatever. But once you step out of that position and you're the manager, it's your job to guide. People up to reclaim the oxygen that you once, breathe.

It's your job to grow people into that space. And this can require giving up a lot of ego on our parts, but its necessary. It's the best way to grow people up is to have a manager who actually understands what they're going through and can help guide and nudge and suggest opportunities for them. So, I feel like this is a change that needs to be fought on the ground Company. By company. We need to explain to our Executives, why it's better for everyone.

We need to build this into our org charts. We need to stop talking about management as a promotion. A promotion is just a change of career. It's a lateral step. What that means is that it's not a demotion. If it manager wants to go back to engineering. You don't want to have managers who are resentful or who miss being engineers and who like try to reclaim that glory and try to either one, who's always talking who has all the best dancers, you want your managers to went

to be managing people? You want the people who were better as Engineers to try it for a while maybe and then be able to switch back to engineering without any loss of status without any loss of ego. So it's really We important that management is not a promotion to change a career. I think anyone who's trying management out. You should ask for a two-year commitment. Otherwise, it's too rough on the team's. If you're like, shuffling manager around constantly.

You literally have to become a different person to Be an Effective manager. You have to retrain your own instincts, your own reactions of the way that you think you're pacing. You're pacing on it on a daily. Weekly basis is so much different as a manager. You have to gain the Instinct and intuition know when you're doing a good job when you're doing a bad job. Takes at least two years to build. And so in the early days as a new manager. Everybody is all my doing a good job or not.

I do I like this or not. It's like chill out. You don't know, you have no way of knowing just turn the voices off. Learn everything you can and get back to me in two years and then we'll have a conversation about it. But I feel like the industry would be so much better as a whole if we demythologize management. If pretty much anyone who is interested in trying management, gotta shake at it. And then if we ask ourselves, honestly, is this good for me?

Do I like it or not? And there should be a path back and forth. That is well-trodden. I feel like so much of the pathologies that I see on teams come from the relationship between management and Engineers. If we could just make it less of a thing, this make it. Another thing that you can try and it's not a big deal. It's not a promotion. It's not a big ego thing because you're not in charge of people. It's not like that. It's a support role.

I hadn't really thought of this until now, but I think in the way that the interval between, when you write the code when you ship the code is a Building block of a healthy software digestive cycle. I think that pathway between senior engineer and manager, being an equal one. And a well-trodden, one is a fundamental building block of a really good social side of the socio-technical organization. I also think like many Engineers, of course, aspire to

try out management. So the swing from the engineer to manage arrived, and they should because even if long-term, they want to be a senior technical leader. There are things that you will learn as manager, that will give you so, much more empathy will be Q so much more effective. As a technical leader from doing a two-year stint as a manager, I think it's a good thing for engineer to try management and maybe what are some of the tips for like, first, new manager out

there from engineer. They want to try manager, how to become effective in that new rule. Yeah. I think that a lot of it comes down to self-awareness. So number one, I'd say, go to therapy. This is get a therapist and go to therapy every week. Learn stuff about power dynamics, especially if you're a white or Asian, dude, Tech. I think that there's a lot of stuff that you need to learn to

be sensitive to that. You've perhaps never been sensitive to before you owe it to your reports to learn about those things. So, start following feminists and some people of color and learn about power dynamics learn how to listen, all of the advice that I would give for new managers is really about self-awareness and humility and communication skills, which is another reason that I think it's a really interesting two or three year journey to go on as an engineer who's now in your 30s.

Maybe has a family now. I think it's a nice. Change of pace. I think that it dovetails really well with just kind of growing up. And I think one of the reason the pendulum back from manager to engineer, I think it's about the concern that once they go back to individual contributor. For example, it's probably difficult, especially in this region.

At least the way I see it is that you might not be able to get back that management role, especially when people see the numbers of experience the bigger team that they managed before. So, what do you see in terms of this pendulum back? How could someone assure themselves that Create safe. I can go back to I see and I can always go back as being manager. Choose your company's carefully, especially if you'd be worth in

your in your career. You should be interviewing them just as much as they're interviewing you ask them about their philosophy of management. Is it a promotion? How do they think about it as anyone ever gotten back from being a manager to, I see, would you discourage them? Talk to them about it. You really want to find a place that has a culture that matches your own aspirations. Also, like just Comfort yourself that if it's been two or three years, you can go back. It's fine.

Especially if you're a dude, right? Nobody's going to even question you if it's been two or three years. Yeah, it'll take you a month or two. You might have to study a little bit for the coding interview. But, you know, you'll be fine. You might have to change jobs, if you aren't a company that values, this, but it is honestly much easier to go back from being a manager, to be an engineer because this is a superpower, everyone wants Engineers.

There is so much more demand for engineers and there is availability. So, if it's only been two or three years, you're going to be fine, but if you're concerned about that, then make sure it's all For three years, don't go to five years or Beyond, or it could actually be very difficult. So it's been a pleasant conversation. But before we end our chat here, normally I would ask all my guests to share their three technical leadership. Wisdom.

So charity. Do you have some wisdom that you want to share with all my listeners here? Yeah. I think number one is your career is an enormous asset. It is the most valuable thing. You possess. It is a multimillion-dollar appreciating asset and you should treat it that way in my twenties. He's I didn't think of my career that way I just lurched from one thing to the next, you know seat of my pants and it turned out okay for me.

I was pretty lucky but I think that a little bit of introspection a little bit of where do I want to be and pointing your nose in that direction.

It has a way of making you a tune to opportunities that open up. If you're like someday, I might want to be a manager, then you might be more likely to take a role at a company that has some obvious management roles opening up. So I just feel like treat your career, the way that you would want a friend of yours with a multimillion-dollar appreciating asset to treat their assets. Think about it for the long term.

Sometimes this means taking a job with a little bit less pay because you think it makes the story of your career look much better. I think that being in a position where you are not spending all of your money. Every month is a very good tip for everyone. Have some fuck you money in the bank, don't live paycheck to paycheck because then your trap, you can't take any jobs that pay

any less than that. And sometimes you're going to want to look at your career as a long term asset and think about the story that you wanted to tell. And it's not something you have to take up a lot of time and energy, right? Just like a couple hours. Year, I think will really help. It's really valuable though. Like, we are so fortunate to live in this time and be doing this thing because this is the only growth industry of my

lifetime. We live in a different economic reality than most of the world does. Most of the United States has been in a depression for 15 years. Not in Tech opportunities, abound. Yes, Tech has problems with sexism and racism and yet I often feel like I want to just like tone down, so the shit that we tell the kids about because I'm like, if you're telling them, not to come to Tech because we're two, Here, where do you want them to go?

What is this? Magical mythical industry where they will be highly paid in valued? Where they've not going to have to deal with you? This bullshit. This is called living in the world, guys. Like it just is this way and so I think that we owe it to ourselves to get as many women and underrepresented minorities in detect, it helped them survive and tough it out because we're so fucking fortunate and blessed. So that was two or three things. All in one. I guess try to be financially

constrained. Try to be an ally to In general other genders and races. The advice that I always give to young women who ask me for advice is like when you're a junior engineer, just toughen up. This is problematic. I absolutely agree. Tough it up. Try not to take it too. Personally, put your head down, learn as much as you can just survive, become a senior engineer and then get sensitive and then learn to be sensitive on behalf of other people, use

your power for good. Anyone who's a senior engineering Tech has an absolute responsibility to learn to be sensitive and to use our power for good on behalf of those who can't, Wow, thanks for sharing all this. I think it's a message that we all need to learn especially in these days where so many things happening, especially in Tech and around the world. So, thanks for sharing that. So charity, if people want to connect with you or find you more, where can they find you online?

I am on Twitter as Nifty Tipsy. I EverQuest enchanters name and Trudy. WTF is my blog and honeycomb. Dot IO is where you can sign up for a free honeycomb account and use the build event stuff to visualize. Cic pipeline. The way I was talking about. It's super fun. So one day, I'll make sure that I'll try that. You expect less than an hour. It's super fun. You could see all your time is going. It's great.

Although thanks again for your time, charity, honeycomb, and everything that you do. Thank you for listening to this episode and for staying right till the end. If you highly enjoyed, please share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you're new to the podcast, make sure to subscribe and leave me your valuable review and feedback. It really, really helps me a lot in order to grow these podcasts better.

You can also find the full show notes of this conversation on the episode page at technology. Know the death website, including the full transcript interesting quotes and links to the resources and mentions from the conversation. And lastly, make sure to subscribe to the shows mailing list on technology on of the deaf to get notified for any future episodes. Stay tuned for the next technique Journal episode. And until then. Goodbye.

Transcript source: Provided by creator in RSS feed: download file
For the best experience, listen in Metacast app for iOS or Android