The first time we properly, like,
met
No way.
It was It was everything's on fire. Right? It it sounds like I'm making it up for, like, company origin narrative. But, like, genuinely, I think the first time me and Chris spent a lot of time together was sort of over the table, nerding out on some go concurrency.
I remember Tom just just turning to me and being like, I find these instances really, really exciting sometimes. Yeah. Everyone assumes it's, like, officially owned software that is which team owns it? It's, like, it's me. And I've programmed it mostly when I'm watching watching my kids at swimming. He'd done
a dragon in ASCII, and it was just huge. So it was, sort of you had to scroll past this huge dragon in ASCII. And it just said, like, you know, warning, traveler. Like, you know, beware dragons, etcetera.
Our single highest traffics, the blog post.
This episode was actually recorded more than four months ago. And even though it was really, really good, I thought we weren't gonna release it because the audio quality got completely corrupted. But because it was so good, we went back and had a look, and I think we've been able to salvage it. You might notice a couple times where there's a little bit of AI in there. But other than that, I think it sounds great, and it really is such a good episode.
I misunderstood what instant was for ages. My initial impression was that it was, like, kind of sentries, you know, get the errors, but it's really more to actually respond to it, making sure that the right people are on it. It's like kind of when it goes wrong, it's like kind of like your special forces just like come here like you wanna respond as fast as possible.
I want to explain. Every time I explain it to, you know, a family member or relative and things and they're like, what what is it you do? And you sort of explain it. They're like, you you fix things for people. I'm like, no. Not really. Like, we're we're a tool they use so they can fix it themselves. Yeah. And then it it it honestly, it got easier this year. If something goes wrong, a system's gonna pick it up.
It will fire an alert, and that alert has to get to a human somehow. And we're the software that basically fires out their mobile phone, makes a, you know, loud noise and wakes them up and allows for scheduling and all that kind of stuff. So, yeah, it's getting easier over time.
The place we've ended up is sort of not miles different to where we started, but we started in a very deliberate place, which is kind of what we saw as the gap. Right? So we already had PagerDuty. I mean, Chris and I both independently used, you know, tools like that for for donkeys years. Right?
They've been around forever. And I think the thing that always still felt messy and, like, there was nothing out there was that bit where I've been paged, I've been woken up. I'm now sat sort of in Slack, bleary eyed. Maybe if I'm lucky, I've managed to get a coffee, but, like, now what? And and that tends to be sort of every company starting point is if you're lucky, there's a process where it's written down in paper somewhere.
No one's really gonna follow it when everything's actually on fire. And so there was this big gap of, how do I find the right people, get them all together, coordinate response, make sure we're communicating effectively. Like, all that stuff was still very messy, unstructured, inconsistent. And so I I guess, like, at Mondo, they did it very differently, large part due to, you know, the work that Chris has done. And that felt like, you know, when I joined Mondo, was like, oh, yes.
This is how it should be when you have, like, well orchestrated and well coordinated instant response. And it just it just felt like that was the gap. And so we just went for that first. Right?
But I
think that the long term plan was always, you know, why would you only do that? Right? If you're if you're now coordinating the middle bit where everyone's responding, why not go to the world after and do insights reporting post mortems, things like that, and also then go upstream. Like, why are you not handling paging and alerting? Because all these things work so much better when you own When I join the the whole life cycle.
Right? It's like, it's a better together situation where, like, you know, there's some really cool stuff we do, I think, where, you know, because we own the paging product, the Thesis Fader product, and we own the response product, you can kind of stitch it all in. And that just doesn't work if you're trying to glue stuff together with, like, some random, you know, script or bot, which is what most companies end up doing. So, yeah, that's been really fun to see it all come together, but, yeah, very deliberate starting point.
This episode is brought to you by WorkOS. If you're building a dev tool, at some point, your customers are gonna start asking you for enterprise features. Work OS offers you single sign on, skin provisioning, and audit logs out the box. Can I just ask you about Work OS?
Are we doing the good version or the bad version? Yeah. There's not there's not a bad version.
Yeah. Yeah.
So you guys use WorkOS, right?
So we actually use it for several things. So we kind of came across WorkOS when we were going into enterprise companies where things like SAML and single sign on and authentication become like a big, big deal, particularly around user management and things like that. So that was the starting point and massively sped up our development process, so massively worthwhile investment there. And then we adopted it later on for other similar security and enterprise needs, like audit logs. And I think because we predominantly use it for our enterprise customers, the margin cost of that is reasonable.
Yeah. They've been, like, super receptive. I think, you know, it's not a, certainly not a cheap product, but sort of, you know, you get what you pay for. And I think, you know, it's stuff that I just haven't had to worry about or think about as a CTO. There's a ton of things my team haven't had to build, and, you know, that's that's a godsend when you're an early stage company for sure.
Go to WorkOS.com to learn more. For me, at least, like, until fairly recently, I was using a lot of, like, you know, Heroku Digital Ocean, Fuzzy, Firebase. And it's like
Yeah.
Largely, the incidents are not necessarily like a lot of the time the incident is like just wait for it to come back online basically.
Yeah. Off and on again. Yeah. We we started on Hiroki. We we rolled plenty of dinos in the early days. And I go,
and that fixed a lot of issues. Oh,
but it now that I'm kind of, like, trying to play around with my own stuff, it's like, it it gets, like it's pretty scary. You have this really interesting podcast that was, like, super helpful for me preparing, where you're talking to the person that is, like, responsible for Netflix's infrastructure. And it's, like, do you I think you're I think Louise in your team
Yeah.
Asked, like, what is it like when You're the person, like woken up at 3AM to put Netflix back online, basically. And it like, are you scared? Or, like and it it must be, like I guess it's why, like, invest so much money in this space because it's just, like, it's such a like, nothing else matters at that point where it's just, like, just get it back online and you wanna get it back online fast. And and everyone knows how, like, if you've been in that situation, like, you know how horrible it is. It it's it's a very interesting area.
Yeah. I think it is. I think it is. And I think, like, it matters differently for different companies depending on different things. Like Netflix, for example, like, them having an incident and isolated incident doesn't really, like, translate into, like, lost dollars for them in the same way as, an ecommerce company.
So an ecommerce company is downright. They literally can't transact. No one can buy their their stuff. Netflix is sort of a trust thing, I guess, which is like people people just expect it to always be there. You're not gonna no one's gonna cancel their Netflix account because it wasn't available.
They just mean we pissed off with it. Yeah. But, like, regardless of the thing, think you do as a responder of an incident, you end up kind of wearing that that weight on your shoulders, and that that is really why software like the software that we built really does exist and people care about it is because there is enough stress being like, Netflix is down, and, like, you know How many millions millions of users are gonna be using it, can't use it. And so you've kinda got that burden, then you're trying to fix the thing. And then you've got all of the, like, I mean, like, a lot of the time, like, bureaucratic, but, like, often important processes as well inside of organizations of, like, how do we do we communicate with people?
How do we, you know, make sure the right things are happening legally? And the fixing of the issue is at the tip of the iceberg. It's one of those problems where everyone's like, sure. Like, Netflix is just, reboot the database or, like, turn that thing off and turn it on again or, like, go and get three teams to work together to fix the game. That's the small part.
Until we've worked in that space or you've seen it at that kind of scale, it's very hard to conceptualize why it is so important to have streamlined communication, coordination Yeah. Like interactions with all these tools and all the different, like, disciplines and domains inside of a company, all kind of collaborating in one space. But I promise it's it's really important. You know? You should definitely definitely buy software like ours.
I really I really love the idea that, like, a Netflix instant involves Turn the database off and on again, like, you just described Yeah. Someone in Netflix, like,
what the
Having having chatted having chatted to their team, I think, is a bit more complicated, but I love the idea that they just have, like, an off and on again swing. Yeah. Push the red button. Yeah. Yeah. But, yeah, it's it's it's it's kinda wild. And, you know, case in point, we were chatting to them, like, recently. We launched some new stuff around, you know, live call transcription. So it's kind of you you have, like, you're chatting in Slack. That's great.
You can see everything. We can see everything. We can do some cool stuff with that. But, like, the moment you jump on a call, you're gone. Right?
And so we were like, oh, well, in the world, tools like Gong and other fantastic products, like, why isn't that for instant response? So we we built that. So we're talking with them about that. And, like, we're used to maybe, for us, an instant call is maybe two or three people coordinating. And they were, like, really excited about this, and they gave us some really great feedback.
And I was like, oh, I have interest. Like, what does a call look like for for for Netflix? I'm yeah. I mean, sometimes it's like 60 people. And you're like, oh, wow.
Okay. Yeah. That's, like, totally different to my experience of smaller instant. I can see how this would be, like, 10 times as useful for you folks. Just in a larger company, there's so many more things to deal with and, like, you know, even stuff down to you know, I know everyone that might be in an instant Instant IO if you're a large company, and I and I've been there as well. Like, you turn up in an instant. It's like, we've never met before. I think, like, when We had
this at Monzo.
Yeah. When Chris and I met for the first time properly, professionally. So we'd, like Cross Boston. Mostly online events, stuff like that. But the first time we properly, like, met No way. It was It was everything's on fire. Right? Because Chris was Chris was director of platform. I was a sort of senior staff engineer over in operations. And so, like, when the whole thing goes down, like, customer support's on fire, platform's on fire, and then it was, like, respond, follow-up.
Right. Post mortem, we had, like, super in-depth debriefs and stuff like that. So it it sounds like I'm making it up for, like, company origin narrative. But, like, genuinely, I think the first time me and Chris spent a lot of time together sort of over the table nerding out on some go concurrency bug that made fans.
I remember. Yeah.
It was fun. It was fun. It was fun.
Yeah. That is wild. Yeah. Yeah. Netflix is an interesting company, actually. It sounds like I mean, obviously, for for other reasons, but I've heard that they're very, like, kind of they will adopt, like, new relatively new startups, like, quite like, openly given how big they are.
They've made a big shift, I think. So this is, my external perception and sort of a little bit of, like, what we've heard from them. Like, Netflix used to have a very, like, build build it yourself kind of culture. So there's a lot of a lot of software that they wouldn't go to market for. And so when they when they when they came and spoke to us, for example, they were like, we have two options really.
And they evaluate, like, us against a bunch of other folks or or we just build this ourselves. And, like, we I mean, like, money's not the concern for them at that point. It's sort of, you know, they'd happily throw two very well paid engineers at the problem and be like, and they could already build something pretty pretty good and pretty bespoke there. But, yeah, it's I think there's a bit of a culture change like like a lot of companies are doing honestly, which is like, just realizing that there are certain software that is just it doesn't make sense to to build a tool. It's like I mean, it's like there's some it's obviously, like, established norms in organizations.
So it's like, if you were starting a startup, you wouldn't be like, step one, before I start building my product, I'm gonna build spreadsheet software because we need spreadsheet software. He'd be like Google Sheets, obviously. And the same is true of, like, OnCall. Right? So OnCall is is now, like, established enough.
So you'd be like, I'm gonna use PagerDuty or Opsgenie. Right? And now, you know, Incident.io, obviously. But, like, where where we started, it was in that weird space where everyone is, like, nascent enough domain that people are like, yeah. I can build a Slack bot.
I'll I'll do it myself. Like, I don't need incident IO. When you start building in that direction, as I did at Monzo, so I wrote, like, a a a really, really simple version of what we have incident at Monzo and then open sourced it. And, like, I started building and say, this is great. I want it to be able to control my status page and, like, escalate to people from Slack and, like, do a little bit of, like, orchestration of process and stuff.
And then it very quickly became I've just launched a product internally, and now I'm a I'm a sideline project manager for you know, product manager for this this product that I never wanted to be in a role.
Asking for you know, you go in, you're like, it'd be really cool if it did this. And Chris is like, yeah. Yeah. It would. It would. I don't have any time to do that.
Yeah. People are making, like, very Cool requests welcome.
And you're sort of like, but also, like, I've got another job to do. And didn't you build this? And isn't this your thing? It's Yeah. Assumes by
no one. Yeah. Everyone assumes it's, like, officially owned software that is which team owns this? You know, like, it's me. And I program it mostly when I'm watching my kids at swimming.
In some in dodgy Python.
Yeah. Yeah. Not dodgy. Very good very
good Python. Yeah. I'm ready pick to differ. Yeah.
Yeah. That's that's so funny. And I I guess, like, you also kind of, like, practice this as listening. You know, obviously, you guys are very, like it's a very technical team. But I think to start off with, like, you were building with Heroku. I thought that was quite interesting that, like, you were, like, just we need to just ship fast.
It was so it was so refreshing. I remember I remember so, like, coming from Monza, where Monza, very, very technical infrastructure, they had, like, thousand I mean, famously thousands of microservices all running on a home, like, built Kubernetes cluster, and, like, you needed a full team to run it. I remember it just being, like, a incident. It's just poor requests and code. Push to master.
Yeah. Push to master. Auto deploy.
Yeah. Yeah.
I mean, to be to be clear, when we started to get, like, real customers, there was, like, some rigor and some process. But in the early days, we were just like, does this thing even work? Yeah. Like, I mean, yeah, you don't need all the bells and whistles safeguards.
We didn't have monitoring for so that was the thing that so one of the big one of the biggest amount, like, amount of time I had to spend at, like, later years at Monza was, like, on the monitoring stuff. So they like, they were, like, collecting and ingesting, like, millions, maybe billions of samples an hour of, like, all these different things and, like, so that everyone could power dashboards and stuff like that. And I remember starting here and be like, that's a bit of work to do that. We're we haven't got customers yet. And, like, when we had the first few guys, was like, even now we've got logs so we can sort of figure out stuff, and it's not really holding us back from debugging stuff.
And a problem.
It will get reported to Sentry, and then we'll get a notification, and then we'll fix it.
I think that's it. It's like it's I think, yeah, early days was all about sort of, you know, trying to sound too sort of self congratulatory. It's like think it was just like a maniacal focus on the product and then build the product, and, like, everything else was just like, let it burn. If it's if it's good enough, it's good enough kind of thing.
Yeah. It was like until someone's paying you and your product is sort of starting to grow and hit product market fit, like, nothing else really is that important. Right? It's like I I think one of the things that so, Heroku, for example, you you know, a lot of people would call out, like, surely for an incident management platform. Like, is Heroku the most, like, scalable, stable platform?
My honest answer is, like, yeah. It was pretty awesome for for the first sort of year or so. And we managed to scale far past where I thought we'd get to. But I think sort of one of the things that really helps is, like, you you can make trade offs like that where you kind of go with a, you know, a thing that will last you for a little while but not forever because you don't know that you're going to need something much more scalable, or you go there precisely because you do know you need something much more scalable and you know now is not the time. Right?
And so what was interesting is you've got you know, me and Steven and Chris spent each spent, you know, a decade plus building stuff. And, you know, I'd seen GoCardless from, like, tiny literally also Heroku, you know, esque shape, tiny tiny, like, needs all the way up to massive infrastructure. And same probably at Monzo for Chris. I think he probably saw some of the most rapid growth. It was, like, you know, the 500 k to 5,000,000 journey and all that stuff.
And, like, when you've done that, I think it gives you the confidence to go, like, well, I've seen it done both ways, and I kind of know that we don't need any of that stuff. And I think it's almost more intimidating. And I started out. It was like, do I need to care about all these things and scale and what have you? And I think part of the reason we went for so many, like, out the box, amazing, you know, put in a credit card and buy it, shape dev tools is, like, we we will know when we need to change.
And therefore, I am totally comfortable taking any risks or taking shortcuts. Because, like, when it becomes a problem, like, we will know because we've done this before. And we did. And eventually, we sort of, you know, switched it up and reached a point where we moved off Heroku and did a migration, and that was all totally fine. But we got probably, you know, massive acceleration boost in the early days.
And it wasn't just Heroku. There's other stuff as well. And some things last you for ages, like Sentry No problems. Still with us to this day. Fantastic software. Like, probably some of the best money I spend as a engineering leader. So yeah.
Yeah. It's it's something that I think is such, like, quite important because I feel like in DevTools especially, there's almost like a bit of I I don't know if it's like it's the right word, proud to be doing something in, like, a hard way.
Yeah. It's like blog post driven development. It's like, yeah. Look how we built a really, you know, really, like, complex thing. And you're like, early days, it was like there was nothing to blog about. It was like, it's a very simple Go app that takes JSON and does some stuff with it and turns it into JSON elsewhere. Got more interesting.
Got a friend that drives I I I think it's back at my time at GoCardless, and he was he sort of boiled down our entire job in, like, a few sentences to sort of, like our our entire day is essentially spent putting things in a database, taking it out and showing it in HTML, and then putting it back in the database. Initially, you're like, no. It's way harder than that. Mean, you're like, actually, a lot of it is kind of just that. And until you get to a certain scale, honestly, like, a lot of it, you know, just isn't that complicated and you don't need it's like, what matters more, like, having it be ready for the billion users that you may or may not ever get, or, like, getting a user.
It's a bit cliche, but it's like it's it's very, very true. Like, the corners that we cut were horrendous, and my team would get a massive slap on the wrist if they did some of the stuff we did in the early days now. But that's because we have, you know, hundreds and hundreds of customers and, like, billions of requests and hundreds of thousands of incidents. And it's like, yeah. I mean, the game changed.
Do know what you just reminded me of? You just reminded me. I just remember a hilarious comment where I was mid incident, and I sort of found this found this comment on this thing, and it said to do, obviously, not this. Yeah.
That was, like, this one. I remember it.
It's just like, I know it is absolute ass, whatever, and I'm gonna
ship it anyway. There are a
lot of us. Obviously.
Yeah. It's just like, if you if you're an engineer in the future, this this was deliberately rubbish.
Yeah. Yeah.
Yeah. Like, please please rewrite it actively. If you're looking at this, that is enough signal that you should redo it.
I love have you seen online, like, the ones where people, like, leave comments of, like, hey. I came to this function probably with the same assumptions of you of, like, I can fix this and make it better. Here's what I did. If this is your set of steps you're gonna follow, don't do it. Like, just leave it as is.
One of my favorite files at GoCardless was we had this we had this file which is a little to do with, like I think it was to do with, like, some payment processing bit. Was, like, very gnarly, very complex, extremely well tested, to be clear. But it was, if you're coming here and changing it, it was, like, very you know it's highly risky. And our CTO at the time, Harry Maher, had like, he'd done like a massive he'd done a dragon in ASCII and it was just huge. So it was like sort of you had to scroll past this huge dragon in ASCII And it just said, like, you know, warning, traveler, like, you know, beware dragons, etcetera.
And I just thought I that was really fun. That was a moment of just like, okay. Yeah. The joke, but also, like, I will be a little bit more careful. Yeah. And then eventually, we had to, like you know, in terms of, like, scaling and getting better, eventually, we replaced this piece with slightly gnarly functionality, and the PR was called, like, you know, slay the dragon. Yeah. It was it was great. It was great.
That's so funny. That's so funny. Yeah. One one of the things I really like about Instant is I think I think you've got, like, a really good sort of brand around, like, you know, I think you know, like, who you are as a company. It feels like maybe that's like if that kinda makes sense. Yeah. Like, I'm even looking at Chris' Chris' merch that's like Yeah.
Move fast when you break things.
Yeah. Didn't get me ordered ordered a bunch of these brand new things for the for the team. Didn't get one. We were on the way, and Chris is like, it's great. I'm gonna wear my new wear my new hoodie. And I was like, cool. Do I get one? He's like, no.
Clarify. You in the company gets so this is, like, annual annual gift for employees around, like, Christmas time. This is this is the beginning this year. Pete had got one. I didn't give him his. We're giving him out this afternoon. Didn't give Pete his because I was like, if we sat here wearing sort of matching hoodies. Matching hoodie, it's a little bit little bit sort of trite, I think. Great. Yeah. Yeah. Well, at this stage, I get to wear it and
Swap mid podcast. We could, like Tell me what we
could do. We could swap
and share mid pod. Like parents with kids that can't share. And it's like, well, you get a bit of time each. Sure. Sure.
But, yes, to your point on brand, to bring it back, we we do care we do care a ton about this. I think it is I think it's, like, often underrated in start ups. It's sort of, like, a bit of, like, what is your identity? What are you about? Like, you know, the the I was listening to the thing the other day, and they were like, you know, brand is brand is like you know, if if Nike was to open a hotel, you know exactly what Nike's hotel would be like kind of thing. Whereas, you know, if, you know
I'm trying think what our hotel would be like. Yeah. Well, yeah. A lot of fire. Yeah.
Fire and, like, lots of, you know, little comedic quips. We sort of have a few, like, sort of principles around this in in marketing, and nothing super, super concrete. But, like, it's sort of, like, be personal. Like, don't use don't use marketing y language. Like, inject humor. Like, we're we're in a very serious domain. Like, incidents is not really traditionally, like, laughing laughing matter.
It kinda speaks for itself. So being way is
up. Right? That's it. Yeah. Yeah. And I think I think being able to lean into a bit of the you know, our website, the h one r website is literally move faster when you break things, and it's sort of, you know, great feedback on that as, like, a little a little thing. But it's like I think, yeah, it's nice to be it's nice to inject a bit of, like, humor and sort of playfulness into that stuff. So, yeah, I think work to be done on it still, but,
yeah.
Yeah. Kinda right.
Yeah. I guess it makes sense, I guess. It's like, I feel like a lot of, like, sort of, like, war like, have, like, dark humor, and it's like it feels like it's, like, the closest thing to that in-depth.
I think I think, you know, a lot of it you'll find in incidents as well. And I I I think, you know, I've been in plenty of instances where sort of, you know, that's that's been well received and plenty others where it hasn't. But I think there's, you know, we talk about, you know, sometimes you just gotta, like, laugh, otherwise, you'll cry. Right? There's there's kind
of You're working on experience.
You know, that I've been in in my career. They're the same, if not not worse in some cases, where it's like, you know, it's it's really, really difficult. Right? You're kind of you know, everyone's super stressed. They're, like, extremely emotional things rather than just, like, super logical engineering y things.
It's like no. There's, like, like, active stress. Like, you you know, if you finish an instant and it's been a really heavy one, like, you need somewhere for that tension to go. And sometimes all you need is, I think, you know, when everything's on fire, maybe just like that moment where you've stopped the bleeding and everyone just needs a bit of pause or light relief and, you know, someone to just, like, break the tension, sort of lighten the mood a little bit is is really, really helpful. Because otherwise, you you can spend nine hours in sort of, like, you know, super frowny, stressful, like, you know, it it it just feels like you're carrying the weight of the world on your shoulders and kind of a little bit of humor goes a long way.
It definitely And that's our positive, I think. You know, it's it's like it's it's all about the judicious use, I think, which is like no one wants, you know, the the comedian in the corner. We're like, no. We really have to fix this thing. And, like, Pete's just cracking one liner.
You fix it first. Save the one liners for after, but, like, you know.
I remember
I remember a really bad incident at at Monzo, one that we wrote about publicly, a big public post mortem. Wrote up about it. I remember sitting in a room that was called the Summer Of Monzo. Like, I can't remember the the reason for that that meeting room being called that, but it had, like, summer themes. It had, like, an inflatable palm tree.
There was, like, a deck chair in the room. And I remember Tom, the CEO, came in and we're sort of this is, like, the sort of war room of the incidents. Had, like, team leads and stuff, and it was lots and lots of teams involved. So it was, like, the central place, and I'm sort of trying to coordinate stuff. I remember sort of the the stress of being, like, and also the CEO is right here.
Was it was really serious. I remember Tom just just turning to me and being like, I find these incidents really, really exciting sometimes. And and that is not that's not to say that, like, he is sort of, like, making light of it. Right? He was I mean, he's gonna be on the receiving end of press and was having to do comms with regulators and all this stuff, but it's like, I don't know, it's just that that moment of just like someone being like, it's okay.
Like, I sort of trust that the right things are happening. The right people are doing the absolute best that they can. And, like, it's not not on me to kind of lay on extra pressure here. And, like, a little pressure release is actually probably a good thing. But, yeah, I I sort of I don't know. I kinda think think about that a lot in incidences to channel a bit of my inner Tom Blom feel from from that day.
Yeah. What a guy. Yeah. Like, stepping back, the bigger point is it seems like you guys have a very strong understanding of, like, your users and the kind of culture of them. And do you think it's just because of your backgrounds, or is it something that you also put, like, a lot of effort into?
I think I think it's a bit of both. So it's like, you know, we are obviously, like it's like building a little bit with cheat codes when you're building for, you know, your past self kind of thing. So it's like, you know, would this feature have been useful for me when I was running an engineering team of, you know, 30 people or something like that? And you're like, yes. I know exactly how that would be there.
So that that's really helpful. I think it can be also a bit of a curse in that you lean a lot on your own experiences, and that can cause you to shortcut the, like, go and validate with customers sometimes. And so, like, sometimes you get things wrong is is another part of it. But yeah. And I think I think, like, the other thing that's sort of a bit of a cheat code is that we're, like, massive nerds about this stuff. Like, honestly. So it's, like, it's not And some
of us are bigger nerds than others.
It was yes. So I think in terms of incidents, I would say I'm I would say I'm, like, a big, big nerd about those sorts of things. And so, like, like and a good example, case in point, I wrote I spent several weekends reading, like, a 84 page report that came out of the National Air Traffic Service about a massive, like, incident that they had in 02/2023 and sort of went deep into the depths of, like, what was the technical fault, which was, like, a flight plan ingestion issue, and then, like, how did they respond to this incident and stuff and sort of
wrote Super interesting.
Into a blog post and actually was received really, really well. And, like, I know I really like that kind of, like, translate very complicated report that's very big into simple thing with, like, some digestible takeaways. The point here is I I think we spend a ton of time just nerding out on this stuff. So whether it's, like, product things we're nerding out on or, like incident y things, both like in engineering fields and outside. I think all of that just lends itself to, I mean, it's like basically at this point, you know, it's like my life is my work and like it sort of integrates seamlessly and fluently with my family and my kids and things like that.
So, yeah, I don't know.
Anything I I I agree. I think that's sort of the bad version of the world, which is like, I don't I'm doing it, and I don't have a choice. And then there's the good version, which is like, I've managed to meld a thing I love working on day to day with family and stuff outside of work. I think on the on the sort of personality and brand side, I think some of that is some of that is extremely deliberate, some of that's just natural and passive. So even if you look at, like, a lot of the, like, marketing copy and tone, right, you could say, like, oh, we've made a really active decision to not take ourselves too seriously, to not use, like, super serious corporate jargon.
It like, honestly, it's not that deliberate in the early days. That's just us being us. And then actually, we found that, like, it's much more natural. It feels better. Yeah. And then that starts to translate into, like, the product. Right? We've sort of tried to get like, harness it and capture it a little bit. And then you do that for long enough, and suddenly the people you hire start to feel like that and emulate that. And then, you know, your branding starts to feel and emulate that.
And eventually, you're kinda like, this sort of feels like everything just reflects. And now it's in our marketing, and now it's in our product, and it's all organically evolved. And then you can start to dial it up, where you're like, okay. Well, like, this is good. We like this. It feels good. How can we do more of it? You know? And then then it spreads into it spreads into your, like, events and the way you kind of run your conferences and things like that as well. Mhmm.
Yeah. That's been just really nice. Versus, and I I never felt like as a company, we had to be something different to what I would wanna be as a founder,
if that makes sense. Like, I
don't have to talk differently now, for example. I'm not like, oh, I have to go put my, like, podcast face on. Yeah. I'm just like, no. I'm just like
He's never a media trainer is what
he's Yeah.
Yeah. You wouldn't believe
that. Yeah.
Yeah. And by the end of this podcast, you'll know exactly what Christmas means. No. But it's like but that's really nice because it means you you you kind of get this authenticity that comes through. And I think to your point on, you know, does it really help you understand your users and does it help you sort of build better product for them?
Think absolutely does. Because if you kind of understand your users and you can kind of speak their language and kind of do that authentically, I think that makes a massive, massive difference. And I sort of, I have a lot of respect for but I'm also slightly kind of you know apprehensive or confused by like companies that are desperately trying to sell stuff to people they don't understand or that they haven't, you know, who would experience this they haven't lived for example.
I'm just like, how do
you do that? Because like I can sit with an engineer or CTO and become literally nothing else. I'm just like ever had stuff blow up in production? It's like, that's the easiest conversation topic starter ever. But like, I was trying to sell something, I don't know, to someone that I'd never
Accountancy software is always the reference point for me.
Like, you know Regardless, honestly, was like, yeah, I found that tough. Like, internally, I really loved it. The team team were whip smart, like product was really great, like super successful business.
Like You weren't passionate about direct business. But but but like,
you know, joining us like direct debit and accounting software integrations and like, you know, those customers, it's great to serve their needs, but it wasn't like, you didn't get the buzz of, like, I know I've solved your problem and I get it and I know why you love this and I love it too and like, yeah, that makes a huge difference. Also just tactically on product, it helps you make better decisions much quicker. You can follow your gut versus needing to research and validate every single decision because you don't know.
I've, you know, followed you guys for a while, and I realized that I've actually seen a lot of posts not from you founders, but just, like, from your team. Mhmm. So I think one recent one maybe this was actually Founders actually. This I'll just say this one I wanted to say was, like, quite funny that was, like, the Christmas tree. There's been an incident, like, you've ordered a Christmas tree that's, like, too tall.
But the team were the ones that, like, you know, did it Created the and they sort of yeah. They created the instant.
Yeah. That was very funny and very on brand with, like Yeah. The whole thing. And then I think also Lawrence and your team wrote this, like, really interesting article about, like, buying a new m free.
So interesting that that over this year, our single highest, like, traffic, like, blog posts we've we've pulled the stats actually recently.
That come it comes up so when you go to, like, dinner with you know, for engineering meetups or whatever, that quite frequently comes up in the in the context of, like, other CTOs being like, yeah. Thanks for that. Basically, just gave my entire engineering team a, like, sort of logical and mathematically backed up argument for why I had to buy them on new laptops. The thing I enjoyed most Sorry. Not sorry.
What I enjoyed most though was the hack it made it made its way onto Hacker News first put on front page, and Pete just got absolutely eviscerated by Hacker They
always wanted totally level headed and reasonably Yeah.
Oh, you got because
difficult and comprehensive argument just to get a new laptop. And you're like, I think you misunderstood what happened here. And we actually had quite a lot of fun. And then the other people who were, like, sort of flaming it were like, this is really dumb. Like, you don't need to do this. What a waste of money and time. And he was like, I think you misunderstood. Like, we're a startup. This kind of stuff is fun. I know that it's gonna go down well.
Yeah. Was it was probably an offhand comment where everyone's like, can I
have an m three?
And you're like, yeah. If you can justify it Yeah. Go for it. And he's gone. I'm gonna know that.
The conversation was actually a massively overkill Yeah. Analysis.
Well, it it was it is more like the the it actually turned about turned about was like we had, like, three people come up to me and be like, you know, what's the policy on laptop replacement? And I was like, oh, yeah. It was like Apple's release, you know, whatever announcement. And so they went into the team and was like, stop asking me for new laptops. You've all got exactly the right laptop that you need.
And then but I was like, but what I'll do, I'll order a test laptop. And like, if we can prove that it's gonna be faster, then sure. Let's do it. It'll be fun. I'm a nerd.
Right? And Lawrence was like, I'll do like a bunch of analysis. And actually off the back of it, I think he puts it in the post, but it's like, you could see it as a sort of gimmicky stunt or just like a fun side quest. But actually some of the stuff we did was really, really interesting was like instrumenting all of our build times and telemetry on all of the laptops, which we still have now. Like kind of didn't have it before, but I can tell whether we're building faster or slower than we were before.
As the app gets larger, it's like, oh, people are waiting like more time for their builds to happen on their local machines. Actually that's kind of cool to know stuff like that. We got actually quite a lot out of it that's then sort of followed us through to this day. So it's kind of partly started as a bit of a joke and a bit of me just baiting Lawrence and doing some work to kind of get some cool metrics
Yeah.
And so it turned into, you know,
a pretty honestly, like, pretty solid decision. My very first company I worked at as a software engineer, it was the product I was working on was like a c plus plus monolith. Right? And it was built in Visual Studio, whatever it was. When I joined the company, it took twenty eight minutes to build. So you'd make a change, and you'd be like, right, off to go and make another you
you literally can count the number. The x k c d. Right? It's like, why are messing around with too? I'd I'd be
like, right. What's like, I would You can count the number of changes you can make in a day and test as a result of that. And so I remember, we had the same thing, was like every time someone got a new CPU, be like, what's your build time? By the end, it was down to nine minutes. A mixture of hardware and optimizations around build, but it's it is meaningful, especially at the time when when Lawrence was asking for that.
It was like right when we were building on call and we were building mobile app, heavy, heavy compilation. So it's not like, hey, we're running, you know, a a Go binary which, you know, compiles in twenty seconds versus, you know, twenty five seconds or whatever else it might be. It's like meaningful amounts of amounts of time.
Yeah. Because mobile dev was the prompt for like, hey, like, this might be actually really good because it's got more calls, etcetera, etcetera. So as a general principle, piece of sort of culture that probably is worth being aware of or like, you know, you don't necessarily have outside the company is the kind of focus that we have on that dev experience, right? And it's kind of, you know, people are pretty allergic, like the build times go over, you know, whatever, like five minutes, for example, in CI and, you know, we have tons and tons of, know, every six months or so we'll do a big bash to like build the pull the build times down and it's that time of like, you know, the thing that I care about is like from the moment that the thing works on your machine, how long until it works in prod And and like how short can you get that when you culturally embed it? You know, you end up with, you know, fun scenarios like that, but they're also kind of, you know, they're material ones.
We we track all this stuff and, you know, I think it makes a difference. Like teams that ship quickly ship more and that's that's great. Right?
That's amazing. But still, the the more combinations of, like, fun but serious. It's like It's great. Yeah. Think we're actually probably slightly getting towards the end. And I realized we haven't talked of we haven't talked about AI and building with AI.
We haven't. We've always do a whole short course. Talk every
day. Yeah.
I think you have some pretty interesting takeaways on implementing, like, Gen AI stuff into into your projects.
Yeah. I mean, there's endless things we could talk about with AI. I mean, there's there's the stuff we've built. There's the stuff we're building now. There's the stuff we're building next. We can kinda cover all of it. There's Yeah. Endless
I think I think if we we've had we've had, like, a I mean, like like most people, pretty fun journey with AI. So, like, we I think if I look at this year and reflect on what we've done, we've done the the sort of sprinkles with AI across an existing product surface. So, you know, and and that that sort of not to do a disservice. Right? It's really, really good stuff.
So, like, you know, LLMs, like, they're they're they're sort of, like, core at taking corpus of language and doing other things with that language. Right? And so when you look at incidents, which are broadly just like a bunch of people speaking about things, it is very obvious that that is a great application of, like, being able to sort of take that, reduce it, sort of munch it into different formats and stuff. And so things we shipped this year were, like, auto generating summaries of incidents. It's like, you know, someone who joins, then can they very quickly get up to speed on what's happening?
And, like, in most incidents, let's say, it's quite before AI was like, no. Because no one thinks to do that because they're optimizing for, like, how do I fix a thing fast and keep moving forwards? And it's really important. Right? So, like, that that's my way of staying on top of incidents happening at our company.
Right? As I jump onto the dashboard, I have a Kanban board, and I'm like, brilliant. So, anyway, like, that works phenomenally well. Like, the graphs of, you know, people now saying incidents and, you know, we sort of ship that with that human in the loop kind of mechanics. So it'll always be like, we're not automatically doing things.
There's, like, zero risk of or or close to zero risk of hallucinations there. So, you know, saying the wrong things. So people have to go, yep. That's right or no. That's not quite right.
I'll edit it before I set it. Things like pulling pulling, like, actions, like follow-up actions from incidents out of the what was said. So, like, there's so many times in incidents where midway through the incident, you'll be like, oh, we should really, really remember to fix this because this this thing's tripped us up many times. And everyone's like, yeah. I'll do that.
And then no does it because you forget, and it's hidden midway in a channel, and we now have a thing where when you close the incident, we'll be like, hey. Here are the five things that you suggested you should follow-up with and do that. Do wanna track them in Jira?
And here are three things that you said you wanted to do but didn't turn into follow ups. And, like, maybe you wanna create these and you're like, oh, Thank you. Click, click, like that.
Just like onto your backlog straight away. The stuff that I think is more interesting is that's like, I think, table stakes kind of things and, like, auto naming of incidents. Right?
And it's like, it's cute. It's really helpful. Functions. And and and, like, super popular as well for as well. These aren't, like, edge case features.
It's I think, you know we had some crazy stat like within like three months of launching the auto summarization update stuff it was like you know 60% of the updates that are being shared had been drafted by our sort of you know AI first which I think is really, really cool. So it's if you you can view that as, like, you know, how much time have we saved people by, like, sort of doing those drafts. So it's it's very material, but, like, it's very much Sprinkles.
Yeah. The cool stuff, The really good stuff. Shipping in, like, private alphas and and sort of betas now. Now next is two two sort of things we're working on, which we're really excited about. So number one, sounds a little bit sort of trite, but is a really, really useful feature, which is essentially inside of an incident channel.
Like, the the sort of status quo is that you have to use slash commands in Slack. So you'd like, slash incident page p, and you'd be like, great. You do that. And it's like for power users, great. But for for people who are like, this is my first incident. I haven't done it before. We now have the ability for you to just be like, incident, can you grab can you grab Pete for me? And it'd be like, great. Here's a button. Push this.
It'll page him. And it's like, incident, can you just pause this, log a follow-up, and remind me about it in the morning? It's like, great. I think a lot of people are shipping, like, these kind of assistant y type things, and they're they're a bit like hit and miss. I think it's really, really good.
I mean, you see how it's changing how the team are interacting with incidents in in there. So that's really good. And we we did that partly because of that sort of human to us kind of interaction, but also as a foundation for the other stuff that we're working on, which is like the the foundation there was broadly, like, can we get an AI thing to understand language and take action off the back of it or prompt action is maybe a more more accurate way to put it. The thing that that leads into is this thing we're calling investigations, which is broadly you know, if you look at the first fifteen minutes of an incident, it's like trying to orient yourself around, like, cool. I think this should be working.
I don't understand why it's not working, but I've been alerted that it's broken. And you're like, great. Well, what do I do here? I say, first, I'll go and, like, look at what the alert is, then I'll go and jump to the dashboard that is behind that alert and look. Are the graphs looking funny? Is there a spike? Has something changed? Has something stopped running? And then you'll maybe go and look at, like, past incidents and be like, well, hang on. There's something similar happened before.
Has anyone shipped anything to production recently? And all of that is, like, a load of time. Like, it takes ages to do that stuff. And it's a it's it's, like, really undifferentiated stuff for a human to do. It's essentially pattern matching against a big corpus of data is that the sort of very reductive way to kind of phrase that.
Will go off and we will go, great. Based on the alert, based on things that we know, we will kind of plan out an investigation for you and we'll go, great. We're gonna we think, you know, we're gonna check code changes. We're gonna check, you know, know, your metrics. We're gonna check all these sorts of things.
And then we can come back with it, like, based on everything we've seen, this is this is probably what we think is is is wrong. You go and validate that. And here are the next steps that you can follow here. So, like, last time this happened, we immediately paged the CTO because it was really, really bad. You'd like, great. I'll do that.
Probably won't do
And so it's still very we haven't haven't taken the leap into, you know, AI runs my incidents for me and fixes things. I think that is something that the industry and the broader, like, you know, world is, like, very, very sort of, like, hesitant about, like, I think rightfully. And so for us, it's very much, like, how do we arm humans to make them much more effective and able to focus on, like, differentiated parts of what they can do best in incidents. So it's super exciting. It's sort of we we have we've been running ourselves now for a while, and there is, you know you know, a bunch of times where you're like, this is fucking amazing.
And then and you're like, this is the dumbest thing ever.
It's a it's a roller coaster, but, like, when it gets it right, like, it's amazing. Yeah. Like, you'd it was like, oh my god. I I can't believe it just did that. Like, basically debugged and it was correct. Yeah. And I think sort of yeah. I guess to to to Chris's point on, you know, I I think we are seeing probably a competitor pop up in this space weekly at this point. It's like which is which makes sense. It's like a fantastic thing to go after.
And I think a lot of people are starting with the like, I'm gonna build like the perfect AI SRE and like, you know, that'll be in your instant and it'll like fit your Kubernetes for you and like, you know, it will replace infrastructure engineers, all that kind of stuff. And I think like as a North Star, I get why you'd aim for that. But I think, like, as a practitioner and someone that's, like, been in the Slack channels and been on the ground, I I don't want that. I don't I I I describe it as my team is, like, the most likely outcome there is that you end up with the effect, which is kind of like the, you know, massively overconfident senior engineer that sort of thinks they know it all and just makes it all 10 times worse. And, like, that's not what you want.
You don't wanna turn up and have someone just being like, just trust me. I've got this. It's
like, no
no no one when everything is on fire wants someone to turn. To, like, leave this alone. They've got it. Like, what you want is a super smart person who has checked all the sensible things at the start of an instant, can download that context really, really quickly, get you up to speed, and then go, cool. Where do we go now?
Yeah. Right? And maybe you go a step further, which is like, and here's some options or suggestions. Like, here's some routes. Like, what would I want from Chris if I rock up to an incident and Chris is there fifteen minutes before me? That to me is like the bar that you should well, maybe you should shoot a little bit higher than than Chris,
but like, you know, yeah, exactly.
Maybe like if, you know, Chris Barra is here, I'd go like a little bit higher than that. But no. But like that, I think is what you're shooting for is like, you know, you want the the kind of amazing partner versus the kind of like, I don't even have to touch it. And I, you know, I think it's going to be a point of contention and debate, but I kind of, you know, sit pretty firmly on the side of like for now what everyone needs is superpowers not to be replaced. And so we'll start there and then we'll sort of see where we where we end up.
Like so far, honestly, it's like super, super exciting. You can think of it as, you know, you build this thing and it's this like tiny little agent and you sort of start with, you know, what are your constraints as an instant responder? Like when you when you enter an instant and I think, you know, step one is can it sort of make certain like scale blockers go away? So it's like I can check some past instance, but I can't scour 50,000 of them. Like we can do that in like seconds.
Right? So you start with that kind of stuff and then you start to layer in like, oh, can you go and like poke things for me or read my dashboards or and bring that context in. And then you start to get into a lot of like, can you make some suggestions? Layer it all on and every improvement you make kind of feeds into the next improvement. More and more context, more and more abilities, and you can kind of start to just remove these bottlenecks for responders one at a time.
It's like, what is actually slowing you down? And it's usually access to context, ability to do a bunch of stuff really quickly, do things in parallel. Yeah.
Yeah. Final things that I wanna ask you. So two quick rapid fires. So one is what takeaway you would have for other DevTools founders Mhmm. If there was one thing you wanted you would advise them. Yeah. And the second thing is if there's any DevTools that you're using at the moment that have been really handy that might not be on people's radar.
I'm gonna say the second question first because I so I I I I think it qualifies as a Dev. Maybe it doesn't qualify. Linear is the thing, and we're like, I'm so I've I've come from original days of this company building oh, then we use Linear for that. And so me, Pete, and Steven will be like I was like, oh my god. It's like it's like Jira, but someone's reimagined it better.
Gone away. I now work in, like, the commercial side of the business, mostly in marketing. And I've just introduced Linear for the marketing team, and I'm so flipping excited to be back on it. It's like we're we're sort of considering, a wholesale company operating system exists around Linear. So big fan
of that. You can have the wholesale customers. Yeah. Just saying. Very good customers as well. Yeah. Kind of quality bar is extremely high over there. So yeah. Yeah. It's neat.
There's there's tons. I think like I don't know if you I don't know you can qualify as a dev tool, but think one of the things that I've seen a massive uptake of in the team and a huge accelerant for us recently has been, you know, the team have all been using Claude. And, like, I kind of initially was like, yeah, sure. Like, here's a credit card. Like, you know, knock knock yourself out. But actually, I've I've sort of sat with a few of them while they've been pairing and using it.
And using cursor or are they using?
They're using Claude like a sort of Attribute to you. Yeah. So they'll kind of jump in and they'll be like they'll screenshot a piece of UI. They'll we've pre trained Claude with, like, our tailwind rules, our style guides. Like, we've basically told Claude how our entire app is pieced together, and then they'll just, like, ask it a question, throw it a piece of UI, and be like, can you make this on the right versus the bottom?
And Claude goes, yeah. No problem. Here's the code. Here's the diff. Like, it's it's kinda wild. And then, obviously, there's a bit of tuning. That's massively sped them up, and I and that never existed when I sort of was doing lots of engineering. I think that's it's really, really cool. Mhmm. I think, anyway.
I've been using it, of course, so having the image stuff seems really cool. Yeah. Okay. And then final piece of advice that you would have to other DevTools founders?
I mine would be don't neglect UX. I think there's a real a real tendency that when you're building DevTools to focus on, utility and, basically engineers are super accommodating of of things. You know, linear versus Jira, sort of functionally, could stack them up and tick boxes in a check sheet of, like, all the features in in both, and the UX makes the difference. I think that's how we think about products all the time as, you know, it's incidents, but it's built and friendly for anyone kind of thing. So, yeah, that'd be my one.
Yeah. For sure. I think that there's a great I think it's the folks at Evervolt or something like that have, like, a really, really lovely website, and they're kind of they're an API, right, for for security purposes. I think someone there was saying, I'd spend so much time on that, like, visuals and experience in your website and, like, how you present your brand and, like, you're, you know, you're a very, very technical tool. And they're like, well, like, how do you express the level of care we put into our products when you can't see it?
And you're like, well, you sort of, you know, wrap it in something that sort of follows the same level of care. It's a really nice framing. It's sort of like, you know, people generally perceive that if something's really, really well put together on the outside, probably a similar care put on the inside. So yeah. I think I thought it was really interesting.
My advice, I think yeah. Probably the the main thing that I think put us in a really, really good place with Instant I, aside from kind of building for ourselves, like, get someone to pay for what you're trying to build as early as possible. I think, like, we had a hard rule when we started, which was you know, remember, know, talking about it, which was, like, we're gonna charge from day one. Yeah. And the the one of the things about DevTools is, like, you can nerd with another engineer and say, isn't this cool?
And everyone agree it's cool. And the more complicated it it is, probably the cooler we think it is.
Yeah. Yeah.
But, like, the real test is where where you pay me money. It doesn't even really matter what money you pay, but, like, you need someone to put a credit card in. And at that moment, like, two things happen. One is it feels incredibly validating.
Yeah.
And two, like, you know it's worth something to someone versus just a curiosity. We didn't have a free plan for first three years of the company. We've we've introduced one in March for OnCall just because we wanted to see what would happen. But up until that point, nobody got the product for free. Everybody paid.
And, you know, and also when you're only dealing with paying customers, it's just massively focusing. Whereas, like, when you've got 200 free customers, it's like, do you build for them? Who do you listen to? Whereas when everyone's paying you money, it's like a, cool. I'll build for this person because they want to give me more money or they want to expand.
It's like a fairer trade of value. So, yeah, my my my stance is like, you know, don't build in a silo for fun. Like, charge someone for the thing.
Thank you so much, guys. That was really, really exciting. And, yeah, thank thanks for coming along.
Thanks for humoring us. I know.
And thanks, everyone, for listening.