Software Blueprints: Standardizing for Success with Anthony Alaribe

Hi everyone. My name is Patrick Akio. And have you ever imagined software having a blueprint? It's what we discuss as well as the challenges of a technical founder. Joining me today is Anthony Alaviva and I just had a blast of a conversation. I love his perspective, small companies and big companies. So enjoy. I love that you're saying you're combining travelling and working here and there.

I I'm lucky enough to do that at a company that it's not my own, but you actually have your own startup, so you have the flexibility to do that. Basically what what are you doing at the startup mainly currently? So we're we're doing monitoring for for tech companies. And when I say monitoring, it's like 1 aspect and people think about different things when they think about monitoring. But I I realised that you know when we build a lot of back end

services, we're coming. Maybe you have a mobile app that is communicating with your, you know, your systems or you have other customers that are communicating with your systems and they do that through this API interfaces and a lot of

things that go wrong. I usually reflected in there and I started to think, I mean, I've had a lot of experiences that forced me to do this, but I realized we're we're not doing enough monitoring at the level of those APIs, at the at the level of looking at the data and learning what the data should be like, and maybe detecting when something changes when something goes wrong. You mean the request and response data specifically? Yeah, but the payloads

themselves, what is in there? Yeah, Yeah. And I think the decision to build this was from a previous

job. I used to work for Delivery Hero in Germany in Berlin. And I was responsible for a migration of some legacy service that had served the company for like 10 years before. And we had to hit a performance limit. We needed to rewrite it into a new technology stock. And of course we did this like you should. You know, we wrote a lot of tests. We spent some months on this migration, but when we rolled out, put this change in production, of course things went wrong and we lost a lot of

money. I think we lost like something like €2 million worth of orders really. Yeah, to that. How long was the downtime? It it, I think we're down. It wasn't really a downtime. I would even explain what happened, but in some countries, specifically in Scandinavia, there is a requirement that you have to include the dietary requirements of every dish. OK. That you're selling and delivery heroes like this company of food delivery brands.

Yeah, in like 50 countries. And only the Scandinavian countries had this issue, of course. So when we're developing and building, we're testing against, I don't know, Asian countries and European countries, I mean, like Germany, but we're not really testing against Scandinavia. So this issue happened in Scandinavia. So pretty much all of

Scandinavia was down. And those are kind of tricky issues because when someone says, oh, something is wrong and you're like testing, but you're still testing with the other countries. Yeah. So that is what happened. And we realized the issue was basically that there was a field missing the payload which should be present in Scandinavia. Yeah. And no one, everyone who worked on this project in its early days, they're either the Co founder of the company was like,

not. Yeah, I'll touch your code. Yeah. And I think the other people, like, no one knows anyone who worked on this project anymore. So there was no one who could have maybe stayed. Did you check this? So it was just a painful process of figuring out what is missing different payloads in those specific countries. And and I realized that this is the kind of work that computers are good at.

Computers are very good at pattern matching, you know, finding patterns between things, and I decided to be on API

token. Good stuff. It's really funny. And I understand the use case at some point when you're talking about 50 countries and all of the countries or at least cross country, have different regulations in what needs to be done, even on a technical level, you accommodate for that. You make changes in the future. The history of all the people is gone.

That knowledge might be gone. And then when you create changes, and I mean the team still makes the test suite, is that if the test suite is not rich enough, then all of a sudden you can, I mean, release a bug in production that just covers a few countries and then you're in the big deep unknown, I feel like, yeah. It felt like that. Yeah, right. Feel like the unknown unknowns because you have this we're not having. So we had this other tracker that, you know, we we kind of

have. We know exactly how many others we should be processing at every moment. Yeah. And it's just like, yeah, others are down in these countries. Why? Oh, like, shit, why? Yeah. And you go to the web page, it's working. And you're like, you go to this, I think on the mobile app or something, it's crashing. And you're like, OK, why We're actually doing what we should be doing. And there's no one to ask to say, OK, do you have an idea why this is happening? Yeah, that's hard man.

How, how different is it than working on let's say your own start up versus working in. I mean Delivery Hero is a is quite a huge company basically now you're first of all in control of what you build, but you're also responsible. I mean you have a team, so it's basically a lot on your shoulders. How's that difference been so far? It is. It's a different skill set and as an I mean, I'm an, I'm an engineer. All of my career was writing

code and building systems. I realized late that I had to learn some skills because, I mean, I had this experience. I had some other experience from previous jobs that also enforced the decision. Yeah, and I just went out and started writing code. Yeah. Yeah. That makes sense. So it's like, I think maybe the only thing I did was I had a landing page with a wait list, OK?

Which, you know, I shared around and people joined the wait list and I was like, oh, I have people on the wait list now. I can take my time and build and they're. Waiting for. It and they're like, six months later, I'm like, Oh yeah, I've built something now maybe all these people on the wait list would sign up and cricket. So I realized I had to learn, you know, learn how to talk to

customers, talk to people. Like don't just have ideas and go implement, but let let the people who would use these things play a very important role in, you know, what you're building. Yeah. And then have to learn, of course, managing people a little bit and just a lot of a lot of things that we don't think about as developers. I can imagine. SEO, pay per click, advertising. There's just a lot of things. Yeah, yeah. It's, I mean, it's basically

running a company, right. Normally you have people that are specialised in like this small part of running the company either when it comes to selling the product or marketing the product or actually building the product.

And now all of a sudden, I mean especially when you're starting, it's like everything you can do everything and anything and then you have to decide and I completely understand that you default to comfort because of the the engineering side is like, OK, bread and butter, we can do this, we can build the product, but then it's like, OK, how do you sell the product and if no one knows, how do people know. So the marketing side of the product, walk me through kind of

getting that first customer because I think that's always a big win when you actually land your first customer. My, I think my So we have free tier, and then we have, like, people who pay. Yeah, we had, like, people on the free tier pretty much immediately. Yes, I shared what I was doing on Twitter and everywhere, and a lot of people just have had fun with it.

That's great. But the companies who want to use, who really want to monitor critical systems, they're not just going to use any random tool out there. Trust becomes very important. And gaining that trust was really the hard part. I think it took, it took it took a year from hey, you know, you know we're launching to actually having someone drop money and say OK and I want to like really, really pay and I don't know what I did to get that

person. I think I, I just posted a lot more videos on YouTube and I used to someone who had signed up like a year ago but was just watching and just watching me post and share things and eventually paid. Then it just slowly, you know more and more people started to use it and share. I have had a few, actually a lot of word of mouth to be honest, just because at this point I didn't know anything about sales, I didn't know anything

about advertising. So a lot of the people who came in was word of mouth and I I found this niche, kind of sub

niche of users in fintech, but more like this younger fintech companies, OK. And the reason is that companies that are like financial technology companies, they depend on a lot of other third parties. So it could be MasterCard, it could be. So have some Nigerian customers, fintech companies as customers. And for them, there is this national Interbank settlement

system. So when you want to send money to another bank, you're basically communicating with that system and that system is very unreliable. Oh, really it is. OK. So for these fintech companies, it became very important for them just to log all their communications with third parties, with those third parties that they depend on.

Because when something goes wrong, they want to take a proof and say, hey, like I called you with this payload and you responded with, I don't know, something else. And I think I found a very good niche there because they were actually looking for something like that. You know, some people of course rode years, maybe did something with like Elastic search and, but I mean how much can you do

on your own, right? So there are a lot of people who are really searching for something they just ignore. It's like a it's like a product category that you don't know what to search for. It almost doesn't exist. So everyone's just rolling, rolling something themselves. So what most of these people would come looking for just something that logs the requests and they would stay for the

other things. For example, we would if there's any change, like a breaking change in any in your APIs or any APIs you depend on, we would like detect it and just send you an alert that hey, we've noticed the breaking change, this field is missing or this field was updated and then you can go meet that that provider and say, hey, you updated this field, why? So they changed, Yeah. So they would come for just log in and then stay for the other other things.

I love that. I think that's it's really cool that you found a niche where the auditing is really important and probably then holding those third parties accountable for any breaking changes basically because especially when you're talking about financial technologies, I mean anything money people will put extra money in to keep safe basically. So it makes a lot of sense that that need was there. Yeah, there. It's just a very I worked in

fintech, you know, years back, maybe 5-5 years ago, actually, maybe more like 7 years ago. But the niche is just so fickle. It's money. Money's super important. You don't want to lose people's money. But stories you hear like I know a customer who lost basically all of the they raised funding and they lost all the money they raised. No way, due to basically a

provider issue like this. So when you when when you send money, say you're sending 10, I don't know €10 to someone, what would happen is that person would get 1000.

So those two zeros at the end and why that happened was that third party that they relied on updated their APIs and said we need you to send with two zeros at the end of course something like that because I mean that's what you should do anyway and they did not do that before, but we just made a breaking change overnight and these guys lost a lot of money because they were sending extra money to to to users.

So there are lots of those little stories of breaking changes and problems and and it's nice to help these companies figure that out. Yeah, it's. I don't know how it's grown. I think a lot of financial institutions have like very old, old software nowadays. There's practices to, for example, generate your documentation or make sure you have a contract and then generate your APIs. So then breaking changes are kind of out-of-the-box because you have that detection just by

virtue of your way of working. But I do recognize that a lot of, let's say older institutions, I mean at some point if you have a team and that team grows and then people lose, you lose people basically people leave you all of a sudden have AP is and you might not even know what AP is are for or which ones are there out there and then especially what they do. Yet you have to create software, you have to create change because otherwise the business

value is stagnant. And from a business perspective, you need to create value basically to keep customers and to attract customers. So yeah, it's this kind of every evolving problem then, from a legacy perspective. Exactly. There's especially in large

companies, there's also the problem of many separate teams that don't always know what they're doing. Yeah. And one use case that someone like I don't know, solving with APIs, OK, which we did not like design to solve. This use case, yeah, is that they have a documentation team and they have, of course, the engineers. And the engineers are supposed to tell the documentation team that here we made this change that should be documented. OK, but. How does that work? How does that work?

Yeah, it's it's a very common thing that these engineers just forget, you know, and it's then the documentations are always different from what is actually out there. And this this document. Like the head of documentation in this company just realized, like the dev guys, they really don't care about a lot of these other things. But I just want to know if there's a breaking change that I can tell my, you know, my team to update something.

So that's keeping, I don't know, keeping track of what's going on between these departments ends up being useful. Yeah, of course. Yeah.

You mentioned you had a lot of skills to learn in kind of creating this start up. I'm assuming the, the engineering side was just more so comfort you say like the biggest learning was there, was it more on the sales side, marketing side or anything beyond? It was on the sales and marketing side and it still is to be honest, cuz I'm still learning and it's it's, you know, coding really is a comfort zone.

Sometimes I find myself just writing code and doing stuff and then I have to remind myself like why are you doing this? Is it really that important or you should be talking to customers or I don't know, figuring out marketing. So I had to, I think one of the biggest things I did, which I mean, you know, you know Dennis, she ended up becoming some kind of so Dennis from SAS lunch, but she ended up becoming a kind of

like a coach in some sense. You know, because sometimes we would like talk about what I want to do with regards selling and marketing and then, oh, we have this customer bug report that also needs to be done. I think I'm going to spend the next two weeks fixing this bug. I was like is that really what you should be doing or could someone else on your team do that or could you outsource that? So that kind of reminder and that became or was it's very

important. And for sales, it's just a lot of learning to be honest. A lot of podcasts, yeah, like this one. And a lot of books and also a lot of lost money because you know, you, you, you, you listen to people talk about the strategies that are bulletproof and you pump money into it and realize, no. There's holes everywhere. Yeah, OK. But is it then the the learnings, do they come from how

for example you find your customers or how you talk to them or what you say, how you hone kind of your your pitch or what on the sale side was like the biggest learning there? How I talk to customers evolved a lot, so when I started out I read this book called Running Lean. OK. I don't remember the author's name, but it's it's like a book based on lean startup. Yeah. And he, he does a very good job of listing out like a kind of a road map on how you can have

this customer conversations. Yeah. You know you when you get on a on a conversation with a customer it's it's very easy to just start rambling and and selling. You know and oh we do this. We do that. We do that. And sometimes that's not actually that is not even how you sell it turns out and how you sell is really you jump on these calls and you try to learn about them, learn about your use case and help them solve a problem. You know and sometimes that solution might not even be what

you're selling. And I've had a lot of that where I someone comes on a call and says oh want to do this, this, this, this. And I'm like, OK, that's an interesting problem you want to solve. Have you tried this other company? Have you tried this other company? And then they're like, Oh yeah, I didn't know about this people because it turns out as a start up founder, we are like we know a lot about that niche just because we're always researching. We're always looking at

competitors of course. So we end up being the ones who know. Like if someone has a problem in a related niche, we likely know all the other players that might perfectly solve that problem. So why do we keep that information? Because, you know, we just want to sell what we are offering when you could actually point someone to the correct solution. And I learned that along the way, and the conversations became a bit nicer.

Yeah, of course. Because it became lesser selling and more of like just getting to know someone and and more pleasant and maybe more genuine as well. But also just the questions to ask, you know like because at the end of the day you still want to sell, so you want to ask questions about what problems it's solving and lean, running lean helped a little bit with those questions.

On the customer acquisition side, I also learnt a lot and I'm still learning, but one of the things that I learnt is there's just so many approaches to customer acquisition and. All of it works, but they just work for different people and different different people, different niches, okay, different kinds of problems. So what you do when your B to C is different from what you do when your B to B?

Yeah. What you do when your B to B with like a huge sales team is different from what you do when you are like, I don't know, a team of 10 people. And I'm still learning about this and realizing that some things I like more. So even those certain ways kinds of marketing might work, some are just more suited to maybe my my team's personalities and and yeah, our strengths. Let's put it that way. Nice.

I I love that you go into those calls with the mindset of helping people, right? Because if you're in that mindset with this scarcity thought that OK, I have to sell, basically it's a must, then probably you're not going to make friends with people. You're not really going to help them and you can't like you, you don't have a silver bullet. Basically your product solves a specific niche and if the customer doesn't have those problems, then it's just not a

good fit. And I love that you have this mindset of I'll still help them out and I'll, I'll say, OK, you have these options or you can go to these companies because you are in this space and you're very much aware of what happens around you. I think that's very lovely. That's really cool. Yeah, it is. There, there's this lady, April Dunford. She has a she does some talks, so she has some YouTube videos which are already just good information already.

But she has a book called Obviously Awesome OK and a talk called April Buys a Toilet OK and she basically in, in, in, in in the talk she talks about how hard it is to buy a. Toilet. Yeah, to buy anything. She used the toilet as an example. Because, I mean, a toilet is something you've been using your whole life. You would expect that you're like the expat of toilet if you've used it your whole life.

But then she goes to buy this toilet and I was just like learning about all of these features in terms and floppers and payload size. And I was just like, what the hell is a flopper, you know? And so she, she, she gave this example of just the toilet. And then I made a kind of a correlation to our, I don't know, the software things that we're selling. Yeah. To show how it's really hard for like the people who want to buy things to just they're just all these terms.

They don't know if they even need those things. They don't. So you who is in the niche you, you know all of this. It's your job to like, filter just the information they need, you know? And yeah, well, you don't need to know about Flapper like you know. You need this toilet. Yeah. Where do you, where do you want to put the toilet? Is it in the basement? So maybe you know this kind of toilets that you know you don't need something that's super strong.

You use it once in a while in the basement. So maybe this category is good enough for you? Or is it something, Are you like putting it in? AI don't know, in a in a fancy hotel, maybe you don't. If you're not, maybe you don't need this like jade go toilets, you know, it's like just help with with, I don't know, position in what products or what solutions your customers actually need. Yeah, yeah, for sure.

I'm I'm curious since you have the experience of a lot of production companies, you also have the experience of creating your own product and solving customer problems. Basically where those customers are companies and I think mainly they are developers because you solve developer problems. There is not really a a silver bullet when it comes to starting up a product. When it comes to documentation, I think everyone kind of does what they think is best and then

they learn along the way. They take that experience and they still share. But there's not really, let's say, a blueprint of how you create software. You can find a lot of them and then you have to pick which one works best for you. When it comes to starting up, first of all, then making it scale, documenting observability, it's very hard, I feel like, to find kind of the right way to do things nowadays because there's just a lot of options. This is a This is a rabbit hole. Yeah. Yeah.

No, I know. It it's something I've been thinking about a lot. So my background is in medicine, laboratory medicine, and in in, I don't know, in the medical field. We have SOPS, the standard operating procedures for literally everything you want to do. Yeah. So almost anything you want to do, you don't have to be

creative. You don't have to like you, you can just go, oh, my patient has, I don't know, this disease, you know, and you go get SOP for diagnosing that disease and just follow the SOP and you'll be good. OK, what that does, You know, it sounds like, yeah, you're taking away creativity from the doctors. But I mean, first of all, how much creativity do you want from your doctor? I want procedure. Yeah, exactly. But it doesn't mean these doctors are not creative. It just means they can.

They can stand on the shoulders of giants and focus on the other things. I think we need more of those in software. If if I, when I speak with veterans in the industry who have been doing this since, like the, I don't know, the 90s, the things they say they were doing in the 90s are almost exactly the same things we're doing now. What has changed? The tools have changed. Maybe the standard in your user experience has changed, but it's

more or less the same things. And I think because we don't have this kind of standard operating procedures, a lot of people keep reinventing the same, like same things over and over the solutions to the same problems. And I think there's a lot of room for that.

Then I think if that happens, then as engineers we can focus our time on being creative, you know, thinking about more creative ways of solving problems, like the actual problems we want to solve, not how do I write from a database or how do I read database or how do I debug this thing. Can we get there though? Because I feel like a lot of

companies see this challenge and they're trying to fix that. Like I've seen even my friend has like a code automation tool where he generates based on a a blueprint kind of the API landscape based on best practices, any connections to databases. But there's not one tool that like every developer uses because developers love their creativity more than doctors. Way more. Yeah, yeah. Yeah, I think it's hard. I think, like you said, that developers are very religious.

Yeah, Yeah. You know, it's like we, we fall in love with our tools with this particular database or this particular. So when you have this, like, what your friend does this scaffolding generator to generate something, yeah, Then one developer is going to come and say, oh, this scaffolder is using this database. I hate that database. Yeah, yeah. I don't want this. Exactly. So I I I'm thinking of it not even from a tool perspective, but more from a run book perspective.

We're having more run books now in software, but most run books are within companies. So in Delivery Hero, for example, we had like run books for pretty much any kind of incident. If there's a database incident, this is the run book for that. If there is, I don't know whatever incident and what makes that useful is that I can hire pretty much junior developer right? And telling me these are all the run books, these are the alerts we have, these are the monitoring platforms we have.

When you see this error just go pick this run book and just do everything in these steps like step by step. And so I think what would be good is if this runbooks go outside of this company world guide. I know Google has similar list of runbooks and a lot of big companies have the same, Microsoft has the same.

But if these are most standard standardized, you know, across the industry, so people are not solving the same problems over and over, then the details maybe in one of the steps, you know, maybe you can have like generators or something for like some of the steps in between and maybe have some creativity within the steps. But the general runbook is kind of the same across the industry. Yeah, interesting. When you when you mention your runbook example like I I have

when I started from university, I went into operations before development. That's where we had the most runbooks because we would always analyse and this was in a company where OPS was separate from development as a whole. So then we would have runbooks when you have a specific error either. How to fix it? Or just how to debug basically and when there is a hole in the runbook, you you add to the runbook. Basically you have to figure it out on your own at that point.

And the only runbooks I've seen mostly from the development side is not really on the creation side. There's always ideas of OK, this is kind of the latest way to create this. Whether you're doing contract first or you're generating documentation like there's options. I feel like sometimes too many options, but they're separate runbooks I've seen when it comes to releasing. All right, we have started from scratch. We have not gone to production yet. This is going to be our first

release. Then you have a runbook of how to get your stuff to production and then people follow that when it comes to releasing. And that becomes easier and easier to the point where you don't need a runbook anymore. So then the only runbook you have is, let's say a first time release where you make sure that all your production keys are there, your configuration is updated, all your feature flags are enabled, and stuff like that. So I see it from the OS perspective.

It's from the development side by virtue of having so many options. I don't know if a runbook would work. That is the. Yeah that that is something worth thinking about. I still think it could work. I mean even in in medicine it's not that there are no options. Yeah. There are lots of options for things. For example. So I, I, I, I specialised in laboratory medicine before I left. Yeah. And in if you want to do a malaria, some family tropics. So malaria is a big thing.

So when you want to test for malaria, you know, you take a piece of blood, you put it on a slide. You, I don't know, you stain it just like a dozen stains, each of them with like slight differences. You put it in the microscope, how you it, basically. There's a lot of options. But the runbook is, yeah, we know there are lots of options, but this is like just this basic

option that everyone can follow. And then you know maybe you're doing something very special or maybe you are in, in our case maybe you might be in a village without electricity, then maybe you know then you can change one of the steps for something that works without electricity. But the basic can be there. That's what I think we need as well in, in the industry in what

I'm trying to do with API toolkit now. So now the performance evolved a little bit such that we, you know, we monitor the the errors. We also we look at, we're now analysing a lot of things. So we're looking at the headers, we're looking at where if there are runtime errors in the code, like with Sentry, we cannot associate those errors with the payloads that came in and out. And the next step is that from this data, like, can we give you a step by step list of things

you can do to resolve an issue. So for example, if we look at the headers and see that you're not following some best practices, maybe you're not passing authorization properly. Can we tell? OK, this is exactly the industry standard for this. Yeah, I maybe give you some links to, you know, figure out how to do that.

So some it's thought that it's still taking shape, but it would be nice if not just API took it. But like any of the tools out there, you know, could when they detect something or they realize something about your systems, they can also give you steps. They can tell you this is the industry standard, and then maybe these are steps you can use to fix it or implement that standard on your own systems, yeah.

Yeah, very interesting. I I love that it's on a smaller level, let's say when you have an operational concern that not not do you only like detect an anomaly for example or an issue, but you also give like the step by step on not how to resolve it. Because I think sometimes it's very context dependent, but at least how to investigate it or

where to find documentation. And I recently saw a talk where it talked a lot about engineering culture and very, very cool to see that one of the biggest factors in engineering

culture and I'll I'll probably have to find this research and put it in the show notes. But a lot has to do with knowledge sharing. So not just from a tool perspective to the people, but also across people where they find how interesting an issue was or a problem they solved and

they share it with other people. Because I think if you have a passion for creating value solving problems when it comes to coding, people will share that passion with each other and it like becomes this huge bonfire of the flame that is your engineering culture. So when tooling also then provides that provides them with insights that they might not have known or might not have found. I think that can be very valuable in that way. I think, I mean, I agree with you and I, I don't know.

There's a thought that has not taken full shape in my mind. Yeah, but I feel like developers build this amazing things for every other industry, but not so much for ourselves. Yeah. Because there's always, you know, things that we do over and over, things that we are analyzing. And maybe we just need more people thinking, you know, like, how can we, how can we solve things for ourselves as well? Yeah, yeah, I think.

I think people are trying like in the last few years I've seen more and more companies tailor towards developer experience. It's been a bigger and bigger term. This problem solves so and so and it increases your productivity and it really leverages so and so, so you can do this easier. Basically I just think developers as a user are super picky. First of all, I will not pay anything. So if it's not free, then I, I I'm not even going to choose it

as an option basically. And then when there's a free tier I can see oh, maybe if I spend so and so I get these extra things and I'm hooked. That's the only way. I don't believe in paywalls anymore, especially as a

developer basically. And then still I'm very picky with the tools I choose because you don't want to have too many tools and then you have to choose which tools are the best basically for your use case and do you really need a tool or are you then going to be reliant on that tool? Like it's. It's so many thoughts and I love the kind of flexibility of, yeah, being able to do things and switching on the fly as well. Those are the thoughts I have, so I think other people might

have them as well. Developers are difficult. Yeah. Sometimes I wonder if it was even a good idea to target that niche.

But also developers would see something and feel we could build this in the house. We can do better. Yeah, we can do better. Yeah, they don't do better. No, no, no, no, no. That's the hard one. But it just stops them from deciding to use something that could solve their problems. Yeah. And I I like what you said about like being very picky about like adding new tools and maybe think of a a joke.

Someone shared that. How, you know, a very veteran software engineer is that in his house, he doesn't have any technology. He just has one printer and A and besides it in case the printer makes a wrong sound. I was like, yeah, I I remember because we know how things break all the time and we just don't trust anything. Yeah, it's funny. I when you said when you were

talking about blueprints, if I were to start a new project, I I do have kind of a the things I've liked from previous experience that I would start with like the last project I worked with, we had a contract first approach and it was GRPC. We need to make a mobile app. We needed to stream data and the documentation. If you're contract first, it's just there because you look at the contract that's what you use in production. That's the kind of your API

landscape. And we didn't get to a scale where it was a lot of contracts, because I think starting is easier once you have it at a certain scale with multiple teams, that's a whole different beast. And I've never actually experienced that to that degree. But that's how it would start because then I feel like documentation to, at least from an API perspective is there. We also had some conventions in place to document and generate, for example, architecture diagrams, which was very easy

when it's close to the code. But yeah, when it comes to then scale, I don't know how you would scale beyond that. Basically, I haven't had that experience. What have you seen at scale that like works really well when it comes to kind of conventions and still being productive? I don't know. So I think in I don't know in in. It's still manual in every company I've been in. To be honest.

I think documentation, I mean actually I was going to ask what you're using in like this architecture diagrams. Is there like something like mermaid or something like that? Yeah, we've done. We've done a lot with mermaid, actually. Yeah, yeah, yeah. But there's a few other options as well. I just don't know them. The mermaid one is the recent example. And so you had something that would look at the architecture and then generate the diagrams. Yeah. Did people did your engineers

actually use those? Things. So The thing is we had a lot of annotations in place and also when you create a new API, you would have to put an annotation on there. First of all to this was another project to create then Swagger docs because we had third parties that relied on our software. But because of those annotations, we also had kind of an automation that would see if mermaid diagrams changed based on what you had in the code versus what was generated and

then you would have to update. So there were some conventions in place, but still, like there's a human factor in there. You can forget it and then your automation might tell you, or you might forget everything as a whole and then your automation cannot tell you because you created the automation. Like the human factor is always going to be there. Which is why I I really think the challenge of scaling is is the hard part when we're talking about a blueprint of how you do

things. Yeah, it's the people, you know, you need to get everyone, everyone to buy into. You know, this kind of this processes. Yeah. And that is the hard part. I think that is also why anything that can be automated is always preferable. Then you don't have to depend on everyone remembering that they have to add some annotation or something like that. Yeah, but I've not seen at scale, you know, in practice at the larger companies it ends up

still being. I mean, you have a lot of squads or like smaller teams and it everything ends up being at the team level. Yeah. And some teams might just forget or might not care about setting processes as much as others. And so I'd scale, yeah, just how it is. I mean, as an engineering leadership, you can enforce it and say, oh, everyone has to do this and this and everyone enforces it for, I don't know,

1/4. But, you know, I mean, people live in the companies, new engineering management or managers come in and they don't share your vision. I don't even know. Yeah. So yeah, I've not seen it work. Very well at. No kind of scale, yeah. Yeah, it's the it's the humour

factor that that is the most difficult one. And I was thinking when like in this kind of booming age of AI, especially the last year, we have a lot of.

Very interesting models that might even be able to help with that because we have metrics when it comes to how complex your code is. When it comes to like your nested if and for loops for example, I've had coding conventions that would tell us OK, your pipeline fails because your complexity is too high just by virtue of you going to the right instead of staying in a straight line. Cyclomatic complexity. Exactly like especially with Go that that was easy to put in there.

You had go. Yeah. Yeah. I love Go. I know you do as well. Yeah. Yeah, Yeah. But that's not. That's a different episode. Yeah. Yeah. Like, I like Go because it keeps things simple. Yet the. So we have tooling that enables us to observe what might be wrong with our software, how complexity might grow and probably with AI we might get more options for that more information, more out-of-the-box information by virtue of kind of having this blueprint that analysis your software.

The problem is humans will still need to change the code, I feel like. And then if you if you choose to ignore that because even with conventions I've seen people say, yeah I don't, I just don't agree and I'm like, damn it, we have conventions in place in the 1st place which means we aligned on this as a team and you cannot just say we don't agree. So then we have this discussion. The human factor of change is

always there. Yeah, so, so I'm also using LLMS for some of these, but I I think it's about where you want to analyze, right? So there are lots of tools that analyze at the code level. Yeah, you know, cyclomatic complexity. And those are very important because you want the code to be maintainable. Yeah, you don't want a ball of mess that no one wants to dive into. But I I think I've been thinking about it more from the user perspective.

Yeah. So what we've been doing is we take like we take the requests that users are making, we kind of know the average request and we pass this into some LLM to figure out what is the user's flow, the user's journey, you know? And I think that is useful because then you know, OK, when a user comes to this platform, they hit this endpoint which is owned by this team. Then they get this data and they hit this endpoint which is owned

by this team, you know. And that is a different way of like applying LLMS because you're just looking at the data that users are doing and not the code. And it's a different set of insight that they would give you. Yeah. For the code level, I don't. I'm not. I don't know. I've not been impressed by all the code level LLM that I have tried. No. Me neither. Not yet. Yeah, not yet. Not yet. I am hopeful though, for the

future. Yeah, I am hopeful too because they shine very well when you give them like maybe a function or maybe one file and say OK, I don't know add this new function to do this, yeah. But when it comes to looking at the entire code base, I have not had any success. No, me neither. Not yet.

I I like that you take kind of the the code as a boundary and you say, OK, we go beyond that and we look at interactivity because then I feel like something like anomaly detection, which humans are just it's really hard when you have a complex system, when things are tied together and interfacing and even sequencing of messaging sometimes either in parallel or linearly just becomes a lot to comprehend. First of all, from an engineering perspective, how's

this work? And you can look at the code, but that's still the boundary of code. And when it comes to then the messaging, the interfacing in between, I think that's really important when it comes to anomaly detection. It's not just anomaly. I mean there are lots of things we are not doing. But you know when you work in this field you start to see the opportunities. For example, if you're looking at this communication and end users making with all these

services, yeah, you start. At least you can as a user. You can figure that maybe LLMS can figure it out too. Opportunities for improvement. Maybe this particular endpoint is always sending the same kinds of data. Always. And maybe should just be cashed. Or maybe it should even be cashed at the CDN level because the last time the data changed was six months ago.

So we don't need it, yeah. Yeah, maybe it just had code some CDN rules to you know that this rooms for these opportunities you see maybe some endpoint gets called very much and I don't know should be it's too slow. But what is what? What you need in this endpoint? You're already getting from another endpoint by another team. Maybe this endpoint should not

exist. You know that this kind of things that I imagined like LLMS could be like pay programmers or partners in just looking at this patterns and recommending things that you can do. I like that a lot it it

triggered me and I thought of like Google's Lighthouse where you just open up your browser, it runs through an automated set of tests and then it gives you some suggestions based on how your website performed. With the history and kind of the capturing of requests and response over time and the knowledge of OK, this might not be a good idea. Yeah, it can be very helpful to have kind of an assistant there that recommends you stuff to improve.

My problem is I don't know how much sometimes we need to improve because some stuff like when you're talking about speed, I think a lot of software is fast enough, yet we do a lot of things to make it even faster or we over optimize for many, many users and we might not even reach that point. So then it's this balance of OK, where's the value? Like exactly with your challenge of creating your own start up, should I really be fixing these bugs or should I really optimise

these things? Or should I actually talk to customers and do these things instead? That human decision factor, especially when there's an assistant that tells you these are the best practices you can improve on these things, is going to be even more crucial then I. Have to decide on which hats to wear? Yeah Yeah, founder hats. Like you said, most apps are fast enough. Probably should be talking to customers, yeah. But if I were to use a hat, I don't think most software is

fast enough. Like every day I'm frustrated with the things that I use and now we're in the engineer hat. I think part of the reason is that these days we're building micro service architectures and platforms. So your service on its own might be fast enough. It might be 500 milliseconds. But then from a user's perspective, to satisfy what that user wants to do, you have to hit like four of those services.

Now the 500 milliseconds stacks up, and now you have two seconds or three seconds you're waiting for a page to load, you know. So I don't think systems are fast enough, to be honest. Yeah, Fairpoint, it's just, and this is The funny thing, the stuff I use might be fast enough because if it's slow enough, I'm not going to use that. Yeah, But I love thinking about these kind of how the industry can improve when it comes to either just more knowledge sharing assistance or

blueprints. I do think there's a long way to go and I love that you said that whatever we were doing 30 years ago, like the landscape, it might be different. The context is definitely different, but what we do is not that different and we should definitely work more on standing on the shoulders of giants, yeah?

We're sorry, I spoke with this engineer who used to work, I don't remember which company, but he worked on these telephones in like the 80s or something and he was just telling me about like some of the micro services that they were doing at the time. You know, I think it was service oriented architectures at the time. And I'm like, why did we spend like the last 10 years? We're inventing. We could have just read a book from like from the 90s. Yeah, we're going back and

forth. Yeah, it's just ping pong, ping pong. But we need to like, there's a lot of new stuff that we should be inventing. I think, you know, we should. An ideal situation is if we actually automate ourselves out of our current jobs. Yeah, I mean, it looks like it's going that way, even though I don't like. There's different opinions on this, but I think a human will still be in the loop.

But it does look like code automation and code generation is getting to a point where it it can just sometimes do it better than a human. I would love that, yeah. I I think what would just happen is if the code generation gets better than us, it's getting better than us at the things that we're doing now. Yeah, yeah. Which means there will be new things for us to do and figure out. I'm I'm in the same train of

thought. I I think then your role will evolve rather than be obsolete because with this now automation tool that creates stuff, then you're going to be like, oh, so if we have these options and everything's super fast, how do we create and tie everything together to create value basically? And that's then going to be the right race, I feel like. I mean, if if you think about our jobs now already, we what do we do most of the day?

We're connecting APIs together. You know he does service for XYZ, he just says. So we're basically just connecting things. Yeah, so it would just be a higher level of that. Maybe no longer just connecting APIs, but actually connecting maybe algorithms or use cases. But I don't believe we would be out of job. Do you know how hard it is to describe project to someone?

Yeah, yeah. I think half of our jobs is developers is just we're listening to the specifications from a product or from management or whatever and then try just trying to make sense of it. Yeah, yeah, Yeah. So I don't think anyone's automating that out of. We might get prompt engineer. I've heard that. Yeah, cool man. Let's let's round it off here. This has been a blast entry I must say thank you so much for coming on and sharing. Was this kind of what you expected coming into this I?

Mean I had a lot of fun. Yeah, that was good. That's always good to hear. So I really appreciate you having me here. And also I appreciate Denise from SAS lunch for I mean the connect. And yeah, he made this very fun conversation happen. Awesome, awesome to hear. Thank you for recommending Anthony Denise. I I always love it when people do referrals, cause usually they're really good and I had a blast for a conversation. Thank you so much for coming on.

I'm going to round it off here. Thank you for listening. Please leave some comments and some love behind. Below the like button is something I'm trying to work on to make it more interesting than just the comment section. But in any case, thank you for listening. We'll see you on the next one.