You always want to look at what is your business need to do differently or your organization need to do differently than other organizations. If you can, outsource it. If it's not something that makes you different, you should use a service because you will always be asked to do more things that you can build as a developer that are differentiated, that are like special to your organization. So don't add in things that aren't.
Hey everyone, my name is Henry Surya Virawan and you're listening to the Technically Journal Podcast, the show where I'll be bringing you the greatest technical leaders, practitioners and thought leaders in the industry to discuss about their journey, ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our journal. Hello Joe, it's great to have
you here. Welcome to the Technical Journal Podcast. Great. Thanks for having me. So Joe, I always love to start my conversation by asking my guest to actually share a little bit more about yourself. Maybe if you can share any highlights or turning points that you think we all can learn from. I'd be happy to.
You know, I think my main journey started as a tech lead, and I think the main thing I've learned across my career is how important it is to understand the business goals and the business itself that I'm working with or nonprofit organization or whatever. I think when I started, I really wanted to work on cool technical things, and that was my focus.
And you know, I realized over time that the quality of the business, the quality of the organization and the quality of the leadership and the culture really mattered a lot more than how cool the technical problem was. And so I've kind of developed a rubric for myself that it's much more fun and rewarding to work with people you like working with in an organization that's grown.
Because the organization that's growing is creating more wealth, essentially in many different ways that can be shared. And so it's an organization that there's like a healthiness to it that makes it a lot more pleasant to work for. And that that's nothing that I optimize for. At the beginning of my career I I'm on my 6th, starting my 6th company that I've been with now for about 5 years.
But every company along the way I've made mistakes around not thinking deeply enough about how growing and how great is this garden that I'm working in as opposed to you know, how cool does it seem? That's the primary piece of advice I'd love to go back and give myself if I could from earlier in my career. Hey, thank you for being part of the technicianal community. This show wouldn't be the same without your ears, and you are the reason this show exists.
If you're loving TLJ and want to see it keep on growing. Consider becoming a patron at Techligional dot Dev Patron or buying me a coffee at Techligional dot Dev Coffee. Every little bit helps field the research, editing, and sleepless nights that go into making this show the best it can be. Thanks for being the best listeners any podcast could ask for. And now let's get back to our episode. Yeah. Thanks for reminding us as well about choosing the right place
to work with. And it's not all about cool technologies, right? I think there are still some of us technologies who love to play with the new techs, right? Especially these days, there are a lot of advancements, new things all the time. So always understand about the business, the quality of the business, the people you work with, the culture. I think that's also important. And yeah, working in a growing organization, I think that's
also very important. I was just to say, you know, if you pick a growing organization, there's going to be plenty of money and advancement and things for everyone. And so that one I think is a real key. It is a way to get everything. You have a good culture and it's not growing and it's not going to create those benefits for you, but if it's growing, it will. Yeah, not to mention when the organization grows, there's so many challenges you can learn from.
I mean like it's a double edged sword. Yeah, lot more challenges means a lot more headaches probably. But also at the same time you grow much, much better as a person or maybe in terms of skill set. So you wrote a book, Serverless Game Changer, How to Get the Most out of the cloud. So maybe before we start talking about your book, tell us a little bit more your relationship with Serverless. How did you bump into serverless? And yeah, what made you started
to write this book? Yeah, you know, I've been on a journey, I think my entire tech career, of trying to leverage the benefits of advancements in technology. And one of the things that I've realized over the past 25 years is that we get really big, important advancements in how to build software about every 18 months. Now, every 18 months, it's not so much better that you should, like, throw away everything you're working on and like, just
change everything. But there's a compounding effect to this, like every 18 months, big changes. And this really hit me hard in about 2007. I had been building software for more than 10 years at that point in time. And we had someone, I had started a company and we had a company looking at making an investment in US and they sent someone to do technical due diligence on us and was looking at what we had done.
And he said well you know, what are you doing with continuous integration And we were feel like I don't know what continuous integration, what are you talking about? And he recommended the book, the sort of the key book on continuous integration, to us. And it really changed everything in terms of how I thought about not only like, wow, we need to do this, but also wow. I had no way of knowing this, like in the way I had been practicing.
And so at that time I started asking how do I learn about new things? How do I go on my own continuing education journey? And that led me pretty quickly to a hypothesis of we continue to figure out what parts of building software are undifferentiated for us and we continue to be able to hand those to other companies and pay them and really pay them for use. Even in 2007, this was true. I mean 2007 we had S3 and was on S3 as a great, wonderful way to just store data that it would
stay forever. It was very cheap, I mean relative to anything else at that point in time. And when EC2 came out the next year, we had a company that really needed not so much to dynamically scale, but we had a scheduling problem. We would get all this data in and need to process it and being able to like flex out to north number of virtual machines and then close them off when running, it was a perfect need that we had.
Then Hadoop came out and we saw wow then this all of the stuff we were building ourselves we could actually like outsource that even a bit more elastic map produce was Amazon service at that point in time. And so I saw this trajectory of I can take these parts of these programs that we're making or the workloads that we're running and processing for our companies and we can hand them more and more off to people and we can
break them into chunks. I mean from the beginning was let's break the application into pieces and then use services for the pieces. And so when Lambda came out, I think in 2014, it was just another progression to me of how do I break an application up and just not have to worry about
running things. And I was already at that point in time I had started four companies and the amount of time and pain that we had to spend as a fairly small staff just keeping things up. I mean we used RDS early, but like we had cases where the volumes filled up and it took us down and so there was never a case where those services were fully like I can sort of just delegate the running of these.
And so really feeling like I can delegate my uptime to a managed service it, you know, starting in 2014, 2015, I was like, this is it, this is perfect. But you know, interesting immediately what I saw was people, in my opinion, designing things poorly like looking at Lambda and saying, oh, we're gonna like put every function in a different Lambda, they're
gonna call different lambdas. And we're gonna like take an application which is here and we're gonna like spread all that complexity out across like network latency calls to all these lambdas. And so I started talking.
So I gave a talk at the first serverless conf I think it was in 2016, 2015, 2016 hosted by A Cloud Guru and started writing more and started arguing more about not only like this is how we should build in serverless, also building serverlessly on Firebase and even building applications like on Google Sheets and Google Apps Script. Really trying everything and interacting with a lot of skeptical developers who would say, oh, that's nice, you get to build toy applications that are
very simple. The real developers don't use serverless. It doesn't work. It's too expensive. And so I started this insurance company. Branch is a full stack insurance company. We bear the risks. Sell home auto, renters, condo umbrella insurance. We're the fastest to be able to sell those bundles by orders of magnitude over any other carrier. And I was in all these conversations with people saying you only know how to build toys, Serverless is just a toy.
And so I said OK, I think I need to address all of those complaints and skepticisms about building serverlessly and really write down now like no, we built a full stack in a financial services carrier that has lots of regulatory compliance requirements on it. We built it fairly serverlessly. There are no VMS or no containers. There's no Kubernetes anywhere. And so how do we think about it? How do we do it? And why does this actually work?
Why is it not a toy? And so that has been the arc, but I feel like it's been this sort of constant attempt to take advantage of We get these new technologies and like they can really change how we develop software and make it cheaper, faster and better. And so that's been my focus and I'm happy to share it in this book. If you're very skeptical of it, this book is designed for you. It's not designed to say you believe in serverless.
Here's how to do it. It's a book that's for the skeptics to help you understand how this actually works. Yeah, thanks for sharing your interesting story. So actually when I read the book in preparation of this conversation, I'm not a skeptic of serverless, but I get more intrigued by reading what you are trying to explain in your book, including Branch, right, which I'll probably will talk more about later. Like what makes Branch unique in
terms of adoption of serverless? But what piqued my interest in the beginning when you said right, so technology keeps advancing every 18 months or so, right? And the gap, I think you mentioned in your book, the gap between the best software development teams and average software development teams is getting more enormous. I think it's related to all these advancement. Maybe if you can pick a little bit like why you think the gap is getting much bigger. Yeah.
I mean, I think if you think about like, if there's like a huge change over 18 months, there's this compounding effect. And so startups that get to start today get to take advantage. There's no legacy, right? They're not relying on any existing technologies. So they can just use the absolute fastest, best way to
build something today. And if you started a company like we did five years ago, you're already gonna be committed to some choices that if you were gonna rebuild it today, you would build it differently and it would be cheaper, faster and better, you know, if you had all the knowledge that you could have. And so we, you know, we look at
that ourselves even. And so in the book, I give an example of Instagram, when it was acquired had 13 developers and doing about the same thing roughly Facebook had hundreds of developers. And then before that, you know Friendster and Myspace had over 1000 developers. And so there's. And actually Instagram had 13 employees. I think they might have had like
5 or 6 developers. And if anyone has gone through the process of like how much can one developer do and then how much can 4 developers do, There are these step points where you lose massive efficiency per developer. I mean, I tend to think of it as kind of like 1 developer, 4 developer, 12 developer, 25 developers, 50 developers, 100 developers. And once you're at 100 developers, there's like a core inefficiency that's in the whole
operation. You know, you're getting an order of magnitude less productivity than if you have a solo developer. Now, you can't run everything on a solo developer. It's not a sustainable thing for a company. But there's also a reason why Amazon has these like 2 pizza teams and trying to keep these services small. It's the only way to keep really good quality and velocity together.
And so when I say that the top teams are orders of magnitude better than the average teams, it's largely that the top teams are leveraging technology so much better than the average teams that they're just not bogged down.
And I feel that very much at Branch, the company that I was working on 10 years ago built fax which was fully in the cloud infrastructure as code, but like running on VMSI mean it's still probably is. You know, in Amazon load balancing, we were using right scale at the time to do some of the orchestration. There was a core inefficiency that we had in running and a core inefficiency that we had in building new features. That branch doesn't have
branches, just faster. And so the average developer at branch is just producing more for Branch and at a higher code quality than the average developer was at build facts. Yeah, it's quite astounding when you mentioned about this. Instagram only have maybe 13 employees or five developers in total, right? But can build a world kind of scale, you know, like with users from all around the world. And I think what you say is right. So for people like me who has been around in industry for
quite some time as well, right. I mean the reason, yes, you can actually see that, oh, there are so many technologies that I'm not even aware of. But actually it's really cool to probably prototype new applications, right, building something from scratch and even like for example, you mentioned. So you can actually build something like only Google or Amazon can do last time, right? And now you can actually tap into those technologies to build similar things like them.
And I think what you mentioned as well, right, If the developer can produce more, that means they're focusing a lot more on the right thing, right, which you mentioned as differentiated versus undifferentiated things. I think this concept is very important for you to actually adopt as technology leader or technologist in general, right. So tell us more about this differentiated versus undifferentiated?
Yeah. I think you always want to look at what does your business need to do differently or your organization need to do differently than other organizations. And what are things that it's fine for your company or organization to do as well as other companies do, like the best other companies doing it. And so let me give 2 examples that are both undifferentiated, but that will generally, in my experience, get very different
reactions from lead developers. So let's start with the simple one where I think we'll all agree. So if I talk about sending text messages, sending text messages is undifferentiated. You don't care if you send text messages as well as the best other company sends text messages. Now, by the way, there's probably some listeners who are going then no, if you don't register your 10 DLP campaigns, you know, you'll get the messages rejected. Yeah, yeah, yeah.
But if you do it as well as the top people, you're fine, right? If you can send text messages as well as you know a quality things, you will be registering 10 DLP campaigns. You know you'll have your short codes and you'll use Twilio or something like Twilio. But Twilio is a great product.
I know it's a little expensive for what it is, but it's a great product and so I don't run into a lot of lead developers who will tell me no. I need to buy, you know, separate boxes and hook up phone lines and have telephony cards and like run them in a data center. I don't find that although I did that in 2005. I ran IV Rs that way so I remember that it was awful.
I would never do that. And so I think everybody agrees that sending SMS is undifferentiated heavy lifting and you should use a service like Twilio to do it right? And in fact, you should probably use a service like Braze or Iterable or Customer IO to like actually trigger those, cause those help make it even easier and you can do templating and have people do that. And if you don't know these services, you should look at them up because they'll make
your life easier. And the back of my book actually has a whole appendix on you should know all these services 'cause they're very useful. Now let me get to a controversial example and I don't know why this is. I mean, I know sort of why it's controversial. Most lead developers want to run Elastic Search or like Amazon Open Search themselves as their search index. When there is a service called Algolia, they will do all of the
painful stuff for you. I will tell you that running search index that's performing is undifferentiated. You do not need to do it better than anybody else. I mean, unless you're competing with Algolia as a business and so you can outsource a lot more to Algolia. It works phenomenally. It's really fast. Elastic search, open search, they're very finicky services. You'll have to be responsible for their up time. Indexing to them is kind of a pain in the butt.
Generally speaking, front ends can't talk directly to them because the security model is too poor. So you'll have to build a proxy. You'll have to query through your back end. It'll slow it down. It adds more development. But most lead developers will say, wow, I looked at the pricing for Algolia. It's really expensive and I'm not quite sure, oh wait, how do I do it in a dev environment and the right way to save costs? And no, no, no, it's just safer. I'll run it myself.
I know how that works. We'll run these things, we know how to run. But like that is classically undifferentiated heavy lifting. Everything you're choosing to do there to run open search or elastic search when Algolia exists is undifferentiated. And if you take an actual cost of ownership view, like the time that it costs and the people that are needed internally to run that thing, Al Golia is much
cheaper, just period. But most developers will make the choice on behalf of their organization to in house this undifferentiated heavy lifting with something like open search or elastic search because of honestly this misplaced stance of how they should be optimizing. And so that's my favorite example for this
undifferentiated heavy lifting. If you can outsource it, if it's not something that makes you different, you should use a service because you will always be asked to do more things that you can build as a developer that are differentiated, that are like special to your organization. So don't add in things that aren't. Yeah, I like their last line, right. So you are always being asked to do more things that bring differentiators to the business, right?
So I think not just Elastic search or such kind of a technology, people these days run their own logging, you know, messaging bus, maybe even containers, right on Kubernetes and all that. So I think that takes a lot of engineers hours and also effort and not to mention like if you want to make it production scale with high availability, right, that takes even much more effort and actually cost.
Yeah, right. Yep. Yeah. So speaking about cost, right, I think this is probably where the, I don't know, misconception or miscalculation, so to speak, right, like why people are still opting for managing things themselves. They think, oh, it's open source, I can also run it myself, you know, maybe build POC, it works pretty simple, right? And they just start from there. But I think the most important challenge is to understand the real cost, right. In your book, you have some
breakdown of cost. Maybe from your advice, how would you tell us to look at cost differently so that we can actually see? The differentiated versus undifferentiated. I mean, I think that you really need to internally understand how much it costs for you to develop and run things yourself in terms of your people costs. And I think you also need to understand and in terms of your
velocity costs. So in general, the more teams that you have in your company that are required to get code live, the more inefficient you are. And it's a compounding effect. In my book and everywhere I talk, I recommend people read Accelerate and use the Dora metrics. We use Linear B at Branch as a way of monitoring the Dora metrics.
The cycle time is critical. And so you know, what I see is that the more undifferentiated work you do, the longer your cycle time and especially when the undifferentiated work is
running the things. And so when you use a service like Algolia versus you are running, you know, your own Amazon Opensearch or you're running elastic search on containers and you're building a proxy service in there, those are orders of magnitude different impact on cycle time when you release changes to how things go to the index.
And I think if you're honest with yourself, both about the actual cost of people and about the impact on something like cycle time or the other dorometrics, it gets really clear that you just don't want to run. I mean, ideally use managed services or, you know, don't write code, use managed services always gives you better outcomes. Here now always does have an asterisk. I mean, there are services that are very expensive, that don't make sense for you to use.
So like, Algolia isn't the solution for everyone all the time because, you know, if you have an index where you're searching, you know, billions of records, probably Algolia is not reasonable. But the way I generally think of this is there's like 9095% of the time we're all kind of building the same apps. They don't have like enormous traffic. By enormous traffic, I mean they don't have so much traffic that your data transfer costs are like the biggest part of your bill.
Generally speaking. You know, none of us have more than, I don't know, 1000 simultaneous users. I mean, generally speaking, most applications don't have more than like 15 simultaneous users, truly simultaneous. But you know, 1000 simultaneous users, most of our database tables at most are sort of in the 10s of millions at most. You know, there's a way to set that up. And so there's sort of a standard 9095% of the time.
And yet most or many lead developers I find always want in their head to be designing for like huge amount of data transfer like you know millions of like that's the model of aspirations like built this like take. But the problem is like those just need to sit.
Those are all exceptions. And most of the time, actually looking at Dora metrics, actually looking at cost, you'll find that managed services that are used by everyone at Twilio and Algolia, Cloudnary, whatever, are just going to be any sort of honest and fair comparison of actually taking into account the cost. And I'm always shocked at how many people just don't want to even think about how much
developers cost. They want to view them as like developers are free, and so if you're a lead developer, you don't know how much your development team costs and you're making decisions based upon this is more expensive than that. Like you need to find out how much the team costs. I mean, like if you don't know, you obviously can't leave making good decisions on what actually costs too much or too well. Yeah.
So I think speaking from my experience, what I can see in my area, right, industry start-ups, a lot of people, especially during the tough time recently, right, the winter time some people call it right. So they just look at the bill, they will see for example all these managed service or serverless, it costs a lot. The easiest one to reduce is actually you know those Bill, right. And since we have already a number of people, right.
OK, maybe let's just switch that to something that we can manage ourselves. But what you're saying is true, right? So at the end of the day, the cost of the people needs to be calculated equally, right? So you cannot just say I reduced the managed service bill, but actually that cost is translated to your engineer hours, right.
And also probably headache whenever there's an incident or trying to scale things, right, the on call thing will create some kind of a cultural issue within the team as well, right? So the harmony might not be as good as before. And speaking about this managed bills, sometimes I find as well the bill is so high because it's unoptimized, the usage is unoptimized. So maybe also you can think of just optimizing rather than switching gear to managing yourself. So I think that's my experience
as well. We have these built in moments, right, where every quarter or so we take a look at all the bills that are more than $50,000 a year or $30,000 a year and we just go ask you know, what can we do to get them down. We move services a lot. And one of the benefits of Serverless is because you have this really nice container, right? I'm using this service, it's got a well documented APII know what I'm using in it.
It's all isolated over here. It actually turns out to be much easier to switch them and move them out than it is if you build it internally. So we're on our like 4th referral management service and we used airplane dot dev as an automation tool and they shut down with 60 days notice. I think today is the last day they're running and we actually decided to in house what we had put there.
But it was so much easier to do it because we could just copy their API and so we were able to take all this stuff we'd written for airplane and port it over. You know, I mean, I don't want to understate the developers who've been working on that have like been working very hard and like week nights and weekends to
like get that done. But to think that we took a service that we were spending like $60,000 a year on and we were able to rebuild parts of it like a little Shim infrastructure for what we were using of it in about in less
than 60 days. Probably more like 35 days was a result of having used that service, was a result of having this serverless mindset and of being able to say, OK, now we understand the problem so well and we isolated it so well that actually rebuilding it or understanding how to shift it somewhere else ended up being much easier. And so I love being in an environment where everything is
kind of contained and small. There's good separation of concerns, and the best way to do that isn't micro services. The best way to do that is to use a lot of managed services because they've already built all those great APIs for you. And then if, look, the case of it's too expensive or I'm locked in is you just build it, then. I mean, that thing that's so wild to me is people saying like serverless is about locking,
It's terrible. It's actually like a very cheap way to like learn how to build something and then if you're gonna build it yourself, just copy the parts of it that you like. It's actually much faster. Yeah, interesting insights, right. So I think many people are, I mean do afraid about the lock in and all that. But I think, hey, if you can get started very easily and you can use all the power of the people operating those services, right. So they are very well expert in
running those services, right. You can actually just use that because, yeah, it will make you start really, really easily. And I think you mentioned earlier that if you have a choice of writing code versus picking up managed service, you should always choose, you know, managed service. And in your book you mentioned this thing called code is a liability. So I think some people think code is asset, right. Some people think code is the most important thing in your company.
So tell us why code is a liability. Yeah, I mean code is something you have to maintain and so this idea doesn't come from me. And so in the book, there are some good links here. But you should think about anything that I write is something there that's necessary for my organization to work properly. Then I have to make sure has proper testing, like doesn't regress, keeps working, and so I have to maintain all of that. That's a liability. You know, the asset is the
experience you're building. And so if I can build something with 10 lines of code, or I can build it with 1000 lines of code, I'm gonna be much happier in the long run with the 10 lines of code. It'll be easier to maintain, it'll have less bugs, it'll be easier to test. All of those things will be much better. And so the less code, the better. At Branch, we have a kind of a waterfall process where when somebody says, hey, can we build this thing and will you build
this thing? Our first question is actually, do we need to build anything at all? And so we do a lot of why don't you use this software as a service instead? And if we need to make some sort of integration, we'll do that. But like, let's just just go buy a software as a service. That's step one. Step 2 is kind of can we like down scope it? Because if you buy a software as a service like you get all those
features, those are great. If that won't work, how do we like take out what you're asking for and make it really small? One of the things that I've found over my career, what generally happens in organizations is people say I want this thing and they usually make it really big because they know that you're gonna go develop it and as soon as you put it live, you can't work on it again for a really long
period of time. And so they're gonna pack everything they can into it because that's the only way they know how to get. If you reorganize and this is very much in line with the Dora metrics and the Agile Manifesto, if you say, look, two things, one, I'm gonna make you down scope this thing to Tiny. But two, I promise we'll keep working on it immediately. Like we won't stop working on it. It's not a limited time window. Then we work on other projects for a year, keep working on it
as soon as it gets released. It's got to be really small. Then you get all this benefit of the feedback in the cycle. So you just squash down what people want to like very tiny, like just helping them a little bit, and then you get it live and then they can see it, and then they can make that next request. If you train an organization that way. And it's easier as a founder than as someone who's inside an organization with other
founders. But I can tell you, you can train an entire organization to think this way, and as long as you deliver on, we will make updates, you come to us, we'll go get other small updates in as quickly as possible. Everything gets more efficient because, you know when you build these big projects, like they're poorly designed, you put them live, they don't work right. We realize like, oh, that was a bad idea. That's a bad interface.
People don't understand it, like, just build it small and watch what happens and iterate on it. So we have this whole process of buy SAS downscope, use a managed service, use open source, everything we can to not write custom code or to write as little custom code as possible. Yeah, I think it's a good principle thought process for anyone who listen, right. So code is a liability. It's a reminder again. So if you think writing a lot more code and infrastructure code or configuration also
counts as code, right? So you do remind yourself. It does although and we don't have a ton of it. But I do think that, and this may be an unpopular opinion, but I do find that domain specific languages for infrastructure like YAML and like cloud formation I do in my head put that in a different category. And for this reason, my experience is when we write cloud formation YAML and we get it right, we never change it and so the maintenance cost of cloud
formation YAML is very low. And so I I don't really worry about having a lot of that. In contrast, something like Pulumi or CDK, that's like code. I find it gets a lot edits, people keep updating it, it's got a lot of maintenance and so it has a different profile to me. And so I would view CDK infrastructure, Pulumi infrastructure as code, as this code that you want to minimize.
I actually think of YAML as more like a YARN lock file or something like that, Like it can get really big, but it's not a maintenance headache. So I actually don't find lots of YAML to have the same liability problem, because generally speaking you get it up and it works and you don't cheat. Interesting. So speaking about serverless, right, I think we have talked a lot about serverless, but I think it's appropriate to get the definition right.
Some people think serverless is, you know, Lambda. Some people think it's a function as a service. So in your view, right? What is your definition of serverless? Because I think from your book it encompasses more than just a function as a service. Yes, and I give I think like 4 definitions in my book of it. My favorite definition of serverless is it's not my up time and so it's functionally about not having to run that.
If it goes down and it's my fault, it has to be like I misconfigured it or I wrote bad code, but otherwise I don't control the up time. But Ben Keogh also has, you know, Ben Keogh's definition here is I'm only doing differentiated coding, right? Everything that's undifferentiated, I'm handing off and there's a bunch of stuff about scale to zero and things like that.
If your view of serverless is And I I saw a post on LinkedIn recently which had somebody saying serverless is expensive and it'll send you in a ruin and it's too complex and just had all these diagrams of like lambdas, calling lambdas and using AWS services. And like I tend to agree that those are I don't like those infrastructure. I've never built one of those and I think that they're overly complex and I I don't think he under.
I mean he's never read any of kind of the serverless practitioners that I agree with the least about what serverless is. And so to me Serverless is very much about asking how do I use. Most of our serverless footprint at branch is managed services. So it is not. I mean, we use Lambda, we use Dynamo DB, we use Cognito, we use AWS App Sync. These are amazing services. My book explains how we use them and why they're so great. And so we do use Lambda, but Lambda isn't everything in the
application. It's like where we need to run some back end code. We run it with Lambda, but most of the lambdas are getting triggered by calls through app sync. But if you don't know what app sync is like, to me it's like a key part of the infrastructure. I also written a lot with Firebase. I think Firebase is also wonderful. I think what in writing kind of hobbyist apps. I think Firebase is just easier to work with than app sync and
the Amazon infrastructure. So I like that Firebase infrastructure for simpler applications. The Amazon one is just much better for like financial services. At least it's it's more robust in a bunch of ways, but that's how I think about it is really, am I responsible for uptime? As long as the code functions properly, then that's serverless. And so I often will tell people you know it's serverless to go build your shopping site on Shopify.
Like that's a serverless architecture for me, and if you don't think about it that way, then I don't think you're in line with what serverless actually is and how to get the business benefits out of it. Right. And you also mentioned managed service. I think for some people there is a lot of confusion as well. What do you mean by managed service? I think Twilio is the best example for people at a managed
service. It is cloud based API usually that you pay and they have a well documented API. They've got a whole operations team, it's multi tenant as a service and you pay them money and they're just up and they do things for. You. Yeah, and I think in your book, if I remember, I also see like for example, if you have to patch something, right, if you have to choose the size, if you have to scale it somehow, right. I think that's not serverless, right? So. Right, right.
And there are these Gray areas, right? Because in Lambda you will pick a size and so like I view Lambda as serverless and I would just say in Lambda just by the largest size because you get more CPU with it. At least by, I don't know, 4 gigs, they default to this minimum size. That's terrible and you shouldn't use it, but you know, that's a Gray area.
But yeah, I I also think about it like, yes, if you have to patch it, but also if it runs out of disk space, whose responsibility is it to manage that? That's a question I like to ask a lot. And so if it's your responsibility if you fill up a disk like, it's not circles, right? So I think that's a good reminder definitely. And of course when we talk about serverless, there are many skeptics like you mentioned, right?
So many objections in the beginning we talked about cost lock in, some people also think about security, right, Because if let's say you use managed service for those managed services like a vendor, third party, right, you don't know them, your data spreads across different services. What's your take about all these objections? Maybe tell us how would you actually, you know, advise people? Yeah, the most important thing that you need is good third party risk management in your
organization. And I would actually say one of the biggest hindrances to serverless adoption organizations that really want to is that they have very poor third party risk management programs in their company. And what they have decided to do in the company, maybe not even consciously. What they've decided to do in their company is basically make it really hard to do contracts with other companies as a way of security. That's not security, that's just pretending you're secure.
And I recognize a lot of lead developers may not have full control over this, but it's useful to know It's useful to, like, have this in your head. Your company should have a robust way for you to say I would like to use this company and to be able to review it in a reasonable amount of time and see if it meets the security guidelines for the information
that that service will get. You should have a good way of looking at its historic uptime and evaluating whether you think it's uptime is going to be appropriate for your use for it. And then you know your finance department or your legal department need to be able to review the contracts and make sure that they have an appropriate DPA, for example, and have other things. And then if they don't meet those things, you shouldn't do business with them.
But I'll tell you, as a regulated US insurance company, there are a few companies regulated, you know as heavily as you as insurance companies. We use lots of these services and it's not a problem, but it's about having really good third party risk management. And that third party risk management understanding that they exist to help us do contracts and work with other companies because that's a key differentiator in how fast we can build.
So getting the company aligned with that is a top down goal of like helping everybody understand we need this function within our company to be able to do these things. But that's how you do it. I've been in charge of information security, you know, many organizations that have had a lot of compliance requirements and every time I run in, anyone says, well, we we never pass compliance with that. I'm just like, no, you just don't know how to do it.
I guess to quote Twitter, that's a skill issue. It's not difficult, but it does require sometimes creativity and it does require knowing your stuff. You do need to understand like, you know, what is our information security policy? Why do we have these rules? Why do these regulations exist? And you need to be fluent enough in it and have a good Chief Information Security Officer who understands them and cares about productivity. So you don't have all these things that can be very
difficult. But I can tell you it is so much easier to be secure with serverless and manage services than it is with an internal network. And so like we are fully no internal network, fully zero trust at Branch and the levels that we can achieve and what we are able to do on the budget we have which is very low is absolutely phenomenal and would have been completely unattainable for me 10 years ago at Guildfax, which honestly has what would be considered like a best practices cloud
architecture today. I mean it's full infrastructure as code. It's got great network roles and great in VPCS and everything, but managing that, proving that and everything around that is so much harder. And so, you know, didn't leave a lot of budget for other things. And at Branch we have managed to make budget for just amazingly
high quality security. I'm very amazed when I, you know read that branch is a insurance company, highly regulated and yet you succeed in adopting all these serverless right, because I think security compliance is always top of mind especially in Fintech or you know financial services industry, right.
But I think what you mentioned is probably like it's more about understanding the compliance itself, right, and be creative in solving it and maybe you put more proper checks and balances before you actually use those managed service. And speaking about, you mentioned it's not our up time, right. So what happens in incidents where the third party is down, right, I think it's also one of
the most common objection. If those core services is down, then we are also down and we can't do anything. So what's your take on this objection? Well, one thing is we run in US E 1, which is great because if there are problems in US E one, we know a large chunk of the Internet's going down as well. And generally speaking, we go down less than the rest of the Internet. So I've been in USC Swan since 2008 and this is what I found.
So I remember the outage in like 2012 and there was a Netflix outage there. I just realized that, you know, you get a lot of grace when it's a USC Swan problem. So in the last five years, there was only one significant USC Swan outage. It was a couple hours. It didn't take us fully down. It was the one that like affected Route 53 DNS where it became clear that that service is bound in USC one and it's not really free from USC One. And so you know everyone says why would you be in USC one.
But I'll tell you like if USC One has problems, like no one's going to blame you because every all of their other products aren't going to work either. And then outside of that, you know we basically don't have outages. We did have one in January 2022. So we've been selling insurance since the middle of 2019. And so I can tell you other than that couple hour USC one and this January 2022 where this insurance specific vendor we used and so there aren't a lot
of other choices. This insurance specific vendor we used went down. Other than that you know there were kind of partial or degraded services from time to time. Occasionally something won't be available like we have to pull motor vehicle records and those come from the States and those services go down like every other day. One of those services down for like 15 or 20 minutes and so you know you just build a lot of alerting around it. And so our applications are very aware of that.
But in terms of the core infrastructure of app Sync, Dynamo, Cognito, Lambda and US E One, I don't think those have really gone down at all ever. Even in that US E one problem they were all still working. I think Cognito had some issues but was just slow. So I would say my experience with it's not my uptime. I hired a company that is very good at uptime has resulted in we basically don't go down.
We have much better up time than I've ever had running you know in any of these organizations running my own infrastructure and so we're running some of the infrastructure. So I think it's kind of a non issue. It is certainly possible. I I will give one sort of caveat here is I did build an application using off 0 for authentication and that service went down all the time in my view. Not for long periods of time but all the time and like I would never.
I mean maybe they fix this. I'm skeptical though after their acquisition and like everything that's happening with Okta I would never use off 0 again and I would be very cautious in authentication services. But I can personally vouch for Cognito and Firebase off, having been very reliable services. Yeah. So when you speak about up time and reliability, right, when you choose serverless, it's not like any kind of serverless or managed service, right.
You also look at the historical up time you have, Yeah. Can they guarantee a good SLA, right? Is there any compensation that they will give or never? This yeah, I tend to look, SLAS are non compensatory, like they'll never compensate you. So I tend to like an SLA where it's punitive to the company. What I'd like to feel is like if you go down, it hurts EU. And I do think like Amazon use these outages very personally and reputationally. And I think it matters to them quite a lot.
And so they're never going to like compensate me appropriately in the SLA. But I think the historic up time is the thing to look at and just to make sure you're getting an honest look at the historic up time because that's the best way to tell. We do use some services that are in beta and you know, we just do a lot of testing to get comfort with them. But there are occasionally times we'll say, yeah, we're not going to use that yet.
We're going to wait another six months or a year, but we use Neon really early. Serverless Postgres because we really wanted it, and Amazon, Serverless, Postgres are not Serverless, They're terrible. And Neon is a great Serverless Postgres database. And so we started using them when they were in private beta. We tested them out. We were like, how do we feel? And we've just built in some backup caching in case it wasn't available. And that's worked out really well.
Leon's been very reliable. Yeah, apart from the historical up time, I think one thing that is quite, you know, recent trend, right, people updating postmortem whenever there's an incident, right. If the company diligently uploads postmortem and they take it seriously, I think that may also be an indicator that the company is serious about their
up time, right? Yeah, it's not a reason why Amazon is so serious about that in a way that Google and Microsoft are not as serious about it, But I just feel a lot safer for production workloads in Amazon, although again, I run them in the other clouds as well. I think many people here would have been intrigued by Branch, right? Let's talk a little bit about Branch. What makes it unique, maybe in terms of how you run the development team, how you run
the infrastructure? There are a few things that you mentioned in the book. Maybe let's start with your primary development principle, right, where you say when we write code, we optimize for maintainability. I think this is very interesting. Maybe can you talk more about it? Yeah, I'm always surprised at how infrequently development teams will say what they're optimizing for in terms of what they're doing. Like, most teams don't have this
principle. But it's very useful to know when you're like looking at code or when you're sitting down to write something like what is my guiding principle on how to write this? My experience is if you write something for a good organization that is going to live for a while, you know I wrote code in 1997 that's still being used widely in Pearl in 1997. And so I think a lot about what is the most important for the company. And I don't think it's about
making it speedy, right? I don't think it's about other things. I think it is. I want an average developer to be able to understand this and work on this. I think far too many companies are like, we've got these 10X ninjas, we write for 10X ninjas, we're 10X ninjas, we just hire 10X ninjas and all that. First of all, if you have a bunch of people who think they're 10X ninjas, one they're probably not, and two, they're probably writing unmaintainable garbage.
It's a lot better to have people to have the cultural identity of like, it's important for me to write. I do like the idea of like I'm writing this and next developer has to work on this, has a gun to my head like write this thing so it makes sense, so it's easy to read. So that the code review that you have one you should have code review. You know everyone, not senior developers should have their code reviewed. Everyone should have their code reviewed.
And we should have you know an understanding that I'm writing this so that somebody else, an average developer can come along and understand what the heck I was doing and like let's do that And it makes code reviews easier to know. Like this is what I'm optimizing for. It's like having an elinter or having style rules. You just like have opinions about things and everything gets easier. And so that's our primary
principle. And there's a corollary to that, which is less code in general is more maintainable than more code. And so there's a lot of debate about like, should you DRY things up, how do you think about repetition? I think you can obviously go overboard with DRY, but I do think in general, you know, it's very common. We hire a lot of junior
developers at branch. It's very common for junior developers to have patterns that are too repetitive that are just much better and much more readable. These are just everybody knows the pattern of like if I equals one then you know return one, if I = 2 then return 2 or whatever like that. Obviously that's a you shouldn't that's too much repetition. Like, I find more junior developers tend to not have these like ways of like how do I abstract this? Like there's a pattern that I'm
repeating here. How do I abstract that. So in a very normal setting, I find that I do reviews and like somewhat frequently say, hey, you can get rid of some of this repetition, like abstract some of this logic. And, you know, I think that that's the key part of the principle is to have a less good, but not at the expense of maintainability. Yeah, speaking about developers, I think this is also unique in branch, right?
You said that you hire more junior developers and actually you have less so-called infra developers or engineers, right? So tell us like, how do you actually run a good insurance company with mostly junior developers? Yeah, my hypothesis in starting branch was that, and I had seen this at a small scale but not at a large scale before, is that if we build serverlessly then mostly we're going to be
building interfaces. And so my belief was I could hire front end developers and I could basically make them full stack developers as long as we were using the same language, JavaScript, TypeScript on the front end of the back end. And I thought if it's not our uptime, if other people are doing the uptime, then I should in theory just be able to hire front end developers, make them full stack developers and then
that's all we would have. And then everyone would just be a developer, and then everybody could work on anything. They could work on the infrastructure, which is in YAML. So they could work on the phone and they could work on the back end. And then we'd have it all in a mono repo. Oh, and we'd have a mobile app. It would also be using JavaScript and React as well. And you know, everything would be kind of the same. It would all be the same repository. We would deploy it
monolithically. Everyone would have their own isolated Amazon account to deploy it. And so this was the vision from the beginning. And it worked. It worked so well. And actually, let me take a step back. So at the beginning I said I can't hire a typical senior developer because a typical senior, I know what's going to happen if I hire a typical lead who's listening to this podcast. If I'd hire them into that, the average one of them would have said like oh, I know how to build.
It's not like this. We need containers. We need to go build some containers. We need to do this thing. We need to do that thing. And my vision would be eroded. And and I believe in empowering people I hire. I want to hire someone and I want to let them do it the way they want to do it.
And so I said I can't do that. So I said, OK, the safest thing that I can do that is if I hire more junior front end developers who I feel are capable on the front end side, like I think I can teach them how I'm thinking about building this and I they won't know any differently, right? By the way, it'll be so empowering because instead of just being a front end developer who's like dependent on other people, they'll be able to work on everything, They'll work on
infrastructure and everything. And so we did this and it worked really well. And we said, OK, let's hire people directly out of boot camps now instead of, like, with a little experience. And so we did that. And we realized these boot camps aren't teaching people anything useful. Like, we're having to, like, retrain them on everything. I've got lots of thoughts about boot camps. And so then we said, let's run our own boot camp because the boot camps aren't useful.
And then we run our own boot camp. And we did it again. And so you know, my opinion today is if you set this up correctly and if you hire the right people who don't have preconceptions about you know how things should be done, then we have lots of people are very
happy in this environment. By the way, we did a search for AVP engineering fairly early on, about 3 1/2 years ago and found this wonderful guy Ivan Herndon, who had been an engineering manager at Stock X, but mainly front end and but like really understood everything and had taught at a boot camp. And he was like fully bought into. Like this is a great model. I like this, but it took like 8 months to find him. Like it was just like a long journey, but we had built enough.
Then then we went out and hired more senior developers and we basically screamed for like who's going to like this, you know? And we found senior developers who were like, I love this, this is great. Like this is exactly what I want. Like it's so easy and I have so much control. And like again, when our senior developers are given a feature, they do all of it. They can do 100% of everything.
The infrastructure, the front of the back, and like, we truly have full stack developers, but they're full stack developers because the stack is simple, right? It's all the same language, It's all in the same repository, right? It has a monolithic deploy. And so we can make full stack developers if we make the stack simpler. Otherwise it's much harder to have full stack developers because you need to know too many different technologies. And so it's worked phenomenally
well. We just have developers. There's no front end, there's no back end, there's no API, there's no infrastructure people, there's no OPS people because it's not our Uptown. Yeah, I think it's really interesting model, right? If people haven't heard about it, I think this is a very interesting thing that you can
learn from Joe's book, right? So I think how to make it work is something that maybe you need to work on more, like for example training people, make sure your code is maintainable. But I can imagine for example, having everything as a serverless managed service, right? Every developer is empowered to not just code and commit their changes, but also bring it up to the production, right? And even operating it in a sense because they can just see it end
to end. And many people these days try to build platform engineering right simply because they cannot do self-service right. They cannot make deployment themselves. So I think this is a different kind of perspective how you run the engineering team. So if you use more serverless, I think you probably don't need so much platform engineering effort, right?
So you can do it end to end. And I think there are many other things in the book that you can learn from branch, like for example, a few things that I learned, the department head actually is the one who kind of like negotiate contracts with the SAS, not like the central team. And the other thing is like the engineering lead is the one who configure the cloud or be the DevOps engineer kind of thing, right. So I think that's also
interesting. Maybe one thing if you can share right, people might be intrigued like how much Cloud Bills actually Branch is running. Yeah, I mean I think in the book I shared. So we have an Amazon account for every developer and for every environment. And every developer in every environment has a full copy of production. And this is cheap when you use serverless because everything scales to 0.
So like the average developer who's working at least 40 hours a week doing development, their environment might cost four or $5 a month because it's just not that much usage. But yeah, I think I shared a bill where our entire all developer environments, all everything was about $1000 at Amazon for that month. We're now significantly bigger. So you know, that production environment is probably more like the full bill 7 or $8000 a month right now for branch.
But it's every environment, every developer, everything, including like all of the continuous integration testing that we have. So it's so much cheaper. I mean at build fax in 2015, I think dams on monthly bill was probably $30,000 a month and that was doing so much less honestly in that environment, but having to run, you know those VMS and containers. Wow, 78000 a month for all environments. That's really all environments. Yeah. And you are a full running
insurance company as well. So kudos for you to run that. So I think we reached the end of our conversation. So before I let you go so to speak, I have one last question for you, Joe. It's been a pleasant conversation by the way. So this question I called the three technical leadership wisdom. You can think of it just like advice that you want to give to
us to learn from you. Yeah, so I've got 3 here for you, and I'll start with some things I've said before, which is one you should have an optimization principle for how to write code. It's just as important as having linting rules or style rules. You should also have an opinion about what you're optimizing for. So we optimize for maintainability.
I think it would be fine at you know in some places and might be we're optimizing for like the lowest latency execution or whatever it is. But you should have that. You should write it down because it will help solve problems and it will help resolve questions and doubts when you're not there. So you should do that and then you know, my view is optimize for maintainability is a great default. And the second thing is less code is more maintainable than more code.
And that is just true outside of like obfuscation contests for like small lines of code and so you should strive for less code. Again, the book has sort of an interesting waterfall about how do I what is a tactic when I'm being asked to build something? How do I end up with less code there and then? Finally I have a saying that I give all the time, so I'll give that as my final one, which is along the same lines.
But it is that it is better to spend 2 weeks researching and two days developing than it is to spend 2 days researching and two weeks developing. And I'm always just amazed by organizations that don't give developers time to research how they would do something. And I'm actually less amazed at developers who just want to start building. It's the hardest thing I know. I mean, I always wanted to start
building too. But the more you can train yourself and your teams to think about how do I spend a good period of time where I'm just trying to understand what's out there, What are the options and how do I make that a good thing. And like, I'm going to throw away whatever's done at the end of it. And this really you need leadership to help you with this. But I think as a tech lead, you can set this tone.
And I'll give us the example. Our airplane migration, when we found out essentially like January 5th that we were going to have 60 days, we actually spent basically all of January trying to figure out what we were going to do. We looked at a bunch of other services that were options that we evaluated them and we talked to their teams and tried to understand things. We looked at what it would take to build it ourselves and what that would look like.
And so we spent, I think a lot of people would stay. You spent too much time, you know, like not acting because the service was going to shut down and it was critical production functions. But the reality is you just can't make the right decision unless you spend time
understanding what's out there. And today, if you're building new things, one of the things you need to know what's out there is what are the managed services that could do a big chunk of this for me. And you're not going to understand those unless you give yourself the time to play around with them. And almost all of these providers will give you free trials. You may have to talk to someone. And I know a lot of developers really hate scheduling the meeting and doing the Zoom demo
and stuff. You may have to do that. I'm sorry, but it's still worth it. You should do it. Don't say I can't do a self sign up on this thing. I'm not even going to consider it because there's no pricing page. We can't consider it. Don't do that. Go understand it. Even if you don't use it, there's actually real important value in understanding what is
the service. Get access to documentation, understand what that interface is, understand what that price is. All of that will really help you understand what you need to do better, even if you don't use it. And we just don't do that. We don't do any research and planning in dev. It's like, oh, I know one way to build that, so I'm just going to do it that way. And like, don't do that. Do take weeks to research. It's worth it. Yeah, not just research in terms of product, but sometimes like
design, right? Like how would you design your solution? Yeah, And sometimes understanding the problem itself, right. Because sometimes. Oh, absolutely, Yeah. All of them is critical, yeah. Yeah. So thank you so much, Joe, for this opportunity for this great talk. So if people love this conversation, they would want to connect with you or find more about your resources, your book. Is there a place where they can
reach out online? Yeah, I'm Joe Emerson on Twitter. Slash X. And yeah, my book's on Amazon. Serverless as a game changer. I really highly recommend people to read it if you are into serverless and if you're skeptics, I think you can also read this book just for your knowledge, yes. So thanks again for your time, Joe. Thank you. Thank you for listening to this episode and for staying right until the end.
If you highly enjoyed it, I would appreciate if you share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you're new to the podcast, make sure to subscribe and leave me your valuable review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show notes of this conversation on the episode page at techlitjournal dot dev website, including the full transcript, interesting quotes, and links to the resources mentioned from the conversation. And lastly, make sure to subscribe to the show's mailing list on techlitjournal dot dev to get notified for any future episodes. Stay tuned for the next Techly Journal episode, and until then, goodbye.