How'd you like to listen to dot net rocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here. As you may have heard, NDC is back offering their incredible in person conferences around the world, and we'd like to tell you about them. NDC
Copenhagen is happening August twenty seventh through the thirty first. Go to NDC Copenhagen dot com for more information. NDC Porto is happening October sixteenth through the twentieth. The early bird discount for ADC Porto ends July twenty first. Go to Dcporto dot com to register and check out the full lineup of conferences at NDC conferences dot com. Hey there, this is Jeff Fritz, the purple blazer guy from like or Soft, letting you in on a little secret about my
friend Carl Franklin. You know, the guy who started dot net rocks. The first podcast about dot net in two thousand and two. The guy who's been teaching Blazer on YouTube since twenty twenty, Yeah, that Carl Franklin. Well, Carl's joined up with the folks from Code in a Castle to teach a week long hands on Blazer class at Are you Ready Get This? At a castle slash villa in Tuscany. It's sort of a luxury vacation with Blazer
learning built in. Carl's calling it the Blazer master Class. You'll learn Blazer from the ground up, finishing the week with the ability to build and deploy Blazer applications. Since the training happens for only four hours in the morning over six days, you can bring your significant other, your partner with you and you should right This part of Italy is absolutely beautiful. There's so much to see and do, and in Larion Marco from Code into Castle are organizing daily
activities both at the castle and in the area. The castle is in the Marema, a less touristed region of Tuscany, offering both classic Tuscan hill country as well as easy access to the Etruscan Riviera, with sublime local food, wine and olive oil around every corner. Breakfast is included. Every day there will be two communal dinners at the castle book ending the experience, and most other meals and all activities are included. And did I mention you'll learn Blazer
in person from Carl Franklin. Listen, space is limited and for very good reason. This is quality training in a beautiful setting. Go to code in Acastle dot com slash Blazer twenty twenty three. That's b L A z R two zero two three to take advantage of this amazing opportunity to join Carl in Tuscany for an unforgettable week of La dolce vita while advancing your programming skills in this important new technology. Welcome back to dot and rocks. This is Carl
Franklin and this is Richard cavill Man. You know I got a story for you. Oh my new studio is coming along. Have you seeing any of the videos I have seen? You know what I saw? I saw red? Yes, you do? Like the red walls. I do. And it's the same exact fabric that I had at Plato point zero. Interesting, Okay, the same exact stuff and it's that was really two point zero right, No, No, that was one point well, okay two point oh,
yeah, I guess. I mean that the old State Street place you had it was the training space first, then it became the studio space, and then you have built this studios. Yeah, yeah, you're right, like you definitely went through some iterations. We might as well say studio one was my basement in Waterford where night arguably community. And now you're staring at five. Yeah, I just call it. I call it PLoP three point zero. Okay, I'll buy that. Yeah, we'll buy that. The
third version is the best version anyway, everything you know. Yeah, anyway, putting in the floor in a couple of weeks. The split HRAC system is going in on Tuesday. Nice and the band is going to start rehearsing there in just a few weeks. So I'm really excul that's really great. Anyway, let's start with better no framework, roll the clar music at awesome. All right, what do you got? This is um something I know
you're going to be interested. You probably already know this because you know everything, but it's a It's an article in Electrek Dot and CEO. Sweden is building the world's first permanent electric road that charges moving evs that's cool, isn't that cool? Yeah, And there's a picture of a bus and it's driving over this blue strip of charging some things in it. Magnetic charging, I don't know, you probably know. Yeah, it's probably cute style charging or
just impedance or field style charging, which is final. They'll only charge so fast, but you and it's pretty smart because you know, it doesn't have to charge really powerfully because you're constantly going over it. Right. Yeah, if every foot you get a little bit of charge, enough to go at least a foot, then you're positive. Right, Yeah, I don't think I don't think you're gonna be that net benefit, but hey, anything that adds a little range, yeah, you know, it stretches a bit further,
so much the better. I mean, clearly, this project is an experiment. They're talking twenty kilometers, right, so it's one piece of one thing, and who knows how many more they're going to do. But you know, obviously after their experiments they said, hey, this piece makes sense that we're going to keep doing this and we'll see what happens. It's great to see, you know, infrastructure come into play. Not to derail the conversation, but you also saw Ford signing on a now GM signing on with
Tesla to do the North American charging standard like this. Here's an interesting point. The model team released in nineteen o one, first gas station opens nineteen thirteen. Wow, what do they do for twelve years? Yeah, I don't know. You bought your gas in containers from the chemist. That's what you did, chemists. Yes, well you know what we now call a
pharmacy, but basically exactly. But I mean to that point, you know, watch how these things emerge, the idea of the standard eyes gas filler that you know, one gas station had different tanks for different kinds of fuel. Right, you can get diesel, you can get we have to do a new geek out on electric cars and electric transportation perhaps, but you know they we're at this point now where it's like, hey, we want standards around this, we want these things to work correctly. So again, the
experiments continue. This This electro tech article is a great example of that of more experiments. Well, you know, it's a nice experiment, but will it scale? That's see what I did there. I like that. I know it's very clever. Are you're the one trying to stay on subject? That's weirded out, all right, who's talking to us? Richard grabbed a comment off a show seventeen seventy eight. That's the one we did back in twenty twenty two about pro micro services and dot at six. That was Sean
white Cell, Rod Richardson and Matthew Groves. And this comment comes from Mark Essex, who said, Wow, what a timely podcast. We're in the early stages of hoarding our monolithic web forms app, which has been in production for twenty plus years in some form to micro services. I immediately went out and purchased the book and I will force myself to make time to read all four hundred and eight pages. Well, I'm sure Sean and Robin Matthew will
be to listen to this show first. Yeah, I thought discussion was so relevant, really hit on some key points that we've been discussing internally about areas of concerning issues it might arise. I'm scared to death. I'm about how to debug this thing, or to test it, or to ensure the same quality that we've enjoyed with our current application. I mean, if you've running an app for twenty plus years, like people are pretty used to it,
right, and as somebody else mentioned in the comments. I really hope we don't have to decide to go back to our monolithic app as that would have a whole other set of challenges. So thanks so much for a great podcast. Yeah, with any luck, Mark, you'll be that. You know that was a year ago admittedly, so you're probably pretty far down the path. Maybe the show isn't timely enough for you. But here's to hoping.
We're certainly having a conversation about you know, there is no one perfect solution for anything, and there's a bunch of ways is all of this problem, and that's certainly what this show will be about. Yes, sir so Mark, thank you so much for your comment. And a copy of music Go Buy is on its way to you. And if you'd like a copy of music Go Buy, write a comment on the website at dot net rocks dot
com or on the facebooks. We publish every show there and if you comment there and everything in the show, we'll let you a copy of music Go Buy. And you should definitely follow us on something. You can follow us on Twitter if you want to, we were there, or Facebook. As Richard said, but the cool kids are over on Mastodon, right now I'm at Carl Franklin at tech hub dot social, and I'm Rich Campbell at Mastodon dot Social send Institute, Rudy Tooty, Fresh and fruity. Okay, let's
bring on Derek. We actually started the conversation before we started recording, so we'll probably pick up there. But Derek Comartin is a software developer. Over two decades of professional experience, He's written software for various business domains such as consumer goods, distribution, transportation, manufacturing, and accounting. Derek has a very active blog on YouTube channel called code Opinion, which focuses on software architecture
and design. Welcome Derek, thanks for having me. I wanted to say I'm a first time color longtime listener, but I was not sure if anybody would realize what that is. Yeah, well we get it. School. Yeah, longtime listener, first time Yeah, I love it well. And I'm a longtime reader of your blog, sir. I often tweet out my favorite of your blog. Yes, yeah, I appreciate that. Yeah for sure. So scaling in general is seems like an insurmountable task when you know
the boss comes and says we have to talk about scaling. Uh, okay, because we can't. We can't imagine to what degree we need to scale. And most companies think, if they're public facing websites, for example, they think that they're going to be the next Netflix or the next Amazon or something, and so it may be completely over architected and completely overextended. And then you have the companies that are small now but think that they will grow
to be these big companies. So we want to be ready to scale, and that may involve a lot of extra architecture developers don't want. I mean, there's just so many gotchas in scaling, and I guess that's why we
brought you on to help clear up some of this nonsense. It is like it's an interesting topic because, like you said, there's this kind of this notion where right from the get go, we need to decide now in stone exactly what it's going to be so we can be the next big thing or that we think we might be. And I think there's just this right now, even though we've been swinging back, I feel like a last couple of
years worth of this. Okay, we've been doing the kind of getting more focused on micro services, like we're talking about the comment, and I think it's just it's viewed the wrong way. I think it's you can go pretty far for if you have them onolith or if you're starting that way, if you think about how you're building your app, not necessarily in a physical sense right away, but more in a logical sense, how you design it that way, you can then decide how you want to scale. And there's a
variety of different ways to think about scaling. I mean, I think primarily if you were to stay cheese, I don't know, fifteen twenty years ago, before a cloud, you would it was a little bit more difficult in the sense that you were thinking, okay, we got to buy new hardware, and that was a little bit more difficult to scale up or scale out potentially. But now it's just a little bit easier. But the kind of
the same thought process is still it's still the same for me. It's thinking about just so the traditional things of are we scaling up, are we scaling out? But they all kind of have implications about what I really call moving the bottleneck. Usually when you're changing one thing down the road, you're likely
changing something else. So it's it's kind of an evolving thing. That's why I think it's a little bit of a fallacy to think up front, oh, we're going to create microservices and that's gonna just solve all our scaling problems. When scaling of water actually we're talking about there, because in most times that's more of an organizational thing than it was necessarily you said the word microservices. I could tell that a million sphincters just tightened. I thought I saw
your eye twitch. Actually, carl As, we've got video because of the sphincter tightening. I think you know, it occurred to me. It's like we should switch to microservices is like the new version of we should have written this in Ruby on rails right Like it's it's the hey, we can't fix this problem, so let's come up with a solution so complicated that you'll leave us alone for six months. It's interesting because we were kind of alluding to
this before we started, which I was bringing up months ago. I can't
remember how long ago it was. There was Amazon Prime Video had a video or had a blog post about kind of their transition from using lambda's aws lambda and a combination of S three and some other things, and they migrated that to just be containers in ecs, and there was seemingly kind of an uproar of various opinions, including my own, about what that really was, which people were saying, well, they went back to a monolith, which even in their own posts they did, but I kind of said, it really
wasn't And there's a lot of people claiming, oh, well, how does Amazon doesn't even know AWS doesn't even know really what lambdas are good for, And it was just a lot of hot takes I guess about Okay, they had this service specifically for I believe it was for kind of trying to determine the quality of a stream and letting know the streamer know, like when quality was degrading, etc. And it initially said in the post that it was
just it wasn't designed for the scale that it ended up becoming, which is okay, great, I guess, and they evolved it. It's like, what's so terrible about deciding, Okay, well this worked at one scale, but it's not currently how it is now in the volume that we need. Yeah, it's funny how people want the one right way right, like they well, if it doesn't work for this, and it wouldn't work for anything, right, I mean I read the original post and what my immediate thought
was, why were you using servi lists here? To me, the reason you servillists is you you have an elastic load that has to abruptly scale up a lot and abruptly drop back down. The audio video consumption on Prime Video is probably pretty high all the time. It probably goes from high to really freaking high, Like what do they need that, I'm out of elasticity for well, as LAMB does though they they can shut down, right, Yeah,
that's the whole thing, right is it very elastic? But it also takes time for them to spin back up again, so that could you know very few people, you know, the first person that has to wait for that has to wait. Yeah. And another piece of that was from what I got from it was just the cost in the issues of moving data back and forth from those landas two S three like back and forth, like pull them that data from there. Yeah, so going to ECS and being a
container, you're pulling it once they're running their detectors. It's all at that point in memory, once they fetched it once, rather than the however many times they needed to do it. And in theory serverlists should have if you're used to service properly, it saves you money because you only use instances when you need them. But it's got to be the shape of the workload, right, And I guess as they understood the shape of their workload, they
could implement a lower cost solution for that workload exactly. It's just it's I always say context this king like at the starting of it, maybe I don't know what their situation was other than what they wrote in the post. It just it didn't seem like the initially the solution work because it it'sn't at the scale that they were headed towards, and they had no intention to it. So it evolved. It's like, yeah, okay, great, you evolved. It like makes sense to me. This is like saying what you're a
teenager, childhood's a waste of time. Yeah, it's a great point. And it is really like like you said, it's a hot take and then and a hot takes in general are incorrect. I think anybody's actually built and cared for software, Like I spent a lot of time helping people scale, and you always start the conversation with corridulations. You have a good problem and
enough people are using your stuff that scale matters. If nobody was using it, it'd worked great, exactly when you kind of in for I would say, I'm making the assumption here that it's not an abrupt oh oh my gosh, I need to scale like from zero to one hundred. It's usually kind of you know, I mean over time as you're gaining traction, if some type of sas product, whatever the case may be. And usually when that's
coming up, that's when you're trying to determine what needs the scales. So, like I was mentioning, if it's it could just be scaling up kind of that app service that you're running. So let's say you're just running some web app, some ht B API, whatever the case may be. Maybe that first step is okay, just scaling up, giving it more resources,
giving it more memory, whatever it's kind of starving on. But like I said, when doing that, usually the next step it might not be instantly, but it might be okay, well, now we can handle more inbound
requests. So what is that going to affect downstream? Right nowadays, it's not just usually some app running, it's app running with obviously a database, some other infrastructure, there's a lot of moving parts, and usually when you scale, when you fix that one bottleneck, let's say it's at the very beginning of that app compute, you're likely gonna start seeing a bottleneck somewhere else
and that yeah, in some other infrastructure. So then you're kind of tackling that piece of the puzzle if you need more performance, right, like, yes, absolutely, yeah, And there's a lot of ways of handling that. It's I think it's just it's not immediately. I think when people get into it, and it's kind of a slow process of having to fix each individual place that may be experiencing performance issues if you will or you can't handle
that type of load. I think intuitively people start figuring it out of where what to fix where, or whether it be like it said, scaling up, scaling out at adding more instances. But I think that a bigger challenge oftentimes is more related to infrastructure that you don't necessarily like it's not your app, so that whether that be your database and depending on what type of database you're using. Right, So it's easy to say, okay, well I'm
just gonna scale up my relational database. It's not so easy to all of a sudden say Okay, well we're going to introduce a read replica and that's just gonna be no problem, right, And that's a major re architect Like that's it can be your data quite differently, you want to talk through a read replica or folks who haven't dealt with it exactly, Like, it's not all of a sudden, it's well, I always say this about cashing. It's like I always get the sense that there's this people talk about it.
I just I mean, add some cashing problem solved, everything'll be fine. It's like I always like people say that cash invalidation is a hard problem. I don't think it's like it's it's a hard problem, don't get me wrong. But in the sense of an application like these types of systems, it's I find it more of an issue of just dealing with a cash in general.
There's so much that can go wrong with it, sure, right, so well, and it has direct impact on business, right, Like every airline consolidation site you've ever used doesn't check with the airlines every time they show you a flight, they've cashed all of that, And every so often you're going to click on a flight for a price, and then it's gonna go, oh, sorry, I don't have that flight. Then maybe it's one percent of the time they're going to annoy you that way, and they've decided
that's sufficient. I dealt with an inventory system that was too slow and they wanted to add cashing to it, and I said, that means occasional you're going to sell stuff you don't have. What do you want to do about that, right, And the answer was, we put in a back order system, Like we accept the fact that we don't have it, and so we now have. We never had back orders before, we'll add back ordering.
So if you've ordered something that wasn't in the we're still in the cash, but it wasn't on the shelf, it's like, hey, really sorry, we put you on back order. If they choose to cancel whatever, but at least we're doing investor can and we literally change the business flow to compensate for performance exactly trade offs, right, And a lot of the stuff is usually built into in most cases, like if you're talking about whatever industry.
I love that you brought up that example because I always said that if you oversell something like in distribution and warehousing, it wasn't a sales problem. It's a purchasing problem, right, so they usually have stuff in place for this to begin with. But whether it would be a read replica that you're introducing a cash when we're talking about something that's hivan stale data and how you're
going to deal with that. But even just like to touch on the cash again, it's even from a technical point of view, is it always available? Is it responding to you like you're using a cash typically because you don't want to reach out, you want to offload some of those requests from your database. What happens when the cash isn't available or it's not as responses of you do want it to be right, right, guess what you're doing.
You're usually going back to the database. So right, so if all of a sudden everything's sailing great one day and then there's hiccups, there's performance issues with your cash. Now all of a sudden, these requests that we're hitting
the cash or cash misses and they're going back to your database. You just flood your database with a pile of cash miss is that it normally isn't expecting Yeah, let's talk through the read replicub model, because I think it's really interesting for folks, and I don't know, I don't know there's a product off the shelf that can let you do read replica, Like you have to code for this, this idea that you're writing to one database that then is
synchronizing to one or more reading databases. Yeah, I mean there's options to have some type of proxies there so and then rules built around that so you can be offloading them. But it's just still a matter of if that re replica is a synchronous and it's eventually consistent, then that's more of the issue
than necessarily like you need to know it. The whole world has been taught that by Facebook because Facebook clearly uses read replicas, and when you didn't immediately see your post, you would post it again and then you see them both. Like everybody knows about that problem now just because they learned Facebook. Yeah, I think there's definitely something to it in terms of note like people are a little bit more familiar with with it, whether they're technical or not,
that can sometimes probably guess what's happening. Yeah, but still it's not the greatest obviously, and I think it's similar whether it's a re replica or whether it's a cash they're both are potentially stale at any given time. It's like it's contact specific. Most users, I would think, in your example, if they performed some right, which they may not know they're doing, but yeah, if they're performing some right and you're immediately going to provide them with
some read of that right, they're expecting it. Yeah, and you And there's lots of ways to cheat on that too, absolutely and say I already you won't You just want to see your data, so I'll keep a copy of your data around to show you right away while it spends X many milliseconds
getting synchronized on the back end. I mean, we're maybe we're getting a little nitty gritty here, but I found the bigger thing for performance in the rereplica model was that the right database A was insert only and B had minimal indexes on it because you didn't you didn't need them. You were never going to read from them, and index has caused the law of a blocking and locking that they then you would later synchronize it later being milliseconds later, would
synchronize to a read replica that was orgoned. It only had one place of right path coming from the right database, and then did have all that indexing to optimize for read that as well as which is another option as we're talking about options for scaling, is creating what I'll call like there's different ways of determining that calling this is like a materialized view, right, creating some type
of projection where that's specifically for that's something that's query optimized. Right. It's that kind of separating these concerns of Okay, I have these rights, like you're saying you want them to be fast or kind of off floating indexes. There that contention and doing the same thing on that read side, which is, let's optimize for reads, whether that be indexes, whether that be more denormalized in regardless of what type of data store, it is just more structured
to the way that you want to fetch that data. Well, it really salient point. It doesn't all have to be relation databases, like one could argue, and I have made the argument before and on this show that when you've received an order, which is basically a glob of objects, now, making the customer wait while you decompose that in the rows and columns is kind of daft, like just store the object. It is the source of truth, send the customer on their way, and do your decomposition into the read
system separately. I'll take it a step further and say where we're going with this is offloading work. So that offloading of that decomposition to whatever immune secreture process exactly. But that cannot be just like that view and composing that view.
It's actually composing the work, like actually performing the work. So like I said, like if you're placing an order, it's I don't necessarily need to persist the actual order and perform the payment in everything, It's okay, here's my message dropped a CE. Do this asynchronously where that I can perform that work. And in terms of scaling, that's like I would say, kind of the I think it's a way that people immediately can think of in
your system. If you're not using any type of messaging, not using queues or anything, and you want to start somewhere down the road, that way is just look at something simple that you know can be asynchronous, like an email. Right when you click that button that says do something, and it performs some type of tasks it's say persisting data and then needs to send out
an email, It's like nobody expects that email to be instantly there. Right, So that's kind of the immediate place you can look at your system. Where can I offload some of this work? Where can I run this asynchronously? Yeah? Yeah, And I love the expectation part. Nobody expects it like they're used to. This is asynchronous. That's and it's going to have some latency and that's fine, right, And I think it's how you're how you're providing it to the user. Kind of then I hate to use user
experience, but it is. It's it's how what their expectation is. Like I said, was reading your right is do they expect to see immediately the receipt with their transaction number of their credit card? Like not necessarily so that it's just a lot of this work can happen. It's how you're presenting it to them. Thank you, We've I mean, we've accepted your orgale. Will let you know when it's processed. I like e commerce systems like that, like okay, I've received your order, all right, we've yeah,
we definitely have. All another emails is I've we've picked your product. We know we have it all. Now I've build your card, here's the receipt like and that I a lot of ways. Like I like that better because you see it's clearly a workflow. They're not making anything up, they're validating as they go, and more importantly, we'll step off if there's a problem
and they're using known methods to communicate email, text, whatever. What I don't like is when you may want to track your package install this app. Yeah, no, go away, thanks, no spying, no spying, thanks very much. Yeah. The queuing model is you talk about a programming intensive modification to a piece of software, is to go from a synchronous database connection to a que approach like a takes some time to get your head around that, but it takes time for the users to get used to the fact
that all of that is now different too. Yes, for sure, an existing app depending on like what we said, is depending on how you're implementing it. But like I said, I think there's a lot of places that you could probably look in your if you think if people are listening or now thinking of where can I do that, it's it's just trying to find those natural places that are asynchronous. And I think that's just an easier place to start, especially, like you said, wrapping your head around at how it's
working in your system. But it's there's more places probably than not that you can that are really good fits. Yeah, there are some building gotchas with that kind of system too, such as let's just take a very simple example. You have a model and you want to add this data to the database through this model, and the model doesn't have a primary key because there's no primary key created in the database. Yet. You might have a you might
have an identifier for that model, but it's not a database identifier. But then you can't really use it until you get that and you get that from the database, right, the database is going to assign you and tell you what that unique identifier is. So in the meantime, you can't do any edits, you can't do any updates, and you can't even do a delete. I mean you really kind of have to for that record that you're you just created. Your user is in limbo. Yeah, that is one.
There's a bunch of gotcha's here, for sure. That's probably the one that I think people run into most. But oftentimes a solution there is depending on where you're making the requests from, is just providing that identifier from the get go. It may not be the primary key, but it's some other way that you can look up separately with what you've generated. Yeah, right, So if it's some type of guid, I've moved to GUIDs for primary keys
instead of letting the database do it for me. I mean, anybody can generate guid at any time and it's going to be unique. Yeah, exactly, It's just you're at the client, and I don't mean the client as the end user. I mean is wherever that needs to initially make that request is essentially providing some identifier that wherever else it needs to be doing those lookups like you said, so it doesn't necessarily need to know when the request was
processed. Right. The example I love to use in terms of visually two is if you're in AS, you're you're in AWS wherever, and let's say you go to stop, restart or do something with some type of resource. You know that's not happening immediately. You're not just like making that request. Then know this VM is stopping like it just stopped like right, Like it's an asynchronous process, you know for sure, that that thing behind the scenes
is queuing that up for that request to be process. Now, in terms of feedback, like we're talking about, if that is a thing where you're behind the scenes providing some identifier, it may be using i mean some type of web sockets connection to get pushed back down of that update happening, which most of them do, so that you kind of get that instant feedback,
but it's not really full proof if you've ever used it before. So you are refreshing to kind of see the latest, and there's usually your little refresh button there for that purpose, but you know it's in the indicated to you. It's like it's stopping, it's restarting and shutting down. It's you know, it's not instant. Yeah, absolutely, And Derek coming in and run for one moment for this very important message. There is always something new from
our sponsor, text Control. As a developer, do you need to integrate PDF generation, document editing, or electronic signatures into your asp net corp or angular applications, or you want to learn more about the differences between electronic and digital signatures. Text Control is offering a free consulting service to educate you about digital document processing and how text control products can help you add these features to
your applications. Go to text control dot com, slash contact and request your free personal consultation and we're back. It's done at Rocks. I'm Richard Campbell. That's Carl Franklin, Hey, and talking to our new friend Derek O. Martin, who writes his great blog posts on Code Opinion and as an excellent YouTube channel, talking about scaling the monolith. That being said, have you seen scenarios where micro services were the right choice? I think so.
I think, Oh well, let me start by saying I can't deal with the micro part of the word there. Yeah, it's just silly. Yeah it is, but yeah, I know, um, but I think it's I think it's applicable. I think I wish the industry would get more on thinking in terms of monolith and micro services, but rather thinking about logical boundaries
and physical boundaries. I wish we could just term it that way. So I mean by that right is thinking about if you're defining, like if you have some type of large system, define logical boundaries, which I say are incredibly important to do, yet they're really difficult to quote unquote get right. Because of what right kind of changes as things evolved. Sure, but it's thinking about what does your system do, like what what pieces of functionality?
Where's the value defining those piece of functionality, grouping that altogether logically, and then deciding, okay, once we have all this, how do we deploy this? What's the physical aspect of this? Right? Can this all to be deployed together? Can this be all composed into a single unit? And maybe this is just deployed as a container, maybe always built separate correcting because
a unit yep, deployed as a unit. Maybe we are using messaging and queuing and it's still underneath the hood using the same like all that same code, but the entry points are different rather than let's say we had our first piece of the puzzle here was an htb API or some web AP, but there's still a queueing component of that, which could be just a separate container that's listening to whatever queuing message bus or you know what I mean, broker
that we're using, still using the same underlying code. So there's just this aspect of we could compose this however physically that we want to. One logical boundary could be deployed independently if it needs to scale and it has different resources. That's a certainly situation I've run into where we realized we're scaling this large set of classes, most of which are not being called in each of the
scale instances. But you know, there's just one problem a child that gets called a lot, and if we peel it out of that larger ball of goo, it'll scaree. It's easier to ramp that up, it's lighter weight to do so, and the rest you don't need is a bunch of the other stuff. So it's just like pull the too part together in exchange to the complexity of now we have another security boundary, we have another API boundary,
like all of those things had to be addressed in exchange. Absolutely, Like it's trade offs, right if you all of a sudden want to decide, Okay, we're gonna peel out this part, and a lot of this comes down to how you're coupling between these logical boundaries, right, So that's kind of a big factor in whether you're going to be able to extract something, but how you're doing that. But if you decide that is something you
need to do, like you said, for scale, purposes. Then you have a lot of options about just in terms of how you're communicating with the other logical boundaries. Whether that's something that you hopefully from the start maybe went down a more loosely coupled route, but I realized that's not always the case.
But you realize that, Okay, if you weren't, and you were doing in process calls, say between logical we'll call service A and service B, and now you're going to do that over HTTP, gRPC, whatever the case with some request response that's latency. There's a whole lot of that's not like, oh, we're loosely coupled now because we have two services that communicate
over the network. No, you just created a huge performance bottle, exactly done, exactly so, thinking that's why I said I don't love the monolith versus micro services. I'd rather the conversation was around, Okay, let's define logical boundaries and then separately, let's talk about how these things are deployed.
Right, So, what are the logical boundaries? I mean, scale is one of them, but I'm sure there's others in well, So the way I often think of logical boundaries, the way I describe it because I lived in warehousing and distribution for a long time. And my explanation of this of how to think about it is, Okay, yes, you have a warehouse system. It's enormous, right, it does everything, It does a whole
lackload of things. It runs the entire company essentially. So what does that mean though, Well, it's if you think about it, if you were just to think about the business itself, there's a lot of departments, like they're shipping and receiving, there's sales, there's purchasing, accounting, the warehouse itself, people working in the warehouse, and if you want to think of it that way, at a sense, that is a logical boundary, and
that would be a logical boundary in your system. And kind of the way that I like to use to describe this is if you were working in that system and you're talking to anybody that works in that business, and you're to go to them and you say, like, what's a product, they would all give you different answers because each one of them has a different concern about what a product is. Right, So if you're talking to somebody in sales,
they care about the price. If you're talking to somebody and purchasing they're talking about like the vendor cost manufacturer. If you're talking to something in the warehouse, they care none of that. They just care about where's that product physically in the warehouse or warehouses. They just have different concerns. So I think that's where the kind of the I think people get in trouble is thinking of quote unquote entities as there's like one singular like be all end all of
a product, right, But there's not. It's it's logically it's the same product, but it just exists in different ways, in different logical boundaries. Yeah, and that's normal. It's always been true. It just becomes more
apparent as you press against these kinds of problems. And that's the thing is that when you think about when you think about them that way, and you think, okay, okay, I have this subset of functionality that we're performing on, like in sales, for example, about how we sell products, their prices, their discounts, all those types of things, and you have some other like let's say purchasing. As was using the example and thinking about
vendors. It's kind of liberating in a sense because they're different ways of thinking about it and not conflating two ideas together, which then in terms of your codebase, you're starting to think about, okay, well it's not just this what I call entity services where it has all this mixed behavior about all these things. It's no. Then you become a little bit more focused on what the how I say is, like, what the capabilities are of that logical
boundary. The biggest part is usually is inevitably these things need to be composed in some way. Yeah, that represent a product. Like if some users on a screen, they might want to see, okay, well what's the sale price and what's the actual purchase price? Right, So how you're composing those things if you're in process together, well, maybe you are just making API calls, I mean in process from as it was. Let's use the
example of you are making these things different projects within a solution. Maybe it's just some interface that you're calling, some type of class that you're calling to get that data. But there's still that boundary there physically that you're doing. It's just how you wanted to then do that. If like you're saying, if you're kind of ripping it apart, you need to be aware about when you're going ahead of this, like ahead of time. Okay, we're gonna
rip this out and then we have all these in process calls. That's going to be a problem. Yeah, where where is it going to? Where is it going to impact things? You can't just pull it out and hope for the best. Do You need to have some sense of organization and where where the code dependencies actually exist and decide if it's actually worthwhile too, Right, there can be cases where it's like the performance gains just not enough for
the complexity being added. Yeah, that's one of the thing with complexity is it's it's you're moving. You're getting pretty high when you don't need to if you don't need to move stuff out of process. Right, It's just I guess there's this sense that moving stuff out of process, are starting that way immediately for the sake of scale. Is I'm genuinely curious sometimes that people really know what they're getting into in terms of all the like just deployment in I
realize things like open telemetry are awesome, but not that simple. It's not that simple. Well, this is this is the developer credo. We we did these things not because they're easy, but because we thought they were easy. Yeah, I want to hear you say that with a John F. Kennedy accent. Not because they are easy easy, but you know, we have this terrible disease in software development where we keep calling stuff easy that clearly has never been easy. Like virtually none of this is easy. You may
know how to do it, but that doesn't make it easy. All everything is effort, and everything comes with a cost. And and you know, and you're and you're shifting problems around. There's no free lunch here, not a bit. I think it's a good way of thinking about as shifting problems are like just creating I was creating problems you're yeah, I mean you're trying
to come up with solutions. But everything we've mentioned right in terms of cashing or read replicas and queuing, they all come with a lot of trade offs, a lot of complexity that you're adding that you need to be like that it's got to come with value, like you actually need need it. And if you're unfamiliar with it, some of these things that you're going to get a root awakening, right, Well, that's an overhead too. If how
long is it going to take to learn this to do it decently. But also I don't think there's any way to predict the outcome, Like you can't do a calculus and go, hey, if we do this, we'll get sufficient performance for effort. In some ways, there's almost no way around putting out the effort to get to a point to even know this has a return. And that's certainly the challenge I had was scaling anything was we're going to try this, well, you do it as quickly as we can, but
we're not really going to know the results to we do it. I mean, hopefully we measured before we started so that we can measure after we've done it to say there was a return. The number of times I've seen stuff people just do things and go does it feel faster to you? LIKEE know, faster. That's the thing brings up a good point in terms of measuring, because a lot of this now for me is it's it's purely based on that of knowing, just having metrics, having some type of visibility, and
how your systems performing. Besides the well, this seems slow, It's like yeah, is it? It only seems slow because the CTO keeps telling me it's slow. Yeah, everything seems slow. Nothing's quite fast enough generally. Yeah, well and is that return right? Like, how much faster can I make this? Yeah, I'm used to. I'm used to using an I nine computer that I built myself. That's just the fastest thing I've ever
used in my life. And the other day I set up a webcam using one of those little Intel Mini stick computers for gigs of RAM or something like that. And if you know, watching that thing just crawl, you know, just sit there at a prompt thinking and thinking and thinking, and I'm thinking, wait for computers like this and we don't even realize it because they were they were so slow, they weren't realize there's a slow until we've got
something faster. Well, I think when we move back kind of in the conversation and we were talking about, I mean being elastic and having those options, it's it really is incredible now to be able to do it in terms of land as in anything, even if we're talking about just the ability to scale up quickly with minimal minimal downtime kind of gives you that option when even if it's just for a moment of time where you don't really know what the
heck's going on and you're having issues, you can do it and at least kind of weather the storm a little bit so you can investigate what what you're going to do. But I generally find most of these things they're they're kind of if you have metrics, if you know what's happening in your system, you can kind of tell slowly over time where things are going to start going south, whether like what infrastructure it is, what resources it is it is it your app compute, is it your day to base, is it your
cash whatever services you're using. Ultimately you can kind of start telling. But it's I find one of those things that's usually you know the metrics, usually when it's somewhat too later, you don't have the foresight to know what you need to measure now. I mean when you really go back and read that post by Prime Video, you see exactly that. Over time as we measured it, we saw performance problems, we saw them with problems, and we
saw costs increases. Like they didn't know when they went into this. I mean one would argue they built it server list because it was quick, it was an easy way to go, and it's only after it ran for a while and it got more popular. It's like, huh, now, let's and they literally say, let's revisit architecture because you have that option and got great results from the revisit, Like this is actually a good news story.
I think it was, And especially at the end of it, I believe at the end of that post they were saying how most of the code was
reused. It was it's basically pulling it out of landas that were howard they were being invoked, and it's putting it into that process that's being executed in as a container in ECS, like they did a platform shift, like this is actually wicked cool that a lot of this code was just fine, but when you shifted the platform around, and if you shifted the platform around, you got rid of a bunch of contention, it reduced costs, Like there
was all these benefits as different architecture. Would it work for you? Done? O, You got to go measure what you're doing right now to try and compare against other architectures. I think that's the most difficult thing. I think you alluded to you earlier. It's just like we want this cookie cutter solution that this is just going to work all the time, and it's just not the case. It's just like the one right way. Please. Yeah, it's just not the case. It doesn't exist. It's very it's very
funny just that. Yeah. I wish it was that simple, but there's too many different kinds of workloads, and there's too many different ways at things evolved. You know, I wonder if this was a great solution when they started. I think it was. I how I remember it correctly is like I think their exact kind of message was we never intended it to be at the scale that it was. Yeah, so when and I dealt with lots of e commerce sites like that, we thought it was only going to have
this many customers, and then it got all this much bigger. It's like, wow, great problem, Like part of this is his whole You ain't gonna need it. Like how do you go into a complex architecture having no idea if the product's really going to take off or not? To build it as simply as possible, because delivering it is an important value too, you
know. One way to make sure it never provides value is never finished the darn thing, because you made an architecture is so complicated it takes forever to get it built. Yeah, it is. And when we go back to the kind of the one way that this is going to work is it is it's it's starting simply based on what you're aware of kind of that space.
But like I said, if you if you focus to me, if you focus on defining logical boundaries, trying its best you can to understand the coupling between them, You've got a lot of options about why you can choose. Then you can choose how you want to do this stuff. Like you said, if there's one particular part of your system that is heavily overloaded, yeah, maybe that's where you choose to remove it and peel it off and scale
it on its own and see if that's sufficient. You know. The other thing about that good architecture, that separation and concerns is the experiments are easier too. Yeah, well, I to based on experiments. I would also say that not everything needs to be created the same way, right right, So that means that if each if each boundary has defines how it wants to
do a storage and that could be still the same underlying physical database. Instance, Let's say it like, hey, we're using this relational database for the entire thing, the exact same data we got it. Yeah, we're not gonna like split these things up more costs also just to use the exact same instance. It could be in the same physical database. But hey, these
particular tables are owned by this particular boundary. Don't go start muddling in my stuff, right, Like, defind some type of API we want to exchange this if we need to. Yeah, but just define those boundaries and then from there maybe you decide later on, you know what, like this particular
boundary that relational just wasn't a good choice here isn't needed. So yeah, so like when you're staying like experiment, it's like you get that option, right, It's and it's not like you're doing this because you want that option. It's just you're getting it kind of. It kind of is a little free lunch. And that's it comes at the complexity of you have to define these boundaries. It's not just one massive thinks about it. Yeah, but also now that you have that boundary, it's like, hey, would this
be better served in an object store instead of a relational database. Well, let's let's see what it takes to do that. You know, maybe a little spike on it to do some experimentation, and it's like, hey, this works pretty good. I'll finish implementing it, and then let's do some benchmarks. You know, I've added that they you know, then we'll maybe
deal with some of the complexity around it. Like it is. These are fun explorers too, right, Like I found I had pretty good patterns for scaling websites because I did a ton of them, and I was often right. But you'd get into more esoteric systems, more interior type architectures, you're like, yeah, I see that this is likely the bottleneck. I see three ways for us to try and scale us. I can't tell you which
one's gonna win, Like, we've got to try exactly. And I think kind of my point when I wrote this post even some of the videos is just there's just you have different ways of dealing with it, right, So whether that be scaling up, scaling out, knowing where you're going to kind of move that ballneck move that problem, whether you're off floating work with queues, messaging, whether it be materialized views for queries, and the other point
I want to make is, at least in my experience, is generally there's hot paths. Right, Like, especially in big systems, I always kind of use eighty twenty rule, is that you're going to have the twenty percent that's causing all your havoc basically right, and just being able to optimize those So whether that be never mind from an infrastructure point of view, but just let's look at where those routes are, Like, let's look at those code
pass What can we be doing there? Specifically? Are we making too many round trips in the database? Maybe we're just doing something absurd where we're just hitting our database and fetching able to a whole lot of nonsense that we don't need. You'd be surprised how much that happens. So it's even just and it can be cost effective, as crazy as sounds. Sometimes it's like, let's just spend some time doing this, Like, let's just look at what
we're actually doing. The question is do you know what twenty percent or do you have to get out of the field and run it for a while ago,
Oh, this is the twenty percent that's the problem shelves. I think metrics, but I think intuitively in a large system, least in my experience, you know what that core part of functionality is generally I think even from if you were to ask, say, people on the business side, like hey, and the app, where do you think the most action is at the because that's what they're in most of the time, right, Like it's I would say, there's kind of the core part of your system as well
as that fringe supporting more generic stuff. So like when I was talking about distribution, say let's say like CRM or something like that, maybe you're like, you know what, that's not a big part of our business. We have some rudimentary thing for dealing with customers. It's really not that big a
deal. Like our bread and butter is warehouse management, and that's where all the complexity is, right, so you kind of know to a degree, I think even on the business side, they could probably give you some hints. Well, in the web world, nobody ever went wrong tuning a landing page, you know, because it just had so much traffic, like it was pretty much a guaranteed win. It spends some time in there or usually it was just you got to a point where it's like, hey, the
incremental improvement left is so small you can't justifind more time on it. There are easier winds elsewhere, but they may not be as important. Ah, it's a fun battle, you know. It's not like there's any simple solution to this. You've got to go out measure and feel an ound and hypothesize and build some tests and push a little further. But I wonder if we're speaking heresy here, because last time I look, there's a cloud and I just have a knob I have to turn up right, Like, isn't there?
They go faster knob of everything related to the cloud. You just spend more. I mean too, I like we're joking, but like somewhat yea in this sense, right, it's at least being able to scale up in a lot of circumstances, like I can always shift to a larger VM, and they are perfectly willing to sell you a bigger and bigger VM these days. And seeing two is that you can do that with at least with app
services on Azure you can. You can. It's really trivial to add and scale it up exactly like if you if you already are in the situation where all we we scale out, say maybe our web app and we're running three instances, but all of a sudden things are going south, and we'll just add more instances. But like I said, and that may work. You had one instance and then all of a sudden, you start noticing, like I said, downstream start suffering, and then it just bottleneck and you get
a back end bottleneck. Like well, yeah, okay, you also have to be your code has to be able to handle multiple spanning, multiple machines. That's an old talk I used to do from one server to two. Yeah, you know, it's a pretty hard leap just to get to that point, you know what I mean. With Blazer server anyway, which is sort of like in my wheelhouse now that if you're using their um the signal
ar service, it's pretty trivial. You there's affinity is turned on by default, so whatever, the first server that you logged into, that's the one that you're going to use for the rest of your session. And that's that's nice to not have to worry about that. Yeah, I think I think the scale ups scale out just with cloud just making it so much easier is
great, and I leverage it quite often when things start going south. But I guess in experience, I kind of okay, well, like we got some thresholds here where I know, okay, if this is scaling out, this is when it's a little bit too far. I just can't infinitely scale out and everything's going to be going to be great. And to that effect too, A lot of it is things that aren't even in your control.
So if you're leveraging, say some external services, and you're making requests to them, I don't know the examples, like say you're making a call to being Maps or Google Maps or any other service, some type of payment gateway, and you expect that to on the happy time, it only takes like hunter milliseconds, it's fine, and all of a sudden that thing starts taking seconds. Yeah, right, like all of a sudden. It's so,
what the heck's going on? Where what we were mentioning with metrics that comes into play because you know, how matter how much you scale out, you ain't going to help that. Yeah, yeah, the simple So what's next for you, Derek? What's in your same old thing of just creating videos and blogs at all? Kind of pertain to everything we just talked about pretty much for the most part. Well, I'm going to urge our listeners to go check out your blog and continue reading it. Thanks so much for spending
a time with us. It's been enlightening. Yeah, I appreciate this was this was really cool, all right, and we'll see you next time on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online
at pwop dot com. Visit our website at dt nt R o c ks dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September two thousand and two. And make sure you check out our sponsors. They keep us in business. Now go write some code, See you next time. You got a vand