How'd you like to listen to dot net Rocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot NetRocks dot com. Welcome back to dot net Rocks. This is Carl Franklin, this is Richard Gamble and this is the coming out on the twenty first. So this is the last dot net Rocks
show published before Christmas, but we've got a few more to record. Just Geekouts. How you been, man? You know I've been. I've been working on the scripture the Geekouts turns out. You know, it's a lot of stuff. It's been a busy year, yeah, sure has. How about you? What are you well? Last night I did my first recording of one of my songs in the studio, in the new studio. Oh, a new track, A new well. It's an older track that nobody's
heard yet. But the first recording I did was all kind of discombobulated. It's slower, it's at one hundred bpm and and there, and with acoustics strumming and picking and stuff, and therefore it's very easy to get off time. So I want to tell you about the cool stuff that I did that I did this with. I recorded the acoustic guitar in the bass first, and then I used a studio one's bend tool to quantize the audio. What quantize the audio? It basically worked right out of the box, and I
had tried it before, but it hadn't really worked well. Maybe the default settings are good now. But what it does is it finds the transience according to your quantize value eighth notes or whatever, and then it essentially moves the transience and either stretches or compresses the audio between the transience and basically turns it into quantized like it's right on the beat. Okay, so it's just like making the beat perfect. Yeah, And there's no it does not sound artifacted
at all interesting to me anyway. So then I did the whole the same thing with the drums. For the drums, I used my iPad at the drum set because it's all the way across the room right right, and the iPad I installed this thing called studio one remote, and so I set markers in the three places where I wanted to record drums and I could just go back undo record. So I did all the recording from the drum set with my iPad on a music stand and I did two sections, one with brushes
and one without. And that it took a while to record, you know, but it was great because I could just take another takenother take till it was right, and then I quantized the drums. Those drummers are never on beat anyway. I was pretty close, but I wasn't perfect, and I really wanted this to be tight interesting and it really is. So I'm very excited. Everything worked and yeah, what can I say? Very happy the new studio. Yeah, fully up, rational nice. Hey let's get started
with better no framework? All right, man, what do you got? Well? I found this post on board panda dot com. Have you ever seen that crazy site? Yeah? Yeah, So this is thirty of the worst Christmas gifts people ever received, as shared in this online group. One of my favorites is a dish towel. I was eight, It was eight years old, and my parents gave me a dish towel. Nice and it
wasn't even a new dish towel. I learn it was just like they went to the kitchen, found a dish towel, wrapped it up in paper, and then get to this eight year old kid. H it. Yeah, Dad, Christmas presents so hours and hours of fun and yucks and silliness. Silliness, Yeah, which we need once in a while, right, especially around the end of the year. It's like we've been working hard all year. Now let's take some time. Awesome. Anyway, That's what I got.
Who's talking to us? Richard grabbed a comment of of Show eighteen seventy three, the one we just published a little while ago with Leah Melanino from NDC and Portal. We were talking about sustainable development. You know, I'm just thinking about the energy can say. We've got a great comment here from Jackie. You said, Yeah, Hi, Carlin Richard. I want to express my thanks for tackling this important subject. This conversation resonated with me a
lot, as I'm a software engineer working with Azure Cloud technologies. In our related note, I'd like to share my recent experience with an iPhone. I've always been an Android user, primarily do the perception that iPhones are overpriced to non standard devices. I think Apple pretty much dictates the standard. Just look
at out what happened they did to RSS. Often seen as more suitable for users seeking opinionated ux like my mother, or a symbol of social status like my brother Jackie, like your diss in the fan, right it's Christmas. However, when my Google pixel broke and I had to let it in for service, I was left with no choice but to use a backup iPhone fives, a decade old model offered by my brother. See he may be a social status seeker, but at least they'll give you a spare iPhone. It's
not a bad guy. He's not a bad guy. Yeah, you had to reassess this relationship, man. And to my surprise, the iPhone five S was still receiving security updates and all my essential mobile apps function flawlessly on it. This is something I couldn't even say about some five year old low end Android devices, not that any iPhone is a low end device. The only drawback was battery life. Yeah, battery is not going to be great.
This experience made me realize that even though iPhones may not be considered entirely sustainable, due to a lack of adherence to certain standards like USBC or Place Little batteries. Their longevity is great right up until you drop them. Therefore, I've decided to offer an iPhone model with a USB C port. Thank you EU who demanded that Apple start using USB c's. This all allow me to utilize most of my Android gadgets and extended lifespan on the phone. And
thanks again for your dedication to the dot net Rocks podcast. It has been a valuable source of knowledge and inspiration, especially during my career transition to dot net c sharp looking forward to future episode, and I can add to that the iPhones are inherently more secure than Android phones just because Apple is such a closed system. That's one of the benefits actually of having It's sort of security
by obscurity. Nobody wants to try and act them. Really, I don't know about obscurity, but security by by iron fist control over everything, everything that goes on that phone, right, So yeah, I like I That's that's one of the reasons why I have an iPhone. And the guys at Security this week Patrick Hanes and Twain Laflat say the same thing. Yeah,
I get the iPhone. The phone is more secure the Android phone. I mean under the hood androids Linux. Everybody knows Linux is terrible, especially Daniel. He knows that that. I'm just gonna poke at the Linux guy the whole day. So we're gonna do but we're gonna poke at each other at Christmas time. Come on, hey, Jackie, thank you so much for
your comment. Glad you really liked the show. And a copy of music co Buy is on its way to a un If you'd like a copy of music Cobe, I write a comment on the website at dotnet rocks dot com or on the facebooks. We publish every show there, and if you comment there and read on the show, we'll send to your copy of music code by and you can follow us on Twitter if you like. But the real
fun happens. I'm mastadon, I'm at Carl Franklin at tech hub Social, and I'm Rich Campbell at masodon do Social send us a two You might get a mug if we read it on the show. Think so pretty sure you won't? Yeah, pretty sure? You get a copy of music opy. Kind of how that worked? Did I say mug? He did? All right? Let me say that again, Brandon, all Right, senas, although that was kind of funny, we might want to leave it in. It's kind of funny. I'm pretty sure you won't. Let's leave it in.
It's Christmas time, what the hell? All right, let me bring on our guest, Daniel Marbach as a distinguished Microsoft MVP and software maestro at Particular Software, Daniel knows a thing or two about code. By day, he's a devoted dot net crusader espousing the virtues of message based systems. By night, he's racing against his own mischievous router hack, committing a bevy of performance improvements before the clock strikes midnight and he turns into a pumpkin. Yes,
exactly that. What's a router hack? What are you going to do? Low wrt? No, it's just it's a very simple sort of trick because I've been contributing to open source and various things, and I'm just I just like to spend some time with code because I feel like it's sharp and sort of my understanding of just actor i'm working with. And I had appeared in my life where I just couldn't stop right because it's like and then it was at first it was like one am, two am, three am in
the morning. And then luckily I always basically my only one rule was that I will not extend my alarm clock to a later point in the day, so otherwise my days would have shifted. But at some point I was like, Okay, that's it. I need to change something in my life.
So basically I'm switching off the internet around midnights at my whole Oh and that's when you have to hack your router so exactly, because that usually when you're when you're like in the middle of thing and is I should Google bing this or whatever you're using, right, you're like, ah, the internet doesn't work anymore. Ah, it has time until tomorrow and then I switch off and just go to bed. So yeah, I never worked that way for me. I had to turn the bugger back. I put the same rule
in place because I had teenage daughters. And now you hear the audible groans at midnight, right, it's like, oh, yeah, everybody's in bed, sure they are. But me, it's like, I guess got to you know, finishing up to the go to push the code fails. Yeah, you're like, uh, it's me. I got to fix the rider.
So your performance wonk, are you? Well? Actually, I work for the company called Particular Software. I guess you had Dhan on the police before, right, so you know we had a few particular rights and a few others and yeah, so I basically my day to day shop is I'm building a robust and reliable frameworks and libraries for people that sort of want to sort of build it distributed systems primarily based on messaging stuff like ASH, service bus, SQS, SNS, storage queues in the old days, God forbids
MSMQ. Right, but it's still it's still out there thriving. Surprise, surprise, It's quite heavily used there in the industry. Yeah, and it's one of the great things is feature complete, right, it it's just there
if you're still running on Windows and other on Linux like I do. And one of the things that we do there is we want to make sure that the customers that they're using in service bus can focus on their just writing their business code, don't need to write any plumbing code, and that stuff should run as efficiently as possible. Right, So I guess performance throughput was always sort of right from the center sort of in my day to day job.
But they also care a lot about it because I believe especially it came us out in the comment right that you read out Richard. There's like, if you're targeting the cloud, or if you're shifting the cloud, or if you're already are in the cloud, sort of you are basically built by the amount of resources that you're using in the cloud. Yeah, so there's a direct revenue relationship to that consumption, which it leaves giuse of some kind of incentive.
Yeah. Correct, You put down your credit card and then you get surprised at the end of the month. Especially I'm surprising yourself as one thing. Surprising the CFO is another. That's a very loud noise from a large office. Yeah, that's the question. I mean, there's so many ways to tweak performance. One is just by using the latest netstack correct, and then you know, keeping your new get packages updated. But on top of that, you know, are what are kind of knobs are you post?
Are you pulling software knobs, hardware knobs? All of the above. So let me cook before I answer your question. Let me quickly go to what
you said about updating the dot net the dot net version. That's actually really interesting because Microsoft has this block series where they essentially talk about their teams migrating to for example, from dot Net framework to newer dot Net versions or from dot net six net eight, And one of the cool things they blocked there is they the Microsoft Teams Infrastructure team, they basically migrated from dot Net framework to dot net six and just by basically migrating to that LTS version of dot
Net, they were actually able to sort of buy almost twenty four percent reduce their monthly cost expenditure in asure, which is pretty amazing if you think that's amazing about that, right, And that's definitely one way to do it. So I always encourage people to sort of if they can stay up to date
with the with the latest dot Net versions. Definitely, Yeah, I'm a big fan of that series on the dot net blog, just because you know, you talk about like the teams guys migrating to a new version of dot net and the benefits they got from it, and also the things they struggled with on that, like what problems they had. But to me, more than anything, it's like, hey, you know, if these guys got this kind of benefit and we're able to move that big an app you're going
to be okay, like you can do it. Yeah, absolutely. But to come back to your question, Carl, I think one of the things that I try to apply to sort of in my thinking is I want to make explicit trade off as I'm going with things, and so that means I want to be aware of sort of is this code going to be executed on the hot path or at scale, right, so, and how many times a second is that going to be or is it just something that runs on a sort of a background chop once a day or twice a day or because
then usually if it's just executed once or twice a day, it doesn't really matter that much whether it's super fast or not. But then when it's executed on the hot path potentially hundreds and thousand times per second, then it's usually good to sort of become more of performance aware, but performance aware. Unfortunately, this is also something that gets thrown around quite a lot sort of in the industry, and then everyone assumes everyone knows what performance awareness means, right.
But one of the things that I struggled with was where should I even get started to become performance aware, Because apparently if you go down sort of the literature of looking at benchmarking and performance optimizations. You can actually go from just doing little things up to setting up your entire CICD pipeline with dedicated hardware doing regression testing. Right, But usually we don't start there. Usually we or somewhere else. And that's one of the things that I'm trying to apply.
So usually I ask myself a bunch of questions when I look at the code, right, So, for example, I I go and look for, well, what could be the CP and memory characteristics? What could that be for the specific line that I'm looking at that I know is on the
hot path? And then I usually start thinking about, so, are there any sort of low hanging fruits that I can sort of apply to this, maybe do some more efficient string splitting options and stuff like that that I know from reading the performance blog posts, and I can apply there and that is and I'm presumed that's Stephen Taub's post, Yes, of course, yeah, that the book the Book of the Taub right y, yeah, exactly. And then most of the time, right, so there are a few tricks
you can apply. So, for example, if you're allocating a byttery, right, and you know that you you actually just need it for every iteration. What you can do is you can sort of move that away from the hot path allocated once and clear it and then you're no longer having allocations that you're executing like on the hot path, and then the garbage collection doesn't have to clean up a lot of things, and there things get get get better as well. But then what how do you measure that, Daniel, Like,
how do you know it got? Like, that's a tricky one that the impact of garbage collection, Like, how would I measure that I reduce the amount of garbage collecting? Yes, that's that's a that's a very good question. So usually what I do is before I so I call this performance loop. So, for lack of a better term, I call the performance loop. So what I usually start with is when I have ip I hypothesis
about a piece of code that needs to that it creates uh garbage. Then I what I do is I write a test harness, and essentially what that harness sort of does. It sort of takes whatever I'm looking at, takes it into specific context of my suspicions, and then it executes it in that specific scenarios. And then I attached profilers to that piece of code, And what I usually do is I take at least a memory snapshot and also create
a CPO snapshot. And of course, if you have an io bond system, like you have a database in place, or you have HDP called stuff like that, you also want to look at your you want to do io based profiling as an example, right, because usually when you look at your iosystem like the database, that's you can basically achieve orders of magnitude of performance improvement by tweaking your sequel querits right before you will start thinking about memory allocations
and stuff like that. But assuming you have sort of removed that part, then essentially those sort of profiler snapshots give you an indication of where could you voteocus on And here comes the next problem when you attach your profiler is that you might see lots and lots of allocations from different subsystems and component on that
specific cultree and where should you even start? So I usually try to sort of apply a combination of I call it the one percent improvement philosophy vverasus deliberate contextual based optimizations that I want that I want to do so because for example, I believe that if you do enough little performance optimizations over time, that's the one percent improvement sort of philosophy, then eventually they will end up making a big impact. And we can see that Microsoft applies it as well to
do dot net run time. Right, They're doing lots and lots and lots and lots of small changes all over the place, and the compounding effect of these changes they essentially are massive when you look at them in sort of the greater scheme of things, right, And what profiling tools are you using here? Is this just like the built in profiler. Okay, you made a joke about me. You're running on Linux, right, so so well, so I'm a big fan of the sort of chet Brains tools I've been using
for a year alone. So I'm using usually dot trace and dot memory sort of those two tools, and Writer also has some built in analysis, so for example, they do when you execute your tests or you execute your solution. They also do some dynamic program analysis where they show you sort of the
allocations that your stuff had or the the CP that it wasted. So I tried to use those tools, but primarily dot trace and dot memory to get to get an overview of what is actually going going on, right, And I mean, yeah, there are built in profilers, but if you're willing to pay for one, there are better ones. Yeah, I mean, to be fair, a visual Studio is great, right, It's like visual Studio has depending on the license, I'm not entirely familiar with the licensing terms
there, but it has great tooling built in. Or if you are sort of very advanced and mostly on Windows, you can also use perfew, right. So perfew is a very powerful tool that you can use, although I struggle with it a bit, I must say every time it's like using WINDPG. Every time I use these tools, I have to sort of get some cheat sheets onto my machine in order to remember the complex commands. And that's one of the benefits those too. There are tools you need to learn,
and I mean I suspect you use them more than most people. And if you can't keep them in your head, then nobody can. I mean, I've done a lot of performance tuning over the years, and people already surprised. It's like why are are you reading the docs? Like don't you know this? It's like, listen, there's a lot of knobs on these things, and if you don't go through the steps you can waste a lot of
time, yes, and actually wasting a lot of time. That's actually a very good, good sort of comment that you made there, because I think even if even if you are more familiar with performance optimizations and benchmarking, it's like and profiling at the end of the day, that's not my day job. My day job is building robust and reliable messaging frameworks and middlewares and the
platform at particular, and not doing performance optimizations all day long. That's not my job description, right, And I guess that many people that are also listening to that podcast have the same thing, right, They would like to dive into profiling and benchmarking performance optimizations, but they only have a limited budget
in order to spend on those types of things. And that's why I always recommend start with a test harness, reproduce the scenario, attach your profiler, and then use your domain knowledge of the things that you're working to basically sift through the noise of allocations and CPU and on the call stack and then figure out Okay, probably here is where we can make the biggest impact on sort of reducing the numbers of CPU cycle spent or reducing the numbers of garbage allocations
that are happening there. But sometimes, like I said before, it's also where you think you have the most knowledge in and then applying sort of the one percent improvement over time in order to sort of make things better and better and not trying to gold plate so everything out of an existing code path. But I guess we now have sort of touched a little bit on the how
would I even know where to get started? Right? We talked about profiling, We talked about doing CPO memory at least right always get sort of two views on the code base. But the next thing is then I mean improvements. Of course, that goes more into the territory of knowing your stack,
knowing your language, knowing the libraries that you work with. Like Carl said, also looking for has the library and you release that we can pull in or maybe reach out to the maintainers and say, hey, by the way, we did the profiling snapshot and we found out that this library allocates that
much amount of memory. And guess what when you reach out with a profiler snapshot to third party tooling providers, they are like super happy because you're then in the one percent sort of customers and then it's like, hey, now we have data from the customer that we can see what's actually going on. You've now described a workload in a meaningful way to them correct correct, right,
And so for example, I myself have done that as well. At some point I st over sort of memory inefficiencies in the ash service bus s d K, and then I saw I had a hunch. I wrote the test harness attached profiler was able to show that when you access the bodies of a service bus message that allocates unnecessary memories every time you essentially access the body. I was able to sort of show the profiler snapshots show the memories.
And then I, even because I was lucky, I already knew the library a little bit, I guess, and I was able to contribute a fix for that memory allocation problem to the Azure service bus. Yeah that's a good way to get an email from Clements. Yeah, yeah, I've I've I've had comments from from Clements originally on on some of my pull requests as well. And then and there in lies the point, like you get into it.
And of course the kicker, of course is at its open source, so you could just contribute a fix yes, yes, yeah, but I guess, I mean, nobody expects you to write but at least having a memory profile or a CPU profile, and like you said, Richard, showing what's going on in production, is it. Yeah, I just think it's always challenging to get down to the brass tax like that. To me, most of my profiling experience has been trying to optimize an e commerce site where
it's like we're just you know, we're running. We're now looking at a buy of more servers because the site's so busy, like an optimization can mean a lot of money and the prob I was a nance guy at the time,
and that was the tool that showed me. I mean, this may not have been it was always that balance between this is a very complicated method and so it's consuming a lot of resources and it's a very simple method, but it's called hundreds of thousands of times, and so the fact that the tool would help sort out that weight of often called, and so were thought minute optimizations will make big differences versus rarely called but complex enough that you will
get some return on that, you know, didn't I'd never worried about optimizing in mind calls because it just didn't call that often. But all of that mainstream shopping cart recommendation engine, you know, custom render pieces, ad pieces like those, were all the things where it's like these get called a lot even though they don't look that big, and just playing with string compilation like
those kinds of things made a huge difference in the end. But the challenge I think for a lot of folks is they just want to get into the code and this idea of you snap the harness on first and get a baseline set of profiles in place, and then as you said, the magic word, the hypothesis says, if we do an optimization here, it'll make a difference. Now you go tinker, then run the benchmarks again and it's they
did we make a difference? And King and if you didn't revert. Yes, And because there's no performance code i've ever written them was easier to read than the original. Yes, ever, absolutely true, And I think what you said is super crucial. That is sort of the performance look that I
that I apply. Is so when you have the harness and then usually that reproduces this scenario right, and then like you said, you do the improvements that might be several iterations of ideas that you tinker around with, and then you might execute several benchmarks to sort of look at those sort of optimizations that you're doing. Maybe it's even several micro benchmarks sort of measuring sort of little improvements in that call stack that you came up with during that tinkering phase.
And then at the end what you do is you bring it back into that harness, right, and then you look at sort of the end to end profile again where you look at again the CPU and memory profile at least actually to actually see the before and after right, and then you see on your graphs, oh, we spend six hundred and fifty megabytes of memory on that
specific scenario before. Now we're at six hundred. Now you know that you actually have gained something, and you also have the numbers that from the benchmarks that you run against each individual part of the callstack that you try to optimize. I have one question to you, Richard. You said you use the
NDS profiler. Do you also happen to sort of because I had a period where I used several tools because I had sort of I know it when I see it type of investigations, So I used several tools that have had different sort of dashboards and overviews that sometimes gave me sort of a slightly different view based on the preferences of the tooling against the test harness, and then I was like, oh, there it is. Yeah, I mean different problems
in different spaces. And granted a lot of my experiences are from a while ago where there wasn't as many tools as there are today, but yeah, definitely there's a difference between tweaking a piece of code that you know sits in a call stack for a web page and understanding a sort of end to end run where it's like, oh, the real problem here is that there's a repeated call to a database enough that it's doing a reauthenticate in the middle,
or it's forcing a recompile of a stored procedure. By the way, on the day you find one of those from a method call and you got all the way down to but we call it this many times and so forces is recompiled and that creates this overhead like that's a very good day because those are hard to find, like just a tough place get too, but you know
your point's well taken. There are Each tool provides its own viewed to that and we ended up I think it was a dying trace where we were only able to see all this is a multiple database interaction problem before we really saw the behavior correctly. And guys, I want to pause for just a few moments for these very important messages, and we're back. It's dot net rocks. I'm Carl Franklin, that's Richard Campbell, hey, and that's Daniel Marbach.
We're talking about performance, squeezing performance out of our applications. And Daniel, right before the break, you were going to make a point about memory allocations. Memory allocations. Yeah, So what I wanted to say is I feel like I need to sort of clarify one thing because I talked a lot
about sort of memory allocations. I also sort of highlighted a little bit the CPU stuff, right, But people might get sort of the message that all that matters is memory allocations, and I definitely don't want to say that way because I just feel like for me, I've always sort of started looking at memory allocations because I've seen that these are the areas in the applications that I worked with and the systems that I worked with where I can make the sort
of the biggest impact to reduce the GC overhead without sort of going into sort of the algorithmic complexity and stuff like that that sometimes comes with tweaking algorithms where
CPU cycles are spent. And I remember, I don't know the exact quote, but David Fowler once sort of tweeted or is it still called tweet, I don't know, but he shouted into the interwebs that essentially, apparently memory stream two array and other sort of two array are still the biggest source of memory allocations in dot net systems out there, which kind of shows how important sort of thinking about memory allocations actually and certainly in this case of scale that
you know, when we're dealing with lots of federations. Again, I come from the e commerce space. The other thing I ran into was we typically had to build our load tests to run for longer because you needed to get into multi generational memory to actually understand behavior in production. That lighting up a load test that ran for ten minutes and wrapped up didn't give you the same results as what was actually happening with your server, which was two days in.
Because the way that memory gets allocated over multiple generations became a huge part of the problem not that we had to wait two days, but often we had to go a couple of hours before you actually get that Gen two, Gen three reshuffling a memory enough to say this is fully fragmented and restacked memory a few times, and that's when those orphan long duration objects created problems for
us. It was like, there's this chunk of memory in the middle of the pool and it's been there for two hours, and what the hell is that? Screwing up every GZ But you only found that from these longer runs, Daniel, is there anywhere that replacing a task with a value task will be a problem. I mean, that's a common way to reduce gen zero allocations is to use value tasks. Value task is definitely an interesting, sort
of a newer ish addition to dot net. I know that this is a little bit of a controversial topic because some people are like, yeah, you should be using value task everywhere. I'm more sort of in the camp of using it where it was designed for, which is for io bound paths where you essentially have the maturity of the calls or sort of getting a cached value and then only like I don't know, out of ten calls maybe one or
two are actually doing the actually operation. That's where I feel like value task is sort of a new sort of approach to sort of make sure that you
don't have that many allocations anymore. But to be honest, I think in most systems, when you have to sort of do that type of optimizations, then you're already super far because I bet many sort of applications systems that they're running on top of maybe ASPO, net Core or some others, they have other problems like memory stream two array, unnecessary bitary allocations, stringifying stuff that
doesn't need to be stringified, all sorts of that stuff. Instead of really thinking about switching from task to value task, how about switching from web API to gRPC. It's a good one. Yeah, I mean even for example, switching from newtons of chasing to system text chasing right or using source generated I'm sorry, which ones faster? Are you going to touch that? No? Let's not go there, let's not well, yeah, I mean you
open the door, you might as well walk through it. I would much rather to talk about a little Swiss cheese or something like that than talking about that. Well, I mean, we don't want the listeners to get the impression that just by switching from Newton's Soft to system text Chason, you're going
to find a performance improvement. Did you really mean that? So we actually have seen when we switched from Newton's Jason across the board quite hefty improvements, especially when combined the system text Chasin's US combined with source generated approaches, where you also sort of are more AOT friendly. As an example, you have far faster startup times on ash of Functions or aw ISLAMDA. So there is definitely merits to to to that. But I don't wanna I don't want to
say Newton's have Chason is bad. I think it has its place. Well, the same guy's working on system texts Chason. I mean it's James Newton King, yeah exactly, and g RPC web right right exactly. Yeah, being part of that new version of dot net and integrated with that team. Like there's a lot of performance people there. You just have the resources. Yeah, Like if the eye of sore On that is Stephen Tabb is paying a chance in your code, your coach is going to be faster. Yeah.
Yeah, that guy's amazing. But again, I mean, if if you if you have your harness and you attach it and do some profiling, and you find out that the civilization subsystem is actually the problem that makes everything so much slower, then that change makes sense. But I guess there are also other areas of improvement that you can sort of leverage. But I would like to, if you don't mind, I would like to switch a little bit into sort of the benchmarking stuff if we still have some sure for you.
Because Richard and I we talked about this as well in recently in Warsaw and in Importal about benchmarking stuff, and I think that was a point where I said, all right, we need to make a show about this. Yeah. So one of the things that I found really interesting is when the first time I sort of got into contact with benchmarking was I read lots of blog posts about benchmark net and sort of I was looking at sort of these
benchmarks out there's like ads, it's easy. It's like a unit test, right, It's like I've written plenty of unit tests with x units, god forbid MS test or whatever. Right, So it's like I know this, it's it's not going to be difficult. But I was quite surprised that essentially you required to have a different understanding in order to have a good benchmark. And one is a unit test has two days, right, it's either red
or it's green, so fast or failed. Right. But a benchmark is something really different because what a benchmark does is essentially especially when you use a benchmark on net by the way, excellent tool, shout out to all the people that have been involved there, it's it's like you are executing a given scenario under hundreds and thousands of iterations, right, And so what it means
is there is no pass or failed. You basically get sort of standard deviations, you get g C, you measure the GC involvement and stuff like that. So that's what the benchmark is, right. But it doesn't just start with understanding what the benchmark is. It also goes to what how should I even put my code under benchmark harness? Right? And that turned out to
me the really tricky part because I was only reading about this. Oh, here is a before and after comparison between string concatenation, a string builder and the value string builder, which is the fastest, right, super easyst scenario. But when you're actually taking your production system's code that you had under a test harness, and you sort of filtered in and say, okay, there is a bunch of things that we need to improve, and then you want
to measure it before and after. How do you take this code which is probably not just a public method on a static class that you can take and
then put into a benchmark. How do you actually take all the sort of make sure that you can measure what's going on, remove all the side effects that you don't want to have in your code, so that you have reliable sort of benchmarking results, and that you can compare the before and enough to And that was the thing that I struggled tremendously with it, and I found my way sort of to do it, especially sort of when I was still
sort of growing on the before becoming performance aware. And that was essentially I went with a very simple, simple approach. I essentially took sort of the components that were sort of on that code path. I usually copy pasted the code into a sort of a dedicated source repository, stripped away all the unnecessary stuff. For example, when you when I had an IOC container in place, I removed it and I added anew of the thing that I wanted to
new up. Or if I had IO bond stuff and I didn't want to measure the IE bound stuff, I removed it with a task complete the task stuff like that, right, and then I had sort of a dedicated code base that was in a specific state, in a controllable state where I knew all the noise that is that I don't want to sort of measure is gone, and now I can focus on that specific sort of benchmarking scenario that I'm
looking at. But it is the unit testyle test. Effectively, it was like just bench this piece so that I know I've gotten results from that. I'd also say, as a corollary to this, that you do get decreases and performance with later versions because you're doing more and so often it's like, you know, the whole conversation with the PM, because I've gotten a situation
where we literally had benchmarks as part of the CICD pipeline. Now they're coming back and saying, hey, the new version is slower than the old version.
It's like the old version didn't do all these things you asked for, Like this is the overhead for the feature you asked and that was getting into SLA rules and things where it's like the customer expects us to deliver this in x many second fractions of a second, it's like, well, we're getting closer to the limit because the customer asked to do more update the SLA.
It's interesting that you mentioned that because I think one of the sort of benefits of the approach that just described is that you can easily get start with not even thinking about what is how can we actually capture regression or where should we execute those tests? What is a reliable cic the environment in order to have sort of measurable and statistically relevant results from this environment, right, But what
you're essentially talking about is sort of more regression testing as well. And there's actually there's actually a lot of great guidance a little bit hidden in the dot net Performance Repository. I think it was also driven by Adam Sidnik and some other people from his team. I actually talked to him at a little bit over the course of I was preparing for a talk about this specific topic.
I talked to him a lot about it, and they have a tool that essentially allows you to when you use the benchmark dot net, what you can do is you can actually execute benchmark dot net against the specific version of a benchmark and the specific version of the code, you can store the artifacts, and then you can execute it again against the changed version, store the artifacts, and then you can use the compared to that they have to sort of
sort of create a diff between the before and after version, and then you can define a threshold that sort of determines when it was an unacceptable sort of regression, and then for example, you can fail your CICD environment but kind
of performance because of performance, right, but CICD. It's also interesting because andre Akinschin wrote an excellent blog post about this topic because he did as part of sort of investigations, he looked, for example, at git ub action runners and the result there is that you essentially cannot use git ub action runners
to actually do regression testing for performance because they're just so unreliable. So basically you basically have up to three times different sort of three times different execute memory and CPO difference between builds. Right, So it's it's insane. So wow, that's crazy. Yeah, yeah, it's it's quite fascinating. Right.
So that means you just means if you're going to benchmark issue this way, you have to control your CICD pipeline record, right, and you have to have dedicated hardware that you sort of put somewhere or rent basically bare metal hardware where you and have run their infrastructure that sort of allows you to sort of
put those tests to that reliable hardware. And you can't use sure. I mean you're saying the clouds not an option here, Well it is definitely it is an option, right, Yeah, it's just you're gonna you can't use the built in infrastructure for this. You have to set it. You have
to write your own YAML YAMO you have done this. Like one of the one of the issues we were dealing with is we were up to I don't know, three or four thousand different web tests we wanted to do, and they took a day and we needed under fifteen minutes, and so we would light up twenty instances of the site in the cloud and parallelize all the tests. My goal was always by the time you got back to your desk from coffee, the test results were back and you had failed. Yeah, right,
like that that was because instead it's still in your head. Like what we realized was that every minute that goes by after they've pushed the codes leaking from their mind. And if it's and if it's a day, it could be anybody, like they're going to have to start over picking it back up again. But if we could get it to them in under an hour, like fifteen minutes was the magic number, they knew exactly what Oh, I know what that error is, and off they'd go again, like it just
saves so much remediation. Yeah, so I jacked up the test bill because it saved the dev bill. Amazing. Yeah. So, and again I think it's it's all. When we talk about ESHA DevOps run or get up action runs, usually there is a shared sort of infrastructure that you have there. And yeah, so you just it's not good for benchmarket. You don't know what performance you're going to get, no repeatable results, which oddly are necessary. But if you're talking past fail, that's fine. You don't care
if it ran twice as long, half as long. It's just pass fail. For functionality, that's just fine. Correctly, no big yeah. But
again my message here is I want to hammer this home. I think becoming performance aware, I think it's a journey and you don't need to basically end up where we just oh, well, you will end up eventually there where we just talked about with having potentially your own hardware if you need to do regressive testing, but already having your harness in place, understanding the profilers so that you can sum in where you should actually make those sort of performance improvements.
Then using a tool like benchmark dot net, which saves you a lot of headache because it's sort of sort of is already designed to mitigate the most all the stuff you'd end up writing for yourself any direct So just use it and a bunch of smart people are working on correct and it mitigates, and then you can start there, maybe copy paste your code at the beginning, isolate the things that you want to do, get started there, and then
at the later point in time, where when your company is already sort of more performance aware, you can start slowly introducing sort of more sort of mature ways of actually doing performance testing and regression testing all the way along. Yeah, but you're you know, what you're applying there is like you're pretty far down the path at that point too. Yes, to me. The big
thing here is when does performance creep into the requirements? Because a lot of folks you know, early days of projects, it's just not even on the radar, right, but you know, to actually file performance problems as a bug to get them on the sprint, right, to be part of the conversation at all, Like that's already that's arguably the starting point of any of that path is that it's bubbled up to the point where business cares about it. You know, the line I used to do when I did these talks
was performance is like air. You only care about it when you don't have it. Yeah. I'm a big believer in having non functional requirements in the design or and the architecture sort of built in and having explicit discussions about non functional requirements. It's also prioritize them with your business stakeholders in order to make the right trade offs. Well, and you know the sneaky part about that is let let them tell you it isn't important, so later when they decide
it is important. But you said, you know, because again, performance is one of those things where nobody cares about it until they do. Yes, yeah, right, if I suck the air out of the room, you're suddenly really interested in air. Absolutely, And it's the same thing. It's like you never thought about. You know, you can debate all day a render time of two seconds versus four seconds. I'm sorry, I'm so
web centric on this stuff, and that's not a big deal. Everybody knows that thirty seconds is bad, right, and so that's sort of these kinds of thresholds, and it's hard to talk about that in out of context. You kind of have to make a slow page for everybody to start getting hey, slow page bad, set requirements for minimum performance and figure out where they
are. Any good news is, of course there's lots of written documentation on its like you don't have to invest And one of the things that I also really appreciate by doing these types of investigations is and small improvements is you learn a ton about the cold path in question, and that gives you a lot of insights into a potential redesign in the future, right, because so many people are just throwing out there, just rewrite this, right, but they
have make it, make it faster. You're not going to make it better if you don't know what they are like this, like, yes, I also find that most folks who spend time in the tuning part understand the x you should have, like the behavior of software in a deeper level than a lot of folks that wrote it in the first place, because often you're just trying to get to the deliverable. Does the future do the future requirements that
were there? I know we only have a few minutes left, but at one point in your investigation, do you consider re architecture, which is obviously the most expensive and risky thing way to improve performance. But you know, I mean, as you're going through a project and looking or a tool or something and looking at every little thing that you can squeeze out, and something jumps out at you, Oh, well, you know this should be refactored
or maybe even completely re architected. How often does that happen? It's a difficult question to answer generically, but I can give you a complete example. So I wrote a lot. It contributes a lot to the ASHER service bust on that st K, and essentially I think it was down sort of twenty pool requests on the sort of the path where the sort of the ASH service bus sort of takes you get the byte the race from ASH service bus and
handed over to ASH of functions or to your code that is running. Where I did lots of lots of tiny improvements until I actually understood sort of how the body management of the byte payloads actually really really work. And then only then I came up with a better idea how to sort of manage that that body work from different aspects, and that then led to sort of even more orders of magnitude of improvement how the body is sort of managed and less allocations,
more efficient in CPU cycles. But it was like I think, Okay, I contributed in my free time, whatever that means these days when you were constantly online, right, But I contributed in my free time, I guess, over a year to this code base, until together with the team, we realized, oh, there are there are actually things we can sort of really refactor and make things even faster. So I get I'm actually I have the tendency to go a very long time on a specific code path before
I even reconsider re architecting or redesigning. Of course, small impress can also sort of sometimes mean you're not doing up something. You're making something a singleton that previously wasn't a singles or something like that, right, and if it's still not performant enough, that's when you think about re architecture, because what's also great is right when it's running in production, it's making money right,
and it gives you insights about the good about what it's doing. And when you're architectu and redesigning for that period of time, you have no validation whether the stuff that you're changing towards will work. And with the small improvements you have constant feedback loops. And I think I feel that's super super important. So what's next for you? Man? What's in your inbox? Well? Christmas time of course, drinking way too much graft beer probably over Christmas time,
a delicious stout or something like that. What is this? But but next year I will be I will be in dot Net Day Romania at eating US conference. I will be delivering a workshop about reliable messaging in ASHER, sort of deep diving into ash service, bus storage, ques, event hups
and event grets. That's going to be really interesting. And what else I don't know yet from a conference perspective, but definitely sort of increase a little bit more of my contributions to open source stuff because I still have a few to contribute to a few open source libraries. Yeah, it sounds good, well, Daniel, thanks for spending this hour with us. It's been great, Thank you all right, and we'll see you next time on dot net
rock. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot com. Visit our website at d O T N E t R O c k S dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September two thousand and two, and make sure you check out our sponsors.
They keep us in business. Now go write some code, See you next time in the band. Sadly seen a summer time that means hard than my Texas in Line Revell