How'd you like to listen to dot net rocks with no ads? Easy? Become a patron for just five dollars a month you get access to a private RSS feed where all the shows have no ads. Twenty dollars a month will get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot NetRocks dot com. Hey guess what, it's dot net rocks. I'm Carl Franklin at Amgard Campbell and we're representing four time zones today. Nice. We have two guests. Everybody in their own time zone.
We're in their little corner of the world and their little corner of the world. However, electrons do not observe time zones, so we can all be together on this call. I think they observe it a little bit, maybe, you know, maybe a little bit. You know, Apple has some software. There's an old reference for you story. That was a great story call. Apple's got some software coming up, all right, speed light, hard to beat. Yeah, it's kind of the law Buffalo funny. How
that works? Well, Uh, let's get right into it with better no framework awesome? Alright, Well, I found this series on Microsoft Learned. Oh that is really cool. It's and you know what's great about this is that it's a it's fundamental stuff, leveraging branches with GitHub and Visual Studio and intermediate series. It doesn't talk over your head and it doesn't talk down to you. But you know, there are those guys on the team that are afraid of Gethub and afraid of get you know, I used to be one
of them. Sure, it's so heavily integrated into studio now, like, yeah, may just make your life pretty easy. Yeah, so this, you know, this is required learning. And so even if you know this stuff and you think you know it, it's good to brush up on it, especially if you're in charge of, you know, teaching other people on your team all about it. But anyway, I put a link to it.
It's since this is episode eighteen eighty seven. You can go to eighteen eighty seven dot pop, dot m E and then I'll get you there. It's had what twenty five hundred views since January fifteenth. Yeah, it's just it was just put up a little while ago. Yeah, it's very good. It's cool and a lot of thumbs up and I watched it and it's
good. No learning love it. Who's talking to us today, Richard, I grabbed a calm on top of show eighteen sixty nine, which is from last fall October twenty twenty three when we were at n DC Porto an on stage interview with one Charity Majors from company called the honey Comb that we put a terrible pressure on Martin or anything, but she was awesome. I don't know if Martin is gonna have and and see we're gonna have as many F
bombs however, No, but that's not about it either. And just to reinforce the point, and g Houseworth had this comment which he said, I never missed an episode of dot Net Ross because they absolutely rock, and this episode with Charity Majors was none, if not the best. Yeah, no pressure, the insight and energy was awesome, and we did have the distinct advantage of being on stage with her. Two you're at NDCA Porto at the
time. But you know, we're talking about observability more and more, and I'm really looking forward to getting into this one because we've got a couple of different viewpoints coming at it. But if you want to get started on this subject, that show we did with Charity no fool and it was great. It was great. G thanks so much for your comment and a copy of music Code by. It's on its way to unit. If you'd like a copy of music Code buy. I write a comment on the website at dot
at Rocks dot com or on the Facebook to publish every show there. And if you comment there and I reading on the show, was that your copy of music go by? And uh, you can follow us on Twitter if you want. But the cool kids are at masadon and you know, I put all of my social media links at Carl Franklin dot com cool so we don't have to just rattle them off. I got a bunch of them up
there. I think I've got a Richard dot Campbell dot me, you know, one of the about me pages that has to catch that stuff up there too. If I actually looked at it recently, I don't know. I think I did it when we when we were doing the mass account thing. Yeah, but yeah, you know it is. But you're at rich Campbell on Twitter and what's your masdon and pretty and rich Campbell at masadon dot social you know, yeah, yeah, oh no, it's our Campbell dot me.
There you go. R Campbell dot me. All right, got them all. Well I'm going to just start talking about that. Carl Franklin dot com. Just go there. You want to do that now, Okay? Like yeah, okay, shorter, Yeah it is shorter, and uh you know. Then then everybody has a lot more to read that they didn't want to so too long, don't read? Yeah, go away. So let's
bring on our guests, Steve Gordon and Martin Thwaits. Steve is a plural site author, Microsoft MVP and engineer at Elastic maintaining their dot net libraries. Steve enjoys sharing his knowledge by presenting talks at user groups and conferences, writing on his blog at Stevejgordon dot co dot uk, and creating videos. You can find him on most social media platforms under the username at Steve J. Gordon. Martin Thwaits is an observability evangelist, dot net developer, Microsoft MVP.
He works for Honeycomb dot Io, an observability vendor. As a developer advocate, he travels the world spreading the good word about open telemetry and observability engineering. He'll talk to anyone about observability, even the likes of us Jamos that's not very nice. So welcome guys. Hey, thanks having us, Yeah, thanks for having me. So this culminated at in the fish Powl at NDC London, right, Yeah, we were all hanging out, you know. I was taking it easy because a co host wasn't there, so
so at some point somebody just declared it Richard's office. There was a few other foot I think Miss Manners came in and recorded some shows like there's a few folks came and went and used it. I did some run as as well. I was there. Yeah, you could. Mostly we sat around. I do believe a couple of bottles of whiskey may have just disappeared into that room and never we're seen again, as they do, as they do. Well, Steve was in the UK and it's afternoon there or evening,
right, so you've got your whiskey there, Steve. That's right. Yeah, I'm I'm declaring it post six pm. Fine time to drink, right, I am going tonight to Texas Day Brazil with Robert Ramsey. Nice Richard, Yeah, and we will I'm sure have plenty of meat and whiskey. So anyway, what do you guys think of following up with on Charity's Talk here. Are you scared or are you up to the challenge? Martin said that very well, don't you get it? Very well? Answered that question.
I mean, F bombs is kind of our company policy. No kidding, you know, I'm not saying he's part of the interview process, but I'm not not saying that either. That is so cool. I mean, you know, let's think about it for a second. I mean, who are you going to offend? Right? You go to any movie, even if it's PG, and there's F bombs everywhere, right everywhere. But I mean we do get some email from folks on set when they show up. We do try to bleep them too, Yes we do. But I appreciate
the sentiment either way. It's like there's so much of observer your ability that falls into the what the heck is going on in this app or in this case, what the f is going on? And it's you know, trying to be Canadian here. Yeah, we have a we have a you've heard of MTTR the Meantime to Recovery, and so we have a thing called mtt WTF, which when you're debugging systems, it's how long does it take you to go what there was that No, that thing doesn't call that thing,
that doesn't happen. That's not right at all. So what I haven't used this Honeycomb, But what I remember distinctly about that conversation was it's architecturally different from a lot of other, you know, well known telemetry kind of things that you hook on or add on side cars and to get any kind of insight and observability into what's going on in your application places where you just can't
put a break point. Yeah, I mean, I think this is the whole tracing debate versus blogs and APM type stuff, where tracing is the thing that's different. It's the evolution, if you like, of that idea of just log everything, throw it in a big bucket, try and run some log stash things on the back of it, or fluent stuff to build it all and try and get a picture. It's really the tracing stuff that is the sort of next evolution that isn't is not new, and that's the thing
that really gets me. I was going to say, we've had tracing for a long time, but it's new about honeycomb tracing. So well, Honeycomber is just a back end, we just take spans and allow you to query them in weird and wonderful ways. In rapid fire, you know the idea of an outage going on and you're waiting five minutes for telemetry data to get through for you to ask a question, to say, did I fix it? Is it now fixed? That idea of just rapid fire, ask questions,
Ask questions, because that's really what you want to do. You're on that conversation with your production system that's like, you know what's happening? Who hurt you? Yeah? And that's really what we're trying to show me where the bad man? Yeah, it couldn't help, but notice your pain. But that's what we try to do is really try to allow people to just
ask weird and wonderful questions. The example we users is, you know what happens when the French user is currently in Norway using the French language pack in the Norway region on iOS fourteen point one, and that's the era that's happening. It's only on those users, because that's that's where we are with systems now. There's no longer this system that you use inside of your organization that only your users are only your organizational users. Everything's global. You know,
people are working from home, working from anywhere. So even if you've got just an internal system, these people are from everywhere now, so you know, we're starting to get into a world where things have changed. It's not about I've got six month release cycle. I can wait months. I can do a QA cycle. And my my whole thing at the moment is if we don't change the tools, and I'm not talking about the vendor back ends.
I'm talking about like open telemetry and tracing and profiles that are coming up and various other signals that are coming up, we don't give people better tools, but we're saying you need to do things better than you did before. Like it's not it's not okay, yeah, yeah, the the yeah, the reliability strategy of be more careful next time. Yeah, don't do that. Yeah, don't touch that. It's yeah. The real point here is the tools should lead us to the path of success. They should resist.
We resist as are making mistakes, and so we get to still be human and have reliable software. What a concept. Yeah, And I mean that was I think from my experience that was one of the problems with just sort of relying on logging as your observability. Logging in a few metrics was was us doing observability. But logging can be quite good if you spend a heck of a lot of time and I use heck there heck of a lot of time getting the logging in that you need. It tends to be quite retrospective.
You figure out there's a problem in a particular code path, you put a new log in, you'd ship to production, You wait for someone else to hit the error, and it logs something else that gives you another clue of where you want to log next, right, And so it's it's very
retrospective and looking back. Whereas with traces, if you design them well and you put tracing around your key points in your application code, you get the full, you know, waterfall effect of what's happened, and you've added hopefully decent attributes on there so that you can, as Martin said, go back and answer any weird and wonderful questions and combinations of events with particular attributes on
them. Awesome, And Steve, I got to ask you, like I normally resist two guest shows because you get up with a lot of voices and I but we're talking of observability, and Martin works for the company. Arguably that makes one of the most popular observeability tools out there. Does Alaska you does Elastic use Honeycomb or is it just you know, you're trying to understand
what your own product does. What's the story there? So we provide an application performance monitoring solution as well, so we compete in a nice friendly way. So we have a different back end. Our back ends built on top of Elastic Search, which is kind of Elastic's main product, and our APM server uses that as the data store, so you can ship your telemetry data to us. So you can use our APM agent to do that, or
today you can also use any open telemetry capable agent as well. So the sort of built in what we call the Vanilla Open Telemetry SDK can be used to ship over the open telemetry protocol directly into Elastic APM server and then we'll stare that for you and then give you the again a different form of you to search over those traces and those logs. I'm just staggered at the idea
that we're talking about Elastic for something other than search. What I know, I thought they made other products, but when don't we ever discussed that? So okay, so this is the elastic has their eight PM product, which is another aspect of them. But you're both guys are in observability in a big way. That's right. Yeah, I think that the really I mean, one of the reasons why I thought it was really good that we're both
here is because different back ends provide you with different query and capabilities. Right. You know, the idea that you just buy one tool off the shelf and everything just works is just not going to work in today's world. You really need crazy you know, you really need to think about what's going to work for your organization. You know, what are you trying to achieve with observability, and then choose the back ends that work. And that's where open
celemetry is really awesome because you can send it to two providers. You can send that same exact data to two separate viders and say, well, this one's really good at metrics. It's really good at showing me some dashboards about pre aggregated data. Awesome, that's a really good tool for that thing. It does infrastructure metrics. There's one that does say costing and takes your metrics and can project costs based on previous metrics. Those are really cool tools.
There's other ones that focus on, say Servilus, there's other ones that focus on you know, everything's just they've got their own niches. You know, Elastic has their own niche. We have our own niche. So open celemetry just makes it easy as your needs change, you can change provider. There's no I'm going to change provider. We'll install the new agent on fifteen hundred machines. You know, well, I'm nobody's going to do that. Or
now it's I'm going to move vender. I'll change three lines of config right now. Speaking of that, I mean you bring up pain right. I mean, let's say I've got an app in production and I've got you know, thousands and maybe one hundreds of thousands of lines of code. How does one just retrofit this kind of tracing into an app like that? Obviously there isn't any kind of wizard where you can just snap your fingers and it's instant insights, right, ten lines of code, ten ten lines of code in
not for every object in every page. And so I mean the thing is with open celemetry, we've got three care signals, we've got tracing, we've got metrics, and we've got locks. Tracing is predominantly back end. It's focused on this idea of we've got a context. So a trace is a context, a thing that's been done. So it really works best in the back end because you've got an inbuilt context of a web request. It starts and it finishes and things happen. It doesn't work well currently in the front
end, so it doesn't work well in the client side. What people call rum or really use monishing, so it doesn't work well in that way. Open celemetry is trying to build a signal for that. So in the backkend it's actually really easy. You just add say ten lines of code to your startup and say I would like to see a span for every HDTP request that I get into my site. I would like to see a span for every
HDDP request I make out of my site. And when you say span, there's several definitions of that word, but in this context, what does it mean. So we're on a dot net podcast, so let's use dot net terms. You can create an activity, so activity and this is where we say we've had this for ages because activity has existed since I think dot Net called two point one I think was when they first started bringing in activity.
It's been there for a while, and activity kind of represents a sort of unit of work, if you like, and an actor you create an activity, you create one for your HDTP inbound request, one for your outbound request, one for your when you hit a database. And these are all in built into the libraries already. Espnut core has It's equal client has it, Myscore client has it, MPGSQL Entity Framework, you name it. All of those have been moved up. Even the Azure client SDKs have now been changed
over to using activity sources and activities to create these spands. So a span being this, I've got a start time, and I've got an end time, and I've got some attributes, and it allows you to build these really rich waterfalls that allow you to see how things transition through your system. But you can do that with about ten lines of code in the startup of your
application. And with a spire, they're actually promoting this idea of these shared projects where all of that setup happens once, so it's not even ten lines of code per application, it's ten lines of code in a common startup, So you can get essentially all this automatic data just by adding ten lines of code and then putting in a little bit config to say where you're going to
send it. You're going to put in the elastic APM end points, you're going to put in the honeycombman pints, you're going to put it into any of the end points, and if you want to go further, you can
you can do zero lines and code as well. So there's also this kind of concept of auto instrumentation where by flicking a few environment variables to configure the profiling hooks that the dot net run time has built into it, if you can point it at this auto instrumenting profil and that can go in and wire up whether it be the open telemetary STK or the elastic APM agent wire out
for you in your application with no code changes at all. Just deploy the environment variables and restart the app, which can be a really good way to get started with. That's amazing actually, Yeah, with basic instrumentation of what's happening. Guys, you're talking about the easy part collecting too much, Like
that's really easy. Yeah, you know, I'm a guy who who spent time in the sequel world where we used to do full traces of every transaction going through a SQL server because it was really great at filling up hard drives a a bit. It's the real question is after you gather all this stuff, and I emphasis on stuff, I can think of several other words that would apply begin with s, Yeah, how do you mash this in the
submission? And you know to the line I would use when working with the team was create actionable items, things we can do to make the system better. That is the rub really, you know, and that I think is what has been tracing's downfall for probably the last sort of five years really because people see tracing as a I have a trace ID and now I can see a trace waterfall for that thing, but I need something else in order to
tell me what trace ID to use. And what that means is you end up with say ninety nine point nine nine nine recurring percent of your data not being used and just sitting on hard drives and costing you money. And that I think has been one of the big downfalls that people have said, well, tracing is not really useful because I need to know what trace it was so I'm just going to use logs because I have to keep them anyway,
And I think that's been one of the big downfalls. But I think there's there's a movement now around tracing, around things like sampling and doing various different mechanisms for how we reduce down the amount of data that we store and only
store the interesting stuff. Yeah, that's assessing what's interesting is not a trivial problem either, right, Yeah, I mean interesting is subjective at best, you know, you know, but the general rule is, you know, you keep every one of your slow traces and keep everyone that has an error in it, because there was at least going to be the interesting ones. But you do that, and then you can start to think about how do I query my trace data, not how do I query my logs and my
metrics, and then get back to tracing. How do I start with that tracing data, because it's really rich. And I think that's where we're seeing this big change now with open telemetry, because the back ends, all the back ends now are starting to look at this trace analysis type stuff, whether it's ours or Elastic or a lot of the open source ones that are normally based on ClickHouse as a data store. They're starting to do this idea of well, you know, how do I get and ask questions of my trace
data? Because it's really rich, it's really wide, it's got loads of attributes, all that interesting stuff, and that really is the rub. Yeah, so always a question of what's actually interesting, because you're always afraid of what you shave off because that might have been the interesting data. But yeah, we've had so many people who come to us with, yeah, I don't really like tracing because you know, every time I wanted to ask a
question, the customer got an error. It was sampled out because or discarded because somebody took one percent of random sample data. And the era happens, you know, once in a million times, so they never get the trace
that they want. And that's where we're doing a lot of evolution right now in the open telemetric collector and some of the stuff that even me and Steve were talking about at NBC London about how we roll up stands and write tell something processes that are more bespoke and you know, this is the sort of conversations we're having now because it's vendor agnostic and so you've got so many more
people. You know, me and Steve, we don't work for the same company, but we're working towards a common goal of reducing the amount of data, making it more actionable, and making it easier for people to implement. So you've got, you know, hundreds of developers all working on that same
code base. And that's the major advantage of open telemetry as a goal, isn't it for the industry that rather than every vendor focusing on their way of doing something and having their own little pieces that they can use in sales and marketing, it's actually what's the right solution across the industry for how this should be done in the best way for the organized relations that need this information to be able to store this stuff in a sensible, cost effective kind of way
to do this level of sort of advanced sampling, sort of intelligent sampling, if you will, on the on the tail of things coming through. And I think we're starting to see that shift with vendors now kind of all starting to adopt finally, I think open telemetry as that solution. Yeah, it's always a question of mentioning this data together into like do you make sense of it? They and what's that you see mostly it's the same stuff over and
over again. It is what you already know. I'm tapping on the book I've never freaking finished and still hope to one day. With the history of dot net, where the seql team had a tool where anytime a unique execution path happened inside its equal server, it would stop. It would hit a breakpoint, so as long as it was executing the normal path or normal paths over time, it would continue running. And so it was one It was
like a tool designed to find the one in one hundred million case. So you're iterating rapidly and it's like initially it's breaking on each unique it's breaking on each unique path, but after you say, okay, that's a fine path, keep going. Then eventually it's like running for days before it's finally like think how about this? You're like, now, well the heck did you get there? And the reasons in the story of the context of dot net
is because that's what made dot net two point zero. Making dot Net run inside a sequel server, made them test dot net in away never been tested before, and they found rare and unusual bugs because they were running into context of sequel server. Here end of the history lesson. I love the idea of like the system just going hold on a minute, can you just get your supervise it this road before boys, let's have a little chat. And speaking of that, my friends, let's take a brief break for this very
important message, and we're back. It's dot net rocks. I'm Richard Campbell, it's Martin Waite's and Steve Gordon and my buddy Carl Hey and talking a little bit about more than one way to get observability here and the fact that
you know, now you're really creating trouble for me. It's not like I was get too much telemetry out of my Honeycomb stack, but now I'm gonna go get elastic eight PM as well, feed the same data to it twice because I just own too many drives, Like let's try and kill them all. Like doing analysis between those data sets could be interesting too. I mean,
we do that law, you know, the Bakoff type stuff. You know, once you've got an open celemetry collector in place, you just sort of send it to five vendors, see which ones do the best fit out what you want to hear. You know, in accounting they call it reconciliation, right, Like you got to add up the numbers more than one way.
I wonder we're not there yet in our industry. Like if I really wanted to be a pain in the butt about a performance contract, the fact that I could ask her to the telemetry to be sent to two different sources and we'll reconcile the performance metrics against it from two different places and see if they actually agree. When they don't, wonder why, Like, yeah, you really press against that contract. Yeah, I haven't been in there be CTO in too long. You've given me new ideas on behalf of myself,
and I am sorry to the industry. Yeah, I mean it's all Yeah. The other side of this is like nowurkicking, we're doing multiple analysis on the same sets of data. Hopefully obviously different tools, like you said, have better insights in different areas. But yeah, I'd love to have a standard set of comparisons. Says oh yeah, no, we reconcile the same way we agree on stuff, so you get more confidence. There's also different
users as well. You know, we've got the you know, just in the sort of reliability space, you've got developers who are using this information You've got srs that are using this information to optimize the infrastructure. You've got platform engineers that are using it for the same things to build tools. You know, that's just three users just in engineering. You know, you start to
think that what else could we use this information for. I know of a company that's running at the moment the idea of taking metrics to you to do cost projections. Right, So take the metrics of your kuminettes poor to how much you might need to scale in the future. Take that metrics data. Don't allow people to quer it and do dashboards, but take it and do
projections. You know, this is now into the thinops world. Well, this is my mid life, right, Like all too often we'd roll out a new feature and it would tip the system over a week later because we weren't looking at the additional resource impacts. And as we you know, as e commerce site started getting really big, we were dark launching features and running
them on one server just to see what the load differences were. So I knew how much hardware to order, which just speaks to how old I am, right, because back then it was like, see, we love that feature, but we literally need thirty more servers to roll it out. Yeah, and then you end up with one server running on a previous version and then you bankrupt the company, right yeah, well yeah, we don't scare
me Martin like dinged. But I have used that line in the boardroom where it's like, I think we're about a week away from bankruptcy unless we turn this thing off, because we're gonna we're gonna crush the whole thing, Like we have to think about it. Yeah, it's uh, it is it it? And it's that whole part about the telemetry part was the it's very
hard to estimate the impact of a feature. Not only you don't really understand the resource consumption of a feature anyway, but you certainly don't know what the user utilization is going to be, like you're just making this stuff up, right, So it is only telemetry that really tells us how much is it being used? What's the actual impact on the systems? At least now we're living in a cloud where you can just flip a knob and the CFO find out about it in about thirty days, right, Like, you not your
problem. You kept the system up. That's my metric. I'm up, Like I'm good. Well, what is this about a bill? And this is the the other flip side of that is what we call the nines don't matter if uses unhappy, it's like, yeah, ninety nine time ninet nine percent O time awesome, twenty second response times though, that's a good yeah, but yeah, this is the game we're playing, right is looking through
all of these things. And really we came into this conversation talking about what's the app up to, like we're not even you know here we're talking about portant business cases, but mostly it's just trying to diagnose weird behavior, like I don't know what's going on, and I have a customer that can totally use this Right now, after upgrading to blazer dot Net eight from six, all sorts of weird things are happening the and we're still using the same services
and stuff, just the newer versions. So it's probably just some versioning thing, but weird unhandled exceptions that are evading the error handlers and closing the whole thing and not giving us an opportunity to find out where they are. Like this this is going to be a game changer for them. So what I should mention at that point is it doesn't work well in Blazer. I'm really sorry Stack or is there something inherent to Blazer. So it's more to do
with the front end side. So if you're using Blazer web assembly, you have a single thread and telemetry should inherently be done in the background. So the Blazer server has some problems with the gRPC connections and various other bits that we're working on it. I'm currently talking to somebody who runs one of the Blazer University things who's trying all this stuff out and trying to to work out
what what that should look like for Blazer. But there's a lot of things that are just they're new, they're not they're not in the tracing paradigm right now. Durable functions is another one that has some really big interesting nuances around how do we propagate a context, So how do we say that this function execution was related to this other function execution, and how do we get that data between the two because it's not as easy as just past some HTTP headers.
So you know, we've got as your service bus has similar nuances. But we're seeing we're seeing right now a lot of open source libraries that are going I need to add tracing because people want this, They want that internal information. They want to know how the cash server is working. They want to know the messaging systems and how long it takes me to post a message to my messaging system, not just how long it's going to take to receive
and process it afterwards. So they're adding it. Rabbit MQ recently in the new version is going to have tracing as a first class citizen in there, and then it's going to have metrics as well. Fusion Cash was another one that I worked on with some people, and we've just seen loads more people going, yeah, tracing, I need the tracing now. So it's just it's really good to see, right. Yeah, we're at that kind of
tipping point of adoption, which is yeah, very encouraging. I think as people get benefit out of the box, when all of these libraries they're using and they're already getting useful traces that tell them a lot of stuff even without them instrumenting their own code, that's the benefit of everyone. I mean, we recently instrumented the client library for the language client for Elastic Search, so that's a separate package you pull in if you want to do you know,
dot net code to talk to elastic search. So we've added some instrumentation down in the transport layer so you can kind of see what's going on there. Added our own useful attribute, so out of the box, if you're using that library and you're using an open telemetry capable you know, collection process, then you immediately get that information and you can you can diagnose what's going on at that level. Yeah, and Aspire, you know, you had David
on talking about Aspire. I think that is one of going to be the real big game changer in a option because one of the big blockers is people like logs, Well, I can see them in my console when I'm running, why would I use traces because I can't see them in my console? But now with Aspire, it's like you've got a nice, convenient little dashboard that's just running there that takes your metrics and your logs and your traces.
And I'm seeing loads of people use that. They look at the logs thing and then look at the traces thing and say, well, the traces are way richer than the logs, So why would I use the logs? And I'm like, I've been saying this for five years. Thank you you're right.
You know. The one thing I came away with while I came up with many things from David's show, but it was this, we're trying to make all the right defaults for you when you're building a cloud native app, and one of them is you use open telemetry, like, you use that level of instrumentation. So you know, it's exactly that you have to fight against it now to not have all that stuff in place. Yeah, it's the pit of success in it. You know, Yeah, it's right.
But you know, it's an interesting point because there's also folks that are like, I don't know why we need this, and it's like, yeah, you know, because it's too many choices. You know, if you're perfect, if you can get everything right every time, you know, if you never need to be more careful because you're always careful, then fine, you don't need it. But I'm just not that careful. It's like unit tests.
Spend several days trying to analyze logs in your head, and you know, correlate everything across any even a good logging system, and you'll you'll soon realize that actually a trace might have saved you a few sort of head against all moments. Yeah, just like unit tests. I don't need unit tests unit tests with people who don't trust that they can write code right for the first time. Yeah, that's true. I know I can't write code fight
right the first time. A certain quote from a certain Billy Harleist comes to mind. Yeah, you might be addicted to code. You might be. Yeah, and you must suck as a code if you need two people writing ten lines of code for every line of code you right. Yeah, I mean I even speaking to an extreme case, but it's like, look, people make mistakes, they get tired, they work too late, they don't
quite understand the problem. Like, all these things are real, and you look at all the work we're doing lately around shortening the internal cycle, the impacts of GitHub co pilot, they're all speaking of the same thing, more correct code and an aspire to me looks like this tool that says, okay, so you're going to be cloud native and you haven't done it for thirty years for some strange reason. So now we're going to give you a tool set, you know, a framework that leads you to that path. It's
a distributed debugge. Yes, you know full the cloud that's you know, I joked in some of my talks over the years about you can't attach a debugger to production. You know, nobody nobody attaches their visual studio instance to production. Well sorry, nobody does it more than once. But you know nobody does that because well, for a start, it blocks all the threads,
and that's you know, probably a bad thing. But also, now we've got you have to attach a debugger to fifteen different services because you've got five different services running three replicas and you don't know which one is going to hit. And you know, we're in this world now where we need that debugging experience. Nothing's changed. You know, we still need to debug our systems, but we need to debug them in an environment where they're replicated and
they're distributed. Some of them are using the event driven systems, and you're like, I can't, I can't correlate these logs across an hour's worth of time window. And yeah, it's just it's hard. And that's where I think people are getting to now that they're starting to add tools and tools and tools on top of the old logging systems that they had to try and meet
the demands of these distributed systems. And then they realize that actually know there's something that's more native that they can use instead, and it requires a paradigm shift. And I think that's where as an industry we're bad at acknowledge it. The telemetry and logging aren't the same thing. They because they're not weirdly enough. But you know, I've been saying for a while now that if you can answer all the questions that you need from the logs that you've got,
you don't need traces. Yes, stay still fine. Yeah, but the question is can you and I don't think you can or you're not asking good questions you Yeah, I mean that's that's the number of rises above none. That's an old scalability problem where I found that. I found an app whor it's like it's a long one person was used as it was fine, but as soon as two did not, it's so good. That was a
long time ago, My goodness. M I mean, I've hit that problem so many times when people use singletons instead of transients, and you know, they still stay in a singleton, and especially in multi tenant systems, and it's like, yeah, when we've got one tenant, it's fine, but as soon as we get two tenants. Everybody's got everybody else's data, and
that's a bad thing, I think. Holy man, Yeah, easy mistakes to make too, right, Like, you realize we don't think about that anymore, that the framework is just doing that for us now most of the time, as long as you follow the rules. Like, we're definitely working on a different tier of problems now as we get you We're used to distributing across multiple machines and now it's you know, in different clouds, with different
platforms. We definitely build way more complicated software. It's no question that we need to change the way you measure them. Uh. The debugging story is separate though, right, Like you just sort of casually mentioned the debugging on top of it. But telemetry is not necessarily for debugging, is it? So it can be for telemetry isn't a debugging tool, but you can use
it for debugging. You know. One of the big things we talk about at the moment is the difference between three pillars of observability and just telemetry signals. So the three telemetry signals being logs, petrics, and traces, they allow you to do lots of things, and they allow you to build dashboards. They allow you to do monitoring build alerts, don't. You don't need one or one or more of these. You can use one, you can
use three. You don't need all of the signals. But when you're starting to do the debugging side, you need to work out which signals you need to do it, and you can correlate them. Sometimes one of the key ones is the user request is taking too long and it turns out it's actually just one pod and that one pods has its CPU spinning out, and you look at infrastructure metrics for those things. So debugging or do you call it
debugging working out what the problem is in production the root cause analysis. Yeah, Debugging to me speaks to I am fixing code often because I'm looking in an executing environment for what's going on within that code. I could see having the telemetry sitting beside me because we're seeing a behavior in production, we're not quite sure what it is, and so now I set up a debugging environment
to try and capture that. And I suppose so I've been playing with this narrative, and let's try this, which is I think tracing is kind of like conditional breakpoints that you can use with a time machine. Now, when I discovered conditional great breakpoints in dotnet, it was like a revelation. When you're running a SPA app on the front end and it's going through this line of code, and you want to know when it hits this line of code.
But when it comes from this end point is when I want to be able to hit it, because it hits sixteen times and I don't want to have to hit play every time, and I've got multiple threads going on.
If you think about that, but being able to do it in production and then say what happened yesterday when this person hit this end point for this particular method or this particular database call that I think is that this revelation of you, now you are doing deep You're you're using that telemetry from production to do debugging, yeah, to construct an executing environment to recreate the problem in production. Like it gives you the cues to say, we need to be in
this state to be able to see this bug. Yeah, to be able to see those constraints. Because you still can't set a break point in production. Well you can, you just won't have a job after it. Story Steve, I stepped down you there yeah, no, I was just yeah,
I'm totally on board with you know, not this point there. And I think it's it's the ability to apply constraints to the data that you have that would be very difficult with just pure logs, where you want to set up this series of criteria around I want to look at this coming from this in this region, you know, deployed on this particular piece of infrastructure, and then see a waterfall visually of Okay, what did that flow through in
the code? Where are those you know, lines on this waterfall wider taking longer? And then you narrow down, Okay, well that's this block of code or this library. That's where I need to go and look to try and find out why that's taking so long that piece of the puzzle with these given sets of criteria applied to it. And that's something I think only only well defined tracing can can really give you. You could, you could sort of get there with logs, but you you would take a lot longer to
find there. I think, yeah, there's always a question of do I understand with any given log set, it's like these six logs, where is the trends this particular transaction across these logs, like we've always had a tough time joining that stuff up and tell them she does a better jar with that, yeah, and then relating that back to infrastructure because you know, we don't run our code in isolation. Infrastructure is a thing that we use.
So yes, it ran slower. But was it running slow because that Kubernetti's node was overloaded? Was it running slow because we just switched slots in service? You know though that infrastructure stuff is just as important. Yeah, it's you know again, I've spent a long time on the ops side of this because why is this slow? And it's like it's your hardware, No, it's your crappy software. Like you go back and forth on that a fair
bit. I have had a situation where we had a hundred based tea hub in the in the network we'd forgotten about and literally were throttling all the network and traffic. So you know, it looked like the database was slow, and it's like, literally it took a long time for the data to come and go from that because this network mistake was there. Like, those things happen, and the trick is to not be mean about it. Like everybody sit on the same side of the table in the boardroom with the projector on
going. What is this data telling us? And that's correlation? Yeah, you know, that's the idea of you know, the ops people are looking at their infrastructure dashboards and you know there's a there's a thing here, and then the developers are looking at their log aggregation dashboard that show them some stuff and they've got some graphs that they can It's like, whoa these two graphs they go up at the same time. Ah, maybe there is something here.
Is there a knob here that we can wiggle, like, you know, just kick it out? Can we change that number for anything? Like when we do this, does it make a difference. So at least you have a sense that, yeah, there's a there's a constraint here and it's related to this value, and if we modify it, we can improve it in some way. Convincing that we have to buy a new hardware is challenging.
And again Cloud took a lot of this away because it costs you nothing to turn that up for a test and then turn it back down again like a buck. It's just you solve a lot of problems just by doing those tests. Yeah, I just run a workshop on Kubernettes for some attendees, and I span up a Cubernettes cluster for every one of the attendees for two days. And it costs me I think four pounds per attendee to run a
three note Kubernettes cluster in a KS for two days. I mean, can you imagine if you were having to do that on premise and ship in all that hardware three three hundreds just for each intending and a rack whirring in the background, but with hal blinky lights, So that'd be a thing. Yeah, yeah, we'd have blinky lights. But you know you can get a get a bunch of r G B L e eds, put them in a box just called the computer right anytime the wire goes back into the cloud.
Might do that for the next workshop. It's like, yeah, you're all running those blinking lights in the corner. That one it's off. That's Carl Carl oh Man. What if we what have we not talked about here? Because there's a lot to doing a good job here. What are the learning ingredients? Like how do people really get You know, it's quick to install, but do you ten ten lines of code? You're up and running but there is some things you need to know. I think start with aspire.
You know, start with a local dashboard. Start by turning some things on locally in your local environment, development environment, and see what some of these traces look like and see whether they're useful before you start thinking about where I'm going to put them, what observability back? In? What scale do I need? Am I going to use open source and host it myself? Am I going to go with a SaaS vender? Am I going to go one in built into my cloud? Use the spire? That's that's what the real
cool thing about it's available as a container. Now that you can just run up locally. You don't need to use the DCP and all the other bits that go with it. You can just run it in a container and just experiment with it. What data do you see? Is it going to help you? You know, the value of the value to work ratio should be really small anyway, in like ten minds of code. Great, but start with something like that locally because you don't need to choose a vendor. I
equate this to you. Remember when they brought an eye logger, And the great thing about eyelogger is it saved six months on every project because the first six months of any project was deciding between n log and love pnette and the arguments. The arguments that people had. I mean, that was the true
flame Wars, you know. But they brought an eye logger, and the problem was that you needed to make that decision really far up from because if you didn't, then you'd have to retrofit it across every all of your code develop everybody this one. They brought an eye logger and they were like, great, yeah, everybody use ilogger. It'll just send it through eyelogger and then you'll choose your vendor. And I think open celemetry is the same thing
is we've got a gimme. Now. It should be in everything, whether you're to wire it up or not. It costs you nothing to have it there, and when you do wire it out, mental help. Yeah, and that's the place to start. Well, And I like this tipping point angle that Steve brought up, which is that and more and more vendors see it that way. So the information you're getting out of open Columbatry just keeps getting better. Yeah, this whole thing works best as more and more vendors
get on board. So that's the library authors, it's the you know, the various different language run times, it's the open Telemetry team. It's people within your organization, engineers within your organization buying into the fact that, okay, we've got this. I mean in dot Net it's super easy because we do have activity built in. It's it's there. If you're using dot net modern dot net, you've got the libraries, you've got the APIs. You
need just to turn this stuff on and start instrumenting code. But as all of these people shift towards that general direction, I think that's where why open telemetry now is really starting to look like a good solution for everyone. What's the app Insights story? Should we just rip it out? And so Application Insights is currently redeveloping all of their SDKs to use activity source and activities. So it's not wow, I'm getting those now saying you're going to have to
change this soon. So it shows that you Microsoft's listening to the open telemetry story. I mean, they had the experimental flags in the as your SDK last year, I think, and I think those are being removed now. So all the AZ your SDK is anything that you use from Cosmos to service bos if Angrid. All of those things will be instrumented with activities which means they're compatible with open celametry to send it and as your monitor. They've got
their own exporter. Now they don't accept OTLP yet. Whether they will, I don't know. I don't work on that team, but they do have a distro so that you can do add as your monitor exporter and then you can just use open telemetry and decide later away're going to send it or just add the one line of code that says add to a your monitor and it will appear in app insites. Everybody's doing it. And to go back to this Blazer thing, do you recall hearing any kind of timeline as to when
things will be more copestic with tracing and everything else that's going on. So I had a long conversation with the other Steve NDC London about exactly that, the more famous Steve. So they're looking at it, they are aware of the issues that are coming up, and they are you know, wanting to solve them. So I would imagine that yes, they will be solved soon and you can do it. It's just it's not as easy as ten lines of code. You know, you've got to go through a few more hoops,
especially with the whole render mode thing. If you're using dynamic render modes, You've got some other problems that have to get solved, first state problems and things. Something I'm currently working on. But I'm hopeful. I'm hopeful for a future where this is the default. You know, this is well, I just add those ten lines of code, or even just add one line of code when we've worked on the SDKs a little bit more, and just add the one thing that's says send my stuff, and you know it's
a what is it services? Don't let me debug my things? You know that that kind of simplicity just becomes the new cargo cult if you like, I don't just cargo cult thing. I well, I do cargo cult thing, but it's the new thing I cago cult. Well, guys, great conversation, Thanks so much for coming on absolutely and what's next. Let's send your inbox Marten. So I'm doing a load of conferences. I'm doing some
meetups around the UK. I'll be at Cubcon, the Kubernettes Cloud Native Conference in match so but yeah, I am flying around the world spreading the good word about this sort of stuff. How about you you? Yeah, conferences coming up, so I'll be in Sweet Sweden next week for Sweet Tug, which will be in the past by the time this this one goes out. And then yeah, day to day we're going to be focused on where Elastic
and the engineers can contribute to open telemetry. Really, I'm really starting to poke around in the open telemetry sd K repays and think about how we can make that experience good for good for everyone. Say, yeah, I think that will keep you busy for a while yet. Well, this is all great news. Thanks again, and I'll catch up with you at some conference somewhere. I'm sure, sure, all right, We'll see you next time
on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot com. Visit our website at d O T N E t R O c k S dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives. Going back to show number one recorded in September two thousand and two. And make sure you check
out our sponsors. They keep us in business. Now go write some code. See you next time. You got javans and