How'd you like to listen to dot net Rocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot net rocks dot com. Hey there, this is Jeff Fritz, the Purple Blazer guy from Microsoft, letting you in on
a little secret about my friend Carl Franklin. You know, the guy who started dot net Rocks, the first podcast about dot net in two thousand and two, The guy who's been teaching Blazer on YouTube since twenty twenty. Yeah, that Carl Franklin. Well, Carl's joined up with the folks from Code in a castle to teach a week long hands on Blazer class at Are you ready to get this? At a castle slash villa in Tuscany. It's sort of a luxury vacation. It's Blazer learning built in. Carl's calling it the
Blazer Master Class. You'll learn Blazer from the ground up, finishing the week with the ability to build and deploy Blazer applications. Since the training happens for only four hours in the morning over six days, you can bring your significant other your partner with you and you should right This part of Italy is absolutely beautiful. There's so much to see and do and in. Larry and Marco from code In to Castle are organizing daily activities both at the castle and in
the area. The castle is in the Marema, a less touristed region of Tuscany, offering both classic Tuscan hill country as well as easy access to the Etruscan Riviera. With sublime local food, wine and olive oil around every corner. Breakfast is included every day. There will be two communal dinners at the castle book ending the experience, and most other meals and all activities are included. And did I mention you'll learn Blazer in person from Carl Franklin Listen.
Space is limited and for very good reason. This is quality training in a beautiful setting. Go to code Inacastle dot com slash Blazer twenty twenty three that's bla z O R two zero two three to take advantage of this amazing opportunity to join Carl in Tuscany for an unforgettable week of la dolce Vita while advancing your programming skills in this important new technology. After building software for a while, you know it's only a matter of time before you see an HTTP timeout
or a database deadlock. In software, it's not a case of if things fail, but a case of when one mishap like this and valuable data is lost forever. And these failures occur all the time, but it doesn't have to be this way. Introducing n service bus, the ultimate tool to build robust and reliable systems that can handle failures gracefully, maintain high availability, and
scale to meet growing demand. For more than fifteen years, end service bus has been trusted to run mission critical systems that must not go down or lose any data ever, and now you can try it for yourself. End service Bus integrates seamlessly with your dot net applications and could be hosted on premises or in the cloud. Say goodbye to loss data and system failures and say hello
to a better, more reliable way of building distributed systems. Try end service bus today by heading over to go dot particular dot net slash dot net rocks and start building better systems with asynchronous messaging using end service bus Hey, Antwerp. It's dot Rock Holy crap, there must be fifty thousand people here. Yeah, who knew we were in a stadium? I know, right, we are in a state. We're actually in a movie theater, which is cool. It is cool. The last time we did a dot Net Rocks
in a movie theater I think was in Sofia, Bulgaria. Oh yeah, do you remember that? Yeah, a few years that was dev reached every And the funny thing was the Bulgarians thought it would be funny if Richard and I announced the names of the winners, the Bulgarian winners of the swag at the end of the show, and that was funny for them, for them, Bulgarian names need to buy a bow Okay, it lacks where snudge and they were just laughing their butts off. That's you. Oh you were there?
Okay, all right, well, uh it's we're glad to be here obviously. Yeah, last day of the show. Fun to do a live show. Yeah, it's a live show. We have some good stuff coming up here. Layla Bougria is here and we'll be talking to her in a minute. But first we have to do this little thing called better know a framework roll the crazy music. All right, buddy, what do you got? Well, I saw this come across Twitter. It's a tweet and while
I'm linking to it, Boston Dynamics. You know who those crazy people are, the guys with the robots. They make the robots that dance, they do backflips. Now, yeah, a little parkour with robots and stuff. Yeah, they used to have a little uncanny heathered robots, robots that were tethered, and then they got gas powered robots. Now they're battery driven and they put out videos every once in a while of things that look like animals,
like cougars and dogs. Well, anyway, they've put chat GPT into a robot and now you can talk to it and it will talk to you back. And so there's a tweet about that. It was from April, but I thought it was so cool and a little bit scary. But they're asking a questions like, you know, are you function It's like data, are you functioning within normal parameters? You know? And it would say, yes, you know, I have this blah blah blah. What my levels
are? You know these kinds of things, my battery level? How many what did it say? How many interactions in your last mission, and it will tell you about its last mission and where it went. Just as they used the word mission, I don't even know if it was mission, but that an extermination mission. It could have been just wondering. It's pretty scary, but it's cool. That's awesome. So that's what I got it. It's a tweet with a video. Okay, yeah, Boston Dynamis videos are
always amazing, fun. Good one who's talking to us? Richard grabbed a comment off a show seventeen fifty three. That's the one we did with Mika about Visual Studio twenty twenty two productivity back in August last year, and Mark Wansel had this great comment. Mark has a lot of great common spells, and he says one of the things that Mika noted a few times was telemetry. I think that this would be an interesting show topic. There's a popular
GitHub project called open telemetry that might be a good starting space. There seems to be an art of telemetry. What to collect, how much, performance considerations and primacy considerations. Check out open telemetry dot io. What do you think that that's not a good idea? Not want to do that? Noah, we won't do that show. Yeah, sorry, Mark, I wish
we could help you, but we can't help you. Yeah, but I will send you a copy of music Cobi And if you'd like copy of music by I read a comment on the website at dot net rocks dot com or on the facebooks. We publish every show there, and if you comment there and ever reading on the show, we'll say, do you a copy of music by? And you should follow us on Twitter. But the real fun happens on Mastodon. I'm Masodon, I'm at Carl Franklin at tech Hubs Social,
and I'm Rich Campbell at masodonda Social. Send us a two let us know you're listening, and that brings us to our show on open telemetry. What yeah, how sorry, that's the topic. Lila Bougrie is here. She's a software engineer with over fifteen years of experience in the dot net space and currently works in particular software where they build nd service bus. Maybe use it maybe forard a bit, certainly don't know the show a few times.
Yeah, She's a Microsoft MVP and frequent speaker conferences and interspare time. She loves to knit and crochet. Welcome, thank you. How about Lila, huh, we were lying when we said we weren't going to do a show about open telemetry. That's kind of why we read that. Yeah, yeah, I figured if I've got a comment literally somebody asked her for to open telemetry, now is the time to read it. So, Lila, what's
the elevator pitch for open telemetry? Well, the elevator pitch. So we've we've had multiple telemetry signals in software for years, right, the oldest one being logs. I think we all know logs. We've also mostly used metrics. Maybe I guess distributed tracing is a newest signal, but I would say that the elevator pitch for open telemetry is really correlating them all together. And that's what makes it really interesting for me at least, because you know,
each signal has its own value. And we've also been logging for years, and even though we can see, like you know, tracing might be the better option today, but then we still have all of those logs we don't want to go rewrite and higher applications though. We do have log providers so you can plug in however you want to do your logging, but that's not really that's taking the same source and putting it in different places, isn't it.
You're talking about ingesting different sources into one place exactly, Yeah, and then being able to connect it. So basically, imagine, you know, over the rainbow that you get lowered and there's a metric that looks out of whack, and it's like, okay, something is up. What is up? I don't know, just the metric is out of whack. So you have to go figure out how to do that, and that's usually a challenge because you then have to go figure out, Okay, how do I connect
that to those other signals that I have. But what if you could look at the metric and say, okay, I can see that it's correlated to these traces, and then go look at that, and then the traces would also be connected to the logs and you would be able to basically, yeah, paint that entire picture of what's going on, and you wouldn't be losing all of that time doing that thing. Manually sounds like a Wikipedia rabbit hole, right. One thing leads you to another to another. So I'm trying
to distinguish between all these different things. I mean, logging means a particular product like sequel server spitting out logs about how it's functioning, possibly versus metrics being more things like the state of a server, like the hard drives pinned or running low memory. And then when I think about traces, I think about there's tools that specifically are about following a button click from the client to
the server to the database and back again. Right. Yeah, I like to think of that like following a business transaction, right, right, like the workflow. Right. So it's true, Yeah, that's definitely true. But at the same time, we've also been logging in our applications a lot,
and that's still then connects together. This is as a developer writing code to push messages onto a log right, So that's usually I'm a developer, right, So I'm always looking at things from the application perspective, and how do I make this application observable? And there's multiple signals, and you would choose each individual signal based on whatever your scenario is. It could even be that for a specific scenario you might come to the conclusion that using multiple signals
might be useful. So for example, let's say that a failure occurs, right, you want to keep track of that and have a trace that reflects that failure. But you might also want to have that reflected emetrics so you can do the alert de alerting and all of that. Yeah, So that meaning to trip it up in through the system means to say there was a failure that occurred as well as what shows up in the law. Right now,
there's a lot of third party products out there that do telemetry. Or does open Telemetry allow you to use those as sources and then pull them together, or does open Telemetry have its own things that you can plug in or both? It's both, and that's a good part because I think if we look at our applications and we want to make them observable, there's a lot
that we can do. But I think with the Open Telemetry project, and also like all of the effort that the entire community has basically put into this, they've made it easy for us, right because you could basically say what second are using, Oh, I'm using a speed on at core and I'm using the Azure is thek whatever it is, right, and you could just
use those instrumentation libraries that are available from those frameworks. It could be built into the framework or could be a dedicated package and you can turn them on and just by doing that, you're already collecting a bunch of information, and specifically in the distributed system, it's usually going to give you insight into that interservice communication where we have the blind gaps. So it already like gives you
a lot of information. So the cool thing is that you can then intercept basically those traces that are being generated by those libraries, and in that that I would be calling activity. It occurred, right, and you could add you could create your own activities, which is, by by the way,
the sort of same thing as a span. But yeah, they basically what they did is the activity API already existed, and instead of creating a completely new API to match the naming of the open telum Try specification, they just implemented the specification inside the activity API. So that's why the name is a
little bit different. But basically, what you could do is take the current activity, which could be omitted by an instrumentation library, and say, I want to add some information to this that is specific to my application, to the workflow I'm running in so that I can get even more insight. What's the voodoo that allows us to go between the different tiers and an app and
say these are all part of the same transaction. Right, that's basically a propagation mechanism really, because if you think if a trace, it's basically a bunch of spans that are connected to each other. Now, what happens is at the beginning of the trace, we basically get a trace ID assigned and that's going to be carried across all of the spens and then each span has a unique ID. Now, in order for that information to propagate across multiple
services, we need a propagation mechanism. And there's multiple protocols that are basically supported by the Open Telemetry Project, of most of well known. One is a W three C trace context for HTP headers always feel like I need a breath, yeah, And then there's another one for gRPC. So it depends on what you're doing there. So I have, for example, and as your web app, right, I have Application Insights turned on, I've got all the switches lit up, and do I need open telemetry at that point?
What's it kind of give me over what Azure Insights already has. So the way that I look at that is what Application Insights provides you is also sort of known as black box instrumentation, so it's basically independent of your specific
application code. Yeah, but obviously the things that we are doing in our code is usually the interesting bits, and sometimes we need a little bit more insight to understand, you know, what pieces of code are we executing there, what are we doing, what is like the cause of latency or whatever
it is, what pieces slow? Application Insights doesn't provide that well. I think it's it's quite different to to compare because at least to my to my understanding, it's more of an overall view that you get, and you can still use the Application Insights as AK directly and still emit like application specific telemetry. But but then you're tied, right, You're tied to the vendor, right, and the thing is right right exactly, and if you don't want
to change to another vendor, you're have that vanderlock. It's the point, right, Yeah, So if you use open telemetry, then you you could just wire up a different exporter. I mean that would also app Insights has good features for dot net specific absolutely, Yeah, if you've got some other code written in other things that aren't ornet related, yeah, happid sence is only going to do so much for you and certainly not going to work if
you're in a container on AWS, is it. Yeah? So, I mean it certainly opens the door to working with more platforms, even more places and open hey, hey, there's a concept for you. Yeah, and it being available cross platform and cross front time as well. Like there's implementations for instrumentation libraries in many languages for many framework So that's really cool, especially
if you think of like multi stack applications. Right. Sure, if I've got a group of Python developers going to build a data importer for me, the fact that I can instrument it the same way as everything I've gotten built and dot net, that's pretty compelling, right, and bring everything under one roof and measured the same way. I mean, that's the real problem is that often we're chasing problems that transition between different systems, and because they're measured
differently, it's very hard to associate restuff together. I'm just thinking in terms of how much code I need to write as a developer to take advantage of all this and how much of it comes of the box. Well, that's what I That's why I mentioned the instrumentation libraries, right, and even if you know things like event counters and stuff like that, even that has dedicated libraries available already, so you can basically just turn them on and like I
said, you could plug into that and add to that information information. So usually what I say is, look at what it already gives you. Turn on the instrumentation library. Yeah exactly, don't reinvent the library. So look at what's out there. Turn it on and see what it emits, and take a look at what type of insight that already gives you, and it
probably is picked up by open telebras you're just fine. Yeah, yeah, Well, for example, let's say that you have aspe core instrumentation enabled, right, so you're going to see the request, but you don't get a lot of insight into the request. But I mean it has hooks that you could then plug in additional information and expose whatever is you know, interesting to you in that scenario, so that you could understand like the sort of business
context of what's going on. And that's what makes it really powerful. I spend enough time on the firefighting side of being assisted min where there's stuff's being spewed out in the logs that we're looking at. We just don't know what it means, right, right, like anything there? Yeah, it's clear. It's like your own little Internet. Everything you need to know is there, you just can't find it. Yeah, So you know, how do
you add the additional information that helps someone see? This is where we're having right. That's where defenders really come in, right, because they are then going to offer capabilities that allow you to querry, to analyze that information, and to basically get to actionable insights, because that's the whole point of telemetry. Right. That's a bunch of data, But then what do I do
with it? Right? So you want to basically have tools that help you get pointers on how do I fix this problem or how do I improve this latency issue that I'm seeing, or maybe even see like which feature gets used a lot? And things like that. She is almost more of a profiling thing, right, Like what are the functions are we called the most often? And we know perfectly well why the system is slow. It's the database is fault. We just blame the DBA. Then we're done. Life is
good, all right. You know Another one that I always think of is because I'm you know, I'm like an observability enthusiast, I would say, right. And the reason why I'm so enthusiastic about it is because, well, my sort of core focus has been message based systems for two years are to trust. Shoot, yeah, it's just clim messages split across Q and then you get out of order messages and it's like, what's going on?
Why am I seeing this fail? And especially in a message based system, usually the problem is happening like for the up stream right, right, So how do you get to that? Right? So? And how do you connect at even outside of the messages that are being sent, because it has to maybe connect back to, like you said, a click somewhere on a user interface. So being able to have that full visibility across all of the
subsystems is really really powerful. Yeah, and again I'm still worrying this a lot of codforma, right, But you're telling me that when when you're using a library that has protocol understanding, it's going to insert a lot of that information automatically for us. So it's selfol together. Yes, Yeah, And because of the sort of nature of how distributed tracing works, that information is going to be connected together through that same trace idea that's basically being propagated does
have substantial overhead. Is there is there any reason to only turn it on when you have a problem or can you leave it on all the time? Okay, that's that that's going to be a long question. As you can you can go for the It dependspends definitely. So yeah, it definitely depends.
But so usually what I tend to say is make sure that what you're collecting is useful, right start there, because if you just turn I don't know, every instrumentational library on the planet on, we log all the things and capture all of the traces and all of that, you're going to have to sift through that all of that information to be able to understand like what's
going on. Right, So it's also not a thing of oh, look at all of these instrumentation libraries and then turning everything on, because you're going to be incredibly overwhelmed, to the point that as a developer you might feel like, okay, this is not useful. Yeah, let's just turn it all back off. I mean, I've also had the problem where I've said, Okay, I'm not going to measure this thing, and then I never get data for that thing, Like it turns out I'm looking for the wrong
in the wrong place. Like that's not a number that moves. So some of these telemetry products that are out there have ways that they can work on a background thread or they can attach as a sidecar, you know. So do you have those kinds of things where you can sort of stay out of the way so if there is something that takes up some more time, it
can happen on a background thread. Yeah. So that's where the open telemetry project is also really interesting because if let's say that you look at the basic samples that are out there for dot net right, what you're going to see there is that you can basically, in a service enable open telemetry at an exporter, which means that you're collecting let's say, for the sake of the
example traces and sending them directly to an observability back end. Basically what that could be as your application Insights or Jager or Honeygo, whatever it is. So, but the thing is is that there's many problems to that. First of all, like you said, there is overhead for that service because it has to collect all of that information. There might even be some processing behind the scenes happening as well. We had a service bus, well, well
that's we'll get to that. It will get to that, yeah, and then you have to export it. But then imagine that the observability back end is not available for a few seconds because it's you know, it's the network.
It's the network. Yeah right, so well no, that's usually that's handled by the libraries themselves, but it is adding that pressure to the services telemetry, right, yeah, you don't want to have that disconnected information and then looking at half of the story, right, But there are ways to solve this, and that's where the open telemetry collector comes in. And then
you have multiple deployment options on how to run that. The first one is, like you said, a sidecar, So basically you're going to have a sidecar for each service that you're instrumenting, and then immediately you're offloading all of the telemetry. Well, you're collecting it and sending it through to the sidecar. But there are any processing that needs to be done, like redacting information because remember sensitive information, you don't want that you're in all of that telemetry
you're collecting. So that's just down the road. We can be fined very easily. I call that digital white out. So then you have all of that processing and then you could export it then to the observability back end and you'd be able to handle all those communication issues and all of that in the
side car and your service is not affected. Now that's one option, but you could also set up the open telemetry collector as as at the gateway, so it's a central components and all of the services can basically send their telemetry information to that central component, which then will take care of processing all of that information to the to the back Yeah, and it could batch that information and it's a pretty powerful thing, and you could even let go crazy or
if you need it right, but you could have a sort of hybrid model in which you have a sidecar per service which then sending their reformation to the central collectors. Is there anything special in the storage on that and there's just blobs or text files or are they actually using a database on that? Well? I think it depends on the signal, sure, because usually metrics go to time serious databases and then logs. Honestly, I don't know. It's
a good question, depends but it depends. Yeah. But well, actually about those time series databases. That's a sort of interesting topic on its own because at the beginning I was talking about TELME tree correlation, right, and basically adding the trace ID to the metric, just called them exemplar in open CELM tree naming, so that you would be able to connect that together, so you'd see a spike in the metric and see, oh, that's caused by those traces. Right now, the thing is that you have to be
aware of what's known as cardinality explosion. So I'll try to when a bomb goes off in the Vatican. Dude, but did I say that. I'm sorry, those are cardinal explosions, not the same cardinality explosion. Cardinality explosion. I'll try to explain that, but usually I do this visually, so
okay, I'll give me a bit to try. But think of the exemplars basically a label, right that you're adding to a metric, and that's going to give you some insights on the context in which that metric is being collected. So let's say that I have a metric called failure rate, because I want insight into that and to have a little bit more background information. I want to know which environment that that happened, bill, development, tests,
production, whatever it is. And I also want to know which hp status coode came back. Now that HSP status coode in a production environment is going to have like for the sake of the example, thirty possible values. And we have three different environments. So that's three possible values for that environment label. Now the cardinality is basically all of the possible combinations of those values of every label that you add that is a cardinality. So it's a multiplier exactly.
So we're fine with the environment and then having the HTP statoscope and then I add customer ID. Oh boy, so why let you do that? We're like one hundred combinations, and then you threw in twenty thousand customers a ruined everything more. Yeah, how do thousand customers a million customers? Somebody
would ever do vouch? And that's how we then basically get cardinality explosion because what happens in the time series database is that every time you have a sort of unique combination and your series is created, so your cost goes up right and it becomes really hard to quire that information. So it's also important to one field, which just one field. If you just don't do that, there won't be this problem. Yeah, well it seems like a good idea
when you do it right until cardinalities exactly. So it's also important to be aware of, you know, what observability back end are you using and how does that work, because there are some tools out there that do support high cardinality. So it's just something that you have to be aware of. Yeah, and so you can tolerate if you really come to the resolution you have
to do that one way or the other. I mean, custerrity doesn't seem that crazy because it is useful if you've got a customer on a phone to say, hey, I could pull all the transactions, all of those streams for all of that customer and sort of look at where they were having problems. Yep. That's like full production debugability, right, yeah, without doubt. And with that, I've got to interrupt for one moment for this very important message too, and we're back. It's dotting at Rocks. I'm Richard
Campbell, that's Carl Franklin. Hey, Hey, talking to our friend Leila about open telemetry. Hey hey, and we've kind of gotten to that place now, right, Like we we how do we visualize this because you're getting you're probably it a lot of information like what is are the tooling that comes with it? Or I have to write my own what are the dashboards?
Well, that's hopefully where you choose a vendor, right, and then you know, with the abilities of open telemetry, by standardizing all of that information, I usually just say, like, try a bunch out. You see what are the requirements for choices? You imagine you can just look these up to standard graph controls and things like that from your various vendors. If you want a dashboard, yeah, yeah, yeah, there's just so many options,
and I feel like each of them has their own strength. But yeah, for example, if you're if you're running in the Azure stack then and you're already using application insights, it's a thing you know, can it makes sense against and you're using that as well. Yeah, But then for example, I've played around with Honeycomb, and I think that they're especially like the collaboration that they've built into the tool is really cool as well as AWS product. No, no, it's its own company, Honeycomb. Okay, big
big taste and a big big bike. Nice. That's the breakfast Cereal. But okay, I was a kid of the you know what I'm thinking. I'm thinking of that there is a honey Honey something product in AWSS, no code product, yeah, okay, different one, but yeah, Honeycomb is an instrumentation library. Yeah. So, and they're very invested in open telemetry as well, and I think it's one of those tools that supports high cardinality. They talk about it a lot as well, right, yeah, but
yeah, it would sit out to me. There's really the sort of collaborative feature features that they had, because usually if you're looking at a huge problem, you're not doing that by yourselves, right, yeah, you know, and if it's like if you if you have a twenty four seventeen and one and stair shift, you want to be able to hand over where you left
off things like that. Yeah, I'm wondering about So who is this Typically the systems that are going to get these packages in the first place, that that's where the air resh is first show up, where the problems appear, and then they might be passing it to development saying hey, we're looking at
this and we think it's it's this kind of problem. Well that's where I'm also expecting some evolution because yes, usually now it would be a different team when they would be looking at something that looks funny, yeah right, and then get some understanding, hopefully some actionable insights, right, and then being able to bring that back to the development team. But it's it's really interesting to me in the sense that we're the developers, We are the ones going
to be writing that application specific telemetry. So it's it's also really important to get like organization wide alignment as well on the type of telemetry that you're going to be collecting. So this seems like there's an infinite number of decisions to make here, right, I mean you just by using open telemetry, that's just one step of many. Yeah, what are we going to be looking at how what's the granularity of it, How are we going to query it,
how are we going to look at it visually? Like, these are all things that aren't just in the box. You have to think them through. Yeah. Well, usually what I try to give a world try to advise as well, is that to sort of documents and guidelines like what are you looking for with your telemetry? What are the problems that you're trying to solve and like come up with a specific set of questions that you could as a developer when you write a feature, ask yourself so that you could add
the telemetry that is going to answer those questions. Right, I mean, what if you don't know what you don't know? What if you don't know what you want to look for as a guidance? Are right to say? Yeah? Some of this sounds like business related decisions, like we sell widgets, and I want to know when a sale fails because of technology rather than the customer didn't want to buy it, right, yeah, definitely, or simple things like this page took too long to render or whatever. Yeah.
Yeah. And then from the sort of failure perspective, I usually try to look at the code and think to myself, if something were failing here, what I would be What would I be looking at if I were debugging this, like, what what state would be interesting to me? What variables would I be looking at? Have I captured the input enough? Have I captured what's going out to be able to understand and be able to then you know,
go back and try to understand what happened from the outside. Right, there's an easy solution all of this you just put all your code in a try with an empty catch. No problem is that on error resumed next a sort of yeah, it's more like slash day to turn off all the debuggs. I don't want to know. Just keep getting don't tell me about these guys. I mean a lot of this we've talked about very proactively, like we're going to detect the errors before the customer does or before the customer complaint.
Yeah. I think there's another dynamic where the customer is complaining and we're getting a ticket that's like what error. I think it'd be very challenging to say, you've got this ticket is about this customer, it was roughly at this time, and now you want to go dig through the logs to say, can we see what this person's complained about. To do is to have the customer on the phone and enable this sort of thing, but just for them, just for their customer ID and say and then just watch it as
they're going through the process where it fails and now you've got something. But is that impossible? Well? I think so, but it sort of depends
on what type of sampling strategies that you're applying. And also, like you know the type of observability that you're collecting, because let's say that if you want full insight, so basically any error that occurs to any user, right, be able to say, oh, you know that was since we're an antwer that was Shalts and it was three pm on a Friday, and basically be able to find that request and look at what were they doing, because the thing is that users when they open tickets or whatever it is, the
thing is that they weren't paying attention to doing their usual thing, right. They didn't set down See, let's cause an error, and I'm not thinking about every step that I did so I could be able to explain it to you later. I usually get it doesn't work. What can you be more explicit? It doesn't work? Right, So then being able to connect that information back and say, oh, yeah, that was shuts right, and then find that, you know, Tracey's locks metric whatever was connected to that
information. It's like being able to debug in production. Really yeah, And then we think you just described like turning up a lot of data there too, Like that's also you know, we were also warning not to do that because you're getting buried in minutia yea, yep. That's where the sampling strategies
commits. Sure, and there's different ways to go about that. So if you think about traces specifically, you basically can choose between head and tail sampling and head sampling and dare You're basically going to decide whether to sample the trace at the beginning. So let's say when the business transaction starts, right, they're going to immediately make the decision of I'm keeping this or I just don't care about it. Right. Usually that's the most unbiased type of sampling as
well quick order. Right. The thing is what if something fails, right, you don't know that upfront, What if this request was super slow. It's not something that you can know up front, so you could be losing a lot of insightful information. Sure, and that's where you get, you know, the tail based approach where you're basically going to collect everything so the entire trace across all of the services that it goes through, and at the
end make the decision of is this an interesting trace? Do I want to keep it? So? For example, does it carry some specific attributes that I care about? Or was it slow? Yeah? And banks, the question can you turn these things on and off in production without restarting. Right.
So that's where it becomes important again how you deployed this, right, because if you have a sort of direct export and it's really tricky, but if if you had it deployed as a side card, and it could be a thing of changing the configuration of the side card, like tailbase says, I'm going to assess the finish transactions, right, there's nothing special about this, throw it out right. Oh, this one had an unusual value, it took too long, it generated this air. So forth, I'm going
to keep this one. And so that way you're sort of sculling as you complete exactly. That's pretty cool. Yeah, there's a cost that comes with that, sure, because basically you're collecting everything, so you've got those overhead on the workload. Although hopefully synchronous to some degree. Actually that asynchronicsy brings some dishing point that you wanted to have the customer on the phone while you
work them through it, like that one scenario. How much latency do we have when we're doing all that processing separately, Like how soon can we see data from when it tries actionized? You happen, right, that's again another concern that you have to basically be focused though, right, like what is
what type of latency are you willing to accept? And then it becomes a thing of being very mindful of what type of processing that you're doing there, because obviously all of that telemetry is going to have to go through that pipeline of processors, so you have to be very mindful of that. And then you have to get it out as soon as you can so that you can see it as quickly as you can and you can create as quickly as you can. But we're still talking seconds, aren't we really? Yeah, I
hope even less milliseconds. That's been my usual experiences with asynchronous and telemetry. It's only slightly behind it, but it's not holding the transaction while it finished, you know that. That to me, the big sin here is don't delay the customer. Get they get the working transaction done. All telemetry can happen later, even though that later is a few milliseconds, right, That's
why you offload. That's an our components again, Yeah, safer, safer to work that way anyway, which you don't want as a transaction field because you were measuring it exactly exactly that's dumb, that's no no. Yeah, so yeah, definitely quantum. I just got it, just like that, boom, it all makes it just makes sense, even though it still don't understand quantum. I'll leave that to you and sip, okay, we'll work
We'll keep working on that problem. Yeah, that's definitely something to keep in mind is that, you know, instrumentation is at the end of the day, nice to have. It's not a mission critical component, so have to be very mindful of of how you do that. So one of the biggest mistakes that customers are people using the open til I'm train make when they first started. The biggest mistake, well, I turned everything on. Of course
you did. I do that too. I like all the knobs, turn on all the knobs and then yeah, another thing was just figuring out what do I really mean? Right, So it's really understanding what your requirement is, because if you're saying I want to be able to debug any user request that happens in the system and have full visibility into that as opposed to for example, I walked into a project it's massive, there's zero documentation, and all of the people that worked on it, they're gone, right, and
there's no instrumentation. Yeah, at that point, what you would like to have observability wise, it's just insight into how does this thing work? Yeah? Just follow a transaction, right, you know, have one end to end trace once and you'll have made a lot of progress. Yes, Now, the amount of telemetry that you need for those two things, it's completely the opposite, right, it's like one percent versus one hundred. So it's
definitely still for me as well a learning experience. I mean, this is still pretty new and we're basically just adjusting to see still how the project is evolving. Yeah, because I mean many parts of the specification are stable by now, but a lot of things are still evolving. Yeah. Another thing that keeps coming up is sort of the three pillars of observability being you know, traces, metrics, and logs, And I wonder, but what if a first one comes along that might just happen. I can't think of a
fourth one, right, neither find us pretty well. Yeah, when when the only thing we had was logs, could we think of metrics and traces? I guess that's true. Yeah, we started dreaming about them. It's like I'm trying to I've had that experience of I have the log from this machine, the log from this machine, now try and line those entries up.
Would quantum computing introduce a fourth pillar? I think it would introduce to the sixteenth pillars, or two to the two hundred and thirty seconds parallel there. I also like this idea of knowing it failed before the customer does have failed. You know, you get what I'm hoping to, this tail loog
thing of I see these numbers as insufficient. In some way, it kicks up into a system where someone can look at it and perhaps even call a customer and say, hey, we noticed, can we help you with You know, it's not just you're waiting for the for the people to complain or if it has to be on fire. I have never gotten a message or an email like that from a company that I and if I if that happened
to me, I would be like really impressed. So I'm on a website, for example, and it screws up, and then I immediately get an email that says, hey, we noticed you had this problem, didn't work. Here's a solution. M Wow, that would be amazed. I ever see that is like a credit card when I'm traveling, yeah where every so often I use the card and a minute or so later the phone rings and it's hey, are you in Belgium? Right, yep, yep, I'm in Belgium. Okay, that's good. Then thanks, And then you're like,
that's pretty cool. It turns out that was Bob from New Jersey. Just just want to know that happens to me all the time. I asked you about how service bus or you know, service buses and service bus plays into this. Obviously, this is what you do for your job. How does using a messaging system figure into open telemetry. Well, like I said earlier, right, um, building a message based system is is really nice. Well, you know, if you take into account the entire problem space.
But one of the things you can go around around, basically is how hard it becomes the troubleshoot things in that type of system. Now, with the platform that we're building at particular, we don't only have in service bus as a middleware framework, but we also have a bunch of tools. Now
service Insite specifically is basically already that sort of black box instrumentation. It's like you don't need to do anything, you just need to configure that you want basically those platform tools to be enabled, and it already gives you insight into all of the messages that are being sent around in the system, and you can see that in a production environment and understand which exact flow and where that come from and which message led to which message. So we've really been doing
observability for years. Yeah, that's the thing. Well, because of the asyncrency and out of orderness, like you can't count on timestamps, you really need to have some kind of attribute flag to be able to know the related Yeah, those are things that we capture in the message headers, so we can basically know which message LEDs to which other messages, So that that's really
cool. But the thing that we could never solve, and that's the sort of gap that opens telemetry closes, is having that visibility system wise, so connecting back to you know, a fronted or even a database, a web server that restarted in the middle of something like those kinds of things like, that's where you want the metrics to show. Hey, this machine got into crisis and the supervisor killed the process and it recovery and continued, but it
kicked off all this weirdness right, like I can. You're looking at it, You're looking at the trades and going, what the heck happened here? Right? Right? Is the is the program broken? It's just a bug? And it's like, no, this is what recovery looks. Yeah,
yeah, exactly is one of these machines. Some of that is also built into the platform because we have insight into what are the failure how many failures are happening, So we also have messaging specific metrics already in there, and and yeah, now we're working to basically make sure that it also connects and feeds into the open telemetry signals so that if people are using it, that
they get all of that information in there as well. Wow, that's pretty pretty substantially cool, especially when you start throwing cloud in here where it's entirely possible the cloud vendor might move you and it might have impact on your software, like stuff you literally don't have control over, Like hopefully your telemetry can surface that in a way where you're like, oh, this wasn't us. It's the vendor change something for whatever reason, and we should absorb it.
But we don't know how yet because we've never had this happen before and right now we have to look at it and saying what would we do differently? Yeah, yeah, because we have sort of recoverability built in, so retries and all of that happens out of the box. You don't even have to configure them. But yeah, then it's like, okay, why does it take ten times for this message to be processed? Every time a message of that type comes in? Right, So it's you don't want to hide that
away. It could be that maybe there's a database suffering underneath, right, it retries too quickly and it takes that long for that thing to get up to speed or yeah, yeah, along one of the side effects. Yeah, that's also why we have delayed retries. So we'll basically also have this sort of back off mechanism. We'll retry immediately. But if we see exactly that's too yep, yeah, it's the same. It's the same concept. Yeah. Yeah. So what's next for you? What are you doing next?
What's in your inbox? What's in my inbox? Well, uh, we're so a techarama today. Then I'm taking a few days with the family. It's a long weekend, is it? It's a long Weekend's holiday Tomorrow? And then they're basically bridging to the weekend and then on Sunday, I'm leaving to Oslo for NDC. Very great. I will not be joining you there this year, so I heard. Yeah, not happy to hear that. Sorry, just schedules. Yeah. We love Oslo, we love NDC.
We're there all the time. Usually, I'm gonna have to miss it this year. Yeah, I'm really excited to go. It's my first time at Oslo's top sure to get into. Yeah, you know, and I always loved the way they did the show on the floor in the but I kind of tell you this techarama's pretty close to the Yeah, and I have it homies. Yeah, yeah, all right, it's really cool. Everybody give it up for Lila Bilbria and we'll see you next time on dot net
rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot com. Visit our website at dt n et r ocks dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to sh number one, recorded in September two thousand and two.
And make sure you check out our sponsors. They keep us in business. Now go write some code. See you next time. You got a dead middle band