Practical Observability: Logging, Tracing, and Metrics for Better Debugging - RUBY 656

This is one of the things that is really really important to me and I'm passionate about and particularly passionate about introducing this into rails apps. So thanks for having me on.

kind of overlooked. If you're starting up, maybe you get some like you know, bug alerting or something like that in place as like a standard, but kind of anything performance monitoring wise is kind of like a oh no, like something happened, let's look into it now. I feel like it's like the typical flow of things as people start up. Do you want to just give us like a high level like what is observability and why should we care?

actual goal. It doesn't matter how you get there, but the goal is being able to number one understand your rails app in production, and number two be able to ask unusual questions, not questions that you've thought of a day, two days, three weeks ago, because that's not really very useful or interesting if we knew exactly the questions to ask. In the future of our apps, everything will be easy. Just be like, how many two hundreds have we had in last week? Kind of boring question to ask, maybe

a bit useful. I find the more obvious the question, the less useful it is. So observability is the practice of making a black box system more transparent. So I like to think of it. Imagine your entire rails app, all the hosting, everything to do with that app is wrapped up in an opaque black box, and somebody says, how does it work? And why is this thing going wrong? You would have no hope of understanding it if the box is completely translucent and you can see everything, which

of course is completely impossible in software. But in theory, you'd have this completely translucent box and you can ask all these questions and you get instant that's like one hundred percent observability. And of course that is absolutely impossible. And so what we're trying to do with observability is understand what is going on, not just when it goes wrong, although that's the obvious use case. Is we have an incident.

The most you know, the most critical point where observability comes into play is an exact scenario that I landed in two weeks into a new role I had, So it was two weeks in the site had gone down. I was in the U, I am in the UK, and my tea, the rest of my team we're in the US and there were two other engineers in my time zone, and all of us have been at the company for a total of five weeks. So we've got this app. It's down, it's on fire, and we need to put the fire out. And we just the three

of us looked at each other. We're like, what should we just restart the dinos? Yeah, so we restarted the dyno. We crossed our fingers and it was pure. Look that the app came back up. That is the exact opposite of what we want. And we've now moved to a situation where we can ask our app a whole load of very unusual questions and we will get an answer to that. Why are there a peak of four h fours on iOS at three a m. Looks like that a lot of them are coming from this IP address. Okay,

what's that IP address doing on the site? Okay? Interesting? How many users are using that IP address? Five? So only five people using and you keep So that's the point of observability to me, to be able to ask unusual questions that you haven't thought of already dynamically and explore the space and come to some conclusions.

it a quick little flush like reset things. And you're just like, oh, I guess that's how we're gonna do it until we can get like some insight into what's happening, right, And I think that's definitely underlines the importance of observability in general, Like, uh, you know, how do you get those insights to begin with? And maybe that's a great starting point, Like where do you start like looking at it,

like adding this insights? Right, Like what's the Is there a modular approach you could take or is it more of like you should look at doing everything all at once kind of thing?

it was out there in the universe. Okay, maybe I should learn that. I should learn that, and I kept using this. Should I should learn this? Have loads of other stuff to do. I've got loads of other things. I don't know what it is. I know it comes from controls here, and there's a Wikipedia page. It's really complex and really confusing whatever. I've got real work to do. But what I know is that he kept coming across these bugs in bugs, Century air Break, choose your error

reporting tool. They all help you to a degree, but they're not a silver bullet. And I kept coming across these defects over and the story was exactly the same. Come across the defect, I'd see the stack trace in the error reporting tool, and I would look at it and first first emotion right out of the gate, complete confusion. What is going on here? No idea? So I dig a little bit into the code and dig it a little bit into the stack trass. So it's coming from

here and this thing is nil. Classic right, this thing is nil? Where is it being passed in this nil? I don't know. So now I'm like, well, I can't just say I can't fix this, so I now have to well do what exactly? I don't have any information to go off. Well, I guess we'll do that. Let's look at the next one. And this just kept happening, and I would find myself going through all the bugs in the backlog and I couldn't fix any of them.

And I just wasted four hours looking at things, asking questions that I couldn't explain, looking at things I didn't understand, And for years I thought the problem was with me. I honestly thought, I'm just not smart enough. I'm not a good engineer. Blah blah blah blah blah. Bug fixing isn't really my thing. It's I'm just not really good at it. And then, after many, many years of this, I was in a company and I just got really

sick of this. We just released a brand new app and it was it was a customer account app, and we're getting all these weird bug reports, people saying how logging. People kept saying, I can't reset my password. And every time we did this, we would add a little bit of kind of this ad hoc logging and then put the bug back in the backlog, and then it would come up again and come up again. And after a while, I was just like, this is just this is ridicul

We're highly paid engineers. Is it not a better way? So then I started looking into we were using Cabana at the time, or rather I should say we were not using Cabano at the time. Cabana was there, we were paying for it, and I was like, I've heard this is something to do with logging, So where do we do our logging? People like Cabana, I have no idea what this even is. Let's open it up, and there was just all this trash, all this rubbish. I was like, what's this? How is this supposed to be useful?

People like, oh, we don't really look at that. It's not very useful. So so how do you figure out bugs? And they're like, well, we just yes, we just figure it out. Well, yes, but we're not figuring it out.

So all of this was born through frustration. And so what they did back then is what I recommend everybody does now to answer to your To answer your question, come back to the point, John, which is, take a question that you wish you knew the answer to a very specific question, not why is our app not performing as we want? Not as in like why do our you know? A very very specific question. So take your big, big question, and the time this was why are people

being locked out of the app? Why can they not reset the password? They're clicking on this password link and they're saying it's expired or it goes nowhere or it doesn't work. Okay, why are those people like? Why is that happening? So that's quite a general question, and you want to break it down into some hypotheses. So that's the first thing. I have a five step process, and this is step one. I'll go through the five step process in a minute. So Step one is think of

a specific question. So a specific question. This might case might be Okay, I've got one customer here. There's many, many different types of defect. So this one customer here is saying it was expired. I went to the web page and the said it had expired. Okay, when did they click on that link? What response did the app give to them? And when did the token time out? Right? So those are three questions. Now, they're not going to get us to the answer directly, but there are three questions,

very specific questions that we can add instrumentation for. So I take one of those questions, when did the token time out? Great question. So in order to do that, we need to know when the token was created and what the expire of the token. Well, this is just a random example off the top of my head. So you'd be like, Okay, well we need to know the customer ID. We need to know the token. We don't actually need to know the exact token, but we need

to know the customer ID. We need to know the time that the token was created and the spiry time of that token. Is it fifteen minutes, is it two hours? Whatever? So I would then look into the code. That's the next So we've done step two. Step two is define the data you want to collect, user ID token expiry, and an event saying the token has been created now for this user ID. Okay, So that's the second step. The third step is build the instrumentation to do that.

So whatever you have to do, maybe it's you need to actually add structured logging to your entire app. I don't know. Maybe it's that you've got the structured logging fine, but there's nothing listening to it. Maybe maybe the tool just can't actually measure what you want it to measure, So maybe you need to invest in a new tool, whatever it is. And then you build some code to

instrument just that very small piece of functionality. And then once you've done that, you wait for it to deploy and then you look at the graphs, you look at the logs, at the charts, whatever out but you've got, and what normally happens is for me, I look at the charts and I say that is not what I wanted at all, Actually I've misunderstood the problem. I've misunderstood

the data I want. Now that I see it. Ah, just like you would with agility, true agility not agile, because agile mean something else now, But true agility is you do a little bit of work, You develop a feature, You show the customer. They say, not quite right, go back, adjust it closer, but still not quite right. But if you ask them to describe it exactly right from the beginning, it doesn't align with what they want at all. You need to show them, and it's only by showing them

that you get feedback. And the same is true for ourselves. It's only by looking at the graphs and the logs that I realized that actually isn't what I wanted to begin with, or it is, or I'm onto something there, and so I keep. Then sort of I've used the graph, maybe it was unusable, maybe I could query the parameter. Maybe there's all sorts of things that might be happening there.

So then the last stage is improved, and so from improve you can go back to the very beginning, ask a different question, or maybe you just want to a trade on the instrumentation a bit, deploy it again. Oh, that's more like it. Okay, So now we know the token expiry, what's the next question we want to ask? Well, why did like, when did the user actually hit the site? Was it after the token expiry or before? Hmm? Okay, sounds like an obvious question. But maybe maybe it's after,

which would indicate the token really had expired. Oh it's before. Huh? How could it be expired when it was before? Oh? Hang on, what's the time zone of the token? Now we're getting into it, right, so you've logged the time zone. Holy how the time zone of the token is out of sync with the time zone of the user? That's what it is. Yeah?

resource like Cabana, and it's not very useful. But once you start to abstract the ideas and use cases and how people are actually like using the thing that you've built, you know, you can definitely isolate what it is that

you actually care about. And I think that I think you're right, like that is like kind of the whole importance of observability is identifying the use case and exposing what you actually care about as far as all of these things that are because I mean, you know, there's HTTP logs, there's like all kinds of logs and information available that's just like emitting all the time. Like how do you know and identify you know, which are really important? And I think it just depends, right, like what do

you you know? What are you trying to capture? So it's a it's a great like step wise way to just like start to figure that out, right, because yeah, just depending on your role and depending on what you know your responsibilities are, that could change and that could be different, and your observability needs will change with that. So identifying that is probably most important, I think.

You should be blair, you should be blair. You'm sick of it, absolutely sick of all these gurus telling me what to do and what I should be learning and what I and very few of them talk about, well, what's the benefit to me? And in order for me to do anything, in order for me to change as a human being in any way learn anything, I have to feel the pain of it. If you're not feeling

the pain, don't bother. But if you are feeling the pain, if deploys are really glitchy, if you keep asking for me, the kicker is if I keep asking questions I don't have the answer to, that's a concern. And if they're just mine, oh, like why did I wake up ten

minutes late today? Who cares? It's not important. But if the site's gone down for the fourth time this month, and every time the site goes down, we lose at least five grand, ten grand, maybe even more, and even worse, every single type in the site does go down we just kind of get it back up more by luck than good judgment, this kind of feeling of, oh, we kind of got away with it that time. That's okay. I know there was this weird thing and it's still

not really figured that one out, but that's okay. We'll just put it in the back blog. It's the operational risk. You've got to decide are you comfortable with that operational risk or not? Is it big enough? And in my experience, you've kind of got to hit rock bottom with this stuff as I did. There were loads and loads of bugs that I could have investigated and added logging for and fixed. But you know, it's pushing a boulder up

a hill. It's not actually worth it. And it was only when it reached my threshold of pain I was like, you know what, I have to do something about this now. This is just ridiculous. We're professional people, were being paid a lot of money and it's not working. The app that we've delivered is not working, and what's more, we don't know why. But also I do just want to add, and this may broaden out the conversation a little bit. You may want to maybe we may want to keep

it narrow on rails apps. But I've realized that observability principles go way beyond how does our web back work. It applies to any black box. So as an example, a few years ago, I was working at a company and their SEO wasn't great, and they just kind of were like, oh, you know, well, we'll try and fix it. And they had several attempts to fix it. None of them really worked, and every attempt was the same. They would get some expert in the expert would give us

a list of one hundred things to do. We would do eighty of the hundred, and then nothing would really improve. And then they'd be like, well, we did everything you said, and then they move on to another and rin some repeat, keep doing that, and then one day, within four weeks, twenty percent of the site traffic disappeared, and nobody could tell us why. Nobody understood why observability. Now, Google is a black box, so you know you're not going to

be able to instrument Google. But there's lots of tools that allow you to peer into the inner workings of Google, Zen Rush, Screaming Frog, all these kind of tools. They are, in my opinion, actually in to some degree the observability space. They're not you know, everybody thinks of them as marketing tools, SERPs,

engine and optimization tools, whatever, whatever whatever. They're allowing you to make reasoned guesses about why your searchers aren't performing the way they are, and then you can actually take action on that because now you have some data. Oh this keyword dropped from place four to place one hundred, why is that? Okay, let's try a let's try hypothesis A. Put that live and see if Google will respond to that. Oh and now up to you know, position eighty whatever

it is. So the idea of observability goes way way beyond like data dog and you relic and obviously all of those people in the observability space. But I see it as a much much wider and much more applicable topic.

what at what point do you? Like stop like that kind of mentality and be like, I will you know, every rails app should at least be able to get insight into the logs so that you can see what the application is doing, Like, well, how long do you capture that? Like what kind of time frame do you have? Any like default standards where you're like, well, I know that I'm going to need to look at this at some point in the application cycle, like what are your defaults?

more the defaults change. For a rails app that's mission critical that I'm not even going to say mission critical, but just serving a decent number of hits a month ten twenty thousand. I don't know. I've tried a lot of observability tools and there's no one that yet that I can unreservedly recommend. They all got their pros and cons. Data dog is a good option if money is no object. I kind of don't want to get into the tooling debate because it's kind of a bit of a red herring.

I think in many ways there's various costs benefit trade offs there, but in terms of the defaults, in terms of what you observe requests, has got to be up there. So every app that I I have in my care of any significant size, I would always say installed Semantic Logger. Semantic Logger is the best logger I've found out does Jason out of the box. It's quite extensible. There are many problems with it, but it's the best option that

we've got. So that's number one that will log every like Rails already logs every request for you that will form it in Jason for you. There are some notable missing defaults in semantic Logger, and I'm working on a gem at the moment that will add some even more sensible defaults into it. So For example, I believe that request headers do not get logged out of the box. Certainly request body does not get logged out of the box. Request headers might be the user agent doesn't get logged

out of the box. I mean, this is just pretty basic stuff, right, But so I have a setup that I use is the logs a whole load of things about the requests out of the box. I liked adding user ID out of the box. And it depends what kind of setup you have for authentication. But at the very very least, if somebody's logged in the idea of them should be logged in every single request. That is absolutely you know, absolutely basic stuff. A request I D

is also a really really useful one. I have a complex relationship with logs and tracing because tracing is essentially the pinnacle of observability. I hear a lot of people say logging look like logging's a be all and end all. Logging is a great place to start, but tracing is really where it's at, And I can go into that why that is in a bit. But logging is a great default. Logging is a good place to start. Start with semantic logger. Basically every single thing that's important in

any request should be logged, so that's every header. Obviously you need to be careful with sensitive data and headers like do your rails active. I can't remember what it's called, but there's the filtering module that you can add in and sometimes semantic logger doesn't give you that by defaults you need to be a bit careful. A good default as well is logging all background jobs. Background jobs are one of the most painful areas of observability that I've experienced,

and we still haven't really cracked it. We have some very very basic logging out of the box, and semantic Logger I believe it logs the job class, the job ID, and a few other things, but it doesn't log the latency, which is a huge, huge missed opportunity. And it also I don't believe it logs the request I d from

whence it was in queued. So when a job is in queued, it will by default semantic Logo will trigger a little entry in the logs this job is in queued, and it will tell you what request it came from. But on the other side, when it's picked up and the job is performed, that request ID is missing. So you need to kind of go into the request ID, find the qued job, find the job I D and then take that next leap. So I mean it's a

bit clunky, but it's manageable. So in short, semantic logo gives you you some okay defaults out of the box, but there's some really basics that it still misses. And so background jobs requests those are the two really really big ones to start out with. But as you can imagine, there are a tunnel.

dig into next. Because I have a lot of I have a lot of love for all of the standards that are coming out of this, with open tracing and things like that. I love to dig in there. But also like alerting, like you know, how does anybody know that there's ever a problem Like yeah, I mean and I love I love like thinking about it in these separate groups and categories, because I think it also helps

to think about like the overarching theme. We're just like getting insight but also like getting meaningful insight and like when you want really like the only the only reason anybody ever cares about the ability anyway is like when something goes wrong or something is problematic that causes something to go wrong, and do you want to either catch it early or you know, try and remediate and so like where do you find like, I mean, background jobs are like kind of like I feel like the first

instance where people realize, like, oh, like we need to start looking at you know, what it's doing, right, Like you start throwing stuff in the background, You're like, okay, great, like it's doing the work. And then you don't maybe realize if you're on the same node, like well those you know, slow requests can block the web requests, right and then okay, well if you split those up, finally

got that result. But then okay, well one problematic you know job can pack up a queue that it's on, you know, like where do you uh.

important pieces of like making sure that you you know, are capturing the right segments and the right flows you know in that process.

hate saying that, but it does. So I think the first question is to ask the question, figure out what data is missing, and then choose the right piece to add into your logs. I feel like I've maybe not understood your question.

had happened. Maybe the change resolved the issue, but like you know, trying to figure out what that is and going running through those questions, right, like how do you think about like reconstructing data or reconstructing the state of an issue? Like is that not the right way to go about it? Or do you try and like do something else?

Well it is because you use those every day. Yes, but you kind of missing the point. So thanks to some amazing work by people at Honeycomb and Charity Majors and reading their stuff and reading their incredible work, I've realized that the metrics tracing logs and missing the point. The point is we want to see events that happened at some point in time, and that neatly answers your question about how do you reconstruct the state of y APP? I mean, the short answer is, of course you can't.

If you're not in an event driven to the system, if you're in a crud AUP, if you're storing state to the database. There is no way you can go back in time and accurately recreate it. But we can give it a reasonably good stab. And we can do this by capturing the state of each event when it was. Forget about observability tools and logging and structured logging and tracing. Just now imagine if when that incident happened, let's say my expired token would be would be maybe potentially a

good example. There are several points in that timeline that we want to understand. Number one, when the token was created. Number two when the user hit the website, and maybe there's a third one when the account was created. Let's say that. So imagine if at each of those three points we had a rich event with everything related to that event in it. So when the account was created, we had the account i'd, the status of the account, whether it's pending or not, the creation date, the customer,

the customer ID, blah blah blah, blah blah. And then when the user visited the site, what was the request? What was the request idea? What was user I D? What was the anonymous user I D? Et cetera, et cetera. And then when the token was created, what was the expiry? What was this? What was that? What was the user I D? Okay, So if we have those three events, and we have enough rick data gathered with each of the events, we can answer your question does that make

sense so far? And there's a whole load of more blah blah blah, But does that make sense so far?

don't capture enough data, that is it. The data is lost forever if you're not in an event and assuming you're not doing event sourcing, and assuming you're not in an event driven system. So to the way I think about it at the most core fundamental level is whether it's druck logs, traces, metrics, whatever it is, we need a way of capturing those events. And more importantly, ideally we need to link the events together. And this is really,

really really important. So if somebody create let's say somebody hits our app and it creates the token, well there's two parts that they hit the app. There was a request to our app, and then in the call stacked somewhere the token is created. Those two things are two separate events, but they're nested. We want to capture that causal relationship one cause the other one is a subset of the other. One is a parent a child.

see where they fall short. See which one meets those criteria. So tracing gives us all three you can. So for those of who I should explain what tracing is, because I was confused about what tracing even was for absolutely years. So tracing allows you to when somebody hits your app, a trace is started. So there are two concepts in tracing. There's traces and there are spans, and then there's the data associated with spans. But let's just leave that to

one side. So when somebody hits your app with a request, a trace is started, and so the trace would be like, okay, I've started here, I am you can append any data that you want to me whilst I'm open. It's like opening the cupboard door and then you keep putting stuff in the cupboard, and then once a cupboard doors closed, you can't put any more stuff in it. Very simple analogy.

So we open the door, we start the trace, and so it goes down to the controller level and the controller says, oh, I'm going to glom on some data into whatever the existing traces about the method, the post body, the request blah blah blah blah blah, headers, whatever it is. I'm going to glomb that onto the current trace. And then we get down into maybe you've got a service object. I know some people hate them. I love themlah blah

blah whatever. That's not the podcast about John. So you get into a service object and the service object says, oh, whatever is in the current trace, I want you to know you hit me, and you hit me with these arguments. Cool, I'm going to append that to the trace as well, and then we enqure background job. That event gets added onto the trace, and then even more excitingly, there's a setting in open cemetry where when the job is picked

up and performed that the trace is kept open. And there's a whole load of debate about whether this is a good idea or not, but you can do it. You can keep the trace open until that job is started. And so the job says, ah, I've kicked off. Now it gloms a whole load must maybe you make an API request in the job. It gloms a whole load more stuff into the trace, and then it comes all the way back up the stat and you have this

trace with all this nested context. And when it's saying I'm going to glom this data onto the trace, that's called a span, and a span is nested, so you can have spans nested inside spans inside span. So essentially it's this big tree structure. And you might have seen this before. It's the flame graph that you get in data Dog and new Relic and all these kind of things. And everybody looks at these things and things are really pretty, and they are indeed they are. So that is the

that's the pinnacle of observability. In my head. Traces give it us all and we can say, as you can do in any of these observability tools that support tracing, you can do some really cool stuff. Show me all the requests that were a two hundred the enqueued a job where the job lasted for more than three seconds. Holy cow, now we're cooking with gas. We've got everything

that we need. Show me all the spans that indicated anything to do with the background job where it was a five hundred response but the user was logged in and and so we can start to not only query the spans, but query the parents of the spans. So you've got all of these nested cause of relationship and it gets ridiculously powerful. So that's traces cool. Let's look at logs. What do logs give us? Well, it gives us events. That's all logs are. Really, it's a series

of events that happened. Does it give us the ability to nest events inside one another? Nop, Sorry, your looks out. You can log causation IDs and you can link them together. And obviously you can log request IDs and filter everything by the request I do, but there's no concept in the log of this log is nested inside this other log. So that information goodbye, it's gone. Don't have it, but you have the rich data in every event. Let's look

at metrics. What does metrics give you. Doesn't give you the events, doesn't give you the nesting, and it just gives you some aggregated numbers. So I don't think of them as three pillars. They're three rungs of a ladder. The very top rung is tracing awesome, The next rung

down is logs. Pretty good. Metrics are useless. Now when I say metrics are useless, people get upset with me and say, oh, well, I look at metrics all the time to understand my app Yeah, okay, but if you derive metrics from higher rungs, that's totally cool, totally fine. But what's a really bad idea is to directly say I'm going to send this metric right now to my back end. And people do this all the time. People

think this is a good idea. It's okay, I mean, it's better than nothing, right, It's just depends on the fidelity of information you want. But the problem is there's two problems actually, But the main one is you've sent that data. Okay, you sent it to Prometheus data dog whatever. You sent that one data point. So then you look in the metrics and you say, holy cah, we're going to all these five hundreds. Why is that. I'll sit

here and wait as long as you want. You're not going to be able to tell me the answer to the question unless it's blindingly obvious, unless you can say, oh, well, this other bit of data over here is like correlates with it time wise, and maybe it might be that, yeah, okay, it might be that, how do you know it's that? Well, we're having to guess. Guessing is not a strategy. Hope is not a strategy. I don't really want to be bugged by just flipping guessing. I want to know, and

the only way of knowing is having traces. So the way I like to think of it is tracing is the pinnacle. Logs can be derived from traces, which is why the three rungs of ladder and everything can be derived as a metric from the two rungs above. So if you've got only logs, you don't have any nested context,

but you can get metrics from logs. Fine. If you just have metrics, I would say you're not in great shape because you can't understand why without pure guessing, and it amazes me how many people push back on this idea and think just having some metrics is enough. It's

nowhere near enough, not in my experience. If somebody wants to refute me and come on this podcast or have a chat with me after, I would love to listen to how metrics allow you to debug, like very very deliberately and get the exact data that you need and you can send off dimensions to metrics and then your metrics bill explodes within about five seconds, especially if it's high cardinality data like IP address is I've made that

mistake before. We're going to send a dimension of IP with our metrics so that we can understand what's going on. In a week. My manager usually messages me, usually in less than a week, saying, can you you can turn that off? We've just got a data dog bill of like five grand whoopsies.

where metrics alone can help like identify things. And that's more where it's like the grain aler metrics are the things that you're actually looking like care about, right Like let's say, for example, like back to the Sidekick background jobs example, like if you notice like your queues piling up, and you happen to have your dashboard of metrics just looking at Q size and looking at throughput, like you can easily say, oh, like there's something blocking it and

gives you kind of a point of where to look at in this specific instance or as an example, like also, you know, you can notice like there's a leak in memory by monitoring you know, your memory consumption of the app and just looking at the metrics for that and getting an alert and saying, why is the memory not

stopping growing after a certain amount of time. I mean these are like, you know, very specific examples that I'm giving, but like I agree, like if you're looking for like you know, it's not going to tell you, like if your users are like back to your like token expiration, like are people having a problem with our application that we've made? Like uh, you know, and like we keep getting these, you know, client you know emails coming in like oh I can't like sign into your app, Like

what's happening? You know you can just like take that and be like, oh, yeah, it's obviously the tokens like expiration, right, like it's your collect customers emails aren't going to like translate directly to that, and you're not going to know right away without having your tracing it in it in place.

there's only three jobs and the systems so cool. And maybe you've segregated your metrics by background jobs, which is fair. You know, it gives you a place to look and it gives you a starting point. But I've yeah, yeah, they're useful in the aggregate, and they're useful at giving you a hint. But and yes, they're useful in terms

of like making sure the infrastructure is still running. But I see a lot of people depending on them, and I you know, there's a guy I really respect used to work with him called Lewis Jones, and Hum and I have gone back and forth on this over LinkedIn, and he is convinced I'm wrong about this. Is like we run everything through metrics. Metrics are awesome, You're just on cloud nine if you think you can trace everything.

And there's also a significant weakness with tracing as well, which is you can't trace everything unless you've got relatively low through put or even medium through put. You can you can make it work. If you trace every single request and you're doing millions of requests a day, I dread to think what your bill is going to be. So and then that's where head tracing and head sampling and tail sampling comes into it, and we can get into that if you would like.

because I think what you're talking about is very important. Like, you know, if we're just talking about tracing through like a single request and a reels app, but it's not as useful as maybe what where tracing really comes to play is where there's multiple things that start happening, Like once you start having more than one application, and you know, the data starts trickling from one application to the other, even in sidekick example, right, if you're throwing stuff into

the background, how does that data snapshot transition through the background jobs, especially if you have ones that start depending on each other, how do you then manage the queue like and the making sure that you know where it started and you know where it's going, because sometimes you can catch a problem before it starts by having the traces in play and know where it's heading, right, And so I would love to dig into those aspects like where do you like what tooling or maybe maybe we

shouldn't talk about tooling specifically, but like what aspects of tracing are most important for like holistically looking at your system outside of like you know, running through your your question Like I think at this point we're beyond like having your questions of what you're trying to look at

and you already know what those questions are. And where do you start like setting up tracing because I know, like a doctimity we'd use open tracing is like an open standard for tracing and observability across like platforms, languages, and things like that. Do you find that the industry standards are like heading in the right direction or like where where are the pitfalls there?

can then start extending to tracing. And the reverse is not true, Like you can't take that event based kind of mindset into metrics because metrics is just that aggregation. Right. So, but I have, like recently, I've been doing a lot of queries in our rails up and I've been going to we use new relic. Sorry, we use data Dog at work, and I've been going to data dogs tracing interface and really trying to answer my questions there instead of in logging. So we have both tracing and logging.

Our tracing is hobbled a little bit, just purely because of cost reasons, and our logging is not so hobbled. So are the standards heading in the right direction? Yes, but it's going to take a really long time to get there, It's my short answer. There is a lot of there's a lot of different ways of going about tracing. The most promising, as we all know, is open telemetry. But I mean I read some pretty harsh critiques of open telemetry. There's kind of as a topic that generally

divides people. If you don't know anything about open telemetry, it sounds absolute utopia, and I got really excited when I started researching into it. The more you dig into it, the more you realize how much complexity there is to resolve, and how many challenges that project faces in order to

resolve them. And so, I mean, what it's trying to resolve is thirty, maybe forty years possibly even more of legacy software, right, because that's how long logging has been around, and they're trying to aggregate all of that into one single standard. Good look. It's a very very difficult problem to solve, and they're doing an incredible job, but it's very very difficult. So they have open celemetry is where I would start with the answer to your question. Open

celemetry is one hundred percent of the future. I've not seen anything that rivals it, and open tracing I believe came first and then evolved into open celemetriies my understanding. Apologies if I've got that slightly wrong, and so yeah, I think there's a few options if you're in Ruby, none of which are ideal. So the open telemetry client in Ruby is not ready for prime time. It's quite

behind the current standards in open celemetry. It doesn't obey any of the latest semantic standards for example I have. I've played around with it in an example project and when it's working, it's absolutely incredible. It's next level brilliant that there are a few problems with it. It's extremely slow.

So I tried to use tracing on our test suite at work using this open telemetry tracing and it just it's like I can't remember the numbers, but it really slowed down our test suite to the point where it you really just want practical to use because we were trying to measure the performance of the test suite. So you know, I could have been doing something stupid there. It's very possible that I just wasn't using it in the right way. So sorry open selemorry folks, if I've

got I know. I think a lady is called Kayley who is from New Relic and she and I'm so sorry my name the names escape me. But there's a whole bunch of people in the Ruby space who are working really hard on open telemetry. But it's just that, like the open celemetry project is moving so fast. That's the other problem. So that's option number one open telemetry.

You could maybe fork it and tweak it yourself. And the second option, and what we use at work is because we're using data Dog, we use data dogs trace tool, which is pretty good. But then even with tracing or logging, I feel like we're kind of maybe twenty years behind where everybody else is in programming in terms of observability because one of the questions I often have when I look at this stuff and even think about tracing, I maybe have like five six seven questions that even I

can't resolve, which is what do I trace? How much detail do I trace in? How much is this going to cost me? And we're still in the stone age with a lot of this stuff, So I don't have any good answers for you in that regard. So we use the vendor tooling for tracing. I'm sure you Relic has its own version of that. In fact, I know they do, I know Century does. There are certain other providers that don't have any tracing capabilities at all, So I would say for now, the best option we have

is relying on the the tracing tools. I would say, Yeah.

Stuff that's coming around. It would be nice to see something like tracing specifically introduced the rails that that would make you know, more sense in that ecosystem, because yeah, I mean where do you where do you start? Profiling? Stuff? Is like kind of like an intro to tracing, right,

like if you wanted to see like the requests. It reminds me of was a rack mini profile or tool right where you can just see a little tiny tab that says, oh, it took this number of seconds to load this particular page you wanted to get, and you can click on and expand and see, oh, well what did your application do at each step of the way and see how long each thing took?

almost never enough information. And this is why Charity Majors and the team at Honeycombe and there's Von Jones always talk about and have wide context awear events. That's their mantra, Wide context awear events and events. We've already talked about context, we've already talked about we haven't talked much about the wide. So wide means lots of attributes. So their take on it is add as many attributes as you can to every event and make them high cardinality attributes. What does

that mean? It took me about three months to outmad around what high cardinality means. It means anything ending in an ID. There you go, there's an easy there's an easy explanation. So a request ID anything that looks sorry that was me and my microphone. Anything that looks gooid like anything that is a unique identifier for anything, so that's user ID, request ID. But anything that is a

domain object. And this is the real missed opportunity I think that we have in the rails community and in observability community potentially in general, when there is when something goes wrong, even when something goes right. Let's say, let's take the Let's take the token as an example. When that token is created, the token is a domain object. Okay, it's to do with authentication, so it's not it's not really a domain object in a way. But let's say

that customer is signing up for an account. The account definitely is a domain object. And if you want to understand what I mean by domain object, I just mean an object that belongs to the domain, the business domain in which you're operating. It's a business object, a domain object.

Call it what you will. But when the CETO or the even better the CEO or somebody in marketing talks about this customer account, they talk about people creating accounts, they use that word account, that's your first clue that it's a really important concept in the domain. So that's what I say when I mean domain objects. I mean words that non technical people use to describe your app. Right, So they're domain objects. Why are we not adding every

relevant domain object every event? We don't do it, And so what you'll see is people do this kind of half heart. Oh well, we'll add the ID to the current span or the current trace or even the current log. We'll add the ID and that's okay, that'll be enough. But you're not capturing the state of the object. Why not just take the object in this case the account, convert it into a hash and attach it to the event.

Why can't we do that? Now, there's a number of reasons why we actually can't do that in some cases. If you're build in terms of the size of your event. So if you're build on data, obviously that's going to get expensive fast. But if you're build on pure events, as in your observability provider or observability tooling is saying for every x number of events or every x number of logs per month, we will charge you this much,

but the size doesn't matter. Then this is a perfect use case to be taking those rich domain objects, converting them into a structured format and dumping them in in the log or the trace, and so I've kind of thought about this quite a lot, and I've come up with a few quite simple ideas that people can use starting tomorrow in their rail SAPs, not without their problems, but the first of which is, I don't know if anybody's worked with formatted so two formatted s for daytime strings.

And we have this idea in Ruby, don't we have duct typing. We have an object, and really good O design is that you shouldn't understand anything about that object. You just know it's got form methods on it. And it can be an account, it can be an invoice, it can be many different things. So my approach and them testing this approach out at work at the moment, is instead of having two formatted s, have two formatted

h What does that mean. It means you're going to format the domain object as a hash, and so two formatted s allows you to pass in a symbol to define the kind of format that you want. So it can be short, ordinal, long, human eyes and it will output a string. I'll put a stringified version of that date in these different formats. So my idea is, why can't we have a method on every single domain object in our rails app called two formatted H and you

pass it in a format. That format could be then open telemetry, it could be any one of numbers, a

short compact. And so for every trace, the way I like to think of it is, I want to into that trace add every object that's related to that, and you could format those in open telemetry format for example, or you could have a full format or a long format, whatever you want, and so that way you can say, oh, I just want to I want a representation of the account that is short and it's just got the idea and that's a totally minimal skeleton, and that's enough for me.

But actually here the work I'm doing is a bit more involved. So I want to call to formatted H with full and that will give the full account, like the updates that I created, that everything about it, and then that will be sent to my logs and traces, and I now have a standardized way of observing what's going on with all the rich data of my app app state at that point, with all the relevant domain objects in it. So that's that's my dream that I'm

headed towards with this GEM. So that's kind of the way I think about structuring it, and I think about the people I see people doing all this ad hoc kind of well, this is this is an idea, and then we'll call the job idea a job underscore idea, suppose, and we'll what's the account. We can call that account underscore idea. And I'd just like to think of it as imagine your domain object. So an account has a customer,

a customer has some bank details. Bank details is a bad idea, but address maybe, And so we could have these different formats that load nested relationships or not. And obviously you've got to be careful about the performance problems with that. And so you will have the exact structure of your domain object in your logs, in your traces. That for me is a dream. And then every single time and account is logged, it's in the same structure. Awesome.

So I know that an account is always going to have an ID, it's always going to have a whatever other attributes, account at pending status, whatever it is. And so therefore I can say, show me every trace where the account was pending boom. Yeah.

agree that there's definitely a lot to unwind there. We'll have to have you on more if you you know, when you you know, put this together as a gem or something, because I would love to dig into that.

for traceability and things like that. I think you're on to the right, You're onto a lot here, and then.

The one last thing, and I know the time is getting on, but the one last thing I want to just say on that is the other huge opportunity is adding context to errors. So we have the exception objects in Ruby and people store strings with them, and it's like, what, how do you suppose how am I supposed to understand anything from a string? And then people try and put IDs in the strings and you're like, knows not so at work, I've made this extremely simple, basically a subclass

of standard error where you can attach context. So when you create the error, you pass in structured context. So if our logs are structured, surely our error should be structured as well. Makes sense, right, So you can say this error happened, and here was the account associated with it when that error happened, and here's a user and here's this, so it gets attached into the error and then using rails is new error handling. Rails do error dot handle. If you've not used it before, look it up.

It's absolutely awesome. It's one of my favorite things that they've added to RAILS recently, relatively recently the last few years. And you can basically have listeners to these events, to these errors. Byby pardon, it will catch the errors. And then the context is encapsulated in the error, so you can pass these errors around and then you can do interesting stuff with that context. And all I do is pull out all the context and send it straight into

the logs. And that has absolutely changed changed the way ID bug because when therever there's an error and it has all those rich data, you just look in the rich data and you're like, oh, that was the count, that was the shopify ida, that was a product ID I've got it, and then you just look up the idea in your externals. Oh right, okay, it's outsinc whatever

it is. It makes life so much easier. So that's something I'm really passionate about as well having domain objects encapsulated within errors or we've got structured errors not just structured logs.

right and say this error has these attributes like it has you know, meaning associated with the air. I think more people doing that is definitely going to be making that easier to do first of all, but yeah, and then also getting more people to take on that convention. I completely agree with you there. Yeah, I mean we are getting at time here, is there any lass you know, pieces you wanted to you know, quickly highlight or mention before we you know, move into pics.

problem is not with you, it's with the tools. So if you improve the tools, not only do you make your life easier and better, you level up everybody around you because all the engineers can use the same tools. And that's what we've experienced at Bigger Pockets, and that culture of observability has really worked its way into our culture, so that now anybody is equipped to go into the logs and ask any question that they want. So it is a long road, but it all starts with a

single step. And so if you are feeling that pain, feel free to reach out to me. And I can go through all my socials in a minute, but feel free to reach out to me, ask me any questions. I'm happy to jump on a zoom call for half an hour and help you for free. But basically, it all starts by taking very small steps towards a very specific question. Don't try and add observability because you'll still

be here next Christmas. So take heed, there is hope, and if any anything that I say resonates, please feel free to reach out to me and I'll help you figure it out.

very strange, unusual language. It's maybe not that readable in places, and it's also extremely new. I mean it's been going for five or six years, but what they're trying to do is incredibly ambitious. But look it up. It's Yeah, it's an incredibly interesting language and it will expand your mind and that's what it's so what it's done for me. And so it's kind of a language that's targeted at creating programs that are just much much simpler but actually

more difficult to get your head around. It's a completely new paradigm for distributed computing basically, and it's absolutely fascinating. So I would highly suggest check that out. I know that Dave Thomas at Uruko. When I spoke at Yuruko recently, he was on stage and he was championing Unison and he called it the future of programming, and I could not agree more. It's an incredible language made by some incredibly smart people. So that's number one. Number two, there

is a static site builder. I've used pretty much all the static site builders on planet Earth, and this is my favorite. It's called eleven T. It's a really odd name. But I'm embarking upon this project at work that really is exciting me, which is how do you serve UI components from a dynamic app rails and melds them into a static site builder without having a pile of JavaScript

that you have to weighe through. So I want to author my UI components in rails, and I want to deliver them extremely fast through a static site that's just a blog without having to run that blog on rails. So eleven T is my go to tool for doing all that stuff. It also encompass this thing called web C, which is my new favorite templating language. Yes, I know, another templating language. I promise, I promise, it's really good. It's not another retread of all these other templating languages

that are very, very niche and very whatever. So web C is compatible with web components and it's a fantastic way of making HTML like components are service side rendered. And I would love to see a plug in for that come to rails because it is absolutely phenomenal. So those are my two favorite things at the moment. If anybody is trying to wrestle with you, I components in rails and trying to extract them out to rails components.

Also would love to chat through that with anybody who's interested in that kind of area because I think it's yeah good, there's a potential to really break new ground, how about you.

and it's a lot of fun. I just did it for fun, but I recently refactored it to use the async framework and shout out to Samuel Williams just phenomenal, like so well put together. The documentation is coming along. It is lacking in some areas, but I was able to just completely refactor the code so that it works with async and runs things, you know, as they come in and it's streaming the whisper you know, transcripts. It performs actions in the background just like in the same thread.

All advantaged with ACYNC just I love it, So check out podcast Buddy and check out async. You can't go wrong async web socket now you can handle even web sockets asynchronously just like completely seamless hgd B two and one compatible. Love it, so check those out. And John, if people want to reach out to you on the web or just in general, how can how can they reach you?

this still a blog anymore? I have no idea, but that's where I write. I'm on Twitter at synaptic missapp and my get up handle is John Gallagher or one word. So yeah, Joyful Programming is the main source of goodies for me. I've also got a fairly minimal YouTube channel called Joyful Programming. So feel free to reach out to me connection, request me, ask me any question. I would love to engage with some reuby folks about observability. Tell me your problems and I'll try and help you wherever.