Deep Dive into Metrics and Monitoring with Prometheus and Grafana - JSJ 645

You know, I've been working on stuff like performance monitoring and performance optimizations for a good number of years now, as you all know if you've been listening to this podcast, and probably one of the most important lessons that I've learned is that you need to have monitoring in place and alerting in place before you start any work on improving things. And in this context, I love a quote

from Peter Drucker. Peter Drucker more or less created the field of business management, and the quote is, if you can't measure it, you can't improve it. And I'm really a big fan of that. If you're unable to measure something, there's really no way for you to know if you're making progress, if you're improving things, or if you're actually

degrading things or making no impact at all. And one of the things that I've done in this context is that whenever I, like say, I join a new company or start a new project, I often find myself under pressure to start delivering improvements, you know, from the get go as quickly as possible, and I always push back on that in order to make sure that we have proper data collection and proper moneying in place before we

start making any improvements. By the way, in this context, is probably not surprising that I joined Size Sense, which is a bi analytics company because I'm really a big believer in that. And you know, as an extra benefit if you're doing this kind of work, one of the big benefits of having some sort of measurement and monitoring solution in place is that after you've made improvements, you have graphs to show the impact of your hard work.

And from my experience, that's really beneficial when you're let's say you're looking for raise or advancement or something like that. But really in order to be successful in any project that requires taking a system and trying to let's say, improve it, then having some sort of a monitoring capability place is crucial. Now the question obviously then becomes what do I actually use for that. I mean, you know, it's fairly straightforward to collect a lot of data these days,

but what do you do with it? Where do you put it, how do you process it, how do you visualize it, et cetera. And And in this context, I want to talk specifically about two things, which are Prometheus and Gofana. And first I'll start with a riddle for you, Chuck, do you know like who Prometheus was?

But the no, I can't prove it. But I think that the reason that this project was called Prometheus is because it's about bringing knowledge, because fire in this context is like synonymous with knowledge, knowledge from the gods to the humans. So it's about bringing knowledge to us developers about how our systems are operating. So that's the mythology of what Prometheus is. Now let's talk about what Prometheus

the service is. So Prometheus is free software as an open source that you can install on premises or you can use as a service I think as well that is used for event monitoring alerting. It was originally created something like twelve years ago at SoundCloud when they came to the conclusion that none of the third party solutions

for monitoring were sufficient for their needs. And after they built their system and used it internally, about four years later, they donated it to the Cloud Native Computing Foundation, which is the same foundation that also hosts the cumbernitese project. So you know, this is another successful project from that foundation. I think the people who are mostly working on it these days are the people from the company that does Grafana.

But again it's open source and you can see the source code on GitHub and there are a lot of contributors. I actually personally contributed to one of the satellite projects around Prometheus, which is the Prometheus client for node which makes it possible to connect node js to Prometheus in order to monitor NOE. So I contributed specifically specifically to that part of the project.

So it's a service used for event monitoring, alerting. It records real time metrics in something called a time series database, which is kind of a special type of database, and we'll talk about it in more detail and how it differs from the databases that most of us are familiar with. It allows for something called high dimensionality, which I also will try to explain. It supports flexible queries and real time alerting. And as I said, it's free software. It's

licensed under the Apache two license. So that's what Prometheus is. So let's say you want to use Prometheus in your organization. What you would do is that you would install the Prometheus service and then hook it up to your various services that you want to monitor. Now, it's a monitoring solution for back end infrastructure, so things like no JS or for you know, the JVM, or for something that's say written in Go. Because that's not surprising because Prometheus

itself is actually written in Go. So maybe it's a shame that we don't have aj on the show this time.

the Kubernetes cluster itself. There are connectors for various AWS services and so on and so forth, so you can collect a lot of data from third party services and infrastructure, and you can also attach it like create applicative level monitoring, so you can monitor the behavior of your own applications that are running on platforms such as the JVM, such as no JS or in Go, etc. Well more or less any programming language that you can think of. Now.

The way that you configure the system, so again not very surprising, given perhaps that it's certain in Go the configuration are yamal files, and again this is kind of correlated with Kubernetes. So Yamal, for those of our listeners who somehow don't know, is a configuration format. You can think of it to an extent kind of sort of similar to what we usually do with Jason files, but it's a different format. It has certain advantages over JSON. For example, it supports comments. It's used by a lot

for a lot of administrative stuff. So any DevOps person is likely very familiar with Yamo.

So in the yamal file, you tell Prometheus the address of the various addresses of the various services that you wanted to monitor and the rate at which it should effectively ping those services, and it effectively does an HTTP get to an end points exposed by these services and downloads data from them. So it actually pulls the data from them into itself. Now, the advantage of this approach is that first of all, they don't need to be aware of where the Prometheus server is, so it's all

the configuration is centralized. They just need to open the port, you know, listen on it, and that's more or less. It also means that they can work with multiple Prometheus service servers at the same time because they all just pull the data, because pulling the data doesn't clear the data. It's not as if, you know, they give the data and then forget it it. They retain the data. Those services are expected to retain their data in memory, so you can just hit any one of them at any

time and pull from them their current situation state. I hope that's clear.

At this point, I like the fact that it's like, Okay, I'm gonna periodically check and then I don't have ten million hits on the service on the other end, right, because I'm not pushing it to it, it's pulling it, and so it only does the work that it has to do, right Yeah, Yeah, I don't have all this extra network crap going on.

the firewall. The one kind of caveat to that is that sometimes you have like short lived services, think about something like a lambda. In that case, what they have is something called a push gateway, which is like a standalone service that those show or lived jobs can push data in. Because they are really short lived, so you can't assume that they'll hang around until they're they're pulled again.

So they can push their data into that push gateway, which holds onto that data, and then Prometheus pulls that data from that push gateway. So it's kind of an intermediary service that that you know, for those special cases. But in most cases and the cases that I've used it in, you know, it was with long lived servers or services, and then you know that's just the way it worked, right.

do anything to start monitoring basic system level stuff. Now, the Prometheus server gets the data in and then put saves it into its own percise database, right, and that database is what I call the time series database. And what I mean by that is that Prometheus doesn't store data, Like, don't think of it something like a database, you know, we tend to think of something like a relational database or maybe a low sequel database. That's not really that

sort of a thing. It's something called the time series database. So basically it has metrics, and it basically saves the value of a metric at every point in time, so it's really like keeps on collecting metric data. So there's no such thing really as schemas or something like that. It just has metrics that it collects data into, and it's data collected over time. So like think of I don't know, let's say you're let's say you're a farmer and you've got a field and you're measuring the temperature

in the field. So you've got the temperature measurement. Let's say every minute that you got from from the whatever, you know, the thermonitor device that you use to monitor the temperature, and it just gets recorded into that persistent storage and you can go backward and forward in time and look at any point in time what the temperature was. Right.

system level things and they're collected for you automatically. So as soon as you NPM install the prompt client for node and it is loaded into your project, your node server, then all that stuff is automatically collected for you, and when the promitth your server hits that port that information, those metrics are available from the get go. On top of that, you can add applicative level metrics that you

push into Prometheus using an API. So, for example, if you've got like your own let's say QUES internal cues that you want to monitor the usage of, or your own business logic processes that you want to measure the duration of, you can measure those as well. Okay, so you've got both the system level stuff and the applicative stuff. And by the way, one system level stuff that is really important in the context of no that may be less obvious or familiar to some of our listeners is

something called the event loop LAG. Are you familiar with that?

a mouse click, a keyboard press. Whenever something happens, an event is triggered and that puts the event information in a queue, and JavaScript which behaves is in a runs kind of in a single threaded type approach, pulls data out the most the top data out of the queue, processes it, and then moves to the next item in the queue, the next item in the queue, and so forth until you know. If the queue becomes empty, then it effectively idles until another stuff is put into the queue.

If HDP requests arrive at a too high rate, then then the node service might not be able to process them quickly enough and will get overloaded. Right, So, what the event loop plaque actually measures is the amount of time from when a message is placed into the queue until it's taken out of the queue in order to be processed. So If that period of time is small,

then you know that your service is really responsive. If it gets to be too high, that it means you know, your your system is overloaded and and it's not responsive enough. You know, if think about think about a service, an express service that takes I don't know two seconds to get to pull something out of the queue, it means that the browser, the client side, is waiting for two

seconds before it's it's event is even processed. So obviously that's a bad thing, and it's especially bad if it keeps on growing, because then eventually your service server will just become a response right. So that's information that Prometheus that the system, the prompt client is actually able to extract out of node and exposes that into Prometheus. So that's one of the system monitoring things that's really useful

to look at when you're monitoring a node service. Another thing that's really useful to look at is, for example, heap usage, because if you've got let's say, some sort of memory leak, then you'll see that after a garbage collection a GC, rather than going all the way down your memory, just utilization keeps going up and up and up and up and again. That if it keeps on going. What will happen is that the node service will try to do GCS more and more and more and more

in order to free memory. But to do so and you get what's known as a GC storm, or effectively, the service all it does is just try to free up memory that it can't and it becomes totally stuck. So that's another thing that you can look at in the context of Prometheus.

at the metrics. But for an app, if you're checking the metrics on a regular basis, it seems like you could start to see this lag event loop flag steadily increasing and go, Okay, we are getting to the point where we need to start looking at right, instead of all of a sudden being whoa, whoa, whoa. We're you know, we're way over the edge, right, And so it allows you to be proactive, right instead of reacting to people are complaining it's slow.

You want an alert to tell you you know, system is running hot, you should you should do something before it breaks, right, And yeah, and and Prometheus is great for that as well because you can specify alerts. And so that's the other part. So I was talking about how all the data is collected into the Prometheus service and then saved into persistent storage. But then you can also do queries on top of that data. Prometheus has its own query language. It's called prom ql, and it

looks nothing like SQL. You know, even though it's a query language, it's a totally different query language as well.

So that's one possible usage. Another possible usage is something called the alert Manager, which is another component of Prometheus that comes along with it. It's a standalone service that does regular that every at regular intervals, it runs uh prom ql queries, gets the data from them, and sees if they're if alerting criteria are met, and if they are, it can then push alerts into emails or Slack or page your duty or whatever, so that you can generate

alerts out of the promit out of the Prometheus data. So, for example, going back to that farmer and field and thermometer example, you could say that if the temperature goes above I don't know, thirty degrees on the ground, send an alert something along these thoughts.

pick up some of the patterns. And the other thing is is you can turn around and you can say, Okay, not only do I know that something's happening now, but I can go look at the current state of things or get a snapshot from the Prometheus data, and then start to solve whatever is the issue is, right, whether it's I need more resources or oh I didn't realize, but I built something into the application that makes it memory heavy, and so my heap size is going out

of control and I'm running out of memory or whatever.

the week ends, which is not surprising. But the downside of that was that data over the weekend would fluctuate a lot a lot more because the sampling size was significantly lower. When you think about it, let's say you have let's say talk about Next Insurance. You might have ten thousand sessions in a week, you know, working in a week day in a working day, but only one hundred over the weekend. And when you've got only one hundred sessions, then ten bad sessions can really impact your

performance numbers. So and then we would see that if we weren't careful, we would start getting alerts over the weekends because the data was less stable because there were just fewer sessions. So we basically kind of had to come up with queries that were also dependent on the number of sessions, not just the duration of the sessions, So if there were too few sessions, we would ignore the other criteria of the duration of the sessions. So that might be something. You know, it makes the queries

more challenging. But you want to take these sort of things into account. So quickly going back to when is Prometheus a good match and when isn't it a good match? What type of data would you want to put into Prometheus and what kind of data you would probably want to put somewhere else. So Prometheus is a good match

when you're recording pure numeric time series data. It's appropriate for machine centric monitoring, you're monitoring systems when it's highly dynamic service oriented architectures, because you know, this whole pull type mechanism makes it very easy to adjust to additional instances coming up or going down in a very dynamic

sort of a manner. And again i'll talk about it assuming we have the time something, but it's very important when the data is multi dimensional and both in the collection and the querying, and I'll explain what that is a bit later on. Now, when is it not a good match. First of all, it's not a good match when you need one hundred percent accuracy So for example, if you're looking at stuff like billing, when the numbers have to be perfect, then Prometheus is not a good solution.

Prometheus in most cases kind of averages out data because again, when you're looking at stuff like CPU usage, it doesn't really matter if you're running at ninety three or ninety three point two or even ninety four percent. But if you know, obviously, if you're looking at stuff like, you know, your taxes, you probably need to be accurate. Uh, it's not approprix.

afford to lose any data. As I said, Prometheus pulls the data, pulls the data out of the very services at regular intervals, let's say every one minute. If that service crashes during that one minute, you've lost the data since you previously pulled it. And that's obviously not something that you can live with in a lot of cases where you need persistent data and you know, if it's if you're dealing with bank accounts and stuff like that, you can't afford to lose transactions or stuff like that.

you're lucky, or look at for some other solution. Right, as I said, it has integrations for no JVM, Go, Python, Ruby, and in terms of systems, stuff like Kubernetes, git Lab, AWS, Gira, mongo dB, reddis. For visualization, usually you'd use Gofana, even though it also has its own built in simple visualization capabilities.

AH And it's compatible with open Telemetry, so if you're using open telemetry for collecting telemetry information, you can also use open Telemetry on top of Promethea and have open Telemetry put it data, put its data the data that it collects into the relevant data, the time series data into Prometheus. So there are several types of metrics and they're appropriate for different scenarios. So very quickly going over the different metric types, you've got something called a counter counter.

Think about you know, the if you're let's say, think

about a club. Let's say that you want to go into and there's somebody at the door with this kind of a counter device clicking it every time somebody goes in, like counting the number of people inside, because you know, the fire regulations only allow up to x number of people to be inside at the same time, and they so they need to count the number of people going in and the number of number people coming out to make sure that you know they don't exceed those limits.

So counter is really something like that. It's a metric that that can only increase and you basically add one or add an your end, which is like adding one end times, so you basically just add add into it and it keeps on getting higher. Can you think about things that you would measure using something like like a counter.

at least until you restart the service, right. And the cool thing about a counter is that because of this behavior of only increasing, you can compute the rate of increase, which makes it possible to have predictions because if you can calculate the rate, then you can also predict where you'll be in a certain amount of time.

be set at any value that you want. So you can literally say, say, the value of this gauge right now is x. Again, can you think of when you might use a gauge? This would be like Q size or shuck, are you looking at my slides? No?

called a histogram. Now that might seem surprising why it's called a histogram if it's used to measure percentiles for example, And I'll explain it. I'll try to explain it in a minute. But can you think about when you want something like percentiles.

So you kind of want to be able to get the measurements but then use them in order to calculate, as I said, percentiles right now, though, it's called a histogram because the way that it's actually implemented is that internally you create buckets. You can say, let's say, like

let's say you're talking about request duration. You'd say, I have a bucket from zero to ten milliseconds, from ten million seconds to fifty million seconds, from fifty million seconds to one hundred million seconds, and anything above one hundred million seconds, okay, And each measurement goes into one of these buckets, and then you look at how many fell into each one of those buckets, and then you can use it to calculate percentiles.

So let's go back again to that field example of the farmer wanting to measure let's say, humidity and temperature in the field in their field. But it's a big field, so they're not using just a single measurement device. They've got devices spread out in different locations in the field. So one way in which you might think about it is that we'll create a separate metric for each one

of those measurement devices. But the way that Prometheus looks at it is that we've got a single temperature metric, but for each measurement we associate labels with it, and that label is textual value. It might be the name or the idea of the specific measurement device. So let's say I've got four devices in the field. They're called ABC and D. I've got the single temperature metric, but each measurement is associated with either A or B or C or D. Mm hmmm is that clear?

can come in with any any arbitrary textual value. And then when I querry the data, I can say, show me just the values for the metric for measurement for when the label is equal to A or to A or B or A and B because when you do queries in the prom ql query language, you can either set a label like in the query to have a specific value or use a regular expression.

So think about again, going back to the field example, and let's say that instead of having each measurement in the field have its own aim, it has a coordinate in the field, so it has an x access, an x value, and a y value, So it's a two dimensional space.

If you've got if you've got so I'll given a real example. We were using Prometheus to monitor data associated with the with the page performance at her next insurance, and so one of the labels was the r L.

distinguish between desktop and mobile. And another label that we had was the browser the browser type because we wanted to distinguish between performance at say on chroman on Safari. Right, So think about it. It's two thousand different URLs times ten different browsers, times three or four different types of devices or five types of devices, and all of a sudden, and for each one of those three dimension coordinates, you

need to remember a metric, so it's a number. So it's millions of numbers that I need to that you're holding on in memory, and we literally kind of blew up the memory space. So yeah, so you need to be careful when when you're like Cavalier about the number of labels and dimensions that you're using. So it's called cardinality, and basically you need to be beware of high cardinality. Right. But the the benefit is that that system is extremely flexible.

Like if there's a new if there's a new r L, you don't need to do any modification in the system. It automatically adjusts to that because it just associates a label value with that particular and another label value with that particular metric. Right now, there's all sorts of things about naming conventions. I won't go into that about how you name your metrics and how you name your labels, I won't go into that. I will say that there's a really cool API for working with all the different

UH metric types. So we talked about counter engage and histogram, so you can there's a if you import the prom UH PROM Client uh NPM package, then you get those APIs that you can push your numbers with associated labels into the system and they just get recorded. Right. So again, you might have let's say, UH some sort of business logic process that UH that you know hits does all sorts of things go to it goes to one database, goes to another database does all sorts of things, and

you want to measure its duration overall. Or maybe you've got your own like custom queue for something and you want to measure the size of that queue, or you've got something like that or a pool that you want. Let's say you're you're you're utilizing your own custom pool of resources, and you want to make sure that you've

made the pool not too big, not too small. Then again you could be looking at the size of that pool and how many what's what's its utilization at each point in time, So you can, through those APIs measure your own business logic values. And the example that I gave is we were actually using it to measure core web vitals. If you remember, we've had a lot of uh yeah, exactly. Now, those are metrics that are actually measured on the browser side. So you might ask how

did they make their way into PROMI. Well, we would collect this data on the client side, post it to the node server, and the node server would use that API to put it into Prometheus. Right, So this way you can collect data not just from the node side, but also from the browser side. Again, it could be a system level stuff like corere vitals, or it could be applicative stuff. You know, whatever your web application happens to be doing that you would like to measure and monitor.

fancy name for percentiles and averages and whatnot. And it can be used to answer questions like what's the ninety fifth pers centile latency in each data center over the past months, right, you know, so very sophisticated queries, or how full will the discs be in four days? So here's an example of a predictive query, or which servers are the top five users of CPU? All these sorts of queries, and you can either use these queries in order to graph things or in order to create alerts.

for pomql. There's also on the Prometheus website there's like there's documentation and tutorials for prom ql as well, so we can post links to that later on in the in the notes. Obviously, if this was a talk, then I would actually be showing examples of a prom ql but you know, I can't really do that. But exactly so, you would put prom ql in one of two places,

either in something so actually one of three places. So the Prometheus itself has its own simple web interface where you can put in prom q ol queries and get data either in tabular as basically just numbers tabular representation of numbers or in simple graphs. If you want much more sophisticated graphs, then you can use gfana, and gfana actually has a pretty sophisticated editor for prom ql, so

it does automatic completion for you. It knows all the metrics in the system and the labels and the different label values, so it can actually do autocomplete for you when you're typing in the queries. Yeah, it's it's it's really nice. It's as I think I said, it's kind of they're kind of sister projects, so it's it's the same people working on both of them, so they made

sure that the integration is really nice. And you can also use the prom ql in the alert manager where you would type the queries in the YAMO files and then you they get run automatically at periods and if they come back with data that matches the query, then an alert gets raised. So you run a query. If it comes back no data, no alert. If it comes back with data, then there's an alert. So the an alert query might be do I the number of CPUs

that are at utilization higher than eighty percent? Right? And if the number is zero, then there's no alert. If the number is greater than zero than there would be an alert makes sense something like that, And it's actually even more sophisticated than that because you can say you want to avoid like peaks, momentary peaks, so you might say it needs to be higher than eighty percent over a period of two minutes.

this kind of data. But you know, I've gotten more and more into self hosting my own kind of thing and and running through some of this, right, and it sounds like this this is right kind of right up that alley too, where it's okay when I deploy right, it just you know, it makes sure that I have a Grafana and a Prometheus instance setup that it just reports to exactly.

then you want to use something like Centric. You know, you could theoretically count the occurrences of a particular type of error, but yeah, you probably want to use Centric for something like that. Also, Centriy has the stuff that's specifically intended for performance monitoring a certain scenarios. But if you want to do more general purpose monitoring and system level monitoring and and monitoring applicative stuff, then Prometheus could be the solution for you.

too slow mean? Because you know, it can't be absolute numbers, because if something it usually takes fifty milliseconds, and then all of a sudden it grows to one hundred milliseconds, that means it's too slow. But if something always runs at one hundred milliseconds and then becomes one hundred and ten, well that's a smaller change than you might want to

ignore it. So you can't look at absolute numbers. So we I created really sophisticated queries that basically looked at behavior over time and tried to see if something became significantly slower compared to its own specific behavior from you know, previous durations. And also again wanted to only look at those endpoints which were sufficiently used, because if there was some endpoint that was hardly ever used and we probably don't care about it, right, So yes, so you can

create really sophisticated queries like that. And it was exactly because with all these end points, you don't want to have to tell somebody a developer where well, you know, you're in charge of a service that has one hundred different API en points. You need to look every day at one hundred different graphs to figure out if there's a problem. You want it so you want you want an alert to be sent to their right to the

Slack channel. Let's say, of that particular team, if any one of those end points all of a sudden became slower in a way that really impacts potentially impacts the system as a whole. Right, and and yeah that's what that's one of the projects that I did at Next Insurance, and we use cool and we use Prometheus and go fun of that.

to something, to the next thing. So I started to tweet out the collect my list of the top standalone fantasy books, and you know what, I'll use each I'll pick each one at the time. So let's see which one one was my first? Can you think of one? There's the obvious choice, by the way, which is I think the Hobbit's I.

a lot of adults like that story as well. Oh yeah, you know what, I'll tell you another interesting story about about the Hobbit. So there's a So, there are a couple of translations of The Hobbit into Hebrew. But the

one of the translations is especially interesting. It's called the Pilot's translation because because during so there was a war between Israel and Egypt in the early in the early seventies, like in between the Sixth Day War and the Young Keeper War that was kind of it's called the War of Attrition, and various pilots were down and captured by Egyptians and held as POWs for several years, kind of like the American peow's in Vietnam, and they were looking

for ways to pass the time, and one of them got their hands on the English version of The Hobbit. The original Hobbit, and and they decided to translate it into Hebrew as a way to pass the time. So and when they were released, they they took their the translation with them and it literally got published and it's called the Pilot's Translation Translation of the Hobbit from from English into Hebrew. So anyway, so the Hobbit would be one.

So but because that's the obvious one, I'll give another one, and that that book is called Tigana. It's by Guy gravel K. Interestingly, to kind of close the loop. He's the person that worked with the with Tolkien's son on on bringing it on, you know, publishing the Simarillion. So they had a lot of notes over. Yeah, they had

a lot of notes over from Tolkien. So they actually took all those notes and and kind of rounded out and filled up the story and then released it as a Simarillion after Tolkien had died, and he worked with it on it with Tolkien's son, But he also wrote several books on his own. One of them is a

standalone novel called Tigana. You might actually find it especially interesting because it's very much inspired by by the Renaissance Italy, so the setting there, it's a fictional world, obviously with magic and you know, and stuff like that, but it's it's the situation. The scenario is very reminiscent of Italy in the Renaissance, with the little country of warring principalities and influenced by by larger external powers and and so on and so forth, and and the culture and whatnot,

and and it's it's a great book. It's amazing the amount of story and settings and scenery and characters that he was able to cram into this one book. It's a pretty thick book, but still one book. And it's very highly recommended. As I said, it's called Tigana. That's t I G A n A by Guy gabrielle K And that would be my pick for today.

last Sunday with with the kids? I don't remember anyway that I mean, there are all kinds of games out there that you can play. I'll just go with one of my favorites. It's funny because I've never actually completely played through this game. No, I take it back, I have played through it once. It's called.

make to get back to his hideout. And you play it in five rounds, and so the police deputies are trying to block him off, and so they'll go investigate different spots on the board, and there are hundreds of spots on the board, and so they'll investigate a spot and they'll either find a clue, which means that Jack was there, and Jack writes down when he moves through

a space, he writes it down on his thing. And one of my favorite things to do when I'm Jack the Ripper is actually to loop back around and so they'll find clues. They'll find clues all around this spot where I went through twice, and so then they don't know which direction I went coming out of there. But anyway, if you manage to get back to your height out after all five rounds, then you win as Jack the Ripper.

And then if if the deputies investigate a spot in Jack's there and they have to specifically say that they're staging in arrest at that spot that if they do that properly where Jack is at, then then they win.

The reason that I've finished with this guy tried to play it with my family, and it's it's kind of a longer game. I mean, it can run like two hours or longer, and so they just kind of especially my kids, just lose interest. And it's not the kind of My wife likes to play the kind of light, airy games, and this one's a heavier game, and so, you know, all this strategizing and stuff, she just doesn't love. It has a board game weight of two point sixty four, so it is on the heavier side of a game

that a casual player would play. But anyway, it's it's a fun one. So yeah, I'll pick Letters from Whitechapel as my board game pick.

in the comments as well. And effectively, it just gives you a calendar. You can buy workout plans. The ones I've bought cost anywhere from five dollars all the way up to like fifty dollars. I think the marathon and triathlon ones were more expensive. But it literally just puts the workouts into your thing. And then I have a garment for runner watch and so it just sinks it down to my watch. So all I have to do is tell it go into my training calendar, pull the workout,

run it through the watch. Right, So I did that this morning, went for a run.

that I need to be doing. Like for example, when Dan and I were talking before the show, he was like, Hey, we ought to get people like this on the show, or Hey, I was going to reach out to so and so, you know, and I was like, oh, I know them, and so as we were having that conversation, I put them into another system called Motion and I kind of I'll look and see if they have an affiliate program, but I'm just going to put the link in.

It's Usmotion dot Com is the app. And the way that this works is so you put your tasks in let me put it usemotion dot Com. Sorry I'm typing in talking at the same time. So you put your tasks in and then it pulls like it pulls from my Google calendar, and so it knows when I have

something scheduled like this episode. And then what it does is it says, Okay, I'm going to put the other tasks that you've put into Motion into your calendar and so and you can set it up so that it'll actually add them to the Google calendar, which is what I did, and then I told it not to market as busy, so they show up as free time. So it's got tasks set up for Tuesday, Wednesday, Thursday, and Friday this week as well, but they show up as

free time. And so what that means is is that if somebody books a time on my calendar, it'll just shift everything around it, and so it essentially tells me what to work on. And then finally, the last thing that I've been using is Focus Blocks and I'm actually going to do another premium episode with Manny who's the

guy that created it. But effectively, what it does is you get on Focus, you get on a zoom call and that you do a breathing exercise before you start, and you commit to this is what I'm going to do this hour. And sometimes I hit it. Sometimes I don't, and it's things out of my control, or it turns out to be harder than I thought it was going to be or whatever. But it keeps me on tasks because what they make you do is they make you put your phone away from your desk and so you

focus on it. And then at the end of the hour, you do another breathing exercise, you report on whether or not you got your thing done, and then you do it over again. And I usually get two, three, four focus blocks in in an afternoon. I try and schedule all the regular stuff in the mornings, but yeah, I'll do that in the afternoons and then Yeah, so a

lot of my like prospecting. So if I'm trying to find sponsors or stuff you know that's on their first thing in the morning, getting through my inbox is a first thing in the morning thing. So after I go for a run and things like that, that's.

So yeah, that's what I've got for picks. So we'll wrap it up until next time.