Hey, folks, welcome back to another episode of the Ruby Rogues podcast. This week, on our panel we have Ayush Nowatya Hello. Hello, I'm Charles max Wood from top End Devs. This week we have a special guest. We have David Henner. David, you gave a talk at Rails World in September talking about performance, and so we reached out because we thought, you know, hey, it's it's interesting to talk about. You're on the Zendesk Ruby scaling team, so it sounds like
you're working on I guess a non trivial application. I don't know if it's just the traffic, but it seems like it's it's an involved app. So yeah, do you want to just give I guess any more introduction that you want us to have and then we can dive in and talk about how to do performance with Ruby.
Yeah.
I gave that talk, and the ironic part of that, I gave that same well, I gave almost the same talk within Zendesk, and it was it was originally an hour and a half and then at Zendesk I had to break it down to forty five minutes. And then I thought I was dumb because I thought it was a forty five minute talk for Raals World. I found out it was a thirty minute talk. It's all so I've got a lot more content, but yeah, thirty minutes
and that's normal. Probably, Wow, thirty minutes, it's not much time.
It's so funny to me because I talked to people who haven't spoken at a conference or haven't you know, given a talk in other settings. And you know, so for example, at church, a lot of times, you know, they'll have us give like a fifteen minute talk on a topic.
But it's like, man, fifteen.
Minute talks are hard, and they're like, what, you only have to come up with this much content, And I'm like I always come up with an hour's worth of content and have to cut it down.
Yeah.
Right, there's always more I could put in it.
And so it's knowing what to take out more than knowing what to put in and then turning that into a I guess, a cohesive narrative.
And yeah, anyway, having the flow is the hardest part. Yeah.
Today it's like, oh my god, I need to have this and the original flow is so perfect that I have actually tweaked down at my company. Then to asking was like, yeah, I got this, And it wasn't until two months before the conference talk when I had to submit my papers. Oh thirty minutes. Oh oh no, let's get that done. I think it turned out fairly well
though at the end of the day. I was the last talk on the last day, right before Aaron Patterson talked, though, and I will say that is the worst time slot to have, like you're worried about your talk the whole time, and first talk that I actually paid attention to I was still kind of little amped up, and that was Aaron's say goodbye speech. Basically, it's like right, dang.
Yeah, so yeah, I've been there too. I want to just dive in. So you know what, what's kind of the low hanging fruit in the rails performance world.
I think the big thing that people like, at least I don't hear people talking about. And it's like everybody focuses on. First off, a lot of people focus on really hard things, and it's like, hey, let's change it to a micro service. It's like, oh, yeah, that might make it faster, but probably is just going to hold a bunch of crust. But the big thing people don't advertise in the community is just observability and like trying
to actually like find the things you need through. Let's say I use data Dog as endsk, but everybody can also complains about the cost of the observability and it's like, right now, you can actually make your costs lower by turning on software switches. You don't the observabilit one hundred percent of time, for example, and you can actually have like tricks around, like the observability layer just literally going out and like Okay, if we monitor something and something
goes awry, turn on the observability layer. But don't turn it out all the time, because if you do, you're going to be paying half of your infrastructure talks or or infrastructure costs to Data Dog. So I started to talk with that and that was I hope a lot of people got valuable as an dollar figures stuff out of that.
But then like, hang on, because I have a question about that. Yep, Because you know, I've used Century, I've used roll Bar, I've used a new relic. I'm trying to think of all the different ones that I've I've used over the years. I mean most of them they sponsor the show, and so I feel like I have to. I have to feel good about it being a good product before I go Okay, you can sponsor the show and I can go out and say you should try
this one. And so I've tried a whole bunch of them, but most of them you just install the gem and then it just kind of, you know, behind the scenes, does the work. And so my question there is how do you put a switch around that? Do you just put it into your initializer? And then so, lawyer, how do you handle it?
In the talk I gave the actual code, but there is a general. There is a general like, hey, it's on or off, and we always have that general thing on. So there's like the general. But then let's say in the traces that you get a stack trace and the default stat trace. I believe it is just one bar, nothing extraordinary, but you can actually turn a whole bunch of traces on within that trace.
Oh okay, break.
Down to see everything that's going on, and then you can actually identify, oh, this part of the code, here's slow, so bye by turning that on and off, like all of it on or all of it off. If you try it all on, all all of it off. I don't know what our costs that data dog would be, but I'd be willing that data dog would even say, hey, stop, uh, you're overloading us, right, But no, that's the big thing is like turning not the base layer on, it's all
the other little things they're optional. And then what those optional traces are where the valuable data is though, Like it's the stuff that you can figure within your own application that gives you way more insight. It's like, hey, I want to put a trace around this certain method because I think it's slow. And then sometimes you do that and you're like, oh, that's fast. Where's the slum part?
But if you put if you put that around enough thing, enough things, and we basically our main application, at least the ticketing inside of the application, we've we got the observability layer, like just it's covering almost everything. So if a certain customer starts to have slow response times, we just turn it on. Really, oh, we can focus on this part of the code and make the customer faster.
I got you. That makes sense.
But let's see what what am I doing now? Right now? I'm doing something that's not in my talk. It wasn't my talk, but it's also interesting. I think if you're a big company or you're a small company, you might go, well, why is that hard, And that would be we're changing the default scope for our main ticketing object. And the reason why we're doing that we're adding new columns to and it just seems like, okay, we just change the queries.
Well you change every single query. You also have to change every index from out from under the hood, and you have to never have downtime and adding aex alone on our ticketing tables, they're just so huge. You can't just say, hey, DBAs add this. They'll say no to you.
So you can lock that table for I don't know a few days exactly.
You can't do. So it is a juggling act that luckily I don't have to deal with the DBA side of it. I don't know what the problems are. And essentially, when we move from a raw two to a RORA three, it did two things. First off, it made some problems bigger, so it made this a direction we needed to go. But also have you heard of invisible indices? No, So
invisible indices is kind of what sounds like. Essentially, you have an index and a lot of times you're scared, too scared to delete any index because if you delete an index, it's gone and maybe you actually affected your application. So invisible basically says make it invisible. It's still there. The sequel is still populating that index to make it actually work, but you can see are there performance problems
with it? So we have not deleted an index for fifteen years of our application, and now we're actually able to go in and say, hey, if we actually delete these induses, we can add new ones because we the memory profile won't blow up. So that is the the saving grace of allowing us to actually do this whole project. Without that, we'd be too scared because well, having things break is scary, a lout of money.
So I'm curious.
I mean, I've deleted indices off of databases that I've worked on, but i haven't done it on like very large tables. I've added into ses on very large tables and watched the thing go ah, But deleting them doesn't seem to have that same effect. So is there a downside to deleting them other than some queries that use those indicies might slow down.
Right, and you don't know what, like like when your application is I think we have over I'm guessing to have over a thousand repos for example.
Uh huh.
You can't figure out whether you have beat down every location where every query that actually hits this. There's actually I had to go through every query in our main API and we separate our queries by the API queries, and we separate it by non API queries. Let's just that way. And the API queries is a manageable list of things that it's like a couple hundred queries to the ticketing table that actually happens. You can wrap your head around that. But the non to API queries, I
think it was over forty thousand different unique queries. Oh wow, go through each query and say would this index work with this index work with no? No, no, no no.
So so it is complex.
So what's your approach to that then?
Uh? So that luckily the things we care more about are the API queries, so we absolutely make sure that those actually are working, and the non API queries. We're actually moving a lot of that stuff to elastic search anyways. So our long term fix is actually all those things are never actually going to need a query. So, and I don't know how the magic of that works, Like, oh wow, you don't even need a query to get
the data. That's awesome. Okay, I'm guessing somebody's looking at en logs or something like that and they're throwing that information rate from the bin logs into seql. But I have That's that's magic to me.
So I used, do you have anything you want to jump in with?
Uh, haven't seen your talk. Probably probably some stupid questions. Really, Uh, what kind of like where do you normally find performance bottlenecks in like apps? You see? Because I know you primarily are you're working for zenders, you've probably worked on that product for quite a while. But I'm a freelancer, So I look at I look at a lot of code bases, and what I see and it's all it's
all rails. And what I see most of the time is people just haven't thought through the database design properly and they're doing all these funky joins or even worse, getting stuff out of the database and doing things in memory that you could quite easily do in the database. And it's like it's quite common mistakes like that is where I usually find performance bottlenecks. What's your experience in that regard.
So the worst areas are where they add funky joints because sometimes you can't solve the is very easily. The talk was actually more focused on the coding aspect, like just simple things that people do that they should know about that they just don't do. So, for example, just memorization and a little different memorization techniques. And one of the cool things I brought up a zendes that we do is we have what I it's kind of like a singleton, but it's not. It's so our account object.
If you get an account, it doesn't matter how you get that account. In the request cycle, you only need one account object. So if you call it tickets dot account, or if you call user dot account, or if you call just get me the account through this method called fetch account that we have, any way you get the account, it is the same exact object. So one of the
cool things you can do based on that. So if you get the same object, if you memoize your whatever you're trying to memorize based on that one object, you can instead of actually have a request where hey listen, I'm going to memorize it on this object. But then someplace else in the application, your memoir. You want to use that, but you can't. You can't get to that original object. We can actually memorize it on this god object, and wherever we call that method again, it is memorized.
So that was I thought, a really cool way of actually going about like memorizing something is having that god object, and then you don't have to worry about Okay, and the presenter we're calling it this way. In the controller we're calling it this way, and the models we're calling it this way. But we cannot tie those spaghetti pieces
of code together. But because we have that god object, we can memorize it in one place and always access the same data throughout and then never have to worry about That was a cool thing that actually, well that was in the talk.
But really, what are you use it for that or using current attributes or something else.
Far as making it a god object. I wish I knew that was well before I came into But I think we're basically we created a method called fetch account and one we call fetch account outside there, and then in probably active record we override the account method an active record, not even every one of our models has an account method, so I'm guessing we override that and just memoize the account in the instant variable and it always goes back to that incident. That's just a guess.
I've not actually looked at that code because I haven't needed to. And I was actually asked that at Rails World. I should actually really get into it, but the person who knows that best just left send us, so I'll have to dive into code instead of ask questions.
Yeah, I thought current doctributes are probably a bit new. Came and five or something that we.
Just upgraded to. Did we hit Rails eight yet seven? Either way? Like, it's so good to be closer to the edge. We're definitely on Rails seven for application. We're on Rails four for like six years or something like that. Who's so good to be closer to the edge.
Yeah, I've experienced that too. For a long time, I was working on like Rails four five apps, and yeah, the last year or so, I've been able to work on Rail seven and Rails eight apps now. And what's fun is you get to go back now and figure
out all the stuff that's been added in since. So, you know, I got into like stimulus and Turbo, and then figured out that oh look, Turbo actually will work through your what's it action cable channels and so you can you can push updates when you have sort of out of band anyway, just all this stuff that it's like, okay, I don't now need a full cycle to get this these features to work.
So, being at a bigger company and we started on Rails one or something like that, upgrading through there's obviously monkey patches like throughout the years because well rails didn't have certain things and now they do, and now taking those monkey patches away is hard, which also means that really adding the newest features of Rails is hard, in fact almost impossible. But what we can do is like, for example, is like, yeah, this week we upgraded to Rails three to three, and all of a sudden, the
performance got a lot better. But also, like let's say.
In Ruby three yeah, or Ruby three three.
Sorry, Rails three three yeah.
I do remember upgrading to Rails three three back in the day, but.
I remember upgrading the Rails three from two. That was hard.
But that was brutal. That was brutal.
That was the mob.
Yeah, it was yeah, it is the herb Any long before.
I don't have those, cause.
Oh I haven't.
Who'da did a ton of work on that? Yeah, it was it was brutal.
But uh, where was I going with this? Oh? I forgot my train of thought.
Upgraded to Ruby three.
Upgraded Ruby three. I don't think I'm going to remember what I was thinking about. Oh, and like, for example, with Ruby three, Uh, we had in our application. We were very scared because we allowed, we had and still do allow customers to add their own rej axes. Well, if you had your own rejects, you can really slow down and stop you're you can just kill an application if you allowed the end users. So we had some weird way of actually making it so they couldn't just
kill our system. But it spun up a process, and that's spinning up a process took about a half second second. Let's just say to actually spin up so it would run, and it make sure that rejects would never actually just be infinitely running. It would time it out at a certain point. But doing that was well added a half second every request. For example, Ruby three allows you to actually by default have a timeout in your rejexes. So
now were in that extra process up. So now if you're all most cases you're just running in rejects, that takes a millisecond or two. Those other cases that actually do time out, those do time out just by default. With Ruby, we didn't have to actually do this weird fancy trick to actually make it happen. So that was actually a very very big upgrade for us because we've had those rejects, allowed people to put rejects's in forever, and we've had to protect against it forever, and that's
just just bad. I think I would have if I was around way back then, it probably was like, no, don't let them put their own rejectses and give them an array of things that they can do. Don't give them an infinite array, right yeah, but but yeah, of our upgrades, it is just like looking at the code that's old, and people don't when it is an application is as big as it is. People cannot wrap their head around the whole ticket, save request of the cycles.
Because we have everything gets into the ticket, there's triggers, there's not just every layer, and essentially people can't wrap their head around that. So once you put traces around this, you find blatant like obvious mistakes and you just tackle them one at a time, and most of the time it's coding mistakes. It's not and it's not even a mistake. It was I got to get this feature out fast. People don't think about performance when you have to get
it out within a week. Right. But it's been good, actually, like the amount that we've saved and the ticket cycle and just not even ticket now we're actually since we did such a good job, we've been given more and more freedom to actually work in random areas of the code, and just the amount of improvements that we've made. You don't think of little coding things like, for example, scopes versus conditional associations, like a scope. Every time you call
a scope, Rails makes the query. If you have a conditional association, it makes a query and Rails by default has that query cashed up. Just making that switch will reduce the number of queries, and sometimes queries aren't very free, so all of a sudden, you've just saved time. There so many N plus one queries where that so?
So were you advocating for scopes or for.
Both? Most of the time conditional queries when when when you're trying to like tweak the last bit of performance. I would be advocating for conditional association. Okay, scopes are not evil, it's just sometimes people don't realize. You know, I'm using this in the scope within a loop for example.
Uh huh, And.
All of a sudden, I'm getting like like just query after query after query, where if you don't put it in a loop, and you if you use a conditional association, that loop makes one query and you're you're.
Done right, So it automatically eliminates the N plus one query.
Is what you're saying, Yeah, well, yes, yes, but N plus one query. I always think of n plus one queriers is the things that you can add just includes in and it just goes away. Yeah generally, yeah, but this is an N plus one query that generates itself a different way.
Yeah, if it was to do like some if I've just built like a new feature and I just want to reassure myself that I haven't made any stupid performance
mistakes or or anything like that. Like because again, like with the work I do, it tends to perform, Like performance doesn't really come into it because I tend to work with startups and stuff, like what how do I just go about like even testing like certain things in my local dev environment in terms of like what do I put in my Ruby good to check how long stupp is taking? I don't even know that. That's how stupid this question is, because you know, back to the basics.
You know, we have this debate all the time with internally because essentially the only way to find out is to have our huge customers. I don't name the customers, but we have a few huge, wile customers, and you put it on and you're going to find the problem very quickly. But then there's also the aspect of a new feature. Even when you deploy it looks great because nobody's using it right. Then two years later somebody uses it in a way that you don't anticipate. Reactive development
isn't terrible. I would say, at the end of the day, your objective when you first get out is if you write the code clean, you can clean it up later. If you don't write the code clean, cleaning it up later is almost impossible. So I would say for feature developers, focus on keeping it clean, because once it's clean and it's very easy to read, somebody else can maintain it later. Then I would say that that's the case for like
any small startup. Just focus on cleaning and a lot of different just review code and actually have an old job. I had. There's a guy he he hated me. He loved me, but I was just me because I would always review code and I would be like a stickler on making it really easy to read and maintainable and stuff like that. He was hated me, and then he eventually left that company, and I left that company, and I'm talking to old people and I've heard him say I miss Dave. Oh my god, I hated when he
reviewed our code. But now I'm at a company that doesn't review code, and the crap that comes in here, we can't maintain it. Yeah, maintainability is ninety percent of getting your speed out of it, because if it's easy to maintain, it's easy. He don't want to add the the best trick for speed day one anyways, because if the code is never used, who cares if it runs a little, if it runs a two milliseconds slower, who cares care about the features that all of a sudden
get like traction to make performance improvements? And then somebody, a team like mine is probably always required a big company is because eventually you get big enough where all of a sudden, hey ten years ago, people weren't paying attention to performance and now needs to and well, yeah, focus, done well.
To your point on this.
So we're actually running into performance issues on this application that I'm working on now for client. And what it does is it it does this gigantic calculation across a whole bunch of transactions on a hedge fund. And the people that wrote it, I mean, it's impossible to read, and so you know, you're trying to just parse through it and go.
Okay, what what is it doing?
And it's super hard, right, But the thing is is that it's it's in a psyche job. So it gets kicked off and you know, it runs, and you know, if it's a large fund, it can run forty five minutes, an hour, two hours, and you know, I'm looking at it and what we're you know, we had the conversation where we're saying, well, it'd be really nice to break this up, right so that you can parallelize some of the work because you have multiple workers, and so you
know that would make it faster. And just getting to the point where we can actually figure out where to
break it apart. Has been just I mean, it's been ugly, and you know, yeah, if it had been written in just a maintainable way, like you're talking about, where it's like, okay, I see where this kind of breaks down and you know, where it kind of goes into these methods and comes back out, it'd be way easier to do because it's like, Okay, we can turn these things into these things, and we can do these.
Things over here.
We can put these safeguards in place to make sure that it's not changing anything it shouldn't. But yeah, it's going to be a major overhaul, but it's like the core of the app, and so it's something that we're
going to have to do. And so it's just been interesting to sit there and think about, Okay, yeah, if this had been written in a maintainable, readable, understandable manner, right, we're not even talking about how it executes or how fast it is in any other way, it would have made the fixes that we have to make to it now so much, so much easier. Yeah, And ninety percent of the problem is is that since it's all in
one job, it's it does everything serially right. It does the whole process in one run without outsourcing any of the work.
Yeah, everybody's ran into that application where they have a twenty six thousand line file and you're.
Like what, Yeah, yeah, I don't think this one was twenty six thousand, but it's a few thousand at least.
I'm thinking about a very specific contract they had a long time.
Oh I'm sure. Yeah.
Oh god, my one of my co workers worked at that job too. So if he listens to this, which I'm sure you will, he'll get a good laugh out of my comment right there. Yeah.
But but it's interesting because, yeah, when we talk about performance, we don't often talk about clean code maintainability, you know, and and those kinds of things, because we're just talking about, yeah, what are the tweaks we can make?
And then when we get into it.
This is the other thing that I'm kind of pulling out of what you've been talking about, is that a lot of times the things that we're doing are not just simple fixes, right, because you know N plus one queries, it's like, okay, drop it includes.
On it, right.
You know, you want to make your query faster at an index, right, And most of the time. Those simple fixes will get you a long way, but you know, you get too far down the line, and yeah, all of a sudden, it's I have to restructure this to run a different way, or it's not gonna write in order to speed it up, or you know, there's gonna be some some massive infrastructure changes in order to make this faster, or you know, I have to change the way I'm doing caching or something like that, where it's
not just this straightforward Oh okay, here's a shortcut. You got some speed out of it, and there are those I'm not saying they're not, but at a certain point, you get to that place where it's like, okay, all the all the quick fixes on this are done, and it's still not good enough.
So now what?
And if your code is maintainable, if you know, if you can approach it and understand it and know how to change it, it makes that job a lot easier.
But it's it's also interesting because I've worked contracts where it's like I'm not I'm not going to pay you to write tests, and I'm not going to pay it a refactor, and I'm not going to pay it right and and I'm looking at them and going you realize that I'm going to be faster for the first like month, and then after that I'm going to have to refactor or write test in order to understand what the hell what this even does so that I can make it do what you want or change what it does.
M Oh yeah, yeah. Without tests, I wouldn't be able to do any of my work right. That that is the key. And we don't have perfect test nobody probably does, but we have enough tests where all of a sudden, you change something and you can feel have a relative confidence. And I think that is another thing about performance work that you can't be afraid to change things and break it, which means you need to have software switches everywhere where you're turning it on and you're seeing if you broke.
We've had extremely good luck within my UH group of turning these switches on and not breaking things. But if without those switches and while the monitors that you set up, you can't change anything because of fear, fear develop your driven development. It's like for it talking about before, it's like if you don't have like the indices that we took away, we didn't take it away because we're scared and was actually that was that before the show started.
There's during the show that I talked about the indices forget anyways.
I don't remember, yeah.
Anyways, like yeah, so anyways, feared of a driven development will like slow down performance too because you don't, as you just said, you don't have tests. You're too scared to make any changes, like U will work And I don't know. My first couple jobs didn't have any tests. And then ever since then, I've had several jobs where I came in like nope, we're writing tests. No, I don't get No, we're not. No, yeah, we were writing tests and people are but how it's like, well, let's learn.
Yeah.
Well that's the other piece on the thing that I'm working on is there aren't tests and so it's like it's this mission critical thing that does all the valuations for all of the hedge funds and all of your customers. God and yeah, I mean I was just like you you want us to refactor this, but you know, but then I'm going putting a.
Test on this is it's ugly, is so ugly.
But the flip side is is that making it testable, right, writing the test will force us to refactor some of it.
And when you wrote code without tests, typically methods all of a sudden go from I like to keep my methods within five lines of code and Ruby and Joba. I'm sure it would be different, but five lines of code, now that's not a rule, but you try to keep them small. Whereas if somebody wrote the original code without tests, usually they have one hundred let's just say, hundred line methods are not uncommon. Trying to test a hundred line method is like I felt, right, Uh, there's too many conditions.
There's like seven ifs here I do.
Yep, yeah, I I've heard that. I've heard that rule. I've heard that rule all the way up to like ten lines of code. Some people are a little bit more rigid on it than others. My rule of thumb is is if I'm talking to somebody and I'm explaining how something works, and I try and everything, that kind of happens at the same level, at the top level, right it it retrieves the you know, or maybe not
even that right. That might be part of a lower step, but usually it's it makes sure that these conditions are met right, and then if the conditions are right, then it makes sure that you know it locks the table, right, does this other stuff in a transaction?
Right?
So there's this step right, Maybe the transaction is an implementation deal further down. But if I can say these are the five steps to do a thing right, and they're all mostly at the same conceptual level, then that's what my.
Method should look like.
And then underneath that it's like, okay, how do I do this stage of the theme right? And again at the conceptual level. And so once you start getting lower into the code or when I've written it, what you wind up seeing is is the bottom levels of the methods, they'll have some things that look like their implementation right where it's like I'm actually doing the query, and then maybe there's a and then call this to do the rest of the work because I feel like the retrieval
and the work are at the same level. But you know, and then at the very bottom, yeah, it's just logic and just just being able to approach that and say, Okay, I see what the process is. Okay, I see what the process for this part of the process is. That's how I like to approach it. So yeah, so sometimes I wind up with a twenty or.
Thirty line method.
But at the end of the day, I'm looking at it and going, I understand, I can look at this and know what it does.
I think if you can't look at the method and understand what a piece of it does, it's time to create another method name.
Yeah, well that's another one.
When I first got started, I kind of had this ethos from Rails where it was like, you know, they have convenient methods called like fine or you know, update or things like that, and I was like, okay, so I've got to come up with this short, you know, concise name nowadays is like updates the thing on the thing, because then you look at it and you're like, oh, I know what this does.
I'll just make a guess on how many methods there are that we've created just on our ticket object, and it's got to be at least five hundred.
I don't know whether that's time.
Yeah, a thousand. You can't create little things, little update method and know what the hell it's doing. You need a bit Well.
It's it's also interesting because you're working on an application that mostly focuses around the ticket so everything everything touches this one core.
Piece absolutely, and we created several teams around it, right.
You know, I've worked on other projects that didn't have that, but then they had like this core two or three objects that were the same thing. I've almost never seen an app where, you know, everything's spread across everything. The only exception I can really think of is the way
that Top Endevs is put together. But that's because I deliver several different types of media and so those kind of get the between those and like the membership and role management, you know, it kind of gets spread across all of that because it's do you have access to it?
And then how do you deliver it?
And so you know, I get a wide swath of things that people are going to be generally touching. But yeah, I mean even within their little areas, right, it's podcast episodes have a ton of stuff on them, right, course, lessons have a ton of stuff on them. And and it's for that same reason, right, because within that little focused area, I've got that core piece and so understanding that and then recognizing, okay, so I've got to optimize and make it as easy to understand as possible.
Around this thing, there's some some somebody else is going to maintain it and Chris, and you're.
That's right, but I'm the boss, so that laughs Purson it quietly.
Granted I worked at a company, it will go nameless that I before I got there. There's another topper that only named this also, and they didn't have everything was mixture between tabs and white space. There's always extra white space at the end. Nobody actually had any cleanup around that. So I wanted to want to created a script that basically made it so everything spaces and there's no extra
white space at the Eddybody method. So you do it, get blame in that and everybody thought I wrote terrible code. I'm like, no, this is not my code right what it is, but get blame wise.
Yeah, we've also, Yeah, I want to bring it back to performance a little bit. Kind of hit the maintainability idea pretty hard, and I think it's it's a critical piece of this, right because you're going to suffer trying to make performance happen with poorly maintained, poorly maintained or hard to maintain code. But when you're so you were talking about observability, I want to go back to that idea,
like what are you actually looking for? Do you do you are you looking for kind of the like new relic for example, We'll break it down and say you spent this much time on your query, and this much time rendering the view, and this much time working in the controller, and this much time doing these other things. Is that what you're looking at when you when you see something slow or are you looking at some other breakdown or some other numbers, or how do you approach something like that?
Before we started the team, it was basically, this is how long the presenter is taken, this is how long the controller is taken, and various little methods throughout saying they maybe validations. When we started this team, our first
objective was, hey, listen, we need observability everywhere. So almost I'm not going to say every method, but every validator, for example, is wrapped in a data dog optional data dog tag where you can actually show really so literally every not every method, but a significant portion of the
methods are actually showing. You can see and you can narrow down exactly in the code very fast, because all of a sudden, it's like you turn it down really there it is, it's right there, and you might have like, okay, it's within this section and there's three things going on here one of these three, and there's sometimes what happens is a lot of times it's one of the three. We have the option of trying to figure out where
in the code. The slow part is where we have the option of just deploying more code that will tell us exactly where it is. And we've gotten to the point where there is some empty gaps, especially with the new features. When people add new features, they're not thinking about adding traces and stuff like that. But we can just really just go down and target exactly the method that we're looking for. And we've had things like you look at the code and you're like, oh, it looks fine,
but you don't see a loop. And then way over here there's another loop, but that's in there. And then the way over here there's another loop and that's in there too. So you got I think they call that ncube or whatnottens, like all of a sudden, like, oh, obviously we need to do something with this, and then
you add some for that specific one. I just added some memoization where all of a sudden, this last loop was just literally just it wasn't even a loop anymore, just calling it into a hash table look up and it's just a fan So all of a sudden turned it into an NQ but is still not good. But the performance of NQ versus and squared or whatever is huge. We got a big enough performance game there. All of a sudden, I could run away from that problem.
Kick it down the road a little bit.
Oh my god. Some of the times you see the code and you're like, in it's business decisions that people make too. Do they make business decisions like yeah, I can see why the developer wrote it like this, But why didn't the developer tell you, no, no, we can't do that. It's going to not perform in five years.
Yeah. Sometimes you don't know though, at the end of the day.
The one thing that I didn't talk about today or on might talk that I think is really important is when you first create an application and you deploy it, you can always set tight limits and the amount of data that you have, so product limit limits. Say, hey, listen, you can only have ten options in your drop down. That's obviously exaggeration, but you're allowed to put any ten, but you only get ten. You can always give more and eventually say I'll give you more. I'll give you more,
I'll give you more. But going from hey, you get unlimited. I remember the first thing I worked at at Zendesk was you can have unlimited brands. Well, now the limit is three hundred because we realize that unlimited brands means almost well, unlimited complexity. And you can give me one hundred thousand brands because and that's one hundred thousand domains and that's the complexity around that is just too big.
So yeah, adding limits to your product. And as a developer, I think that our responsibility to tell the product people that there's areas in the code. You go, if we have a hundred million of these, what does it? What's the performance? It's sometimes very obviously we wouldn't be able to have one hundred thousand of these right when we first developing a product. Go back to your product manager to say, hey, that feature, if somebody actually uses it
and use it a lot, it's not gonna work. And they might come back and say, nobody would ever do that. It's like, so, what's the problem with actually having a limit if nobody's ever going to run into it? So I think that's also very key, and that's not programming per se. But I think ninety or a huge percentage of our job as a programmers to go back to the product managers and telling them, hey, listen, in three
years from now, we wouldn't want a drop down. We had dropdowns for example that had over one hundred thousand options in them.
Oh wow.
First off, the UI isn't very good, but just loading them up and for example, for this one, this one was cool and it wasn't actual talk because one of the things that we did well that customer that had one hundred thousand options, they churned because well, we're too slow for them, right. We actually we probably would be able to support maybe even one hundred thousand in the options now because what we did is we changed our queries.
And our original queries was grabbing one hundred thousand active record objects. And you can have one hundred thousand active rectcord objects without a serious time penalty, right. But what we do now is we pluck for the data, so we get the data fields that we need, and we only need like two or three fields out of records, so you're not initializing one hundred thousand active record objects, so you grab one hundred thousands options right in the array.
And then you populate it via the the erase slash hash that you get back, and all of a sudden, this is very fast now. And all of a sudden, that customer that had one hundred thousand options that churned five six years ago, we might be able to maintain them now, but they're not coming back right. And that's another reason why you have a performance teams. All of a sudden, once you start have enough churn, people are like, oh, customers are leaving because of performance.
Yeah.
Do you have any opinions around using things like five ars of falcon in front of reels to improve performance? Do you have any experience with that?
I wish I did. No, Yeah, that's good. Of the answer as I can give you is yeah, no, I have not used fibers at all.
So yeah, I just started a new side project and I've put falcon and not because it's the performance intensive project. It's just a side project that lets me play around with new stuff with no risk. I was like, why not, it'll be fun.
I did that for what I worked for the UFC one time in the UFC thought they were gonna get a lot of different but it just they thought day one is gonna be bam. It's just an e commerce store too, so it's not going to get that much traffic. But UFC is like, but we're UFC. It's like yeah, but like it's still like only very small part. Anyways. Yeah, I remember I was able to play in that and it's like, hey, I'm just going to tweak every part of this e commerce app so you can go as
fast as possible. I think they launched and had almost no traffic, but it was fast for the no traffic.
Yeah.
One thing that I'm running into with top end devs is the search takes like five seconds to run.
So search is a very completely different team for us. For us, it's the exact opposite. So our search team is better than my sequel. Like whatever they're doing, they've made it index things quite fast, so we've done. There's APIs that sometimes you need. They needed to be like no most current data, so you were going after SQEL, But there's APIs where it's like, hey, if it's delayed by two hundred milliseconds, is it that big of a
deal that we don't return all the tickets. We minus all the tickets except for the one last one that they put in right, and in those cases we've found the performance of a LASS search is so much faster, like, well we'll get our response times response times, we're just dropping half for example.
Yeah, that's the thing that I've been looking at lately. Is I put in PG search and yeah, it's just you know, doing a multi search across all the models. I mean, it will give you back a result, but I had to tell the proxy server that sits in front of it, come all proxy, not to time out so fast in order to get those results. And yeah, I've been looking at possibly moving to elastic search because
I hear it's really just stinking fast. I was using another search engine, mealy Search, but the problem I had there was that they I couldn't I could only search one type of object at a time, and so I had to go and say, okay, give me all the episodes, give me all the podcast and it was fairly fast, but I hated making six queries to it to get all the different things because I kind of want just
the most relevant thing to show up first. And so yeah, that's where I'm looking at elastic search, because I think elastic search doesn't make you do that.
So yeah, we have a differ team that maintains that. So that's another place where I just use plastic search and I was fast in our system, but we have That's the thing about being at a big company. There's areas where you know how to use it within your company, but to set it up or anything to be an expert on it. No, Yeah, that's where I missed the being a startup get you get to be an expert, or at least you get to play with more tools.
Right, So, are there any other big secrets.
That you want to share with us or thoughts before we head into picks.
Nothing that's coming in to mind. But I mean, as I said, we have. I mean I'm creating a whole new talk. For example, I think I told you before we started, it's like how graph ql is going to be, uh speed up a whole bunch of things for us, and then just there's so many different areas that I was not able to hit that now I'm going to I'm submitting cocks right now to actually talk about those.
But those are not well thought out in my brain yet, So I don't think I should be a talking about right now.
Yeah, you mentioned graph ql, and I have to say my experience implementing things in graph Ql has been not my favorite thing that I've ever done. But you're talking about a performance aspect of it is. Yeah, I'm a little curious what you're looking at there. I know that with graph ql, the thing that people tend to like is that you make one query and you get all the data you need one go.
So is that where the speed up comes from or is it something else?
No? No, So we don't use Ruby every place, and our graph Ql endpoints we use Java. Then we're actually only using them internally. We're not using them externally yet, but internally it hits our Java service. Java does an amazing job of paralyzing things. So then Java makes like fifty ten probably, but let's say fifty different rest requests to gain application gets them all back way faster than if they was compiling up, and then that comes back
to the Java app. The Java app knows how to configure and put it all in the way it's supposed to put it back, and then serializes it and sent back to the requester of the actual appeatt And because of the paralyzation of that, it's so much faster because rest requests right now in our application, we have something what we call internally side loads, where you can basically say I want anything I want. It's almost like a
poor man's graph QL. Well, if all of a sudden you're doing that serially, it's just the response times here, so graftl speeding us up because it's basically forcing us to come back to true RESTful responses that are really fast.
All right, cool, Well, let's go ahead and do some picks before we get to those.
If people want to follow you online, where they find you, I.
Think get hub or LinkedIn, or you'd find me. Dr Henter is usually what you find me. If you look for Dr Henter, get Hubb or Dr Henter. I think get LinkedIn. Even Dr Henner in my Gmail would work too awesome.
All right, well, let's go ahead and do some picks. Are you sure? Do you have some picks for us?
Yeah?
Hang on one second, I have broken broad for my client, which is all the fun.
I'm flaxing.
I'm on it.
I'll give you five minutes. I'm on it.
Perfect timing, isn't it shouldn't I shouldn't have imrged. I merged up on requests just before coming onto this collision to done that.
Yeah, so I used when he got real quiet over there that that's.
What was going on.
It was like, yeah, yeah, sorry about that.
Sorry picks.
I'm gonna start with a very selfish pick, which is I just relaunched one of my side projects called Skatagun dot Email, which has been in hibernation for like two and a half years because of the joys of sales tax and EU V eight. I was using Stripe for payments on my accounters Like, yeah, you know, you can't do that because you'll have to collect V eight for everyone in the EU. And I'm like, okay, thanks for
telling me that now. So I've moved payments over to paddle, uh, which took a lot longer because I got distracted by this stuff, but it's it's back now. So Skatagun dot Email has relaunched it Tuesday. So that's one pick.
The I.
Usually do like a music or a TV show pick, So I'll go to the TV show this time. Severance on Apple TV season two just started. It's uh weird, and I like it. I like it very much. It's been a very strong start. I think it was quite a cliffhanger ending in season one, and the show has started off really strong, so if you haven't watched it, i'd recommend that.
Just getting the links here for that.
I've got a couple of picks I'm gonna put out there, so, as most of you probably know, I tend to do the board game picks as part of my picks, So I'm going to put out The game is called Cascadero, and it's.
It's got some.
Elements that are similar to other board games that I've played. So there's kind of a there's a track that you advance up, and you get rewards for hitting certain milestones, and you also get points for hitting certain milestones. The way that you advance up the track is you so there's a kingdom and the king has dyed, and so you're basically you're out there connecting the cities and things like that and spreading the word on the new king
or whatever. And so you put little horsemen down on the on the board around these towns and you're connecting up the towns, and you get bonus points for say, connecting two towns to the same color. So the first time you do that for any given color there are five colors, then you.
You get points you advance up the board.
If you are the first person to that city and it's not the first horseman that you've put down in the group, then you get one point. And if you're the second, third, fourth, I guess you can do fourth because other people can connect onto it as well, right, but it's it's per group. So when you connect a group to a city and you're the second or third, or fourth person, you move your thing up the advanced track or advance it up the track of that color
two spaces instead of one. And then if there's a herald on the city, then you add one to whatever you get to advance it. And then there's a bonus for having connected two cities of all five colors with any you know across any of or all of your groups. And then there are also point bonuses for getting all five of your markers past the first it's kind of a finish line three on the second finish line, or getting being the first person to get one all the
way to the end of one of the lines. And the way that you win is you have the most points of the people that get there their piece all the way to the end of whatever color they're playing. So when I was playing, I was playing as Red, and so I made sure I got my piece all the way to the end of Red, and then I tried to get it all the way to the end of yellow as well, but I didn't have to do that.
And then one of the other guys that I was playing with got his all the way to the end of blue, which was the color he was playing, but he did that on his last turn, and if he hadn't gotten it there, even though he had more points than I did, because I got mine to the end of the track, I won.
And so what your.
Goal is is you want to be the person who gets it all the way to the end and has the most points. But the way you end the game is by getting more than fifty points. As soon as somebody crosses that fifty point threshold, all the other players get one more turn and then it ends. Or when somebody runs out of horsemen and can't place another horseman,
that's the other way it ends. And that's the way I ended for us, because we had never played it before and so we weren't racking up the points the way that we should have been.
But anyway, it was pretty fun. And then they're like.
I said, there are different rewards up the track that you know, so you can advance another marker up the track, or you can place another horseman, or you can move a horseman or just stuff like that.
So anyway, it was pretty fun Cascadero.
It came out in twenty twenty four, so it's a relatively new game and board game Geek gives it a weight of two point five to three, which means that if you're kind of a casual player, this might be a little bit complicated, but not so complicated that you're
not going to enjoy it. I have to say that the complicated part is just keeping track of what you've connected with your groups, and then you know, keeping track of how to move stuff up the track, and you get used to that in the first time minutes or so, and then I would probably wait it once you understand those two things at like one point nine. I mean, it's it's just those couple of concepts you have to get used to and then you're good to go. It
took us about there were three of us playing. It took us what forty five minutes maybe an hour, and that was with us learning the game. So I would, I would assume that it probably takes you know, if you have four or five people playing, you know, I thank you forty five minutes. So anyway, it was fun. I definitely enjoyed it. One of my buddies is teaching the hot games at salt Con in March, which is the big board game convention up in Layton, Utah.
And yeah, anyway, so that was one of the games he had to learn.
We'll probably pick up another one later, but yeah, I really enjoyed it, So I'm gonna pick that. My wife and I for TV shows, we just started watching season two of The Night Agent and we enjoyed the first season a lot, so I'll pick that as well. Let me see if I get a link for that.
And then I am starting to.
Put together meetups and stuff for ruby Geniuses, so if you go to rubygeniuses dot com. I kind of so it took me a little longer to get it launched than I wanted to. And the reason essentially is that the stuff that I'm doing for top end devs as far as like delivering courses and podcasts and stuff like that, I've been talking to other people who want to use the system.
I'm using and so rather than do the smart thing.
And just deploy them a copy, I made it a multi tenant app so I can charge them for it. So that that took a bit of work to kind of get everything figured out and learn how to do multi tenancy with Stripe connect on you know, on my app, make sure I'm getting all the right web hooks and all that stuff. But it's basically done. So if you want to go sign up, I'm going to change the
price today. I'm going to move it down to forty seven dollars just because I'm still pretty happy about the inauguration.
Of Donald Trump and he's president at forty seven.
But yeah, so if you want to go and sign up for forty seven dollars, it will go back up to ninety seven dollars in a few weeks and that'll give you access to I'm essentially I'm putting together videos kind.
Of like railscast was or go Rails or Drifting Ruby.
My focus is going to be essentially I'm going to say here's my project, and then you're gonna get a video of me working on the project every week, and it might be interspersed with and this is something that
I fixed on top end devs. Right, So if I get elastic search working for the search for example, I might work that in the first app I'm building is actually to help track bills through the Utah legislature, and so it's going to be hitting their APIs and working around the gaping holes in their APIs.
Because they have gaping holes in their APIs.
So there may be some screen scraping and things like that to get all the data in and then yeah, putting together the alerts and stuff like that, so you know, just you know, you get a notice that says, hey, your bill went to committee, or you know, your bill had its second reading, and so if it has a third reading and passes, then it passes, you know, and just kind of working through all that stuff and then helping people connect with their legislators because I feel like
if we have more people involved in the legislative process and talking to their legislators, regardless of which side of the aisle you're on, I think.
We generally get better.
Legislation because I think most of the things when we're talking about roads and schools and things like that, we generally agree on what we want, whether you're right or left, and So what I want is I want moms and dads talking to the legislators and saying you're passing, you're pushing this education bill, but that's not what we want, you know, or things like that. So so that's what I'm going to be working through and just showing people
how to do that. It's probably going to have some premium features too that involve AI and doing bill summaries for AI. I'm already doing this for another project, and so yeah, you're going to get kind of a wide breadth of things out of this. Also creating groups so people can discuss bills together. I want to give organizations an option for putting together their slate of bills and where they stand on it, and give them some way
of posting that on their websites. So those are kind of the features that I'm going to be putting in so you know, imbedible stuff. So anyway, if you're looking to learn any of those things, then then you want
to sign up for that. We're also doing the meetups and we'll be going through some of the technologies, maybe a little more in depth, so where in the videos I'll be showing you I'm building this feature using Turbo, I may go through, Hey, here are all the features that turbo offers in this way or comal or I've had people ask me for all kinds of stuff. If you connect with me on LinkedIn too, I'll actually get on a call with you and ask you what you want to learn. So anyway, I've been way more long
winded on that than I want. But yeah, so that's all going to be coming out through Rubygeniuses dot com and yeah, so hopefully that's in the wheelhouse of things you want. But yeah, we're going to be doing meetups and tutorials and screencasts and courses in the whole nine yards and you get access to all of that through the membership.
So anyway, David, what are your picks?
Yes, So, I have a coworker actually that I work side by side with that's publishing a book soon and it's called rail Scales. He actually gives more talks about scaling rails than I do. But Christian Polantis is his the author's name, book name, Ral Scales. I'm actually very interested in to see what it looks like and I can't wait to get myself a copy. I hope I get myself a free copy.
Yeah, become a technical reviewer. They give those out for free.
Well, I mean he works with me and he's yeah, I'll tell him. I gave him a shout out. There you go, you go. What else? I actually have one other coworker that he wrote an article on AWS about scaling Zendesk And if you do a search for the guy's name is Anatoly aws Zendesk Performance, So Anatoly Aws Zendesk Performance, you will find his article in Google and he talks about some of the things that we've been doing.
Much more on the infrastructure side. He's a much more of an infrastructure than a Ruby guy, but I work side by side of the He's absolutely brilliant. Also, if anybody's hiring product manager or managers in general, I know several that got laid off, So feel free to ping me on LinkedIn because I feel like there's some good people out there that I know that I'd love to
connect to other people. And lastly, you guys probably don't know Swift, but Swift is essentially an interactive game that you connect to a bicycle a bicycle trainer, and I absolutely love it. I'm I mean, I every night like I do races sometimes and the last few nights I've been doing races. I am not a racer. I am not that fast, But at the end of the day, I'll still go in and do the races. And I think the category ds I'm taught. I'm I'm the best of the worst category.
Oh, there you go.
So as soon as I get the next category, I'll get destroyed. Yeah.
I got a membership to Swift when I bought my bike trainer because I have a Wahoo trainer and or.
The is it the bike? Okay, I got the Kicker bike.
Okay, Yeah, I've only used it once or twice. In fact, I think it expired.
But yeah, it's cool because it's yeah, it's almost like, well I guess it's not almost like, but it's it's kind of fun to just ride your bike.
And yeah, if you're.
Harder up the hills and all that stuff.
And exactly, and then that you're in Utah, so you you suffer from the coldness and you don't want to be biking.
Oh yeah, I don't train outside. I'm training for a triathlon now this year. I want to do either an iron Man or half iron Man.
I haven't decided I'd do that with you, but I think my knees would say no on the running. In fact, I got an MRI scheduled for next week from my shoulders, so man, I'm breaking down. Yeah, and last I.
Watched my dad fall apart, and I just I feel like I need to keep moving, so I don't.
So the last day I was going to give this my last thing to saying go Bills because I'm hoping for the Super Bowl. But well I still say go Bills because I love them.
Okay, so have you decided if you're rooting for the Eagles or the Chiefs?
Have I decided whether I'm watching the Super Bowls?
Fair enough?
You know, I don't hate the Chiefs, but at the same time, I got to root for the Eagles just because they I don't want the Chiefs to win three times in a row. They're not bad, They're not the Patriots where I hated them, but still I don't want to win three times in a row. And I think that their fan base understands that too.
Yeah.
I like both teams, so I'm not sure or which way I'm swinging yet, but.
A right good deal?
What was that was amazing? This year?
All right, well, I'm gonna go ahead and end the stream, but thanks for coming, thank you, and until next time, folks max out