make podcast

Hey Ben.

Hey Matt.

I don't know what we're going to talk about today. I'm feeling kind of like we should record a podcast, but I've had a long day of talking to other people and I'm a bit low energy, so let's talk about something interesting that's going to get me excited. What do you got? Long pause. What are we going to talk about?

Well, we could talk about Make

I Love Make. That's all I was very, very fashionable to make, but it's arcane, it's awkward, it's spiky to use, but it is good at a particular job and that is primary job is to already be installed on any computer you might want to use.

That is the primary feature that Make has, is that you already have it whether you want it or not.

Very, very likely you have it. I don't actually know nowadays if an Ubuntu say machine comes with it, but I would not be surprised to find that it is already on a system. Even if I went for a minimum install, it just sits so base. But yeah, the thing I like about Make is it's sort of self bootstrapping. Once you've got make maybe Curl, oh no, you see now we've got two things, but all right, okay, but with a very minimal set of things make can build, its everything else that you need from it. So maybe you don't like make, maybe like me, you like to use CMake and Ninja and other magical, clever things to build your software, but make can make those things happen for you. Make download the binaries and put them in a bin directory and make sure that they appear. And that is my preferred way of distributing source code for the projects that I'm involved in. One of the things I covet the most is the feedback from new members of my team or folks who are just interested in the code that I've got that they say, Hey, I just get cloned your repo and I typed Make and the whole thing built just fine out of the box. I didn't have to think about anything. I didn't have to install anything. And I'm like, yes, that's how it should be. That's how software should be.

Yeah, absolutely. I love Make, because it can be used as a general purpose Tree of Tasks thing doer, which also can be run in parallel if you like. And with that, and as you say, it's just everywhere. You almost certainly have it whether you know it or not. And so you can get this sort of magical developer experience where you clone repo, you run make, maybe it prints out a little help menu of targets for you which you run, make, and then you just have everything magically installed and it all just works. And it's like, oh, how do I run my test? We write make test. Well how do I start up the system? Well, you run make start or make run or something like that.

And that factors nicely to make file thing where you're like, okay, if dot bin slash Terraform doesn't exist, then this is how to get dot bin slash terraform curl the HashiCorp endpoint and then pipe it through gunzip and then chmod and put it in dot bin and then we're done. And you're like, there you go. Alright. Now anything that needs Terraform will run that version of Terraform and it came down and then maybe you put the version number in there as well so that you can bump the version and just automatically downloads the next version and everyone's on the same version of Terraform. All that kind of good stuff,

Right? Right. And this all gets back to make, because what is make good at, but it's like make this file exist, right? Well, how do I make that file exist? Well, here's the target that describes how you bring that file into existence at this location that you want it to be. And that can be a dependency of another thing that needs that file. And that's just essentially what you're doing when you're, it's like compilation in this context is just turning curl into bits on your disk instead of compiling a file into bits on your disk. But it's all sort of the same thing in a way. Yes. It's funny. So this is something that I've been struggling with for the last week or so actually because I had an engineer on my team conduct a little experiment with his M2 Mac where right now all their development, like you say, is on Linux. We use Mac as sort of a dumb terminal essentially for a lot of the things that we do, which is a little bit of a shame. They are pretty powerful machines, but that's just how they're set up. And all of our CI runs on Linux, all of the servers that we do run on Linux, everything else. But just out of curiosity, and maybe I shouldn't have asked this question. I had him go and run our tests on his Mac and they ran in half the time and I'm like, well, we're moving to Macs

On a, sorry, a laptop M2, the tests run faster than our big beefy servers that we connect to. Yes, holy smoke

A lot faster. And this project is in Java,

Right yep Now there's a double cost to it. There may already be a cost because if you went back to it right now, it may have rotted in some small way, but it should be within spitting distance of working. Whereas there's sort of an unbounded amount of difference between the version, as you say, of grep or some command line flag that you're like, there is no equivalent on BSD oh oh, what are we going to do here?

And maybe the answer is you just delete it, you haven't used it in six months, and if you need it again, well the code's not going anywhere. You can go find it, you can dust it off, you can rewrite it.

We've spoken of this before the immutability of the history of source code. It's there to go and find again.

Exactly. Exactly.

Maybe is that an argument for those funny little tasks, leaning even more into the make is purely a thing doer and say the things that it does are very, very, very limited. They're restricted to running things like Python scripts for which we know are maybe more portable. Maybe I'm saying that out loud now, but now I'm not so sure because I remember having this debate with Java and another colleague of ours way back wanted to do a similar thing with Java when we were working on a system around the time actually when we were discussing whether it was C++ or Java, which I think was a long while ago, but

It was, yeah,

Yeah. Anyway, that time, and he was making the same point, I would love to be able to run this on a Mac or just not my desktop machine, and Java gives me that illusion, and I'm like, that's fine, but the day you come to me with a bug that's like, oh, this weird native library that we are using doesn't work at the same way on Mac as it does on Linux is the day I say stop doing that because with the particular project we were talking about, that might have been a real possibility, some esoteric networking thing or some pub sub mechanism that had a native backend that kind of feel. Whereas I think the stuff that you are talking about is more like AWS interaction, which seems like it might be the same across the two and HTTP libraries and things.

Yeah, right

So there's a little bit of, I mean I suppose I took the other side of what you are saying, which is like, well, no, I guess it's not actually, you said if there was a choice between, okay, we can't do this, sorry, we have to do this manually, or we have automation in Linux, you say, well, we just have the Linux. We can't do it on the Mac, but if you can get it to work on the Mac, then you would. No, that makes sense. But yeah, it's that thing of not knowing where the cross platform troubles lie until you run everything. I mean, I know this is going to sound weird, but what if you had tests?

Well, yeah, we started out this episode with a darth of ideas and now I'm looking at the clock going and be like, if I start talking about this, we're going to run out of time. But yeah, so I mean I have definitely done things in the past where I had a lot of shell scripts that I needed to test. I wound up testing them and then what I thought was a fairly novel way, if

I remember the way that you were tested, it might actually not solve the problem that we are seeing here.

No, it wouldn't. Right.

Do you want to talk a little bit about, let's segue briefly into your way of testing shell scripts

Yeah

And then we can talk about why it's a great idea, but it doesn't work

And it doesn't solve this problem, right? Yeah, exactly. So the way that I did this was, and I know that there are testing libraries for Bash. That exists.

Like actual regular testing things that work the way

Yes, if you want to write a yes, like a regular unit style test for Bash, you can absolutely do that. I've tried those things. I find them kind of unrewarding. I don't really see the productivity gains that I get with other approaches to testing when I test bash that way. And maybe it's just that I don't know how to use them effectively, but

That

Has been my experience.

Maybe you shouldn't be writing that complicated stuff in Bash in the first place, which is kind of the argument about the Python thing we just talked about. But let's assume that you've got enough little bits and pieces that are nothing more than grep and tail and a few other bits and pieces, and it's like, well, why would I, this is perfect. Sorry, I keep interrupting you.

No, no, it's good. And so the approach that I took here was to kind of treat bash as a kind of difficult to work with text-based interaction protocol. So I wrote a bunch of Python that generated Bash and I unit tested that Python and I basically treated the Bash like, okay, this is just the protocol that you use to interact with the operating system.

And so your tests were done at the level of the interaction with the operating system through the strings that would've passed to the interpreter.

Exactly, exactly. And so what this wound up being is you'd have a Python program that you could run and it would basically just be like run this Python program and then pipe it into Bash because it just outputs bash. And then you can unit test that Python and you could do fun things like you could just write the output to Shellcheck, and then it'd be kind of almost an integration test of the Python to say, okay, did I miss any unit tests here? Right, is the bash that I'm generating no good. And you could compose them together so you could have a Python function that generated a bash function, and then you could call a bunch of those functions to generate a whole script and then call pipe together different bash functions to do different things. And it was kind of clever and interesting, and it gave me what I wanted in terms of the confidence that I needed

To know that you were generating the right interactions with the operating system, which is perfect for what you were doing then. But what we're really talking about here is trying to develop confidence that the interactions with the operating system are in fact the correct interactions and have the same effect in both operating systems. And that's almost an integration test definitionally, right?

Yes. I mean, and there might even be differences in the behavior that would otherwise be acceptable, and it would be very difficult to know what those were a priori and say, and it wouldn't even necessarily make sense to do that. It's like if you're in a world where it's like, yeah, we just deploy on Linux and develop on Linux, why would I do anything that supported BSD that wouldn't make sense? So you wouldn't design it that way. Probably the real answer to this, and I thought about this too, is don't try to lock yourself in a room with a box of pizza and a laptop and just hack this out and alright, this is going to be the weekend when you guys come in on Monday morning, we're going to be on Macs and right now we're on Linux. The way to do this, especially because the project is in Java, is to try to move more of those operational tasks into the Java world. And then when you have shrunk it down to a level where you're like, okay, it is just make and a couple of other little wrapper things and I can very quickly verify, I can clone, I can do exactly what we talked about a few minutes ago, which is I can clone the project, I can type make and it'll run and if it works, it works. And if it doesn't, well then okay, try again. You missed a spot.

Reproducibility is difficult, but I've completely lost the thread of where we were going with that now because I got ranting about Docker as is my way. But I say yeah, you were saying about other people, if you pull down someone else's repository and they either have a Docker file or for me, the worst thing is it says, first of all, pseudo app get and it lists this gigantic list of dependencies. I'm like, I don't want to install this on computer. I already have an incompatible version of this thing and I don't to install your version of it over the top of mine.

Yes, I mean honestly, it's sort of like whenever I see that it's going into a store and they've got, I dunno pants that someone else has tried on, I realize that they're not cleaning them and they just fold 'em and they put 'em back on the shelf and it's like, this is how clothing stores work. But it's just like, I don't want to pollute my environment with whatever random packages you need for your things.

And then you're deleting large events who uses make test three or whatever make log tail. You're like, well, let's got some complicated tail minus F thing in it, but what's also as good at doing that, I can write 10 lines of Java that'll do the same thing and then do cross platform.

Yeah yeah

Write once run anywhere as Java is,

It tries to be. Actually one thing I've never done that might I wind up doing as a result of this is these days I'm pretty sure that it is possible to write a Java file with a shbang at the top and then just run it. And I had never really had the opportunity to do that. Because most of the time if I'm doing something like that, I'm reaching for Bash or something like that. But it would be interesting if some of these bash scripts that I have actually turn into Java, they can use some of the other tools that we have in the project. They could be unit tested in the same way that everyone else can, but you sort of use them as you would use a bash script. They're command line friendly. You can pipe the output.

Frankly, even if you can't do it with a shebang directly as Java, you could very easily make that generic Java shim app that just runs itself as argv zero but within inside that jar in some internal directory or more importantly, does make Java compile. And then does that thing so that you are always making sure you're running the most up-to-date version of that and not being confused. That's an interesting thought.

Yeah. Well, yeah, it's like an option to javac where you can say, I'm going to give you a.java file. I want you to compile it and run it in a single operation. And so you put that in your shebang and now you can just kind of write Java like you would any other interpreted language. Right.

So one more

I've never actually done it, so Why do you need a hundred megs of memory to tell me what version you are? Come on guys.

I mean of course I'm very much aware why, because it's a much more sophisticated system, all that, but it does still grate a little bit when you're running four copies of it and you're like, what? 150 terabytes of ram? Alright, maybe not that. That's a bit except even for our beefy work machines, 150 terabytes of ram, it might be pushing.

That's a lot of ram. I want a computer with a hundred and fity terabytes of ram.

Yeah, that's be pretty cool.

That sounds really cool.

What's the largest amount of RAM you've ever had in a computer?

A million.

A million. That's the largest number, isn't it? I thought it was the largest number. I don't even know how much ram, I'm just going to run that now. It's like one of those things where I've stopped thinking about it for the most part. I know that some of our production machines have 512 gig, which is mind bogglingly large. That's half a terabyte. That already is like, I don't know if I have that. Oh, I do have a drive downstairs in the basement that has my videos and stuff on it that is bigger than that. But most ram hard discs I have are not that big

At prev Preco. We had a database instance that had 192 gigs in it and I think that's probably the most that I can remember.

I'm now just running on this. Oh, I dunno how to use a Mac. I've got a Mac now. I know we talked about this the other day and the fact that I've finally accepted my fate that the company wants me to have a Mac. All those things. I dunno how to find out how much memory it's got. I've just type free. This is exactly what we're talking about

Right now. Yeah, right.

DF worked for me to say how much hard drive space I've got and then I did free dash h

BSD tools are like the bizarro world of gnu, right? Yeah. Everything is kind of the same but not really the same.

I've just typed memory.

It's like Solaris. Did you ever work on Solaris at all? What have we learned today?

What have we learned today? Make is a great application developer interface. An adi of you should be able to clone a project and type, make, build or make or whatever it is you do, and then you've kind of got something which any developer in your organization or your friendship group or your listenership who can go, Hey, I know how to build your software. You're like, great. We've learned that Mac and Linux machines are

Different.

Same, same but different deceptively and we've learned that Matt doesn't know how much memory his computer has anymore. What else would our listeners need to know out of this? This is what they come for. The deep, deep insights.

If you're building a tool that you wanted other people to use, please, please, please give me a single binary for it. I'll Love you if you do.

Yeah, that's a good one.

Docker sucks.

Well, you heard it here first, everybody.

Docker. I use Docker every day.

Yeah, it definitely has its place. We've said that before. Bjarne Stroustrup has a quote about programming languages, which I think adequately also applies to a lot of things, including tools like Docker, and that is there are two types of programming language in the world, ones that people complain about and ones that people don't use, and I think that's true. The only reason we complain about Docker so much is actually it's pretty useful.

Yeah, true facts.

Alright, my friend, I think we should call it there, somehow call there too. We've talked for off an hour on absolutely nothing,

Right

Maybe to ourselves because maybe we lost our poor listener along the way.

Probably so statistically likely.

It is. Actually. It's true. Yeah. Alright, my friend. Well, until next time.

Until next time.