Are you set?
I'm good.
Right? Then we should sing this theme tune. Do, do, do, do, do, do, do, do, do, do, do, do, do, do, do do. Bling. Bling. Bling. Bling. No?
You're you're gonna have to do that now. That's gonna be the episode intro.
Oh no.
Is you singing the,
Hey Ben.
Hey Matt,
How you doing?
Great.
It's another Friday. It's another podcast recording day. And you've you came to me with an idea already.
I did. I had a, I had a thought, which is
Most unusual.
Async-await, and asynchronous programming in general and why we do it and threads and all the things there's a whole, this topic could go on forever, but
Like all of our topics really they're deliberately vague because, um, we haven't really thought about them very much, so, that's uh, that's the thing. So, yes, that's interesting. You said, async, await. Is that how you pronounce that? Or would you, cause I would say await await, and I wonder if that's a British English thing. You know, you await something rather than await or is it because you said ay, cause I wouldn't say
That's just me wanting there to be, uh, alliteration. Is that what that is?
I mean, it's alliterative anyway. Right? Cause they both have an "a" sound at the beginning
But I'm trying to emphasize the alliteration.
Empha-size, I think it's pronounced So we better say what that is. What is, async await
Async await. Um, well it's a, it's a, it's a programming style I guess is what I would call it. Yeah. It's a technology. It's a programming style. It's a floor wax. It's a dessert tapping
What's that from?
I think it's like an SNL bit from back in my very, very, very early days. Um, but yeah, it's, it's it's well, it's a solution to a problem actually. And we should talk about what that problem is. Which is if you have a program that has no threads and has no async magic, there's just a series of instructions. Uh, and that program does things, uh, that are IO bound. Like for example, fetching a bit of data from a remote server. Or, uh, you know, performing a database query or even reading a file.
Right.
And it is the, the progress of that program is, is blocked on that information. It can't go forward until
You've retrieved the information that you need. Then your program is sitting there. Or the CPU usually is sitting there doing not very much while you're waiting for this, you know, favicon ICO file to be downloaded from a server somewhere, halfway across the world, uh, or whatever it is that you're doing. Um, and so it could be doing a lot more and you, you could be getting a lot more out of your computing power by not simply just waiting for it to do that, but by running other parts of your program or a different program or other things. Well,
Yep
So that's taking the other side there just, just quickly, obviously in a modern operating system, the operating system is gonna have other things to do. You know, it's got animating gifs to, to keep moving in the background. It's got, you know, dancing hamsters, and the like, um, or other programs will take advantage of your CPU. But if, if you, as in the user, are running a single program that has a single thread of execution and it is blocked then yeah. You're gonna have to wait. And if there's other useful things that that program could otherwise be doing well, they'll happen after the file has come down, not before. So obviously the normal solution to this is like, I'll make some threads and then I'll have all these complicated, um, mutex and work queues or something, something, something. So is that async?
Yeah, no. So I, I think threads are one solution to this problem and async is maybe another solution to this problem. Um, but the underlying problem, I think, is that, uh, for a single sort of program, you wanna be able to maximize, uh, CPU time, reduce latency, increase throughput. All of these things can be achieved by just making better use of the CPU. And, uh, one way to do that is by having threads, another way to do that is with async. Um, and I think that either way you go, there are some sort of like head twisting complicated problems that you can run into. Uh, some might be easier to, to deal with than others. Some might be easier to test than others, but you're, this is the classic programming technique of solving one problem by creating another problem.
Regular expressions. For example, fall into that category.
If the, if the R value of your problem growth is less than one, then you've done a good job
Yes.
But I suppose the, this hypothetical program that you said it's fetching something or reading a file or doing whatever the assumption here is that there are other things that that program could be doing in order to continue. I mean, let's say it's a word count program and we've given a list of files and it's gonna open up the files and like naively, I would write for each file in list of files, open file F read every line, count how many lines there are, add them up and print it out at the end. That would be like, a not unreasonable way to write that piece of code, but for all, you know, there's multiple drive spins. Some of these things are on the network or whatever, and, you know, you can easily saturate, uh, the PCI bus or the network link or whatever, if you were doing more than one file access at a time
,
But you're not, you're just reading one after another mm-hmm
Yeah, yeah, I think that's good.
So, I mean, if it was, uh, this, this, uh, word accounting thing then in, in, uh, one way you could spawn one thread per, um, job per file, and then
Let those threads run their separate ways and then sort of have some kind of collect all the threads together at the end, and look at the answer and add those all up. That would be a perfectly sensible way of doing this, but in the async await world, you need a framework that understands how to do input output operations that, um, can sort of cooperatively multitask. I think that's the key here is the threads means that we're actually using operating system resources to create multiple execution threads at the operating system level. There are literally potentially multiple CPUs could be involved and these things could be running at once on multiple CPUs, but that's not the kind of problem we're trying to solve here because the CPUs are, are not the problem. It's the waiting for files. That's the problem. In the async await world, you are scheduling pieces of work with callbacks
Somewhere deep down in the in the bowels of the, the, uh, the, the framework you've said open the file, but just call me back when the file's opened, don't get don't block
Mm-hmm Don't block, like open the file and read this stuff, but don't wait until you're done reading it to return from that call. Right.
And that's really the, the, the trick behind the back of all of these async await is that now you suddenly have coroutines that are powering this, uh, system of, uh, sort of cooperatively, multitasking, uh, all the different things that you want to do in a single operating system thread one after another, as they become ready, like the file contents become available, and now your, uh, your, uh, uh, your code can operate on it. And then let's, let's talk about like, like actually what the, the syntax of that looks like you typically, um, hence the async await is that you tag functions of your program as being async, which is a big hint to the framework or language that you are in to say, this function can be suspended halfway through and can sort of return early in some way. Um, and then when it hits await within that, uh, async function, what we're saying is, um, do some work, but actually me here and when that work is ready, come back to me.
Yeah. Yeah.
I think the trick there is that that sounds a bit like a thread and logically it is a thread of executions. There's like a, a stack and as a sequence of, of, of instructions that put that, that pertain to a single kind of, um, idea that you're doing a function that you are running, but there isn't an operating system resource associated with it. There's literally just a big list somewhere in the framework of here's all the things I know I need to do when I dunno this file read has completed, or this network access is done, or, um, or somebody deliberately yields and says, Hey, I, someone else's go to, to, to, to run now.
And quite often, those things are implemented with an event loop, right? So you'll have basically a queue of events. And then you'll have basically just like a while true loop that is consuming those events one at a time. And then calling back to the callbacks that, that, that, um, are related to those events. So, you know, you might, um, schedule a, a file read and then take in a take that event and, or, you know, take something and say, okay, when this file read is done, uh, generate an event, put it in the, in the, in the queue. And then when the queue, when that loop sort of processes through all the events and it gets to that event, it says, oh, well, we did this file read. It actually finished about, you know, 200 milliseconds ago, but we were processing other events at that time.
Uh, now we're ready to process it. So I'm gonna call back all of the people that were interested in this and tell them, Hey, your, your data from this file is ready. Now you can do what you wanna do. And this has the advantage, like you say, of being able to run in a single operating system thread, um, so that you don't have to worry about the things that you have to worry about when you're doing multi-threaded programming. You don't have to worry about synchronization. You don't have to worry about threads, you know, clobbering each other's data. And in that way, it's actually much simpler because there's just this whole set of problems. And, you know, I, I will tell you from personal experience, loving tests and wanting to write tests for these kinds of things, it is quite difficult to write tests for multi-threaded code that truly give you confidence that the multi threaded code actually works 100% of the time. Um, it's quite easy to write tests that convince you that it works, uh, and it only works 99% of the time. And then you find, oh, actually no, yes, there is this one case that I didn't think about that it can not work. So from that standpoint, it has, you know, it's kind of attractive.
Absolutely. I think that's a really good way of, of phrasing what's going on. Like at the nuts and bolts level, there is, as you say, an event loop there, you know, on a, a Unix based system, and there's probably a select loop, that's got all the file handles of all the things that are going on. And then as they become ready, as you say, things are woken up and, uh, described like that. It puts you in mind of say how JavaScript and Node.js do all of their work with, with actual callbacks. You know, you say file dot, oh, open file name, comma, and then call this function when the file has opened. And then you end up with this, you know, very deeply nested, uh, callback, because, you know, once the file's open, your, your function is called, which then wants to read. And then of course, that's another asynchronous call.
And before, you know, it, you've got like 18 levels of indentation, or you've got a 200 disparate, tiny little functions, each of which then cause another tiny little disparate function. And that was how JavaScript was for the longest time. Async await specifically. So you can have an event loop, you can have this callback type thing without async await, but async, await is at a language level feature, which hides under some of syntatic sugar that callback based thing by kind of writing callbacks for you. When you say await file.read, what's really quote happening is something like a, the, the function that you're in the middle of is cut at that point and turned into another function, two functions like the, before the await, and then the bit immediately after the await and the bit immediately after the wait is essentially turned into the callback function for the file dot read.
And so you don't need to think of it in these terms. You don't need to nest things further and further and further your code for an individual. Like I'm the line counting function just says, uh, file equals await file dot open file name something like async for, uh, line in, uh, file dot async readlines count++ return count. And so you've just written code. That looks almost exactly the way that we described for the single threaded case with just a couple of magical keywords, like tagged in there behind the scenes. The whole thing is rewritten to be, be callback based or futures based or some of the other techniques.
Promise based.
Promised based, exactly. Yeah. Um, and it makes for, as you say, a much more testable design, because it is not subject to the whims of operating system time slices or, um, multiple CPUs actually executing multiple code paths at the same time, which threads would be, it's only when you say await, blah, does, does your logical flow of instructions stop and someone else could potentially get the use of the CPU. So you kind of have to be aware of that under more advanced, um, techniques, like share a cache, um, for a class and you have multiple, um, like async awaits of your class going on at the same time, you have to be aware that every time you await something, then potentially the cache could be being used by, by someone else. But it's so much more controllable and it is deterministic, which I think is the key for things like tests.
Yeah, yeah, yeah. And there's still, you can still get things like race conditions in both paradigms, right? Like you can have things that are racing, they're either racing across multiple threads or sometimes, uh, racing to the top of the event loop.
Yes.
Right? Like, depending on, on, uh, which events get into the queue first, you might have code that's executed in different orders.
Right. And oftentimes frameworks don't necessarily specify the sequence in which like, if there are two things that are ready at the same tick of the clock
Yeah. Yeah.
Who goes first, in which case, you know, again, that can be something that can be exposed by tests where you're trying to puppetter time and the completion of things.
Yes. Usually a lot of, a lot of, uh, both languages and frameworks are intentionally vague about how that's gonna happen.
Right.
But yet, sometimes accidentally consistent, which makes you, which sometimes will trick you into thinking that you have code that runs properly when actually that it, there are cases where it doesn't. Um, yeah. I, I think what's interesting is, is that there's cert and you sort of touched on this a little bit is that there are levels to this. There's like the, just like, I'm just gonna do a non-blocking call with a callback, right? Like that's like the most basic level of this is there's no async await, syntax sugar. Um, there's not even necessarily like an event loop. Um, there's just, I'm gonna, you know, the sort of select statement is sort of the classic example of this is I'm, I'm just gonna do a non-blocking IO call and I'm gonna have some facility that, that allows me to say, when this is complete, call me back over here. But this initial call that I'm making, I want you to return immediately, right?
Yeah.
That's like the lowest level of this
Good old EWOULDBLOCK in Unix, you know, like, Hey,
Right. Um, and then I think one level above that are things like futures and promises and things that are not, you're not doing anything special in the language. You're not introducing new keywords. You're not doing anything there. What you're doing is just creating, you know, objects, essentially
Placeholder objects.
Yeah. That allow you to make it a little easier to work with these kinds of callbacks, right. Where, you know, you can call a function, the function will return immediately. You'll get some object back that represents the sort of thing that you've scheduled to happen. And you can interact with that object in interesting ways. Obviously you can just, you know, apply a callback and say like, Hey, when you get my data, let me know. You can also gather a of them together and say, when all of these are complete, let me know. Um, you can gather them together in ways that say, if one fail, we'll then call this function and if some pass and call this other function, um, and you can do interesting things with that as well, but that requires no language changes or language support or anything like that. That's just, you know, objects essentially.
And then a level above that is going all the way to sort of full sort of like, yes, we're gonna introduce new keywords into the language. Um, we're gonna bake the sort of asynchronous non-blocking IO things into the standard library of the language. Um, and we're gonna make this sort of like a first class citizen within the language. Um, Node.js obviously is a great example of that, cuz they just took something that had already been in web browsers and we're like, all right, well we're just gonna lean into this real hard and start building other things out of it. Um, obviously you and I have written a lot of Python recently and Python three six?
Is that something yeah,.Three and a bit?
Three, herm-herm Is when they introduced the sort of, um, you know, async asyncio, uh, that introduced those keywords into the languages into the language. Um, but you know, you can also just use like a third party framework. Like I certainly used, um, Event Machine for Ruby for a good long time, which isn't really async await. It's more of that sort of second level that I was talking about.
Right, with, with futures and adding callbacks to the future or something or um,
Yeah. Um, it's really it's it's like there are futures, but it's it's, it's like, it's like one and a half even
Same in Python. Right. You know, you've got like, you know, Postgres and aio-postgres for the same reason. And I think that's the, it that's one of the more insidious is a little bit of a bad word to use, but like one of the things that happens when you start having async libraries is that you get this separation and because it's, it's all pervasive, anything that, that ay library calls that itself might need to be ay also needs to be async. Yeah. And that's like keyword level down. Right? Like, you know, so you, you with, you know, read file and async read file because they are very different operations. One of which returns the contents of a file, the other one returns essentially a, the promise or a future of a, uh, a file. And so yeah, you end up with this a bit like, so like a, I say this in like C++ once you start introducing const correctness, one place, suddenly everything needs to be const correct. Because you can't, it won't let you not be. And it's the same with async, as soon as you get to a point where you can't do something asynchronous anymore, you're like, well now we're we're blocking again.
Yeah. Yeah. We had this running joke for a while there that it's like the glitter of the programming world once you start using it like, it just gets everywhere. Right? Like you can't, it's got a
It's got a half life of uh, yeah, measured in millennia. You never get rid of that last grain of glitter. Or sand from the beach.
Never really Yeah. Microplastic it's the programming microplastic.
Too soon.
See
Suddenly everything everything's async. And you, you feel like you're just scattering the async keyword everywhere and that's not without cost. Um, but before we move on to like things like cost and how it's actually what's going on under the hood a little bit more, I was gonna sort of make the observation that I don't think I made clear when we were talking about the, the word counting example, right. You know, we've made written these like individual async functions that count the number of lines in a file by async opening it and then async for each reading lines or however that works. But then you mentioned gather as like a primitive operation and that is, is a key to actually unlocking the, uh, multiple outstanding IO requests at once.
Right
So in this instance, you know, one could write the code, uh, uh, at the top level that says for I in each of the files, I want to word count, ay, await, that word count of file zero, you know, to running total plus equals that.
Um, but you would still be back in the world of single, single, uh, file happening at once. You're just doing it asynchronously and that event hasn't got any other work to do. So it'll just sit there and go, okay. I just, I'm literally in select now blocking for that file to become ready. And then I call you back and then you are no better off if you're slightly worse off, but the gather primitive operation, you talked about takes a bunch of future objects of some description, be they promises or futures or coroutines or async whatevers. And it says, okay, all of these, we wanna kick them all off at once at once. Again, there's not any threat. So it doesn't actually happen all at once, each one is run until it gets to a point where it yields. But then I come back and do the next one, then the next one, the next one.
And now I have multiple IO requests in flight. And then whichever one of those comes back first I will service and it goes into the event loop. And now that, uh, file is open. And so that, um, coroutine is starting to process the first file or the second file and so on. So, um, that is the equivalent of spawning threads at that point. That's how you get the parallelism is, uh, by, is by gathering a bunch of, uh, async awaitable things. And I think that's, that's another thing to bear in mind is how, how coroutines, um, interact with this because languages that don't have coroutines or something like coroutines make it's it's, its a diff becomes a difficult thing. I dunno actually you mentioned Ruby and I don't have any, um, experience with, with Ruby specifically. Uh, I know Python had coroutines before had the async and await and they were able to be like library level solutions to async await, you just using coroutines. And I know that the async sort of decorator is a spin on always forcing an object, a function to return, uh, a coroutine as opposed to being a function call in it's own right. So does Ruby have a similar thing? How do or was you did mention it's like level one and a half, so maybe it didn't have the same.
Yeah. I mean, if it does, I'm unaware of it. When I was using it Event Machine was, was entirely like a, a third, third party thing.
Right. It was not a language extension, so it didn't need
No, not at all.
coroutines. And again, another sort of aspect that, that makes, uh, writing this, this kind of code, um, more tractable is, are languages that allow you to do arbitrary captures into lambdas and unnamed objects and things. So either you can use those as the callbacks and not worry about, um, who owns what, uh, or, or, um, you use coroutines themselves. So yeah, it's it. I suppose what I'm winding up towards here is that why do other more low level programming languages like C and C++
Yeah.
And you know, really, I mean, so the, the, the trick there is coroutines are a huge unlocker. They're the thing that allow you to suspend a function and, and carry on again, a bit later on in that function. I know I described it as it being syntactic sugar for like cutting a function into two functions. And there is a way you can do that. And I believe the first version of C# that introduced this, had that kind of rewriting technique behind the scenes. It was just literally rewriting the code as blocks of, of, um, of, um, follow on continuation functions, one after another. But, um, modern times, um, these are coroutines that can yield. That means they literally say, Hey, I'm not finished yet, but I'm gonna return a value to my caller. And that value to the caller indicates that, yeah, I haven't finished. So go do something else and then you can call and then it will continue from where I left on. And in that traditional classic coroutine, um, sense um, but, um, the problem with that is that you need to be able to keep around all of the internal state of a function between calls to that function, you need to able suspend and resume.
Right.
Go going back to our original, like operating system level threads. That's what the operating system knows how to do and it's its meat and potatoes. It knows how to suspend a, a, a thread and page in switch in a new thread, you know, restore all the state from that. And, and at an arbitrary point, it's kind of like a hostile takeover, right. At any point in time, the operating system could say, you've had enough CPU. Now I'm just coming in, I'm saving your registers. You'll never know that I'm about to start using the CPU for something else. And later on your wake up days and confused and might even know what happened.
Right. Right.
Um, and what the coroutine does is, is there's like, this is a cooperative choice. I need to be able to preserve enough of my state at this point for you to continue doing some other piece of work. But then when you come back to me, I can continue where I left off and for interpreted languages, that's usually as straightforward. And I'm gonna air quotes that our listeners can't see as storing the interpreter state for that function up to the point up to some point where, you know, you don't need to store higher up. Maybe that's where the, the event loop started or whatever. Um, and then you can resume by just jumping back into the bytecode with that kind of information around usually they're garbage collected as well. So any anything that you had references to will still naturally still exist because they still have a reference count or they still have a, they, they still have something pointing at them.
So you can just carry on. "Just". Uh, in C and C++ you have the problem of, well it's arbitrary assembly instructions, and the compiler needs to know how to hang onto things like the stack. And there is a single stack. It is the stack. It's not a stack. When you have multiple threads, you have multiple stacks by default, but in a coroutine, you deliberately don't necessarily want that. It becomes more complicated. So, so then to the footnote, the big asterisk, I said earlier,
They very, very low level primitives that the language is making promises. And then the STL, the standard template library does not yet have much in the way of support scaffolding for you. So if you, you wanna write coroutines, you can roll your sleeves up and write a whole bunch of like complicated state management code and you can do it. And there are, there are also cool libraries that do some of this stuff for you, but the expectation, as I understand it is over the next couple of cycles of the C++ committee, every three years meeting, then, um, the sort of predominant best solution for, um, a whole bunch of things to do with execution. And, uh, other things will come to the fore. And that will become the way that we think about these, these, these coroutines and how they exist in a bit bigger, in a wider context.
And then once those building blocks are in place, maybe there will be some async, uh, routines and async libraries that we can use. Which will be great. So obviously I just, me steering it my way there to talk about my own thing, which, and again, I've, I I've seen people use the coyield and coawait stuff. Um, in sort of like toy examples, but I've never used it myself in anger. And, um, although this is not async await at all, because coroutines are separable from async await. Um, you can build async await with them. Uh, coroutines are really, really cool for things like writing emulators, where you want multiple, very lightweight processes. Think every single device wants to have, Hey, another clock tick happened. And you wanna be able to write code that looks not like, uh, the traditional, you know, poll, you know, everyone's written poll where you have like a state machine in every single, um, element on that's being polled.
And the state machine is like, Hey, every time you have called poll, you have to do switch. What's my current state oh. Case waiting for this. Oh, okay. Well then I guess I'll do this type of thing. And it's really convenient to be able to just do some right, like a, uh, a video, uh, chip that just says, you know, await next cycle, draw two pixels, await next cycle, draw two pixels in a, in a, for loop thats's as wide as the screen? And then you do the, oh, and now do the thing that happens at the edge of the screen, await another cycle. And, and you just write straight line code, like you are the video chip, and there was no one else in the world, except every time there's a cycle, you say, okay, I'm done for this clock tick. Now someone else gets a go. And, and you don't use threads for that because the synchronization cost is staggeringly high for that. And the work you're doing in between 'em is tiny, but it's inconvenient to write the normal poll stuff. So anyway, that is a complete aside. And I have never, I've not yet written a, a, a coroutine based emulator, but I would like to.
Um, so let's sort of go back to some of the other languages we were to about earlier. I know, I know we, we, we have spoken about Java before now. Um, is there an equivalent for Java?
Uh, you know, I'm a little embarrassed to say that I don't know.
There's nothing embarrassing about not knowing things. Yeah. Don't. You should make that clear.
I I feel like I should know this. That's why, I mean, there's lots of things I don't don't know. Let's be real clear on that.
Right. OK.
But I feel like I, as much, uh, as much as I've done with the JVM and with asynchronous programming, I've never, I guess doesn't Clojure have like an async I think it does.
So there must be a mechanism for, for it. And again.
Well, I mean, it's, well, to your point, it's like, you can build these things on top of lower level languages and they work just fine, but I actually don't know if there's one in Java. Um, the, the one, one point that you made me think of though, when you were talking about the, the sort of, um, you know, the C++ introduction of this is that I would imagine that that means that the C++ community and standard library is gonna face the same situation that is true in Python. That is true in a few of these other languages that introduced this, where you sort of get this bifurcation of the standard library into like, well, these are the like asynchronous, you know, non-blocking calls. And these are the, you know, these are the star-bellied sneetches, and they're between the two, you know, you, you kind of don't want to cross, uh, and so you sort of have to make your choice.
Um, and if you, if you have to go back and, and sort of retrofit it, it sucks, but it, it, it, in a, a lot of cases means you either have to spend a lot of time figuring out how to, um, unify the duplication between those two worlds. In a way that makes sense, or just duplicate it,
Yeah. Yeah. I don't know about that actually. Um, I know that the boost library, asio, which is the asynchronous IO library, um, is one of the things that's been talked about for being standardized as the network library of, um, C++, and maybe, maybe part of the reason why all, all these things are taking longer than people's thought is that there is this kind of unification discussion going on. And, and, you know, I don't know much about that myself, but, um, I'm sure if we found some people, uh, we could find some people who wanted to talk about this kind of stuff, but I it's,
Yeah, I'm very much more a user of C++ than, than anyone who, uh, deep in the details of how it's, uh, how it's designed. But, and I know that this is the kind of thing that people think about a lot. And that's one of the reasons why arguably the C++, uh, standard library is mostly, I was gonna say impoverished, that's a terrible
You know, you can do all these clever things, but if you want to find, um, the, the full stop at the end, or if you wanna like trim the last character off string, then you just have to do it using those algorithms. You don't just have a dot remove last or whatever, or, you know, dot upper case and dot lower case, you know, all those kind of niceities don't come, but mainly because they aren't necessarily as general purpose as, uh, as one might imagine, you know, like, what does it mean to be upper case? Well, what locale, oh, now you've got all these questions. Right. You know, a lot of other languages go, nah, but yeah, we just wanna print something out in shout caps, you know, that's important to me, you not like the general. So I think that is, uh, definitely a factor in the, um, the, the slower adoption of these new technologies is because they want to do it. Right, knowing that it kind of has a certain persistence. It stays on for a long, long time. There is, you know, it's, it's worse than glitter for the C++, uh, uh, committee. They have to make decisions that are basically forever decisions. They never really go back on stuff.
Yeah, yeah. Yeah. Uh, another point that you made that I, I wanted to briefly touch on was the, the sort of performance impacts. I, I definitely have made the mistake myself and I've seen other people make the mistake of sort of just throwing threads at a problem when they have a performance problem, it's like, oh, I'll just break the work up and distribute it across threads, and then it'll be faster. Right. And, you know, the answer is not always, um, if, if the cost of crossing those thread boundaries is higher than the actual amount that you save by breaking up the work and the, you know, cost of resembling it, when you're all done, you, you can actually make it slower and more complicated at the same time. Yeah.
Brilliant.
Which is not, not great. And so, you know, one of the advantages of, of having a sort of single threaded, you know, event loop style, um, whether it's with async keywords or not, is that you don't have to pay the, not only do you have not have the, the sort of programmer, complexity problems of crossing thread boundaries, but you also don't have to pay the performance, hit crossing thread boundaries. And if you can sort of structure your work in, in such a way that you can, um, you know, take advantage of the nonblocking IO, while keeping the CPU busy, you can actually get like very good throughput by doing that. Um, and, you know, sort of avoid some, some tricky performance problems of like, oh yeah. Why is this queue, every time I go to read from the queue and it blocks, like it really slows things down, why is the queue full? And even when it keeps queue full, it still slows down. What's going on here, you know? All those kinds of things.
I think it's, it's an important distinction you make there. Like, if, if, uh, if you are predominantly IO bound, then you can definitely take advantage of a event based system. Because like you said, like if you're keeping the CPU busy, then sort of, by definition, when the CPU is finished, doing whatever it's doing now, there's always something new for it to do because some other IO event has completed. And someone who is waiting for a read has now got it. Or someone who is waiting for a write to drain has got the, the okay, that it's gone. Um, but it's easy to fall in the situation. And this is, I mean, this is what Compiler Explorer suffers from, right? If you hit Compiler Explorer, uh, some of the work that it does on the event thread. So like the, the, the web server itself is running like this, everything in Node.js, which is what Compiler Explorer is written in, um, is, is, uh, uh, continuation, passing style, mostly, although we're slowly moving to async await actually, um, style of, uh, of work, which means every web request is like an IO thread, as far as we're concerned IO response.
Um, every now and then when we've actually got the results of a compilation for us, um, another piece of IO is we ran the compiler. It's another process. And as far as we're concerned, we're awaiting it now. So like we can be serving more web responses. We can be giving the favicon dot ICO to whoever wants it. And then when the compiler is finished in its own process, somewhere else in a little sandbox, we're gonna read the results back in we're are gonna parse it so that we can give something back to the web browser. That's renderable, that is totally CPU bound. And for like large programs that have 20, 30, 40,000 lines of assembly output, it can take us a long while. Meanwhile, we are essentially blocking any other web request that's coming in. That's unrelated to that for that one node. Now we have load balancing and we have multiple physical computers that are doing the work.
So that's less of a problem, but our CPU is 100% wedged and we can't do the other work that is available to us. Now, if we had threads, that would not be true. And so we've discussed having worker threads for exactly this thing, in which case, actually, just to sort of square this off, um, we would probably kick off that, parse the assembly output and return me like the dictionary that I need to send the user that would be a worker thread. And then we would await the thread coming back, you know, like, Hey, kick off the thread and then await the results. And now we're back in the land of async await. It's just a, just like running the compiler. The result of parsing the output becomes itself an async IO inverted commas thing as for, at least as far as we're concerned. So there are kind of ways of making everything fit together. Um, I think the other thing I think about with performance, you mentioned about crossing thread boundaries, as it were obviously thread boundaries are typically not very expensive in the general sense, because in the same process, you can actually affect the same memory as the other
Yes. For good. Or for evil!
For good, or for evil. Exactly.
yes.
Um, so obviously the locking is where it starts to become more troublesome and, and you mentioned queues, so you did of, of cover that exactly as, as one would want, but that hidden inside that queue is a lock that is gonna prevent you from being able to do work once it's either full or there's no work to do all those kinds of things, or, you know, you've got the possibility of deadlock and, and other, other issues like that. Um, but conversely, if you are in, uh, async await land and with gay abandon, you are putting the, uh, async keyword on every single function you can, you can find even when there isn't any actual awaiting inside of that function, then you are pessimising your program because behind the scenes, what that means is that when you call that function, you don't actually get like the, the nuts and bolts are that function, which returned an integer now returns a, a future of an integer and you call it first of all, and you get back like a, Hey, there's nothing to do yet.
And then it gets scheduled on the event loop, or it could be at least if, if someone's awaiting it right now, it's gonna be a scheduled on the event loop, which means there's at least one tick of like the event loop clock before the first line of your function is gonna start running. Or it, it is actually, it depends on the implementation. Some of these things do actually eagerly, um, execute. And there's a difference between tasks and not, and async things and different languages or whatever, but I've at least had definitely some, some experiences where I've called a function and gone, why did that not do anything? And you realize, oh, I didn't wait. It, which meant that all that happened is it called the function to generate like the coroutine context. And then we're like, okay, here's your thing when you want it, it's
Ready. Yeah. Here's this code that I can run for you whenever you like. But not now.
But I called it. You like, you didn't call it. No, you didn't. So there's, obviously
You told me you were gonna call it.
It's not as, as cheap to do that. You're, you're creating these intermediate objects that, that have some set up cost, as well as the actual, maybe you're going once around the horn of the event loop before you are actually making progress. Now that does have some, some positive side effects, if you are CPU bound, and you're just trying in a really naf way to yield up. So this is another Compiler Explorer anecdotal thing here. And I hope that, um, none of my, my, my developers who are actually good at developing code are listening.
Yep. Yep. Uh, yeah. You asked me earlier about async await and Java. And I, I can't, I don't really, I don't really know, but one thing I did see earlier this week that I thought was super interesting as a proposal for Java virtual threads, which might be an interesting middle ground between these two worlds that we're talking about. Um, and the, the proposal as I, as I understand it, and remember it, um, basically revolves around, um, threads that with when you're executing, um, blocking what would normally be blocking IO calls within the context of a virtual thread, the JVM underneath the covers will turn those into non-blocking calls for you,
Right.
And suspend the execution of the virtual thread to allow other virtual threads to execute, which means you don't have to do the thing that you normally do with threads and pool them.
Because of the expensive start up
You never wanna thousand and thousands of threads at the same time. Cause you're gonna your operating system threads, cuz you're gonna starve out the operating system and, you know, make, make the job, the thread scheduler's job significantly harder. But with virtual threads, you can just go ahead and make as many as you want because they don't, they're not mapped to, um, operating system threads.
They are essentially async awaiting automatically at the point where the JVM says, Hey yeah, you called file dot open. Well, rather than file opening, I'm gonna rewrite that as an async await kind of thing and let another virtual thread take over the, this thread capital T thread, like there's whatever execution thread, however, that's where, but obviously you kind of need to opt into that a little bit higher up somewhere where you spawn these virtual virtual threads. And each one of them is essentially a top level coroutine or a asyncable, um, uh, uh, thing. That's, that's a neat solution that doesn't require much change to the code. And you're still writing straight line code. As far as you are concerned, you're still blocking and you are blocking, right? Yes. It's just the point where you would block and rely on the operating system to context switch you. The JVM says I can context switch you much more cheaply. Right. And I'll do that here and get someone. Yeah. That's awesome. That's really cool.
And one interesting side effect of that, that I, I was thinking of is that it avoids the, this problem that we were just talking about where you sort of have to bifurcate the standard library.
Oh yeah.
Like you're just saying no, no, no. Underneath the covers, it might be bifurcated, but that will never hit the API. You'll never see that.
That's really interesting.
Interesting. It's all one thing. Um, which is kind of cool, cuz it's just sort of reduces the cognitive load on programmers who need to understand how the standard library works.
No, that's super awesome. Yeah. That's a really interesting observation. I hadn't thought of that, but yeah, that bifrucation is otherwise a big deal breaker for a lot of people. They're like, oh, which way, you know, which, which way are we gonna go? Oh no, that's super cool. To finish on. I just wanna make sure a couple of observations that are amuse me and that is all these async await style things. And again, mainly the coroutines that, that I, that, that help them may be more, uh, useful are, are the way that operating systems used to work. I dunno if you, did you ever do any windows, uh, windows three point X development back in the day?
No windows, windows NT was really the first operating system that I used as a programmer in anger.
The windows, uh, in the win 32 API, which, which is still around and about has like an event loop. Right. And Uhhuh
You're reaching into the very depths of my, my neurons here, but yes, that does actually ring a bell.
And then you would sort of you'd pass it, all these structures to fill in, then it would return the, the why, you know, what's the next message. Right. And it would be, you know, WM paint and then you'd be like, oh, I better draw the screen,
Or like a mouse click event. Right?
Or mouse click or yeah. Any of these things. Yes. So in, um, in modern times that's just reading off of an event queue, right? Your, your, your thread is, is, uh, goes to sleep and, uh, you can have many threads, obviously they can be do any other things, but your, your sort of UI thread is there just reading the next message off of that. So as the mouse is move through your world, or if people are clicking or typing or whatever, you know, you get those events, but, but way back in the dawns of time at was how the operating system gave someone else a go, you call get message. And it's like, oh, I'm suspending you. Now. There is no preemptive multitasking here. At this point, we are saving all the registers and we're gonna load someone else's registers and memory map, and then we're gonna return to their get message.
And now they get a go on the, uh, the CPU. And, um, and um, certainly the operating system that I was using prior to windows, which was risk OS had that feel too. You would have to like do this system call that would tell you what the next thing is. And, uh, really it was switching and returning to somebody else, which was a really interesting design because you could do that in user mode. It wasn't that difficult to see how it was saving all the registers and reloading them all back out again. And so you could write your own like cooperatively multitasks sort of sub threads within your thread using the similar, the kind of techniques, which was a great way of introducing of like, well, this is how the operating system must be doing it. And I remember having an epiphany moment, um, doing exactly that and just having those two routines that would, you know, call a function, which isn't really calling a function because the calling that function actually jumps back into and returns from the other functions, call a function function, and then, and ping pong backwards and forwards between the two of them.
And it's, it's uh, yeah, just one of those crazy things. So it's kind of come back now, this cooperative multitasking, which is why I described it as that right at the beginning of this whole conversation is like, right, right. It's cooperative multitasking. You get to say when someone else gets a go of the CPU.
Yeah, yeah. Everything old is new again, my friend.
Everything old is new.
Well,
Uh, we've, we've covered some things that we are inexpert in only as like users. So I, I do hope that anyone listening to this who's been shouting at the, at the microphone. The headphones or the, the speakers about how
Our listener is very frustrated by now.
So do tweet us, uh, at twoscp to let us know all the mistakes we've made I as per usual. And uh, until next time, my friend.
Yep. Next time.