Good afternoon, Ben.
Hello, Matt. How are you?
Wow. Yeah, you've confused the heck out of half of our listeners now. Not only did I not say hi Ben, and I said good afternoon. You also asked me how I'm doing. I'm doing really great. Thank you very much. I am still recovering from a bout of COVID that I caught from seeing far too many people at a conference, which is unfortunately the occupational hazard of such a thing.
Oh no.
But I'm doing fine other than just a bit of a cough. How about you?
I'm doing all right. I have no real complaints. I only have zeroth world complaints.
Imaginary complaints that have no real component to it.
Right. Yes. There's no real component to it. They're truly imaginary.
Whereas if there's a mix, it's a complex complaint.
They're 90 degrees out from any sort of real complaint.
Is that right? So, yeah, you've completely derailed immediately, one minute in and already going, what was it we were going to talk about?
Right. Right.
And I'm going to blame the COVID for that because that's what you're allowed to do. Right. Something, something, brain fog.
You've got fog brain. That's like a, yeah.
Yeah, I feel like someone posted something on Twitter to the effect of something like,
Uh-huh. Right.
Cognitive hypochondria or something like that. And I'm like, maybe it is a bit of that. But, you know, even if it's psychosomatic, it seems really not. Anyway, that's not what we were going to talk about today.
Right.
We're going to talk about not my inability to think, but perhaps thinking or getting something else to think for us.
Yes. Outsourcing your thinking to another thinky thing.
Outsourcing it to a thing that can think or maybe can't think.
Right. Yeah.
Let's not get too caught up in the specifics, the semantics, but something which looks like it might think.
Yeah.
And that person may be an intern, right? As far as I can tell, they can think, but I've got no proof in my solipsistic worldview.
Nope.
Everyone else is a figment of my imagination. So that seems, no, no.
Mm-hmm.
But realistically, no. You had an idea before we pressed the record button.
I did. So I think there's sort of an interesting phenomenon going on with, you know, the development of software engineers, not the development of software, but the engineers that make them. And AI and sort of learning things and the way that people think about, you know, costs. I don't know. There's a whole bunch of interrelated things that I think are going on right now. And it might make for a good podcast.
Well, we can certainly sort of explore it, you and I without really any plan as per usual.
Yeah, yeah. Yeah, as per usual. So I think the way... First, I'm going to back up a level and be like, what is programming? Just to set some really basic foundational layers for this conversation we're about to have.
Okay, that seems reasonable, yeah. So are we talking about software development, like the whole process of it, or programming very specifically, or...
Software development, really, because there's a dimension.
Okay, okay. So yeah, tell me what you...
There's an aspect of this that I think is important for this conversation, which is a lot of what software development is, in my opinion at least, is taking the sort of messy, inconsistent, poorly specified real world, not the imaginary world, the real world and mapping it into,
Okay.
the completely uncompromising world of computers where everything has to be specified exactly. There is no room for vagaries. And if you are unaware of the vagaries that you're creating, the computer will create them for you much to your dismay.
Yes. Very, very apt.
You know, let's talk about undefined behavior, right?
Right. Exactly that. Yeah.
So when a thing is engaging this activity of mapping the real world to the sort of digital world, inevitably, one of the things that you have to be able to do is report back to whatever it is that is telling you to, like, hey, map this thing into a computer for me, please. And be like, actually the thing that you asked me to do is physically impossible. I can't actually do that because it is impossible to do that.
Right. Mm-hmm.
Or logically impossible. And so that's sort of like a feedback loop that is essential to the act of programming. I think it's naive to think that anyone really, even an experienced software engineer, can just say, go build a thing, and then the thing appears 100% of the time, right?
Because there is a very good chance that unless you have actually done the work of taking your ideas and mapping them into the digital world, there's a very good chance that they don't actually map and that compromises will need to be made. Questions will need to be answered. Things will need to be clarified. and
Well, if I can, if I can interrupt, even with a well specified problem, where you say solve X, I need something that will do X. There is an almost infinite number of valid solutions to that problem.
you know, Mm-hmm.
And some of them will be more maintainable than the others. Some of them, which may or may not be a goal. Some of them may be more performant than others, which may or may not be a goal. So many dimensions feed into what is a good X, that aren't often specified up front. And you know if you work for a high-frequency trading company, then probably it's a given that when you're writing something to go in the trading system, it should be as fast as possible and do no extra work.
But if you're writing a framework for wider consumption that's, you know, for anyone to come and use, then you probably want to be as generable as general...
Right. Yeah.
as general and a generally... as general as possible without tying yourself in knots. And so there's a huge amount of unspoken requirements that also come.
Generable. Yeah. Yeah. Yeah.
And as an experienced software developer, you like to think you can make an intelligent guess about it, but also you can be so wrong so often on that, and we are.
Right.
So that's not even taking into account the fact that what you are asked may not, as you say, *be* possible because it involves solving the halting problem or achieving something that is unattainable.
Right. Right, right. Going faster than the speed of light or something like that.
Things like this.
Yeah, right.
Right, yeah. So, yeah, there is a back and forth process where you say, maybe even internally, do I do X? Given two equally possible solutions, should I do X or Y? What are the trade-offs between them and all that stuff? And at that point, you may turn around and go back to your stakeholder and say, hey, do you want it fast or do you want it general or whatever?
Right.
Even that may not be the right axis, but something like that.
Yeah, right. But there's a trade-off to be made and it's not clear what the right one is, right
Yeah. And there are, well, there's, yeah, there are many, many, many trade-offs to be made. Yeah.
Yeah. Yeah. Right. So the question is, who or, and it's my belief by the way. So this is a problem, this doing this mapping from the physical world into the digital world is a thing that people have been trying to solve for a very long time. When I was in school, the big thing was like UML.
Are you going to say UML? Yeah.
Yes, exactly. Right. You know, you know what I'm going to say, you know, the deal.
Yeah, I'm a bit older than you, but it was still around after me.
Yes. Right. UML, Rational Rose, all of these tools that were like, you know, we're going to have all these things that let you solve this problem.
Oh my golly, there's a word I haven't heard.
And they didn't work, of course. But um it is my current...
And, know, things like 4GLs and things like that came along and said, hey, you know, you'll never need to write another line of code.
Yes, yes.
You just draw a picture and it'll all just work.
Right, right. So I have come to believe over the sort of decades of this that the least specific definition of what um you need to create in order to actually solve a problem on a computer is the source code, right?
Anything more abstract than that is not going to work because of this mapping problem, because you're going to have things that are unspecified, whether you know they're unspecified or not, you're going to have things that are unspecified that need to be specified in order for it to work.
Yep.
And so human beings have kind of invented programming languages, this incredible, fairly objective, like the most objective thing that we've maybe ever made beside maybe mathematics, of like how you actually solve a problem and that you cannot be any, you can't be any less abstract than that and actually solve the problem. Right? So fast forward to today and we have a new attempt, which is probably, which is turning out to be a much better attempt to solve this mapping problem, which is,
Specifically LLM coding assistant type stuff that people get.
Specifically, LLMs and maybe more specifically agentic coding, right? Although, you know, sort of depends.
Yeah, fair. Yeah. There are other things.
Yeah. Like it is conceivable, and I know that you have actually done this, to completely offload this mapping problem to an agent and have it solve a problem in a way where it creates source code, but you have not even read it, let alone understand it, right?
Correct.
And that is not, and I'm going to reference what you were kind of saying a little bit before the podcast here. That is not completely different from what we've been doing for decades where a manager says, hey, go build this thing. And the developer goes and writes some code and then comes back to the manager and the manager doesn't read the code at all. Sends pull requests, whatever, approve. And then they try it and they're like, yeah, that doesn't work. Do the thing again.
and then it goes and tries again. And then we just, you know, it's like we're doing the exact same thing. You know, what what what was the phrase that you used for this?
Oh, I need to find it out now. this is I'm going to have to say thank you to ah my friend Jeremy, our mutual friend Jeremy, in fact.
Yeah.
It says, managers have been vibe coding forever.
Right. Yes, exactly.
And it's the loop of like what the vibe coding looks like. And at the end, the manager says, good job, but be faster the next time. Or worse.
Right. So I think there was a perception, I don't know, a year-ish ago, maybe a couple of years ago when these tools started coming out. And I know you and I maybe even had some conversations about this of like, you know, is software engineering as a profession going to go away because of this? Is this going to just be, you know, non-technical people or maybe just like junior people who are right out of school who are not managers yet using these tools to build software?
Right.
And if you especially if you look at some of like the job numbers in like the the, you know, numbers of openings and people were are looking for jobs. It definitely seems like what has happened in the last year or two is that it's the junior engineers who are not getting hired because what people are doing is they're just putting these tools in the hands of very senior software engineers who are now able to get way more leverage than they may be used to, or at least the same amount of leverage, but with many fewer people, right?
Yeah. Yeah.
Like you don't have the thing of like, okay, we're going to give you two or three you know less skilled or more junior engineer.
Junior. Yeah. Whatever.
Yeah.
Yeah.
um And then you're go you you have this team and you just do this thing of like, you sort of farm the work out to them and you do basically this like vibe manager coding thing, right?
Yeah. Yeah.
And it's like, okay, yeah, don't do that. Just have like two or three different Claude agents running on your computer at the same time. And there's your team essentially, and you just do that, right?
Yeah.
And so that makes sense. And that's like a very rational thing for businesses to do, right?
But just so of like to go back to your point, I think you know the point we were making earlier on was that the senior developers would be the ones that would be being cut because they're expensive and their experience and skill level could be replaced by a sufficiently intelligent
Right. Yeah.
AI system.
Yeah. Yeah.
And then it just takes guidance from either a junior dev or no dev at all, because, you know, you just have the manager, the non-technical stakeholder just talking into a ah window saying, no, no, make it bluer.
Right.
No, no, no. Now make it do this when you click that kind of thing or more, more generally.
Yeah.
But the surprise here is at least fast forward a year, that does not seem to be the case. In fact, the opposite. It's more the senior folks like you and I who have gone, hey, this seems valuable and we can perhaps get some use out of it as a tool.
Right.
But the junior folks have not been ah able to get the same amount of leverage as we do. But their role is being potentially usurped by the fact that, yeah, you don't need a team of people anymore. You need one guy and and and the patience of Job to talk to. So many forgetful, stupid, ah you know, ah ADHD robots. But nonetheless, that, and obviously I don't think we're there yet.
Yeah. Right. Right. Right. right
You know, you and I both work in companies where there are still are junior folks around, but that was the interesting point.
Right. That seems to be what is kind of happening as as as best that I can tell. And, you know, I'm not far from like an economic expert here.
Yeah, we,
i just see what I see.
Well, and also we only really have yeah our own vantage point from where we are and we're in a particularly niche industry perhaps as well, um you know where technical um and so and and ah technical knowledge and a specific ah domain-specific knowledge is quite kind of important to what we do and maybe in a more general world where
Right. Also very true.
AI has been trained on more publicly available information. I mean, that may be they're easier to replace folks with. I mean, certainly in my limited experience of the last few months where I've been less employed, um fewer fewer employed, fewer employed,
If you were employed.
I've been using AI to help me on my open source projects, and that's all in TypeScript and JavaScript and web. And there's like thousands of websites that have good examples that presumably it's been trained on.
Right. Yeah.
When I go to conferences and talk to people, they have more equivocal results with more technical things like the minutia of C++, especially some of the new things that are coming down the pipeline because there exists no training data.
Yeah. Yeah.
And that's also true of me, right? You know, someone to come to me and say, can you write me a co-routine?
yeah
I'd be like, give me a few months to go and learn how to do that first, and then I'll be able to. But maybe our particular industry and the intersection of the technology we use and the special things we use makes it more likely that we will still hire and use people. I don't know. I'm just spitballing here, as indeed this entire podcast is.
No. Yeah. I mean, what, that's the whole point of the podcast, but yeah, that's a very, very important point is that it's not like we're talking, it's not like I've analyzed a bunch of economic data to come to this conclusion, right?
That is.
Like this is all mostly from just my personal experience and the things that I've seen in the industry. So, you know, take this with however many grains of salt you believe are appropriate, listener.
Right.
But the kind of result of this now is you have, senior people. And I think the explanation is, is that sort of makes sense to me is that it's like LLM still require a decent amount of babysitting and you need to be able to know when they're hallucinating. You need to be able to know when they're going off in a weird direction to correct them. Otherwise they're like, it's the blind leading the blind.
And that sort of explains why you get the most leverage from from having one experienced software engineer, you know, using all these various tools to build the things that you need, right?
And yeah, I mean, I was going to say, i just interject there, I think that knowing about the kind of hallucinations, or I don't know, even if that's such a loaded term these days, but you know, like when, when mistakes are made and assumptions are made about, oh, there must be something that does x and then it says, "Hey, call x", and you know, that's never existed.
Yeah.
it would be great if it did.
Right, yeah.
But like identifying that and keeping it on track, a kind of slightly different techniques or different skills than we've had to use before, because perhaps as an experienced programmer who has led and mentored junior developers,
You know, there is definitely a set of mistakes that I have seen folks fall foul to that sometimes I let them and, you know, you let them and then you guide them through the process of saying, why do you think that didn't work? And, you know, you use it as a teaching point, which is a valuable thing, I think, for everybody to learn from.
Mm-hmm. Mm-hmm.
But those are very human mistakes. And also the process of doing it that way allows them to grow. And LLMs will make those kinds of mistakes as well. And so I think that if you have led a team before that being able to control a suite of LLMs, some of those skills do overlap and you will go, yeah, yeah, please don't do this.
Whatever you do, don't comment out the test to make everything green or don't check in without running things, which you see junior folks do sometimes, you know, like, hey, that's there for a reason.
Yeah, right. Yeah.
Don't just disable it because it didn't make sense to you. You know, always ask me, all that kind of stuff. But the thing that's frustrating is that the LLM will not learn in the same way as your junior devs. You know, the third or fourth time you said, please, look, if I have to do this again, we're going to have to have a talk, serious talk about this.
Right.
Then, you know, the LLM is like, nope, you've told me a hundred times before and I forget every single time. I mean, like a happy puppy, but...
Right, right. Yep.
So I think there are some new skills.
I don't have that context in my window, right? Yeah, exactly.
Yeah, right. Yeah. There are definitely some new skills for for leading LLMs. And I think it's like probably a 75% overlap with the kind of skills that help you with a team of real humans.
Yeah.
But there's definitely, there's a 25% that's new for LLMs. And there's also probably, I mean, I'm probably underselling it. There's a ton of much more important, you know, interpersonal skills. And didactic techniques that you can use when you're trying to help an individual learn.
Yeah. Mm-hmm.
And I've forgotten where I was going with this, but I interjected nonetheless. You were in the middle of something and I put my hand up and you...
Oh, well, no, I mean, it it's just that, you know, the the phenomena now of having these sort of senior engineers leveraged using these tools is is great from an economic standpoint. Like it makes a lot of sense.
Right.
But the thing that I wonder about is if we are, you know, as an industry, setting ourselves up for future pain. Because if you do this for too long and the senior engineers are too successful. So the problem with companies in general is that if you have very talented people and you don't pay them enough, they'll leave.
And if you pay them too much, they'll leave because they're rich and they don't need to work anymore. Right. So, you have to sort of, you know, strike this middle ground of not paying them too much so they quit because they don't need to work.
So you're saying the only reason they don't pay me more money is because I might quit.
Uh, well, let me ask you this. I mean, if you had, you know, hundreds of millions of dollars in the bank, would you continue to work?
Probably, but not, yeah, not for... yeah, yeah, yeah, I think I'd never stop doing, I mean, we have the same, I'd never stop doing this stuff, but yeah, but no, yeah, all right, your point stands, Ben, your point stands.
Maybe for yourself. Right. Right. Like I, yeah, I'm never going to stop programming because it's fun, right? But would I be? Yeah. So this is my point, is you got to pay people the right amount. So eventually, though, they are going to make enough money or they're just going to get to the point where they want to retire and they're going to retire. And the question is, I've just been teaching robots for the last 10 years.
Who's going to take over my job? Who's going to start being the person commanding the robots?
Right. Right. Right
And this is a problem that capitalism is exceptionally bad at solving. Right. Of these sort of like long term systemic problems. And so I think what may happen is companies will start to discover that as the junior engineers who are currently looking for jobs sort of diffuse into other industries because they don't find any of the the opportunities that are there, that they're going to start to have a really hard time finding the people who can lead the robots.
And what it what it might be is kind of like this pinch point of like, okay, well, we we got all these productivity gains from from structuring things this way. And it made complete sense for this quarter and the next quarter. But 10 years on, now we can't find anybody that understands any of this or how to make any of it work because they don't work in this industry anymore. Or those people maybe have just gotten very expensive. Right.
Because there aren't that many of them that sort of survived the filter. Right.
Right.
Yeah.
They will be the COBOL programmers of their of their day.
Yeah. Yeah. Like it'll, just be a much harder, like an even harder thing than it was, you know, 2021 when it was like there were, ridiculous ah salaries being handed out because it's like you couldn't find enough programmers, right?
um On the flip side of this, just thinking of it from like the the sort of trader perspective, it probably also means that in the next five to 10 years, there are going to be some really really, really good quote unquote junior programmers out there that could run circles around anyone else in your organization with 10 years of experience or 20 years of experience
Yeah
And if you can find them, they will be worth their weight in gold.
That's an interesting observation to turn it that way around. Yeah, that's it. Yeah. No, it's interesting you say that this is a problem with capitalism. This is not a politics podcast, obviously, so i won't we won't go too much into the other choices out there.
Yeah. Right.
But no, I see it as almost like it's ah it's a local minima. We're heading towards a local minima, and nobody is going to do the sort of global optimization problem of like, well, now we just have to do the simulated annealing step of just throwing some randomness into it just to see, because we just need to get out of this this potentially local minima.
Yeah. Right, right.
But to sort of counter that, obviously, the the more... AI um ah proponent or a but I don't even know what the right term would be, but someone who is more confident in the role of AI will just say, well, AI will get better. And actually even the senior engineers,
Won't have to be around anymore. We can just, any anyone can ask that question. And I think that obviously that that gets to the heart of the original thing that you brought up, which is that there is a ton of unknown unknowns that something or someone has to say, either make a call on explicitly and say, look, based on my years of experience, this is a database thing we should probably do use blah, versus this is a trading application, we should probably do something different.
Yeah.
Or you have to ask the the the person who's asking the question, should we optimize for so for throughput or latency? And at which point they're going to say, "the what now"? And then, so yeah, but where would that line be? And yeah,
Yeah. Right. I mean, the the the premise here is that the although AI is great, it still hasn't actually solved the mapping problem, where the mapping problem is map the physical world into the digital world in a reliable, sustainable, continuous way. Obviously, you can have somebody sit down that knows very little about programming and interact with Claude and build like a simple app. Right? Like those things are absolutely possible. Right?
Yeah.
It remains to be seen whether you can build more complicated things that way. It remains to be seen whether you can change things that have been built that way and revise things that have been built that way and grow and scale things that have been built that way.
It's possible that maybe one day, know, Claude version 75 will be so good at all of this that it really it really has like sort of fundamentally solved the mapping problem where you can have a completely non-technical person, somebody who does not really understand that computers work at all, sit down and build. A horizontally scalable web app or a trading system or you know a microcontroller for something that's going to go in the space shuttle. And all of that stuff will just work.
It's entirely possible. But i I could also see a world where where that problem is so difficult that it doesn't actually get solved.
It's possible, but yeah, it's hard to see. Well, I mean, that problem is so difficult that it often doesn't get solved even with experienced programmers and experienced, you know, this is a difficult thing to get to get right.
Right.
But there is, i mean, so you mentioned earlier about the fact that I had had some limited experience of like this whole closed loop vibe-codey type thing where I haven't even looked at the the code of something. And that is that is true. um But uniquely in the situation I was in, I had a very clear goal and a very clear output that I was looking for.
And I didn't really care how we achieved that goal. It was unimportant to me. And that... was a surprise to me because normally I have strong opinions about how the software should be developed. And I actually care about that more than the outcome of the software.
Yeah
And I think we've talked about this before. I think there are some people for whom the, the goal and the result is the important aspect. And then really the difference between, you know, somebody who is, I don't know, a quant or a physicist or whatever, and a programmer is the programmer is like, yeah, I want to do, and a particle system, but it's going to be a really nice piece of code and I'm going to enjoy that process. And that's what floats my boat.
Yeah.
And so that's definitely 100% me.
Right.
But in this particular case, I just didn't care.
Yeah.
I'm like, do a thing, make it happen. you know and that was I've done it a couple of times. Another one was like, you know hey, I've got some YouTube videos. Can you just write a thing that tells me when there's a new... comment on them, please, because I don't own the video, so I can't subscribe to them myself and get notifications.
Oh yeah. yeah Right.
But if I run a Python script once a day, then you keep a .txt file, the ones you've seen before, and just tell me. And that was like, again, I don't care how that works, and it /seems/ to work. That use case covers probably a lot of companies that aren't technology-focused need for technology.
Tight
Your ah HR company that just says, I need a thing that goes through all the resumes and gives me a list of everybody's phone numbers or whatever.
Yeah, yeah, yeah.
Right. You know, those kinds of things.
Right.
And so those I can see, they don't care how maintainable it is, or they don't maybe not even care if it's reproducible at all.
Right. Yes.
It's just a one-off and they're done with it. Right. And that's, that's fine.
Right. Write once read never software. Right.
Right. And that's kind of, yeah, it's sort of anathema to everything that you and I stand for, and really.
Yeah.
But also it is pragmatic and it's useful and and it's...
It's very pragmatic. And honestly, I've been doing some more of that myself lately. And I try to be very intentional about it. It's like, I am going to have Claude generate this Python script for me. And I am basically going to test it empirically. I'm going to run it. I'm going to see if it produces the result that I want. And if it does, I don't really care how it works. I might read it. I might not. Right.
Yeah.
And for certain tasks, that makes a ton of sense. It's incredibly fast. And I can get enough confidence that it does what I want it to do, =by running it. Right?
But then there's a sort of trade-off where that one-off tool becomes more and more useful. You start relying on it, and then eventually you have to turn it into a like a proper tool.
Yeah.
And now you're like, oh, gosh, ah do I treat it as a prototype? Do I redo it myself?
Yeah.
It's interesting. But then, you know again, I think we've talked about this before, like I do a lot of open source software.
Yeah.
At least I have in the last you know few months. And to some extent, all of the code that I maintain is like that because it was written by other people that I don't have even hope of contacting anymore necessarily.
So they might as well have been an agent that that fired up, built up its kind of idea, submitted a small change to the code and then disappeared never to be able to be contacted again because I have this weird language that somebody sent for Compiler Explorer and I haven't got a clue what it is.
Yeah. Yeah.
Now I have reviewed the code, but it's still a feeling of like, there is a sort of sense in which as team size grows, either open source or a closed source, you end up looking at code that you don't recognize before.
Right.
It's just whether or not it fits your style, whether you're you've got some overarching idea about how the the the whole should be put together, which we've we've maybe we talked about before. I think, we yeah, we have. I'm sure we've talked about the fact that like, Setting a project up well for success as a senior engineer with all the things that come with it can help everyone, including AIs, come in and onboard.
Yeah, yeah.
Yeah.
Yeah. But yeah, I don't know. I just think that there's a thing that that is likely to happen that I think the very well could happen. And again, you know I'd kind of say you know capitalism is famously bad at solving these types of problems where it's like stuck in a local minima. It makes completely a ton of sense why you would be in that local minima, but there will come a day when you try to
And, you know, like your shareholders will ask questions if you say, yeah, we just hired a bunch of people to do work that we can do with the computer. And why have you done that? Well, because we need to train them up to be the the next middle tier.
Right, right.
And that's a harder thing to to convince people to spend money on.
Yes Right, right.
Right.
Like you might be able to have something in in much larger companies where you have a little bit, but certainly like medium and smallish companies, In particular, and really, like you say, you know, if you have that sort of like shareholder question of like, why are you why are you guys hiring all of these expensive, you know, MIT grads when you could just take your existing engineers and have them be just as productive, if not more by just you know giving them a hundred-dollar-a-month, you know, Claude license or whatever.
Yeah.
But then, I mean, again, taking the other side of that, again, you know, in in terms of R&D, companies have always, ah you know, forward thinking companies have always had like, hey, we've got a department of people who just sit around and tinker with things in the hope that we find something cool and new.
Mm-hmm. Yeah.
The trick with that, I suppose, now I think about it out loud, is that the results of that, whatever they are, positive or negative, are owned unambiguously by the company. Like, you're creating intellectual property for the company. Whereas when you're investing in the brain trust of new grads,
Right. Mm-hmm.
You are investing in that person, which is wonderful and thoughtful and helps the world, but does not specifically help you unless you can retain that person, which is not necessarily possible.
Right, right.
And so you are let you have kind of you're giving somebody a gift, as it were, that your competitor could could benefit from. And then, yeah, back to your point about capitalism.
Exactly. Yeah, i I have had some discussions with ah people who may be in school in a computer science program, and I think we both know who they are.
I, I've no idea.
And one of the thoughts that I had in the course of those discussions is it is when I was graduating and certainly throughout my career, one of the most important things as a new grad, as ah as a junior person, Was that you demonstrate that you can learn quickly, right? Like that's always sort of the thing is like, we're not expecting you to be an expert in any of this.
Yep.
That would be silly. We just want to find people who can learn quickly. Right. And I think that's always going to be true to an extent. I mean, no one wants to hire people who learn slowly. Right? Like, so it's a little silly, but there may be a little bit of a shift in emphasis, I think, and that that is created by this this local minima where it's like, yeah, yeah, okay, you can learn fast. Everybody can learn fast. What can you /do/?
Yeah.
Because we're going to give you, ah you know, your cloud licenses and a development environment and we, cool if you can learn stuff, but what we want you to do is build this thing and it has to be right and it has to be good.
And so ah i kind of wonder if the thing the thing to do is to say, you know if you are a person who's in school, if you're a person who's trying to get a job, and this was always kind of true, but I wonder if it's becoming more true, is the thing to do is demonstrate that you can get things done, that you can build stuff, right? The old Joel Spolsky test of, you know, "smart and gets things done", right?
Yeah, that's. But yeah, and then but then there's still the the oh sort of sort of getting things done does not give you the opportunity to learn the skills that will allow you to get better necessarily. Maybe you just have to balance those things.
Yeah.
I mean, I certainly know that talking to some some junior folks that like some of them avoid using any kind of AI because they perceive that they won't learn from it, which is valid, absolutely valid.
Yeah, yeah, right.
I mean, there's nothing like doing things to learn things, but balancing that with the sort of the changing winds of our industry, that this is an enabling technology and finding the right balance.
Yeah. Yeah.
And I wish I had better advice for them when they come and say, i like, hey, no, I just won't use ChatGPT. I won't use Claude. And you're like, well, you know, everyone is.
I think, yeah.
I was, I was actually chatting with, um, a friend of mine who is a theoretical physicist at a university.
i don't know, man.
And he was, we were talking about AI usage in general. And he's like, I have to kind of essentially ban my students from using it, even though I use it myself daily.
Yeah. Yeah.
It's an incredibly invaluable tool for doing certain things. And, um, he and I actually mooted some ideas. so so I don't don't know if I've already said this before, and I know it rise, we're getting at time here. I have to be a bit thoughtful about not starting yet a new topic, despite the fact that we've just said, Hey, we should probably look at the end of each episode and see if there are ones we could continue. But no, um I wonder if there is a sweet spot for AI, which is a bit like NP-hard versus NP-complete or one way one of the things around those ways, I forget which one's which, right? [[EDITOR MATT HERE - NEITHER OF THEM ARE THIS, IT IS NOT LIKE THAT AT ALL!]]
But my understanding, limited as it is about these things, is that there are some problems that are NP-hard, which means that there is not a polynomial way of solving them. And for a subset of those, there is a polynomial way of checking that the answer is right. So for example, if I said to you, find me a route through America, going through every single city that is less than 5,000 miles total, then it's NP, it you know, it's it's stupendously difficult to search that space.
Yeah. Yeah, I think that's right.
But once you found a solution, I can check it trivially just by following it along going, oh, it's less than 5,000.
Right. Yeah. Prime factorization is another one of those, right?
But if I then said, perfect example,
It's like, yeah, you just multiply it all together and if you get the thing, then yeah that's right. Yeah.
Exactly. Perfect. Exactly. But um there is another family of, I think this is the NP-complete, but again, I'm really - mathematicians who are listening, I'm so sorry if I'm wrong [[I MOST CERTAINLY AM]] - but one of them is not only is it NP in terms of the time complexity of solving it, but also to check the answer it is also.
Mm-hmm. Mm-hmm.
So like the classical traveling salesman problem, which is to find the shortest
Mm-hmm.
In order to show that it's the shortest, you have to essentially show that no other path is is long. So every other path is longer.
Mm-hmm. Mm-hmm.
And so where does the analogy break down with AI? AI, I think can be very useful for the kind of NP-hard things or whichever is the one that can be polynomial time checked.
Yeah, easy to verify, right?
That's where I'm going with this. Exactly right. Thank you for saying it much more succinctly than I. But yeah, so again, physicist friend is like, yeah, I can set it on certain things. And as long as it gives me the citations of papers, which I can click on afterwards and then read the ones that it's found.
Yeah, yeah, yeah, yeah. Right.
There's obviously an error of omission where it didn't read things and they might be important. But the ones it did find or the ones that it summarized in a particular way, I can read through myself and go, yes, this seems reasonable. And it was much quicker than me reading them all.
But, but then there are the the other problems where you, you know, you just have to kind of go, I guess it's right. And I suppose to an extent, actually, if you're a non programmer and you asked AI to make you a Python program and it's generated 3000 lines of Python, you run it and it appears to work.
Right
I mean, mate you know, maybe that's, maybe that is the the first category because you can prove this right, but you still don't really know that it works on your system.
i i would yeah i would I would argue that if you really generated 3000 line Python program, unless there are there's no conditional logic in it whatsoever, and it is just straight through 3000 lines executed the exact same way every time, there's no way that you can actually know that that's correct in all cases, right?
Yeah. Yeah. But, you know, to your point earlier, you said, you know, you just ran it a few times empirically, it's okay. And and that's so kind of almost like the verification "seems okay to me".
Yeah. Yeah. Yeah. And I mean, usually what I, be honest, what I'm usually doing there is looking through the conditionals being like, can I delete this? Is this, is this work?
Oh, that's an interesting way of doing it.
No, I mean, this is what I did with bash scripts forever. Like I would, I would have, I intentionally try to write bash scripts as I'm not, I never found a good unit testing framework for bash. I think we talked about this maybe in another episode.
We have actually, yeah. That was one the early ones.
Yeah. And so my de facto method for writing bash scripts up until the, you know, creation of Claude, which now I just use that, was take a shell script, make it executable, an empty file, i'll make it executable, put it in watch and have it run over and over and over again, and then try to solve the problem by writing as few conditionals as I possibly can, right?
Just have it go straight through. There's only one code path through this and I can just tell empirically that it works because it prints out seven at the end, right?
Right. And then keep editing.
Yeah, and that was my method because it's like, okay, it's dumb, but it it like there's only one possible way to run this program and the one way works empirically. So that's good enough for now, right?
Right.
It's when you have like the, oh, it handles this case and handles that case and has a combination of these two cases. it is is that Like that's when you start introducing bugs without realizing it. And that's when you actually need to understand all of the possible paths through the code and, you know, hopefully structure them in a way where there's not like 2^n of them, right?
Mm-hmm. Mm-hmm.
Because otherwise you're just going to build unreliable software. Right? So I completely like, um I think that, you know, this, the sort of vibe model of coding, especially when you have sort non-technical people doing it, is really like a rising tide lifting all boats. It's making people much more productive.
But if if we start to try to build complicated systems with this, I'm not really sure that that actually works. Because at the end of the day, the context window of the AI can only be so big and it can only remember so many things.
Yes, and...
And when somebody asks the question, why does this not work? There's not going to be anyone that can provide an answer.
..That is more to the point. But yeah, I think like the the context window thing is a whole other thing.
Yeah.
There are obviously technical reasons there, upping those and stuff. And then you know the fact that my own context window as a human is, getting as I'm getting older, much smaller. And you rely much on, right, yeah.
COVID brain frog, right? Brain fog...
Yeah, yeah. I, there could be a frog in there as well. That would explain a lot.
Brain frog is a, that's a whole other symptom of COVID and trust me, you don't want that one.
That would absolutely. Oh, man. Yeah. No, I think that that those those issues will start to be, that's where the innovations are happening, as far as I can tell, in the AI world now.
Yeah.
But um just one final thing as we're hitting towards the end of one of our longer episodes here. In, I think, 1980, Fred Brooks of Mythical Man Month fame wrote an essay called No Silver Bullet.
Mm-hmm. Mm-hmm. Mm-hmm.
And I've been ruminating that on a ah lot recently as I've gone through the arc of LLM sort of usage myself from, oh my gosh, this is mind blowing and I think it's going to change the universe to this is terrible. This is worse. I'm wasting my time on this, to: ah maybe, maybe not. To, now reasonably, it's another useful tool in my arsenal. Sometimes I can fire it on a couple of low-hanging bugs and I can then PR and I'm not sure if that's faster or not. I don't know yet.
Sometimes it can help me by doing things that I'd never think of doing.
Yeah.
Other times it gets in my way. I don't know. It's a tool. It's like grep. It's like sed. It's a bit more involved. But to your point earlier, I think it's probably got the biggest chance of being a silver bullet out of all the things that we've seen so far. But you know, Fred's point was, everything has come and gone and nothing has ever fundamentally changed the problems of software engineering. That is, we need software engineers to do software engineering practices.
Yeah
And I think it's probably worth a re-read
Yeah, there's always going to be somebody who is ultimately responsible for making sure the technology works. And that person is going to become a technologist whether they want to or not. Because they'll just there'll be situations where they're like, why doesn't this work? It's my job to figure it out. I guess I need to figure that out.
Yeah. And whether they work with an AI to like kind of say, why is this not working? And have that sort of debug experience, pair programming experience, but they are still being brought along for the ride on the technology journey and there's no evading it.
And that's just, there's no avoiding that. Sure. Yeah.
And maybe that's hope for us all.
Yeah.
We'll see. All right. I think we have reached a good enough stopping point for now.
Yeah, that works.
And I will see you next time, my friend.
Until next time.