Agile is not an out. It's not a thing you do. It's a way that you do something. It's agility. And so you do things in an agile way. It's a really difficult problem because everybody wants to be told what to do. When it's scary, when it's unknown, people are looking around for that figure that says, don't worry, I've got it covered. Do it this way. It'll be fine.
We are interview creators of tech products about how they've done it. My name is Ilya Bezilev and my co-host is Arnapp Deca. Arnapp and I are co-founders of Metacast, a brand new podcast app for iOS and Android that has transcripts, powerful playlists, and many other features that can really elevate your podcast listening. You can download Metacast at metacast.app.
The guest of today's episode is Dave Thomas, a co-author of the Pragmatic Programmer book and the Agile Manifesto that both have been super influential in the software development industry. On the episode we covered multiple topics about Agile software development. My favorite part is where we talked about what it means to do agile versus being agile.
We also talked about practices of successful engineering teams, what it means to be pragmatic at an early stage startup, and also very reflect back on the book that was published 24 years ago, and what has changed since then and what still stands true.
Oh, and by the way, this episode was originally recorded for our Metacast behind the scenes podcast, which we have since pivoted into being just the podcast about Metacast. And we are re-running this episode on Builders Got a Build, but you will hear the references to the original podcast in the recording. All right, enjoy.
Hello, and welcome to episode 44 of the Metacast behind the scenes podcast. This is Arnab, and with me I have my co-host, Ilya. We are the founders of Metacast. And today we have an amazing guest with us, perhaps the biggest influence in my life as a software engineer so far. And say hello, Dave Thomas. What's very kind of you to say?
I'll let you introduce yourself in a little bit, but I just want to start with, like I said, the influence that you have had in my life and career back when I started at Amazon as an SD, like an entry level engineer. One of the engineers in my team suggested that I read this book, the pragmatic programmer, you've done so many renowned things, but this is the thing that like immediately I think connect with you.
And this book, aside from all the best practices and the teachings from it, I think it fundamentally changed the way I approach software development in that I always need to be a craftsman. I always need to be practicing. I think some of that goes into us creating this company, Ilya and me, because after getting promoted and promoted and promoted, you get away from actually doing the craftsmen things, right.
And after I think getting into that habit, I was not happy with that eventually. And so Ilya and I decided let's go build something else. So that's kind of like the foundational things that you have had in me. Your book also like helped me evangelize unit testing, CI, CD infrastructure, all of these best practices early on at Amazon and elsewhere too now.
So thank you for it. I have the book right here with me. This is a 2009, I looked it up and that's kind of like when I read it the first time, so it makes sense. Cool. Of course, that makes me feel very, very old, but apart from that, that's cool. Yeah, so please introduce yourself to our listeners. Well, my name is Dave Thomas. I'm English, but I've been living in the US now for 30 something years. My wife is American. I started off programming 1972, I think.
And I have probably programmed just about every day since then fundamentally I enjoy it. It's kind of like my happy place is writing code and discovering things along the way. So I had a pretty standard kind of career. I did computer science at university in London. I was doing a PhD program and I got recruited away by what nowadays would be called a startup back then we called them sweatshops.
It was the best experience because this was a company that was formed to write custom software for people and because of the fact they were small and they were just starting to grow. They said yes to everything. The range of different industries and project types and people that I got exposed to was phenomenal.
And the assumption was yes, we can do it. That kind of left me in this position where I really didn't have any excuses. I couldn't say, well, you can't really do that. You know, because we'd committed to do it. I learned how to basically get things done in code and that was just absolutely fantastic. And I stayed with them for a while. I moved across briefly to what was at the time.
I think the United Kingdom's only surviving computer manufacturer and I joined their kind of research place was called C.T.L. or ITL depending on the year. And they produced a media computer which was quite an old fashioned architecture, but it was cheap and it was made in the UK which meant that for various government contracts and defense contracts and everything else it was important.
But it also meant we got some pretty weird stuff we were asked to do. One of my most fondly remembered projects was we got a job with the European Space Agency writing or supplying the system to do the on the ground checkout for the Geotosatellite that went to see how these comment.
And that involved some bespoke hardware, a very, very wide, very high speed parallel interface that could suck data off all of the various test points and record it which meant changes to obviously the hardware but also to the operating system.
And there was some really pretty cool code to intercept it and there was micro code running on the actual controller itself and the high spot obviously we got to go across to Holland and get our stuff working in the test lab which is where everybody wears the bunny suits and it's all white and you have this sort of golden thing in the middle which is the satellite. That was good fun. That was very good fun. What's here was it?
Now you're asking it would be I can't remember early ish 80s I think but I'm sure Google approved me wrong don't ask me dates I have no idea about dates but did that for a while then I formed my own company in the UK well actually that's not quite true. A friend from this company that we're just talked about and I as a side project developed a terminal emulator like as an open source project or.
I mean the idea of a public internet was still five years away opera existed and there were universities that talked on to universities but we downloaded software using dial up modems and we connect services like comp usur.
Look at up folks look at up so no open source but now this is a product and it's claimed to fame it ran on digital equipment micros and it also run on IBM PCs and it's claimed to fame was you could actually connect to multiple computers at the same time and multiplex them onto one screen.
That found quite a big market in the financial marketplace where real estate is really really tough so this got to the point where it actually became viable so we formed a company and we sold that and then we started writing software for people that was really good fun again we were in that lean hungry stage so we would say yes to everything and so we got to do some pretty wild things along the way accumulated like basically experience working on all sorts of things.
And then we got to do something on all sorts of different projects and styles absolutely absolutely also kind of like learning what not to do it's a lot easier to see things go wrong when it directly affects your bottom line and so we got very good rapid feedback when we were doing things wrong and that was that was useful.
At some point we became the UK point of contact for US computer manufacturer called stratus and what they had was a large mini computer that was fault tolerant everything was duplicated everything ran in parallel and you could go up to it like pool boards out it would carry on running with that interruption which is kind of cool.
We started doing work with that again a big market in the city of London and they center cross there a sales support person to help us and this sales support at some point went back to the states and a few years later I'd also left the company at this point and I was doing independent consulting and he phone me up and said hey look I'm now working at a financial company we're doing switching of debit cards credit cards and I promised them that we would rewrite all of that switch software in a camera was like six months.
I said yes he said to me so I said of course we can do that and then I said then I had to call you to make it happen so I flew across and we had a look at it and I said I'm going to need some help and he said well I had a friend from college who's independent too and that's when I met Andy Hunt so Andy and I spent probably in the end about 10 months.
The office they gave us was a converted closet no windows no air conditioning we were working in Atlanta at the time so it was a very close experience but we did it we got the switch working and we kind of enjoyed working together and so we kept doing it and for the next five six years we did that one of the things we noticed that all of our clients tended to make the same mistakes so they were not testing this offer before they shipped it.
They weren't doing version control they didn't have any automation all of these kind of things so we started jotting down some notes and the idea was that we went to a new client we're to give them these notes as kind of like a baseline and say if you look at this that's kind of what we're going to be talking about.
And as things tend to do it started out as being it will be like five pages and then we got to 50 pages and my wife said maybe you should turn this into something and we were like nah and she was she insisted so we thought we'd never done anything like write a book before and we had no idea what it meant.
So we came up with a really clever plan we would take what we had and we would send it to the publisher we thought was the best technical but publisher in software which at the time was Addison Wesley and we said okay we'll send this to them and they will say no but as a professional company they will tell us what's wrong with it and then we can fix it and they foiled us you know they totally ruined our plan because it said yes and so we ended up.
Writing pragmatic programmer for them at the time we had already created like some tools to help us write something in book format and so they reluctantly agreed to let us use our own tools and so we do all type setting and layout in the indexing of the first edition that one that you showed and from that point on we kind of got the idea that hey writing books is kind of fun and we already have tools to do it and so after a gap about a year we did the Ruby book or I did the Ruby book.
We brought in Mike Clark to write a book on automation and we suddenly realized we had a company and at the time the consultancy business was pretty dire it was just after the Y2K thing and people didn't have budget and so there wasn't that much work going around so we decided to focus more on the books for a while and that just went from strength to strength for a long time the only place I would like go to to look for technical books was pragmatic.
I knew that I don't have to like read reviews of this stuff whatever is on here is golden. Oh that's kind of you it was fun it really was fun and I was lucky because I had a fair amount of contacts from doing conferences and this kind of stuff you know if I saw someone speak at a conference and I thought oh that's interesting I would typically go and say hey do you want to write a book and a number of them were silly enough to say yes and so we
actually got some really good authors and some really good titles out of that I was in one of your I think it was the Ruby Conf in Australia maybe twenty thirty or twenty fourteen I was in one of your elixir I think at that time you were teaching Alexa and you were writing the Alexa book so I was in one of those classes yeah.
Okay cool cool that was always good fun so this book twenty four years later we look back at this book as one of the seminal works in the software industry right because a lot of the things that you talked about like you said believe it are not people were not writing tests there was no definition of what a unit test is versus what an integration test is what should different things look at automation observability and those things have become like standards now.
You don't have to influence anybody to go do these things anymore. You're still consulting. What are the new things that you're seeing that? Okay, I need to again repeatedly go back to my clients and tell them to do this or do that. What are those new things that you're seeing now? So I got to tell you first of all, just in fairness, that I'm not actually being paid to consult. I'm kind of busy right now with the bookshelf, but I am talking to a lot of people.
Many, many people. And little bit of history, if you don't mind. Back in 2000, software was in a really, really, really bad state. The vast majority of projects never completed. Even fewer projects completed successfully and actually delivered what they said they would. And when I say the vast majority, I mean, it's like more than I think 60-something percent failed. This was using 1990's technology, which was nowhere near as complicated as it launched today.
So people were responding to that by creating systems frameworks in which you could develop software. I don't mean like software frameworks, but like structural project frameworks. And everybody and their dog had their own framework that you were supposed to use. And none of them worked. And the reason that none of them worked is that they made a very, very faulty assumption. And that is that people know what they want. And people don't know what they want. Nobody knows what they want.
They may think they know what they want, but you give it to them and they'll go, oh, yeah, but that wasn't quite what I meant. And when you're talking about software that takes a year to write, that's a big investment to discover that you actually didn't quite want what you said. So the motivation behind the manifesto of fragile software development was to try to find a way to break that logger.
The realization is that the only way to make any progress when you are not sure of what you're doing is to use feedback. That's the only way you can do it. Literally, it doesn't matter. I don't care if you are walking across a desert or whether you are flying an airplane or whether you are writing a brand new podcast application. The only way to do it is to collect feedback and to collect feedback at every single level.
All the way down from the lines of code all the way up to the how are we doing as a business. I can very small increments. That's really flies into that right. So if you're going to collect feedback, the longer you leave it to collect the feedback, the more the error potentially. And the bigger the correction and a correction is a cost. So the more you can get feedback rapidly, it's a better. Now, this obvious time scales for different kinds of feedback.
So feedback to do with, say, lines of code, you can get pretty much instantaneously if you want. Write a line of code or two and then try running it against the test. And you can get feedback pretty quickly. Some feedback, like going to the client and saying, is this what you meant? Well, that might take a day or two. So that's going to like on a different level. Some feedback on things like, how did we do as a project team? What would we do differently? What did, what went well?
Well, obviously that relies on the length of a project. So there's different scales of feedback. But the rule is never start something until you know what feedback you're going to collect. Always try to minimize the time it takes to get that feedback. So basically, the feedback loop should be part of the structure of the project. Feedback loop is the structure of the project.
In the ideal world, the entire project is structured purely as a feedback machine that just happens to deliver software at the end of it. But here's the thing. In a successfully orchestrated team, it doesn't even have to develop a software. A client can come to you and say, I have this need. And the team can say, oh, you know what? We can follow that without any software. We could just like say, what do you do this, this, and this? And the client go, oh yeah, that works.
And they're successful. That's a successful project. So the feedback is what drives what you do. And one of the things that I am discovering back in 2000, the idea was you could write a requirements document. And that requirements document was then sacred. No one could change it. And that's what the team would develop to deliver against. And that's what the manifesto said was not true. The manifesto said, people don't know what they want.
So we have to develop incrementally, show them and get feedback from them and fix it. But now 20 years later, we are falling back into the same problem. We're talking about having, you know, liaison people, different people call them different things, product managers or whatever else. And these people are now the repository of what the client wants. And they will come down with documents that say, this is what's needed in this iteration. Oh, this is what the client wants in this iteration.
And so we form back exactly into the same trap of thinking we understand the requirements. We don't. In some ways, I think maybe without those people or those defined roles, but this has always been a problem. That the fact is that you can define the problem before starting to do anything about it. That's the falsehood. Absolutely. Well, it's based on two things, I think. First of all, in many facets of quote engineering, that is true.
To some degree, in that I can go and I can find the properties materials and I can measure the distance. And based on that, I can slap together a standard design bridge. And I'm pretty sure that it won't fall down. The problems in classical engineering come when somebody says, yes, but can you do it cheaper? But that's that's it. The whole point actually of engineering anybody can build a bridge right.
If I wanted to build a bridge across a river, I just dump dirt into it until you can drive across. Anyone can do that. But an engineer can do it cheaply and allow the water to flow and boats to go underneath and all that kind of stuff. And so engineers can do it according to a template very, very simply. The assumption is that we can do the same thing in software because we call it software engineering. And we can't.
There are no tables that you can look up to tell you anything to do a software engineering. Every project is different. Every environment is different. Every tool set is different. Every developer bless them are different. And so we are not doing engineering, but the assumption was we were. So that's one problem. It's a problem, but it's also one of the beauties of software development because I am a civil engineer by education.
And after a while, I figured out that I really enjoy the creative side of software development. Yeah, absolutely. But it comes at a price obviously because as software engineers, we have a very hard interface on one side, which is the computer. And by heart, I mean, unchanging. So, you know, we have to write something that eventually will run on this machine. And we have this very ill-defined thing that people want us to do.
And our job is to sit in the middle and basically make intelligent guesses to make that happen. It's like juggling with jello or something. It's a mess. And the fact we can do it at all given our limited understanding is seriously impressive. But by doing some of the stuff in the Agile Manifesto, we can reduce some of the risk because if we reduce the time to get feedback, then we can more rapidly work out if we're doing it correctly or not.
If we encourage proper prototyping and proper experimentation. And if we allow people to say, I made a mistake without punishing them. If we do all of those things, then we're in a better position to deal with this uncertain world. But many large companies do not have the flexibility to be, quote, agile all the way up. So at some level, and typically it's the middle management. If you talk to really senior management, they love the idea of agility.
They'd love being able to say that you'll get stuff out there as fast as possible. They don't like the idea of two-year plans. But middle management lives and dies by their annual budget and their annual forecast and what kind of stuff. And so middle management wants to impose that kind of engineering structure on their software team.
Because of that, and because of that kind of impedance mismatch between the management and the best way of running the teams, a whole bunch of companies have said, hey, we have the solution. And we have techniques that will work that allow you to run Agile in your company. And because it's a big company, guess what? They've got a lot of money. And so they'll give you five, ten million dollars for the magic secret source that will turn your company Agile. Does it work? No, of course not.
And it can't work because by the definition of agility, what works for one person will not work for another person. If I went and I found an Olympic figure skater, and I recorded every single movement that figure skater made, and wrote it into a book, and then gave it to somebody else and said, here, do this and you'll be an Olympic figure skater. Would it work? Obviously not. That's exactly what people are doing with all of these methodologies that are supposed to be Agile.
And unfortunately, people are believing it. And so nowadays, I got really depressed when I looked around and I saw they call it Agile now. And I got to admit every now and then I fall into it. Agile is not an noun. It's not a thing you do. It's a way that you do something. It's agility. And so you do things in an agile way. But these people aren't selling that. These people are selling, do this and you are Agile, which is totally bogus. Yeah, there's something in methodology.
Yeah, and there's lots of different levels of it too. It's a really difficult problem because everybody wants to be told what to do. When it's scary, when it's unknown, people are looking around for that figure that says, don't worry, I've got it covered. Do it this way. It'll be fine. As human beings, we seek to create templates for things because they're predictable. And it's the predictability that we like. They create an illusion of predictability rather.
Right. But I mean, also, it's the illusion that it's somebody else's responsibility now. It's kind of like the Nuremberg defense. I was just following orders and people look at that in a corporate setting. And a lot of these companies were people impose these frameworks on top of them. The project teams kind of go, well, this isn't going to work, but that's okay because now it's not awful. In the past, people used to blame us.
We've delivered what you asked for. The fact that it doesn't work is not our problem. Yeah. Yeah. I don't care if the horse only has three legs. That's what you asked for. So anyway, that got me really, really depressed. And so I started going around recently, actually, to try and take the temperature of what people were actually doing. So I put out a call and I've had maybe a hundred people sign up to chat with me over Zoom. And I'm about halfway through that right now.
And it has been absolutely fascinating. I would say maybe one in three of the calls were from people who were in a bad situation, where they were suffering from some of the kind of things we've talked about, or the classic bad manager or evil manager, or whatever it might be. But the majority of people were doing what developers have always done.
And that is they found ways to beat the system. And they found ways to get their job done, even within the confines of some of these really bad practices. And what they're doing is really interesting. I'm still kind of at the early stages of picking it all apart. But there's a couple of threads that are coming out consistently that successful and happy teams are all doing. Do you want to know what they are?
Yes. So one of the things that they're doing is once a team gets to a certain size, and that's probably about six to eight people. Or once a team starts to have sub teams, or once a project starts to have sub teams, one of the practices of successful teams is that they have at least one person, senior person, whose job is not to be on a team.
But whose job is to basically float between all the teams and just be chatty, be social to find out what people are doing, what problems they're having, where people are duplicating effort between teams, where people are misunderstanding what other people are doing, all that kind of stuff. And this person's job is basically to act as a lubricant between all the different things that are going on and to make sure that things are not rubbing up against each other the wrong way.
They are looking at people who may be struggling and saying, what can we do to help these people? One of the guys I spoke to was kind of interesting. He instituted a cross team pairing program, where for like a conference like one afternoon, week or something, you had to pair with someone on a different team.
He then tried to arrange things that say you had a developer that was struggling with racial databases, say he would find somebody on another team that was absolutely nailing it and say, okay, I'd like you to go pair with Fred over there for an afternoon and see if you can work out where his problems are and help him.
And doing that, you got to know what other people were doing and you actually helped what other people were up to and Fred then knew that you can go and talk to Sylvia or whatever her name was and ask questions in the future. So that kind of role. I think that is often misunderstood. People sometimes call that the project lead or the project manager, but those roles are telling people what to do in a way.
And the role I'm talking about is asking people what needs to be done. I think that's in a way similar to the principal engineer role at Amazon. I think the thing with the principal engineer role at these big companies is that it's so many things, but this is definitely one of those things is you are talking to people, maybe sometimes they're struggling or they're not at an individual level to different engineers and trying to figure out what's working, what's not.
And you're creating those bridges between teams or essentially you are the bridge between different teams. Yeah. I think Apple has a role that's similar as well. I may misunderstand this, but I believe there's an Apple role called staff engineer that does the same kind of thing.
And I think that's something that's important to me that companies that are successful have those roles. That's an important one, I think. The other thing that seemed to be important is giving the developers access not to what customers want, but to what customers need.
And the difference between what you say and what you mean and different companies had different ways of doing it. One company had this thing where when you kick a project off, you always bring either the team to the customer or the customer to the team.
And you actually don't just sit in the meeting room with their manager, but you're actually going to talk to the people that this work is for because quite often the manager will have one idea, but the people actually doing the work will have a very different idea. And you talk to them and you basically say, hey, good news.
We've got like infinite amount of money to write stuff for you. What would you like? What would you need? And obviously you're guided by what the project is about, but you're not bound by that. I mean, I had an example of that once on a project where I went one of my kind of prime sources for information was one of the people who was working on the help desk for these particular kind of transactions because every now and then I'd have to go ask, what does this mean and what does that mean.
And one time I watched and she was sitting there and she had a pad of paper on the side and she'd be talking to a customer and she'd bring a screen up and she would write a 14 digit number down on her paper, then go to another screen and type this number in when the call ended. I said, what are you doing? Why are you writing down this 14 to you? Oh, because it's actually the same system, but these two screens don't talk to each other.
And those are the kind of things that a manager might not notice, pick up on or whatever when they're specifying a system. So by having the developers out there and talking about need and not about requirements, I think is really, really important because the other thing you can do if you understand need is you can deliver against it.
You can prioritize it. And so if you understand what the customers needs are and what the priorities are, you can deliver incomplete projects that actually will help iteratively and also kind of speculative in that you can deliver it. And there have been times where I delivered something in that kind of vein and the customer said, hey, that's good enough. That's what we needed.
And all the rest, which is kind of icing on the cake, but we don't need that. I was curious on your thoughts about, you know, once was in the project where the system was perfect, but once I spent the night with people in the warehouse, I realized, well, I just don't want the new system because because there's fear this later in fear that managers don't recognize.
Actually, in this particular case, the guy was the manager and basically the whole project was almost sabotaged by that fear. I'm curious if you have thoughts on how you do with that because it's all I don't know what the right word to express is like latent fear. Is the fear there because they're being measured against some productivity or something like that and they left to learn a new way of doing this in a new system? What is the root of the fear?
So the context here is logistics company. It's a customs clearance process. People have been doing the same thing over and over again, tens of times every night to clear customer shipments. And then you introduce with the new system comes a slight change to the process too.
And I think with people who just repeatedly do the same thing over and over, there is maybe just fear of new, but I suspect there could also be fear of job security because if you can do things more efficiently, maybe fewer people are needed to do it. The other thing that I've personally found in those kind of systems is that as developers, we love looking for patterns. And so we look at a system like that and we go, okay, so we can have a look at, I don't know what the details are.
But this number and this number and that will tell us what bin it goes into and then this will tell us what form we need and what's kind of stuff. And we'll then design a system based on this really beautiful structure that we put in place, but the real world is never beautiful. And I bet you that the people on the ground handling all of those items know a thousand special cases that they have to deal with every day.
One of their fears is that unless you work that job for five years, you won't know about that. And that we'll produce for them a system which basically doesn't work because of those edge cases. And fixing manually all of those edge cases is going to be way more work for them than they ever had before.
And I've seen that happen quite a lot where we think things should be simple. We make things simple and we have this kind of arrogant clearly everything in the world has to be a hierarchy and everything has to follow rules and it doesn't. It just doesn't. I always think that the best example of that is the world's most popular software, which is the spreadsheet. The magic of the spreadsheet is it doesn't tell you what to do.
True. That's very true. It's very flexible. You can run cache balances or you can do like like you were talking in the other podcasts in 2015. I think those book publishers they enter data about books and spreadsheets. Yeah, there's limits. Yeah. Yeah. No, but I mean seriously, I mean what you're doing with that and with so many other devices is that you're enabling people to do things the way they want to do them.
And I would like to see us as an industry working on ways we can do something like that. And I think actually to be fair, we're not doing too bad of a job right now. I mean, if you look at the fact that you can now do on your phone stuff that you used to have to have a PC and a screen and everything else. But nowadays you expect to be able to cut and paste something from here to there and have its format adjust automatically as you do it. So again, you're in control.
You're telling it what to do and it's working out how to get it done. That is also something that I'm seeing. Is this move from specifying a requirement to meeting a need. Yeah, and AI probably also plays a role here because it well in theory and also I guess that later stages of evolution it can better predict as to what you want. And actually sometimes when I type an email in Gmail, I'm impressed how well it predicts simple phrases that I don't have to that type anymore.
I just press a tab and just complete it for me. Yeah, as an experiment, I tried to I was doing some layout work and I needed like three or four pages of sample text. So I actually went to kind of one of the AIs and I said, write me 2000 words about this. And I swear I could have published that and no one would have known. It was just scary, scary good. I mean, there is a lot of content right now that is being published that way.
I have a question about our startup. I'm almost a bit afraid to ask you this, but I'll do it still. So we are at a very early stage of our company. We have some early customers. We are validating our idea. We're seeing starting to see some validation that yes, these needs are there. And we're trying to figure out the way that we're trying to satisfy those needs is that a workable thing or not.
What I'm trying to say is this is very early stage. So if you had told me that I would be writing like a persistent without integration tests and things like that from scratch, I would have laughed at like whoever said that Arnauble is going to do this like five years later, right? Let's hear we go. We go because I was in a very structured environment in AWS inside Amazon.
With lots of resources, lots of resources. Yes. How do you decide to balance between like the pragmatic way that we're trying to do this? And at the same time be future proof. Oh, you can't be. It's impossible. Okay. There. First of all, the concept of future proof is about as crazy as the idea of perfection. How can you possibly be future proof unless you have some secret magic crystal ball that nobody else on the universe has?
There's no such thing as future proof. All you can be is easier to change than the next guy. You cannot predict what is going to happen. All you can do is say think of some bad cases, bad scenarios and say, okay, what happens if that happened? How am I going to structure my code so that that change is not going to put my company out of business? I watch people who will spend time putting in place three or four abstraction layers in their code.
Just in case this happens, just in case that happens. And this is code that they're going to throw away. This is their first prototype. This is their get off the ground, validate the ideas code. This is code whose entire job is to get out there, let people try things, crash every now and then, but give you the information you know to make the next one better. There is no point in future proofing that code. As a startup, I can pretty much guarantee that you will be rewriting that code.
The trick will be to make sure that you don't lose anything along the way. So you need to make sure that the data formats you're using are not proprietary to a particular technology. You need to make sure that if you had to switch persistence, for example, then you have an architecture to yourself totally around a unique feature of one persistence layer that others don't have. But apart from that, I don't think you have to think about future proofing at this stage.
At this stage, what you're thinking about is, how do I get stuff out there? How do I start making money from it? And how do I learn what I have to do when I do it properly? If you think about it right now, when you say future proofing, that's exactly the same as saying, I know exactly what I want. Or what I'm going to want in a year. Well, what I'm going to want. Yeah, exactly. And I'm sorry to be the bearer of bad news.
One of the interesting thing here is we've made the commitment to build our system on a no-sequel database. But then it also means sometimes I'm like, can you just show this? But then it requires joining two tables, which you can't in no SQL, right? Instead, you have to make either multiple requests or you just don't do this feature at all because it's too expensive, right?
But then if you would go with the last relation database, we know that maybe we reach like 10,000,000 users and it will start to break. And then we'll have to shard it or we'll have to re-architect everything. And the most logical simplest way to build the thing, if you were optimizing for speed only, would be to just use postgres on my SQL and validate it. But then we would have to rewrite the whole thing. But the problem is a relational doesn't translate the right into no SQL.
So it's a kind of a decision you have to commit to early on. Or you commit to the rewrite and that rewrite will take the amount of time that it will take because it's going to be a significant rewrite. Okay, or you choose the current simplest path, which probably is postgres, but I mean, whatever. I don't care if you use no SQL or SQL database, you choose the path that seems easiest to you right now and it will get you up and running.
Frankly, if you get to 100,000 users, then you should be happy that you have to make a change because of that. Not sad. You get something that's up and running, but you say to yourself, okay, I know that should we become really successful. This is going to be a bottleneck. But if your systems are anything like the systems I've worked on, the kind of queries that get you in trouble are a very small subset of the overall.
Most of the system is straightforward, go and find the user record, go and find the podcast for this user. And those are things that you can do equally well, relationally or no SQL. When you're doing the complex stuff, quite often it's in limited circumstances like reporting or you're setting up new accounts or something like that, you know, where you actually have to do the complex joins.
So maybe that would say, okay, as I'm developing the code, I'm going to be cognizant of the fact that this is not going to work as I get bigger either way, whether it's on the no SQL SQL. And so I'm going to make sure that I keep this isolated or at least in its own little source tree or something and then come the place where suddenly you appear at the top of hack and news and you got a million subscribers.
Then you get a very uncomfortable week when you transition that one piece of code to use some kind of cash, no SQL cash or a SQL cat, whatever it might be. And then you move on, but I have that problem, I find myself constantly not starting projects because I'm spending all my time thinking about what could go wrong and how I could fix it.
And I just I have to keep telling myself, stop it, get something working. I understand what you're saying. And I know it looks scary, but I think you can mitigate the risk without actually going over the top at this phase. So you're saying it's also it doesn't have to be binary choice. It's never a binary choice. The only people that will give you a binary choice is a sales person. Yes, right. There are no binary choices in the world. Any decision worth making is nuanced.
So this has been super exciting. We are already like almost an hour into it. And I know, Ilya, you wanted to go into the book and the identity that's tied to the book and that sort of side. Maybe this is a segue into that. Yeah, I actually want to ask a question. So yesterday on LinkedIn, yeah, my depose, I said, we are talking to Dave Thomas tomorrow. These are the things that we want to talk about.
And then you have any questions that we want to ask. So our friend, Goang, Hi, Goang, he is a host of software, misadventures, podcast, which is a great podcast. So he asked, is there something in the book that you didn't expect to still be the case after 24 years, but it's still true today. I think you thought about this for a while. It's a very good question. The opposite question is easy. You know, other things now that don't apply. The answer is compared to the original, yes, very much so.
There's also interesting question where the stuff we had in the book has been misunderstood. I guess we didn't explain it well. And therefore people have tended to misunderstand things. The biggest example of that is the, don't repeat yourself. Idea, dry to my absolute amazement, dry has become like a standard term in the industry. But people quite often apply it wrongly.
They apply it to the idea of effectively copying and pasting something. Don't repeat yourself. Don't copy this code and put it over there. But really it doesn't mean that. It means that you should only ever have one implementation, a particular concept, because that gives you one place to change when the world changes.
A repository process or information. Well, now I wouldn't go that formal. I would just, it's something where if you discover or if you're writing code and you find yourself doing the same kinds of things, ask us or an abstraction that would make it easier. That was a surprise that we didn't get that right. And so the wording in the dry section in the new book is quite a bit different. Like something that you thought would not be future proof, but it ended up being future proof.
There's a couple of things that people still aren't doing that are disappointed at. Nothing like major, nothing like headline. One practice that I believe in really, really strongly is the idea of the engineering daybook where you keep a physical book at your side. Yes, I remember reading the chapter. Yes. Yeah, it is to my mind. It's my life line. I get so much benefit out of doing that.
And we still had to emphasize it and people are still not doing it. In fact, people are doing it less. And people are doing things like, oh, it doesn't matter. I can stick it into notion or obsidian or whatever I'm using. And it's not the same. One of the bad things now is so many of the new programmers have never actually been taught how to write cursive.
It's really hard for them to take notes, but honestly, getting a decent notebook with high quality paper and a nice pen makes you respect what you're doing so much more. It makes you think about it. And the act of writing it down, slowing down gives you this time to reflect on how that interacts with all the other things you've written down.
And I don't know about you, but when I'm writing my daybook, I'll be writing notes and then suddenly I'll draw a circle around a bit and then a line that points up to like three or four items earlier on, which previously I'd make no connection. So I am a really big fan of daybooks. And I'm also a really, I got this pen, which is really cool. It's a fountain pen that's retractable, but it's not this pilot make one, which is ridiculously expensive. It's like 150 bucks or something.
And this one was like 30 bucks and it uses the same nib as the pilot uses. I love it. I keep this with me all the time. I'll have to insert a plug here too. So I used to have a pilot pen. I just you have it. I don't like it, but I have this carbon fiber pen, the fountain pen too. I mean, it's not retractable. It's a regular pen. It's 15 bucks on Amazon. But because it's carbon fiber, it's very light, but also like it doesn't get cold unlike metal pens.
It feels like metal. I guess it doesn't feel like plastic. I tried the metal pen to but metal pen gets too cold in the room. Yeah. Yeah. This is just plastic, but it's fine. You know? Okay. So while we're geeking out on pens. Oh, I don't have it with me. There is a traveling pen by a German company called Koweko. I think I saw it's pronounced, which is about half the size of this pen is about that big.
You unscrew it and then move the other end here and it makes a full size pen. Absolutely fantastic. Nice. So I keep that in my pocket. Coming back to this, something from 24 years ago, I kind of see that a lot of the things that we talked about have become day to day practices. Like nobody would question you if you want to say, like I want to set up a integration testing mechanism in here. Everybody understands the value of it.
The one thing I think we already talked about this, but I do want to reiterate. I think we still haven't fully embraced the agility of doing things because to this day, we are still going backwards from when do we need to deliver this project on and try to try to shoehorn in the process after that. And I don't have a good way like we struggle with it, Ilia, lots of places that you and I have worked at together separately, but this is a problem.
And I don't have a good way of how to solve this because there is a definite need of projecting out when are you going to be able to release this thing. I agree 100% I think that that is one of the big unsolved areas of agility that it is very, very hard at the interface between the agile world and the non agile world. And I don't know anybody that's actually come up with a universal solution for that, which I did, but you're absolutely right.
Okay, so let's differentiate, I guess the pragmatic stuff and the agile stuff, the agile manifesto never was intended to be a set of practices that you had to follow. All it really is is a set of values and you use those values to make decisions. Should I do this or should I do that exactly the same as every other value in your life, you all raise and you learn and you develop your own values to help you make decisions.
And that's all the manifesto is so it's not a failing of the manifesto that that's happened. Well, no, that's not true. It probably is a failing, but it's a failing because it had high expectations that people could value these things, quote, universally and people don't.
All right, I'll accept that as being a failing. I don't know what you do about it though, but there is the other side to that too. So for example, testing right in the first pragmatic program, we went overboard with testing because people were not testing so many projects being written with zero tests.
And what tests there were attended to be if it outputs this string, then it's correct. So we went over the top with testing and I think combining that with the XP practices of testing and all the rest, people got religion and you know, around about the mid 2000s, it got to the point where people were shaming other people if they didn't have enough tests.
I never write code unless there's a test blah, blah, blah. Well, that just makes you an idiot. Testing is a tool like every other tool and you use it when you need to use it. And if you think agile means I have to write tests, then you are by definition not being agile because the agile view would be let's see what happens if we don't write tests.
Yeah, you know, it kind of reminds me of when I and a few other folks, we started a team at Amazon and basically we were free to set up our own processes as we wished. So we were like, let's do scrum the right way. And then we started with doing those points. As you know, points they shouldn't equal, you know, man hours or whatever, mandates, they should be like abstract points. And after a few weeks, we were like, what are we doing this? It just doesn't make any sense.
We figured out that we're spending more time in figuring out the points than the value that the points were giving us. It was pointless. Very good. Yeah. Yeah, but that kind of ceremony, that's the kind of thing that I object to so much. I mean, there's nothing fundamentally wrong with scrum except people do it as opposed to using it as the basis for developing your own practices.
I don't care if you go out and you kill a cat and do care about that, but in principle, I don't care if you go out and kill a cat and read its entrails and use that to tell you how to write plurfer. As long as every day, every month, every year, you're looking at your processes and improving them based on your experience. Because if you do that, it doesn't matter where you start. You will eventually get to a better place.
I think the thing is that we start to create these templates, then we start to quantify the output of these templates with metrics. And then these metrics become the thing that we're driving towards rather than the actual improvement of the process that we were seeking. Yeah. I mean, you're familiar with Taylorism. Yes, Frederick Taylor. Yeah.
Oh, Frederick Taylor was the original time and motion man. He went around companies and he measured with a stopwatch what people were doing and how they were being more effective or not effective. It's like 150 years ago. Yeah. And he then used that to produce metrics standards against which people had to work. And it basically heralded in the kind of more assembly line based way of looking at working where people were became units of production and resources and not people.
I mean, obviously it worked, but it had limits. And that's when eventually the Japanese worked out as is a better way of doing this. And we started to see Toyota come up with their techniques for managing production lines and stuff that work better. But yeah, you're absolutely right. Because what happens is if I can't put it in a spreadsheet, it doesn't exist if I'm a middle manager.
So give me your metrics. Maybe that's a product. Forget about the podcasting. Right approach that just generates random number metrics. The teams can give their managers and make up name make up names for the metrics. And no one actually knows what they are. If you were a part of the company, we might as well have done it, but we have to feed our families. Oh, all right.
We're running short on time. And which one ask one final question. So they view are very well known for the book. I mean, you've done other things too, but this is the thing that is like V main thing that you're known for it. It's been like quarter century since it was written. So how did that define your identity or do it? How much of is it part of your identity?
Oh, that's a dangerous question. So I don't think it defined my identity internally in that to a large extent, it was simply recounting what I was doing anyway. So in that way, it didn't change what I did. It, however, it gave me the ability to interact in circles that I previously didn't have any access to. So I got invited to speak to conferences and other speakers would talk to me. And you know, we could have conversations and everything else.
So in that way, it dramatically broadened my access to cool new things and to people's opinions and to people's ideas. At the same time, we called our business the pragmatic programmers after the book. And we called it the pragmatic bookshelf. And my pretty much universal handle is priceless. So I guess you could say in that way, it is very much part of our identity. I don't know how significant that is. It certainly doesn't hurt.
I know that some people have done stuff like, well, say the manifesto and have then effectively pivoted what they did to be based on that and built in some cases ridiculously successful companies based on the manifesto stuff. And that was not of interest to me. It probably should have been because I probably worth considerably more than I am now. But that wasn't what I wanted to do because it didn't seem to be that exciting, just like taking what we already had and rehashing it.
So in that way, I would say it probably has affected my density less than it has other people. I'm sure it has, but it's not a motivator in that way, not a driver. Yeah, I think one of the things like this can affect identity, right? Are there things that you decided not to do because you are the pragmatic Dave?
Oh, okay. Yeah, there is actually a really interesting little side effect. And that is whenever I'm doing something like writing code that's public, back in my mind, I have to make sure I actually do it the way I said, said we should do it. So yeah, it does affect me that way. It keeps me honest. Let's put it that way. Yeah, makes a lot of sense. So, yeah, Dave, it has been a great pleasure to have you on the podcast.
When you said, yes, I think it took actually I had to work through your assistant or your marketing manager. It probably took a couple of months before actually we got a response. Oh, sorry about that. No, I mean, it's all good. Like, we don't get a response from most people at all. So we were super excited.
This one was, I told you, yeah, this is not like never coming back. So yeah, might as well ask you. No, I mean, honestly, I enjoy this kind of thing. I always learn because I always find that I'll come up with something that I haven't quite thought of that way before. And people ask questions that just make me think about things and it's enjoyable. And you're nice people and it's nice to talk to nice people. It was amazing. Yeah, thanks for coming.