Spend some time looking at the system in which he work, understand how the width is working. Understand how flow is for your organization and then you can work to optimize that you can find. You can look at where those hierarchies are the informational hierarchies, it might not be on the orchestra, it might be in the value stream maps and work to change that.
Hey everyone. My name is Henry Surya, we Robin. And you're listening to the Tecla Journal, podcast the show, where I'll be bringing you the greatest technical leaders practitioners and thought leaders in the industry to discuss about their Journey ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our Journal. Hello again, to all of you my listeners.
Yes, you are listening to The Tackle, a journal podcast, the podcast where you can learn about technical leadership and Excellence from my conversations with great thought leaders in the tech industry. If you haven't, please follow the show on your podcast app and social media on LinkedIn. Twitter and Instagram. And to appreciate and support my work. Subscribe as a patron at technology. No, dot f / Patron or buy me a coffee at technology. Not deaf / tip.
My guest for today's episode is James Lewis. James is a director at thoughtworks and a Of microservice architecture. In this episode, we went back Memory Lane to the time when James first coined and popularized, the micro service architecture. James described, his definition of a micro service and it's important characteristics. He also shared the reason microservice Evolution, including the swing between micro service and monolith in the second half of our
conversation. James shared, his insights from complexity science related to different scaling patterns, particularly he explained how different Archetypes can affect an organization's growth rate and towards the end James gave some tips on how organization can detect signs of suboptimal growth. And what we can do to maintain
organizational agility. This is such an insightful discussion with James and I find it a really interesting story, how he first got the idea of the micro service architecture. I hope you enjoyed listening to this episode and learning a lot from it. And if you do, please share this with your colleagues, your friends, and your communities, and also leave a five-star. And review on a podcast and Spotify. Let's go to my conversation with James after hearing a few words from our sponsors.
Are you looking for a new cool swag? Tackle, it Journal. Now offers you some swags that you can purchase online. These wax are printed on demand based on your preference and will be delivered safely to you all over the world. We're shipping is available. Check out all the cool tracks available by visiting technology, you know, dot, f / shop and don't forget to break yourself. Once you receive any of those tracks. Hi everyone.
Welcome back to another new episode of the technology on our podcast today, we have another thought workers in the show. So his name is James, DeWees a software, architect and director. Thoughtworks is based in UK.
If some of you don't know about James Lewis, he is actually the person who coined the term microservices with Martin Fowler back then in 2014 and I think some of you would have known by now that since then it has been a craze and hype and so many You transfer following microservices and today we will touch on a little bit about the history of microservices and some Evolution since then and we'll talk about some other things as well.
So James thank you for the opportunity to have you on the show. Looking forward for this conversation, hiring your thanks very much for asking me to be on the show. It's a real privilege and pleasure So yeah, thank you. So James in the beginning, I always love to ask my guess to actually share any courage, you need highlights or turning points that you have in your career to be share here for the listeners to learn from Yeah, that's great question. Actually, what wants to create
hearts? Are there's been so many. I've been, I think genuinely so lucky over the 26, or so years. I've been doing this or adjacent roles in technology. I think I use the words luck deliberately, right, because my career has spanned really work. If you think about the last 26 years of what's been happening in technology, expand the birth of the internet and the World Wide Web. I was one of the first generation, the UK to grow up privileged enough to have access to computer in school.
We do. This BBC micro thing that every school was sent to computer by the government or by the BBC SE. So my career really spans from that when I was sort of 89 years old through Java through the birth of the World Wide Web through Apache, through sun, Microsystems, all these
different milestones. And I've been very privileged along the way to work with some bread people, we go way back, I started off doing physics to be honest, I wasn't diligent enough to be a good theoretical physicist. I didn't do enough practice and I'm definitely Not accurate or precise enough to be experimental physicist. So I said fell into systems Administration and programming after that, which I've really loved ever since. And I start off as a sysadmin that I was a Oracle database
administrator for a while. Well, that pl/sql goodness back in the day and then I studied with programming on. Would you believe TCL to control language from things like chill? Control language for vignette story server, which was a sort of very early content management system for the web. Java are going to Java, I guess back into of 2001 ish. I joined for weeks about 2005 as a Dev to don't know what that meant at the time and I still do it.
But that was my title death to. And I've been a thought, which ever since, which is it's meant. I've been incredibly privileged to spend time with some amazing people, you know, forwards, London 2005. Until today has been tremendous amount of talents for which in general forward, slow ball has this huge amount of talent, but for which London, but I joined there were these People who went on to really change our industry day Farley and Jazz humble where they're being they went on to
write continuous delivery. You had in that price and Steve Freeman who with a crisis of J, more from went on to write growing object-oriented software. Guided by tests down North, was there who went on to create Behavior, driven development at area fifty Daniel tools and obviously very evil, very influential and wise figuring our community and, you know, Cindy Mitchell, who is the MD at
the time but was former son? I think provides the swing libraries for Java. So these amazing insects amazing people. I should mention Eric doing a boot as well because he was an old friend before I joined the likes of reunited with him and thought ways but this is a huge talent that really helped me to embrace the fun, right?
The fun of working with technology, you know there's the old adage, if you enjoy, what you do, don't work a day in your life and I think I've been very privileged and lucky to have that come true for me and I enjoy, what I do enjoy working with the teams of people at thoughtworks, I enjoy meeting new clients and learning new. Domains and still do. I still find it fascinating? So, yeah, I guess that's a brief precis of my early career. So you've been at dogs for I
think about 17, 18 years by now. So adapting. That's pretty long, super long. And I think you naming some of the people just now. Right. I think some of these are thought leaders in the industry. They also change some of the trends in the industry as well. So I'm sure. Today, we'll be talking a lot about some of the trends that you also yourself a drive in the industry. So let's go back to the time then. C14 right?
You are the first to coin the term microservice with Martin Fowler. I don't know who came with the term but I think if we can go back and reflect the story, how did you actually come up with this idea? It's a really good question. And actually you need to go fears before in fact. So one thing about thought works is there are so many amazing people working that the weights doing amazing things, right? So I mentioned a couple of them days Farley mentions Jazz humble.
So continuous delivery eventually. Daniel to host North his thinking wearing the A really informs me. I worked on a team, building our services, our retail bank, with him restful services. That really influenced my thinking Jim Weber idea, Robinson, who were Christian practice, along with other co-authors service that really tied together. For me also Ian's observation that the be of the web, not behind the web, this idea that. Why is it in big organizations?
We spent so much time, creating our own infrastructure is our own protocols. We try and reinvent the wheel the whole time and actually there's this massively successful distributed system called the World Wide Web. Why shouldn't we just Use the Technologies from the world wide web and your breasts Walker can choose level 3 Richardson maturity model level 3 architecture.
So I was lucky enough to be in that on the place and the other thing I was lucky enough to experience a really formative experience for me as he was in 2007. I think I went to my first cross disciplinary or cross technology conference it was called Yahoo in all who's Daniel tears nor they'd really pressed me to go to this thing.
Blew me away, right blew my mind, people talking about Cheri different things, different types of integration different programming languages coming, Of where Java came from virtual machines in frustrations just this massive, interesting information so that when it's my home, and I think that started me on this sort of Journey to try and synthesize a bunch of the things that I was seeing that were successful and to come up with some way of doing the all at once.
I guess in the same way that XP extreme programming came about, right? It's a bunch of practices or bunch of things that people were doing that, we're really successful independently. So let's put them all together to do the ball. That's going well, I think my brain. Works that I see curtains and I'm able to phone connections between things. I think there's maybe a legacy of the physics earlier my life.
And so gradually, these sort of things I was seeing in the waiting outside of thought, work started to fit into place, then there was one event. Actually in 2011 was lucky enough to be invited to something called the software architecture Workshop, this was in just north of Venice. Actually, I can't remember the exact name and there were about 30 or so people. There had been invited to all across Europe and the US Africa. Actually, it's an unconference.
So we were talking over 3, Days form your own agenda and one of the themes that came out over that three days. Was, why are things so hard to change? Why is it that these big products? There's some product people that we've got this big product. How do we break it up? How do we change it more effectively? How do we improve maintenance of it? Do you build a new one or do we strangle it all these different questions I can? Honestly say, I had a sort of never, it'll never happen to me.
Again, you'll happened once at least I literally had like a ding in my mind. You like, literally, almost Like a chime, right? I know. It's like a cliche to say, but genuinely what? My thought. Maybe, we're solving the wrong problem. Maybe if we didn't have, the problem of things that were too big, maybe, if we had lots of smaller things that might make things easier to deal with. So I came downstairs, very excited in the morning to Jimmy Nielsen, who was organizing the event.
The Great's domain-driven design expert in voice person. I said, Jimmy Jimmy, I've got this idea, his wife and kids with our he often travels with his wife and children and he was looking at me quizzically and I said Jimmy I've got this idea. Like, what happens if we just have everything as an act of the route and they talk to each other across the network and he looked at me as like, yeah? Okay.
That's interesting. James, but you're crazy, you know, he probably remembers it differently and yeah, and then we sort of went our separate ways and I went back to a client projects in the UK and we started putting some of this stuff in the practice. So rather than doing threading within the monolith or having a structured model, if we decided
to scale via processes. So every time we wanted to have a new capability or subdomain if you like we've created new service, And along with that, there came a bunch of other practices. We have one repository per service and the reason we have one repository service is because I'm stupid, right? And generally, that was the reason. It's like if you put more into the same repository, then I won't be able to avoid finding the abstractions and extracting stuff into shared libraries.
And we don't want to do that, right? We want to have across the wire a pi to 3 Pi communication. We started putting all these things into practice. We had something that looked very much like ansible free ansible in order to manage all this stuff on. Aw. Yes. And it seemed to work pretty well. And I did a talk at the time called Java, the Unix way.
And I first did that I think in Congress that weekend ever walks in Poland, I also then did it at short notice in Q Khan, I think it was coupon 2012 in San Francisco, Jose humble. That asked me to go and do a training course.
I did a training course on micro Services which was a little bit scary because I had the guy who sat next to Roy Fielding Adobe on my training course, I was trying to teach him about resting, like for this is a bit awkward, and then I did this talk and then it sort of went a bit quiet. Martin Fowler, you know, Martin took a friend are, he'd been trying to persuade me to write it up? Should write it out. We should write to that. We should write it up and we funded a internal Summits.
Always microservices some food people in from around the world to talk about our experiences and eventually he decided that the only way to get me to write it up, was to Fly Me To Boston, and to go for a sleepover at his house. So that's what we did. I flew to Boston, I spent a week living with Martin and his amazing wife, Cindy drinking, good craft beer. And writing about microservices, I should also point out that the history here.
Yeah, and this is where the whole thought, waves, diaspora and the sort of Corrections to the rest of the industry coming. There were other people at the same time, talking about exactly the same thing. So whilst I would argue that Martin and myself, we wrote down what became the description of the characteristics Fred. George at the time, was also talking about micro services in a different style, but he was also talking about these really small things that he was told
about functional microservices. If you like and Adrian cockroft at Netflix, he was also on a That clicks to what he was going, fine, grain service-oriented architecture. But then, you know, Adrienne for a George's. Explore wigs Adrienne has been a longtime Ally and friend of thought whips and marries all the people I know. So it's no surprise really that these ideas all came together, so you could probably say that the three of us Fred Adrian.
The myself with the people who sort of put it all together in one place and then I were to that. So yeah, that's, I guess, a very long or very short version of the story depending on cover. All right, I think it sounds really interesting. II didn't know some of these historical moments, right? Like when you shared the lightbulb moment that you got and then you shared this idea and then it became like a thing that we found out from Martins block, right? So, I think it's still there.
Beyond 2014, both of you publish this on his blog. And I think if we have to go back to the definition, right? If you would Define microservices. Now, I know that people take microservices into different types of understanding different types of characteristics, but maybe from you, originally, how would you define microservices? This now I probably and we've talked about going back and writing division to a writing updated as addition if you. Like think it holds up pretty well.
I mean, there's always going to be I think Martin calls its semantic diffusion right where the original meaning of a term is lost over time. As many people start to add their own meanings to it but I think it actually if you look at those characteristics they stand up pretty well and might make more of a focus. She on the business side of things, rather than the technical side of things, I think the technical side of things is accelerated off in all
sorts of directions. Remembered all came after this, right? And kubernetes, these are technologies that have been invented or popularized because we doctors being the other line Tech has been around for ages to help obviously a lot with building distributed systems.
It makes it a lot easier. I think we're, I'm Consulting, the things I see, least the pride are the things around products versus projects and the organization around business capabilities and really embedding domain-driven design Into the Heart of what you do and I think that's because that's actually harder in some ways to solve than the technology problems of breaking things up and promote communication. And why do I say that will? Because it involves humans. Right.
So, whenever we're in a position where we can pull humans, and teams and people and structures, and this may be related to some of the stuff I'll talk about later, but the time scales to change and for experimentation, on much longer, it's much harder to change your developer ization of your organizational structure frequently. Like we can do when we refactor code bases or where we actually redesign. A Nori architecture, because it takes a long time to see if
something's working. I think the other thing is with the characteristics, I'm pretty early with emphasize Conway's little more in a sense because I think that again, is something that's a real driver for a long. The microservices, I'll all the problems I see with implementations, we often get asked a question. Like, you know, we've got these two thousand Services, I how am I supposed to understand all of those to which my answer is? We you know, right?
Yes, that's the whole point of breaking things up into smaller units. Old Rising around the capabilities, rather domains and then split the team's up like that. Conway's Law happened in there. So that all you need to know is about the things that you know about and the things may be adjacent to. You don't need to know about all 2000 or 4000 of these things that I think that's definitely something that I would spend it. Go back and focus a little bit on some Newman is going to
lovely phrase. He talks about, you know, when you've got a big problem, how do you solve a really big complex problem? Break it up into a lot of smaller, but still complex problems and solve the small problem. That's really what We were trying that's one of the hearts of microservices. The other thing I probably would have emphasized the bit more and this is may be due to my journey as a software professional since is the concept of flow. So I probably would have put
more effort into describing. If are these optimize your ability to get things done and to scale teams around these things because that fundamentally is such a huge issue in our industry. Now how do we go faster? How do we get more stuff done here?
Pretty developer experience. How do we get value out the door improve cost of delay, all these things and I think microservices at the heart or a way of doing that, but only if they're done properly, are quoting properly by paying attention to how you split up your teams and that you don't get this, massive distributed model, if that everyone has to understand everything and everything has to be deployed along once, but that just might be more able obsession with flow and flow
efficiency that I've developed over the last few years. I'm not sure. Yeah. So I think what you said is really insightful, because many people actually took to the extreme on the technology side, right? So things like for example, lines of code like how big is a micro service right? Or what technologies that we should use maybe like kubernetes
contain, and things like that? But I think you remind us the important things which is already defined by both of you in the article itself 9 characteristics, right? And things that stand true to the test of time. I believe things like for example, componentization fire Services, organizing around business. These products are projects and things like that. So I think that's a really insightful.
And I like the way you also mention about Conway's law, I think 11 also mention that it's like a law of gravity. You can't escape from that and you just need to adapt to Conway's law, right? The other thing about microservices lately is that people have been swinging, right? So some started with like service-oriented architecture, then they went into microservices, monolith became quite a trend as well because people implemented microservices in the wrong way.
And our people started to think about right size service microservice. So, what do you think about all these different swings right now? What could go wrong here or what is the underlying Trend behind all these? That's a good question. I mean, going back and I'm fortunate enough to have been involved with industry from family or time. You mentioned service-oriented architecture. But what was the driver for SOA to start with, right? So this is pre ATI economy. This is in many ways.
This is pre Thomas. Write the ideas behind service-oriented architectures. It was about how to maximize the investment in our To like to use date, right? If you think about how organizations grew in the 80s and 90s, what you had internally, you'd end up with lots of these different platforms that were sometimes doing the same thing. Sometimes naughty is our big Erp systems that were like sucks. All your data is really hard to get stuff out of, maybe one department.
But one thing, another department. But another thing that they both did the same thing to end up with duplication of Investments and things and service oriented architecture. Really, was this idea of how can we create a set of common building blocks that we can? Using our organization, right? So this is going back to capabilities as well as business capabilities. Were first talked about in the late 50s.
So, this is nothing new, right? But when we talked about capabilities, they were talked about, not in the sense of Purity, is in the software. I adjust the services. They were talked about in the sense of people processes and tools that make up an important part of what your business does and serious oriented architecture was about then almost like, creating these units that will allow these things to talk to one another without a duplication.
So on so reducing cost by doing X, Y Zed, and I think as things change those, then you move into the internet age, into the world of of where everything is digital. Doesn't matter, if your bricks and mortar retail, you're also a digital retailer. In a sense, at least you have to offer your catalogs digitally Etc.
Then the meaning of a service-oriented architecture started to change and we started thinking less about internal integration between systems and how to make that more stable, which is really where my head was back in the day into this idea of K. How do we offer apis? Externally, how do we build software? That's providing these apis that either our channels can use or that we can offer as a service, Google Maps as an example.
So maybe there's been this long term Trend towards more service-oriented architecture everywhere. If you like backing 2014 15, in my training courses, I'd be asked how big should one be, you know, do you think we'll end up with thousands of these small things that are 50 lines of code, each which some proponents of them at the time were saying, you know, there was one classically This is my chance to
chat. It comes with a minute, but his rule was you can write your service in any language. You like, as long as you can rewrite it in half a day in another language, which means things are really small, you know. So there was that movement. Then there's also the idea of modular monoliths do we have to start with microservices? This seems like a really complex way to build simple systems and it is really complex ways to
build simple systems. I don't think anyone's ever said that you shouldn't build structured. Bolus, I think the problem came that over time. I'm we saw this really hard once you go past a certain scale to keep the model with modular. If you've got something that's relatively simple and small, then that's fine.
But at a certain point with a certain amount of tune on the team with a certain number of people working on the platform excetera, you tend to reach a point where things start to become Tangled. Yeah, you end up having to put a lot of energy into managing Tech debt or a lot of energy into avoiding entropy if you like in the morning. So I think naturally So the strong one way, I think.
Then they're actually we're swinging somewhat in the other direction but I also think there's something in the middle, right? I suspect we still haven't settled on that yet. Which probably looks a bit more like fine grains, service-oriented architectures, which you guys agency because at certain points are certain scales, you can't solve the problems without using this sort of distributed system. There are certain things you have to use these sort of patterns for.
And obviously, as an industry we are amazing. Well bit like, uber Asia know this. Snake that eats its own tail. We keep on going back, round and round and round and do the same things. I had that there's a funny conversation with someone recently about what should we use, Simple stack to build like an internal simple internal ever.
That's never going to be exposed to the outside world is going to be a really simple line of business, kind of thing, not many users, and of course, the immediate reaction from many people were as well. You first of all, you start with react. So then you need maybe graph to hourly this massive, massive stack of stuff, right? Just to build something that you could do really simply via so.
Side rendering and I think that's the safely microservices will hit everything with the microservices tournament. The getting that you don't always have to solve every problem that way, I still think they're very useful tools though. Although I do often when I introduce myself as the sort of grumpy old full of microservices, I do say and I'm sorry about that, you know?
Right. So I think the one of the classic things also for people who are building internal tools, like what you said just now, right? So there are only a few developers but they create more than the number of basis that they could handle as a 1%. So for example, one person could handle three to five microservices on his own. I think that's also probably not the right thing to do. So I think, let's see how it
goes in the industry. These days I want to continue our conversation to the other thing that you mentioned, just now conversation, right? We are always trying to optimize ourselves, how can we deliver things faster and Lady? You'll have also been giving talks in various conferences about this topic which I had the opportunity to attend in y'all Sydney back then in December, which I found really insightful. I want to go and maybe cover some of the things that you
mentioned as well. So one of the most important thing that you mentioned in the talk is about most companies right as they get bigger, actually, it becomes very difficult for them to scale even bigger while in the top. You also mentioned that AWS, interestingly didn't have that characteristics as they get bigger actually, they become much, much bigger looking back from my experience as well.
Right. I can see many companies as they tend to become bigger as more number of people join the company's things start to get slow. Our. So maybe in the first place. Let's go there. Then try to analyze why things become slow as most organizations Ruby? Good. Sure you're absolutely. Let's do that. My observation of Max is and I should point out that again, this is matter research, this is me summarizing. The whole water of work.
Really insightful stuff that's been going on in the world of biology and economics in the world, things like City Planning, or all these different areas that generally come under the term complexity science. So it's been a huge amount of work going on. Is Singapore's a big sound reflects the sciences that is an Institute in Singapore. For be the home of complexity science has been the Santa Fe Institute in New Mexico and there's just been some super interesting things coming out of there.
Well, what I find fascinating is they put together a cross-disciplinary group of scientists of academics, right? People who wouldn't normally talk to one another talk to each other, and they get them together and they bought a conversation if you like. So you end up with a columnist and biologists and physicists and chemists and geographers,
and all these dead. Different groups of people coming together to talk about problems in a holistic way or think interesting things that they're seeing in a holistic way. One example would be economics. So Santa Fe Institute is a driver of this idea of complexity economics or non equilibrium economics where I want to murder this definition but the idea is that traditional economics is based on four Librium States frankly because the math is easy. The math easy.
If you're talking about systems at equilibrium, the observation from a bunch of people coming together in Santa Fe was hang on. It most things answer, accurate rewrite, most things are not every dream states like the world is in a non equilibrium state. So why is it we're trying to approximate what's really going on and economics with these massive oversimplification. And the answer is because the economists are economists, then our mathematical physicist, right?
So, the mass, that is easier for the mathematical physicist, is not so easy if you don't have that. So, that's one example of the sort of cross-disciplinary stuff has been happening and they've been doing some amazing things. Usually, agent-based models they can now do things like say, Simulated economy is not growing economies in the large and they can produce by putting in the rules that the difference countries in the world, apply to the capitalist systems of their
end. So the different Market controls just by using these simple agent-based models exchanging value. And these Market controls, they can produce the gini coefficients for Nations around the world so you can say left before in the Netherlands we've got this type of Market control. All right, let's run the model with these Market controls own turns out.
We can get the gini coefficient so that's the ratio between the richest in society in the poorest in society where the wealth is it pops out for the Netherlands? If you put them off in controls, the UK has its unique coefficient pops. Acts, they do some fascinating interesting stuff across the board.
I think what really interested me was how it then related to have some of that research into scaling laws into how things get bigger relates to what we do as technologists, or what we do in the area, or the organizations, in which we find ourselves. And there's Seems to be, I think, in, at all, that talk, about Geoffrey West hooks to paraphrase.
Every time you see straight lines on a graph, in lots of different places, you all these straight lines, basically, replicating across different domains different parts of the world, different types of complex system. Then you should find something interesting in it. I you should be looking like maybe there's something underlying maybe there's something that's like a truth.
We can go after and that we can use scientific methods to try and determine and that's what they've kind of been doing in. Santa Fe around scaling Jeffrey Webb. The professor West, I should say, I don't know them, they're the gentleman. He's been instrumental as well as some others. And what they've sort of found essentially is with different types of network.
And I'm using the words, the network in the kind of mathematical sense, if you like so, whether you've got and like, hierarchical like graph, like directed graph, like networks versus things like social networks, different types of networks, coming explain the different straight lines on graphs, that they've been finding and all sorts of weird. Weird and wonderful ways with wonderful places.
So, in example, the straight line on the graph that is the scaling law for metabolic rate in mammals, you can draw a straight line. There's now is getting bigger. There is the straight line that can be drawn about how many calories, essentially, they need to intake to take in to survive and there's another straight line for how the infrastructure in the city's expands as a city gets bigger, can you water pipes and much more water pipes? Do you need?
And it's a very similar straight line with a very, very similar experience and it's the same A companies as well. And Company, growth revenue is the same and all of these different things appear to be related to a particular type of network, a hierarchical network. So water pipes and cities are hierarchies in. Mammals it our circulatory system is a hierarchy essentially for heart, down to the quarries, in organizations, we have hierarchies of
information flow, right? So, where do ideas come from? Well, is direction from from whether you orders in some senses come from, and they tend to be hierarchies as well, because as companies grow It tends to become easier to create a hierarchy which information trickles down through in the same way that water flows through pipes in the city than it is to do it any other way. But the interesting thing about these hierarchical networks is that they all scale using what's
called a sub linear scaling. Also this scale with its UB exponentially. What that means is as you double the size of a city, you don't have to double the number of water pipes. Instead, you get less, you have to do less than that, and the same as you double. Size of a metal. You don't have to double the amount of calories intake in its scales, less than that. And it's the same for all those ations for physical
infrastructure. As you double the size of an office, the network cables are double, right? I mean, it's the infrastructure in the office, doesn't double. This is what we call or columnist, call economies of scale. So, as you get bigger, you pay less for things essentially, and all of these complex systems exhibit these behaviors. But it does also implies the other things employers. The fact that things slow down right things, move more slowly when you have deeper
hierarchies. So humans experience, life more slowly than a marriage does. And it's down to a mathematical relationship related to the death of our civil HP system compared to the depth of the circulatory system of most all seems to be explainable that way. Anyway, I should say science, we might find something else out. And similarly, with cities have cities, get bigger. The higher Arc is give us anything.
And then the issue thing is with companies, I what happens with clump, I think we experienced this in our everyday lives here. If you work for a big copy as you get bigger, things do get slower, we do experienced this personally day today, you do individually day today but also you can see it in the things like the results that public companies Post in terms of Revenue and so on. So as a company doubles in Size, Doesn't double the amount of Revenue.
Interestingly, the other thing is like, mammals companies die all the time and this is another fascinating similarity between the two which It is also expandable via these lobbyists, your own complexity science, complex adaptive systems, so has mammals get older as we get older we're taking the same number of calories but we're having to direct more and more of those categories to self-repair to repairing ourselves rather than you know vigorously youth without having
to do that. And so therefore we slow down the age, things like brain down and eventually we passed away. Seems like companies do the same thing with something interesting facts in the book scale. Think it's the half-life of a company on the stage. Is 10 years, so every 10 years, 50 percent of the companies and the exchanges will change. So as I say, there's all these interesting things that seem to be related to hierarchy things. Slowing down, things taking
longer as things get older. And so on, which I thought was really interesting. Right, because in my experience, when you plot out, things like development, life cycle is like, you look at the big old organization. You look at the software development lifecycle you drop that as a value stream map, you'll end up with these huge
long processes, right? Massive, where you've got like it takes a year from an idea to get even to the project management function to be able to see scheduled to have some work done, right? Because bouncing around between different review, board bouncing between different architecture groups and the technical business analysis. This is the business analysis at La one. Never know what? Any of these things me then you hit development and then maybe some development gets double.
It takes however long, it takes and then it pops out into some really long involved process of testing, you know, Testing and user acceptance. Testing regression testing performance testing there until you eventually production and you see this in every old, big traditional company that I visit conversely, if you look at younger companies, often they don't usually they have much shorter time to market, right?
They're much more able to be much more Innovative because the time it takes for idea to go from in someone's head that information passing through a series of process steps to make it may. Tends to be much shorter and slowly over time. There's this thing that companies are add all these processes, these constraints, they add hierarchy to manage all this constraints and all these processes and things start slowing down. There's also another interesting idea that larger organizations
do the same thing. Bigger older organizations rather do the same thing that happens with mammals where they have to spend more of their revenue on keeping the company running. In the same way that, as we get older, we spend more of the color if it intake on Ariel cells companies, they spend more money on just being the company. Whereas at the start, they were spending money on Innovation and products, and people and all this cool stuff and that's why
they were successful. Eventually, they end up in this position where they're spending. So much just keeping themselves going. They stop thinking about Innovation or innovation becomes hard R and D becomes hard. I've got this great idea but I don't know where to put it because there's no one interested in me. Because planning, everything you've got on, will be rude to
say marketing, I don't know. But yeah, that's the own observations but the The issue thing is you coming back to what I was saying earlier, the two types of network it appears, there's another type of network which in the natural world is associated with a different type of scaling or something called super linear scaling. So, super linear scaling is when you're in advance of exponential growth.
So as you double the amount of one thing, as you double the number of people, you get more than double the revenue. For example, and in the cities, cities are another example of social networks, massive, social networks exist, should people communicating as you double the size of a city, you would get more than double The amount of socio-economic things out of it. So you get more than double The Innovation.
You get more than double wage growth, other effects negative effects to get more than double Prime pollution disease, these things, right? And all these super linear scaling super linear factors are related to these social networks. So people talking with people, I like to sort of explain it in the sense that if you live in a village of 150 people, right? And I'm a software developer.
I live in a village of 150. People, how many other software developers live in that Village for me to commit to, on a professional level? Maybe one, who knows maybe more than one these days, but probably not that many. Whereas, if you go to, Yo, Sydney, there's a thousand people just at the, I've Sydney talking about Innovation, talking about bootstrapping each other's ideas.
Like the original conference. I went to, and we talked about right at the start of this, that boot strapped on my ride is so you get a bigger city. As you get bigger places where more people live people tend to Together we'd like addressed. All right, so you get this bootstrapping effect where people with like interests get together. That's true for not just software and cool stuff with Innovation software. You get more than doubled, the number of lawyers in bigger cities, right?
You get more than double the number of accountants, etc, etc. And it related to these social violence, these social network graphs for me, that's a really interesting observation that we know mathematically there's this relationship. If you structure yourself, if information is allowed to flow in a particular way, then you get Get like the whole is greater than the sum of the parts. If you like, you get more out of than you think you should.
And if you structure yourself in a different way, conversely, you get less act than you think you should information. You know, if the information flows hierarchical E versus via social network mathematically, there's a choice between the two. So the question is, how to take advantage of this, how to use this knowledge, to improve our ability to get stuff done, because that's really what we want to do. Not necessarily make more money, maybe it's save lives, maybe it's work.
Hospitals and organizing teams such that they're communicating super effectively which they do incidentally. Soon in the UK are constantly for the rest of the world. Paul, maybe it's schools. How do we organize our schools such that? We've got the right information at the right time and stuff doesn't have to trickle down slowly from above or if it's a big organization, how do we organize ourselves? Such that people have got everything they need within a short number of hops from them.
And this is why, you know, we took and go back to microservices microservices at all. About product teams, which have everything there. You've got everything you need on your team to solve the problem, right? So, we've all of your products towards have been about a price, which is it's the same idea and it's related to Commerce law as well. How do we avoid having to go outside our team?
Ask other people to do stuff for us because that's from the hierarchical way of doing things and the observation are making the talk. Of course, which was related to me originally by someone at AWS wasn't that scary WS structures themselves. These Any small independent teams relying on Self Service infrastructure. That's another key thing that I should talk about in terms of microservices more there. So decoupled from one another that they're able to add new things on the side super fast.
So their ability to spin up your product teams is super quick. Interestingly, we also sort of point out, don't know if you've ever done much work with AWS but, you know, their apis you could be using one set of apis. You look at different set of
products and they bear. No relation to one another right, like the user experience that developer experience it Across the different products is very different, but that's deliberate, it's not deliberate per se, as if we want things to be different, but it's deliberate in the sense is we're not going to pay the coordination cost to make our apis consistent because the coordination costs is going to
slow us down. So with allow the teams to be independent and that means we have to put up with the fact that the developer experience isn't going to be potentially the best, which of course has interesting implications for things like micro front ends or how do you build Those apps that are able to use multiple teams to deliver parts of an aircraft parts of a website parts of the page or whatever it is retain that experience because most organizations that they want
their brand to be relatable across the difference. You don't want one the recommendations bit to look, very different from the main product bit, but it means you have to pay a coordination cost to do that. I think that's the direct result of some of this complexity song. Wow, so many interesting stuffs that you brought up here, right?
So things from the type of networks that you have in the organization, whether it's hierarchical social, Social so amount of flow how much information trickle down easily. So one thing that I want to bring up first before we go into recommendations for all organizations or software teams here many times actually, I also read that software teams is also the analogies like it's also a complex adaptive system, right?
So for many people who may not be familiar with this term, they always think like okay it's just another department within an organization. You can just produce whatever code that you can do and then just deployed but I think we all know as a software Engineers that Not that easy, right?
So there are a lot of things that actually going on and some people actually call it complex adaptive system, maybe if you can touch on a little bit and explain why we should also think that software engineering team or product engineering team is also a complex adaptive system. Here are good question, what is the definition of complex adaptive system, right? I mean again from the center face-to-face, Ethel dollar definition which is a complex adaptive systems, display four
characteristics. The first one is that inherently complex so that the whole is greater than In the sum of the parts, that's the first thing. So for example, I normally explain that by saying I am a complex adaptive system. James, if you put James into a blender and came up with like, James soup, that's not very nice idea. And then you pulled James soup into a James shaped jar, would it be James? And the answer is no right now. This is the interesting thing, this is why the history is
fascinating. It comes back to an old physicist called Murray gell-mann. He was fascinated by the idea of pairing. You get such From such Simplicity, he wrote a book called The quark and the Jaguar, right? How do you get something as complex as a Jackie or from something as simple as a subatomic particle pulled acquire right? What makes a j hero and what makes a human? What makes a team or organization or city? So this is idea of complexity, right? The whole is greater than the
sum of the parts. There's also this idea of emergent Behavior, so when we say measuring Behavior, would mean that you can't reliably predict the outputs of a complex adaptive system from its inputs. Every time it's not a functional problem where you can Say, given this stimuli this thing will always happen. Everything coming back to the thing about teams which teams exhibit that behavior all the time. All right?
You could never run the same situation, the same maybe even use a story or Sprint twice, you couldn't do it, you'd get different outcomes if you buy it a second time, the third characteristic, is this idea that they're made up of self similar parts. So in humans, the self similarity is the cells. We are self-similar our cells have self similar to one. Another actually in mammals, and this is where the scaling laws were.
Come from all cells are self-similar to you and Marcy sell or to an elephant cell, will to a blue whale cell and on teams to self similarity, is the people on the teams are self-similar, we were not within limits and then this idea of self-organization. So there's no Central thing telling a complex adaptive system, how to organize itself. Mammals, in our cases, DNA, right? There's nothing, when we're in the room that says, okay.
Now, build a heart. It just evolved over time, and that happens According to some weird magical, biological process. Yes, and there's obviously similar to the teams are as well. And so that's why I talk about teams being complex, adaptive systems example, I used again, the Sprint thing. If you just change one member of a team, you've got a different outcome. If one person is sick at a different time, you get a different outcome.
If people pay or don't pay you get a different outcome, it's almost like every single line of codes could for, your could not be written depending on so many different factors. And that's why we talk about teams has been complex, adaptive systems, and of course, organizations are made up of these components. Made up of these things. So that organization is also a complex adaptive system.
Hope that someone else's look wet definitely and it's not just within internal, the team's themselves. I think the external factors are so, they are so many complexities. What people call the vuca world, right? The disruptions the new technologies that come as well. So I think all these play A Part as well, to make it much more complex. So, coming back to the topic of how can organization avoid this
sub linear optimization growth. So, would you be able to suggest some things like How can we actually first identify? If we are sharing a sign of Aging. So things get slow down. Things are not flowing much much faster as when we are small and how she'll organization start to think about the more optimal way to actually have this super linear growth that you mentioned for really great question.
And what we normally do is I told you that by saying hey I'm just here to provide the information you can do with it what you want. But actually that's not very helpful at all, is it? I mean I do have in that talk some concrete advice, right? Which is for example, you can use Use the metrics from the door reports of the accelerate, Burke the full key metrics. You should become very popular rightly. So as indicators leading indicators, actually of organizational health.
So you have the four key metrics need time to Value. They put a weird definition of lead time is from commits through to production. Whereas real lead time is actually from idea through Soup To Nuts, while don't be used to lead, time to find mean, time to recovery change the failure rates and number of deploys per unit.
Time those things I think are really interesting to track now Now, a lot of people then fall into the Trap of saying great, you've got tools that will tell us this and then they spent eight months trying to automate measurable, this stuff. There's no need, right? We don't need Precision, we don't need the Precision you get from knowing on average we have exactly 30 2.5 hours between each commit. That's not what we're after here. What we're after is saying, yeah, look, accuracy finger in
the air, your tea. Hello, why average does it take to get stuff into production? That's good enough. And as long as you track that somewhere and as long as, Reflect on it. Maybe at the end of iteration or whatever it is. But you doing quarterly training or when you're okay, ours are being refused. And you look at our we better or worse. That's what we're really looking at. You know, are we trending in the right direction? Save me, time to recovery same for the others.
We can use those Trends to work out which direction we're heading. How are you? Use the analogy from biology or Medicine of going to doctors? Start general practitioners in the UK, they'll take your blood pressure, your temperature, your heart rate, that kind of stuff. They'll know if There's something a bit dodgy going on. Right? You got super high blood pressure maybe there's an intervention that needs to be Sage doesn't tell me exactly what the problem is but it might
tell you there is an issue. Now there are a couple of caveats to using the full key matter. It dry before, key metrics are only going to be improvable to the limit of the system, you find yourself it. So, if you're in a deeply hierarchical organization with these massive, long value streams to get stuff done. You can optimize for key metrics, but you're only going to be able to optimize to the limits imposed by System that's
long. Keeping I think that's kind of interesting is that would be one thing. I think the other thing is this idea of signals when we have some of this information, by how we structure our self fundamentally impacts performers to such a degree, which we kind of no, no based on a bunch of data that's out there in the public to me. How do you take advantage and how do we listen to signals from our own organization.
So that's one set of signals. I've just given you which is a full key metrics but the other thing is taking time setting, The organization up so that people in the organization can take the time to listen for the signals. Now I often these days, use the phrase that people are so busy. Doing the work. They don't have time to think about how the work Works. They don't have time to think about the system and optimize the system.
These people are just head down, just doing stuff, doing stuff, doing stuff all the time. And often that's actually why they hire people. I thought Works to come in right because we come up with tool box with a bunch of techniques. We can say hey we'll build you this value stream map. Now we can optimize it, I probably shouldn't say this.
Secret, there's no reason that you for example Henry films do the same thing with the same tools, it's just that often we don't have time to do that ourselves, you know, because we're so busy doing other things. So that will be the other piece of advice I would give is spend some time looking at the system, in which you were understand how the work is working. Understand how flow is for your organization and then you can work to optimize that you can
find. You can look at where those hierarchies are the informational hierarchies. It might not be on the orchestra, it might be in the valley. Streamers and work to change
that. And one thing I think I mentioned it briefly, I think one thing that is often overlooked is self-service and why self-service is so powerful because as I mentioned the problem with hierarchies is you tricking information one way and often by have to ask someone to get something done and waiting on someone else doing some work, right? That's kind of like blocking my flow broking my circulatory system in this Century. So how do we make those systems such that their cell service?
So I don't ask someone to do Something for me, instead, they provide that in a self-service way and that's what's so powerful that AWS is that they recognize that by providing the undifferentiated, heavy lifting was the original term. They use all the stuff that the original AWS platform did when they provide that self-service, they solve a couple of problems. The first is you have to wait on anyone to get stuff done.
All right. So that's a flow Improvement problem and the other thing is you solve the problem of scarce resource because it's queuing Theory, right? If you've got a request coming in, To a service into a queue rather and you've got one thing able to take that request off
that you and processes. The only way it by adding another message in at the same rate, if you've got multiple producers feeding messages into a queue, then you have to have multiple consumers to process, the messages at the same range. Keep the cue from backing up. And it's the same with what we do is exactly the same as what we do when we are asking people to spin up the ends for us when we are asking another team to make a change for us. The only way for them to scale
is for that. Team were asking to add more people, as they get more requests, they have to add more people, it's unsustainable. As we asked, cops to do more for us, they have to add more people and that's unsustainable. So, you invert the relationship and say, make these things self-service make providing infrastructure.
Self-service make, providing changes self-service, things like develop a platform engineering teams, providing self-service access to tooling to release pipelines to databases on, demand, all that stuff, that's all about improving flow, but you don't get to see that. Really unless you are able to visualize your system, unless you're able to step back, take the time and understand where the blockage is, are where the cholesterol is building up in
your arteries if you like. So, I think that's a very good reminder yet. Like you mentioned, right, when we are working in the organization in the team, we tend to just get things done, right? Okay, we have new features, we have new ideas, we have new, roadmaps, whatever that is, and we just get things done, and we just live with the same habit. Maybe let's put it this way. The same habit that we used to do.
Right. Maybe it proves successful in the beginning and we just continuously do that without actually monitoring and review the system, right? How the system looks like after some time that we go through it. So I think that's a good reminder to always look at the systems. Think about how we do the work, right? And I think the concept of platform teams, the self-service thing, it goes back also to the
team topologies, right? I think this work by Emanuel Pious and metal skeleton is really called it in everywhere. I think I would say, you know, the concept of for the Streamline team and the platform team enabling team and things like that. So I think that's really good concept as well for people to look at how you can actually optimize your organization and you kept mentioning about flow, right? Optimizing flow.
I think this is also, another thing that is really critical for organization to figure out how the flow in your team actually moves. Is it fast? Or is it slow? And we can use the key metrics from Dora as one indicator. It's not the true answer of everything and we don't have to be accurate so thanks for sharing that. So James unfortunately, due to time we have to wrap up pretty soon.
I have one last question though which I would like to ask you which I call this a question 3. I think the leadership is Adam so you can think of it like an advice that you want to import for us here to learn from your experience and your expertise. So can you share maybe two or three technical leadership wisdom? Yeah, this is a big word, right? And you may expect me to come up with some deeply technical from a set of answers here. I'm going to go completely the
other way, okay? These are three things that I've learned from people who work within four weeks. And in particular, I want to call out and with what Daniel to his North year because he's one of the people that I've really learned a lot for is a very, very wise person and The first thing is that empathy is a superpower, especially as you
get more. Experienced being empathetic is absolutely key to getting things done, I think from a personal perspective so much so that you know, I'll go back to 2008 2009 with another team with them. It was when he wrote the article that became the bdd article. It so just invented whatever it was he originated bdd Behavior driven development and in that it's all about being empathetic is all about understanding. Not just what your team needs, what you need, it's about.
Understanding what your other stakeholders need that aren't just the business stakeholders that are, your stakeholders, in, security, and operations, you've Downstream or Upstream, folks. Within the large Apollo, the suffered, a long process, but empathy of this is all your empathy with the people you're working with. You don't know what's going on in people's lives often, and be their pathetic as a long way to creating a really fun environment.
And that comes to my second piece of advice or maybe rather it's a statement and I would say that seriousness does not equal professionalism the best projects I've ever. I worked on the most successful products have ever built the best teams I've worked in. They understood that seriousness was not professionalism. Having fun makes a better team having a good time. Being empathetic with your colleagues, sharing good experiences. That makes the team pure as much as anything else.
I think we sometimes make the mistake that if we seem to be having a good time doing something. Then surely we're not doing it right, you know. But actually often I think it's completely the inverse so that would be my second one serious does not equal. And then the final one, which is interesting, only, then this like awesome down doors. It comes from a conversation with you like gold, brass the great Management Consultant, who wrote the goal and Beyond the goal, and a number of other
books. And he was asked about it. Why organizations often fail at adopting new things, new ideas, new Innovations. And he says that in his experience, it's because there are four questions they need to ask, right? And they forget to ask when. So the four questions are, if you've got a new thing, whatever that Innovation is microservices Docker, kubernetes continuous delivery. You should ask yourself, okay, one of the cool things that this thing gives.
What can this unlock for my organization? If we adopt this Innovation is technology and the second question you should ask is OK, what current its limitations in the company? Will this new innovation help us overcome? And then he says, you should then look and say, okay. What rules do we have to manage our existing limitations? And he says, when people adopt new technologies often what they do is they stop there. So as an example, continuous delivery would be a good example
of this. When you adopting continuous delivery, that's going to give you the ability to speed your time to Market to the Ultimate Security audit, all these really cool things built by quality etcetera, etc, etc. And what limitations do we currently have are we've got this massive manual testing process that we could, probably use continuous delivery would help shorten the amount of time it takes to do that, okay? And then they are all.
What rules do we currently have? Well, you know the rules are that you wanted to get into you, 80 years have to pass this barrier in order to integration. Pass this thing about it. And so you adopt continuous delivery and the development teams, go super super fast, but you forget to ask the fourth question. What new rules do? We need and the new rules are the important bits because if we try and marriage new stuff with the old rules, you don't get
anywhere. So, in the continuous delivery case you can say, well, we'll manage our new features delivery process, but using the same set of manual gates to get through to different environments, it doesn't matter if you've got a computer doing it, if it still takes you six weeks to get through, if you do Anything. So what are the new rules look for the new rules that you need and implement the new rules? That would be my third one. I've ever listened to this.
Four key questions that you mentioned right whenever we want to adopt new things. So thanks for sharing that and I left the second one. Seriousness does not equal professionalism. So I think yeah, in some organizations they tend to want to be very strict and follow the process. You always hit the target, right? But that doesn't always equate to professionalism. But you mentioned right? Or like we need to create fun. Fun.
And I think when we work in a fun manner, I think people tend to try for, I tend to produce the best results. So James, I think it's really a great composition. I learned a lot. So if people want to continue the conversation and maybe connect with you, is there a place they can find you online. They're arming, the best place is probably very email. If you want to chat with me very long dreams, don't leave yourself a full which to come.
I'm also on Twitter at Boise be. Oh, I see why I don't like data that. Use it that much. So those are probably the three places of the best. Thank you for this opportunity. James, I hope people also learned a lot from this conversation. So thank you again. We thank you so much variation, actually, pleasure talking with you and hopefully I'll get to see you in Australia again before too long. Yeah. Looking forward for that.
Thank you for listening to this episode and for staying, right until the end if you highly enjoyed it. I would appreciate if you share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you are new to the podcast, make sure to subscribe and leave me your valuable review and feedback. It helps me a lot. In order to grow this podcast better.
You can also find the full show notes of this conversation on the episode page at technology Arnold death website, including the full transcript, In quotes and links to the resources mention from the conversation. And lastly, make sure to subscribe to the show's mailing list on pack leader know that deaf to get notified for any future episodes. Stay tuned for the next technology, no episode. And until then goodbye.