Welcome to the IBM podcast. AIM is the chartered body for the project profession. My name is Emma De Vita and I'm the editor of Project APM's quarterly journal and your host. In this podcast, I'm speaking to Darren Dalcher, Professor of Strategic Project Management at the University of Lancaster.
Few people have started project management as closely or for as long as Darren, who's also a director of the National Centre for Project Management and Co editor of the 7th edition of OPM's Body of Knowledge. In a career spanning more than 25 years, Darren has become very interested in why projects fail when it comes to Agile. Darren was there from the start when it was being experimented with in software development. Who better than to explore the origins of Agile and its impact
on projects? Listen on. Welcome Darren, Thank you for your time. Could you tell us how you originally got interested in Agile, what its origins were? I was involved in software development originally as an engineer developer, and then I moved away from that and at some point I developed an interest in failures. As a practitioner, I'd worked on
a number of on IT projects. They're all kinds of challenges around them, and personally I was trying to make sense of some of the challenges, some of what I was seeing around me. The whole notion of success and failure were a little bit dubious, as far as I could see. We had fixed deadline, fixed ideas about what what we needed to be produced, and sometimes we delivered on time, sometimes we didn't, Probably around 19818283. I was working on a number of
projects. We were using this new idea called prototyping when we were spending a lot of time with the users. We were trying to understand what they were after and essentially a lot of people didn't have an idea what computer technology could deliver. So working with the users was an opportunity to introduce to them different ways, different options. Very often we didn't know that much about the users world. We would have a client coming in
getting us to do a system. We came from the technical side, so we had an idea about how to develop things, but not necessarily the user's world. And essentially it was a way of interacting with people. So for me, prototyping was a way of interacting. Prototyping made the project longer because once you started interacting with the users, once you spend time with them, it was all about the dialogue, all
about the understanding. This is where you'd realise that the requirements that you were given or the the need statement that was passed along wasn't exactly what the users needed. But we've kind of gone the complete circle now and we recognise that we need to get the users on board. There's no point delivering if nobody uses the system. We deliver systems, we deliver projects so that it can be used. So the results need to be deployed, the results need to be
used. There's no point delivering a bridge if no one's going to go across it if no one lives. In the area so, so it's all about the use, but we were learning that as we work it to computer technology was becoming more and more enabling. There was more that you could do with it. And it became all about creating the interaction, creating the relationships and understanding what people wanted. And how they could do their job better and what we could achieve for them through computers.
So really my journey there starts with an understanding that we need to gain a better insight into what people are trying to achieve. And it's only when we understand what we're trying, what they're trying to achieve, that we can deliver something meaningful and useful to them. So is that really the core of what Agile meant? So this to a large extent predates Agile. This is prototyping, It's experimentation, It's something that people have been doing for a long time.
We have been experimenting, we've been trying things out. So rather than have a fixed idea about what we could deliver, what needs to be done, it's playing with it a little bit and and trying to make sense of what we're getting and seeing if we can, if we can find a better solution. Tinkering with the solution being empiricist, being playful developers. The way we develop things has a has the sense of a problem solving cycle.
We move some problem towards solution and we try not to engage with this solution too early. We go through different modes of development and to a large extent computers enabled us to do a lot of things that we couldn't do before. So they they enabled us to to process large volumes of data, to do so very quickly, to do amazing calculations, to change how we how we implement systems,
how we deal with people. They enabled us to do a lot, so I was involved initially in trying to utilise computers in useful ways, but also trying to make sense of what is success and failure. And for me, prototyping was very much an early precursor of of Agile. We were doing exactly what our journey is doing. The challenge was as soon as you introduced prototyping, as soon as you engaged with the users, the user they wanted more.
So the challenge was not not a technical challenge, but there was a management challenge. How do you control this, this effort? It can go on forever. And then the real challenge is how do you pay for it? How do you structure it? So it became about. How? How do we control this prototyping activity? Do we only allow? 3 increments however long does increments are. So we could say, well we will do 3 versions of the system we will
go through in three times. Or we can just do it for two months and at the end of the two months what we have is really the the answer unless you want to pay more and then we we will do another increment. So it was really imposing some kind of a managerial structure around it, and I was trying to make sense of what seemed rather arbitrary at the time, the notions of success. Failure, because if you say, well, I'm going to deliver a system in six months.
And then you spend 10 months with the users because they're engaging with it and you're delivering a better system. Are you failing or are you succeeding? More so? This is kind of in the 80s. The Agile Manifesto was published in 2001. So what was it trying to achieve and how did it fit with what was going on with prototyping? You know, if you cast your mind back to them, what what? What did you think its value
might be? To set the context in a way that we are still dealing with project failures, they're quite a few challenges around projects, primarily because we are doing things that have not been done before. So computers enable us to seventies 80s, really. Even before that, some amazing things are happening through computers. They're really exciting projects. The problem is it's they're quite difficult to predict. They're quite difficult to control. It's back to this control
question. So there is a perception of failure. There's a perception that software projects fail a lot. And something needs to be done about it. And they're various efforts, various communities are formed and they're looking at different kinds of solutions. So some of them are looking at various ways of introducing quality. So all kinds of quality structures, some of them are looking. At what would later become the
idea of benefits. And some of them are looking at ways of creating a craft, a discipline that is more controlled. So software engineering is born around 1969. In the NATO conference, which brings together a lot of scientists who concerned about suffer being out of control. So the idea of engineering software engineering is to impose engineering structures.
So bring in that control over what we're doing and software engineering allows us. To plan, go through the requirements, design, develop and deliver. And deploy meaningful software almost in an engineering kind of way. So there are different communities all over the place and the different communities are engaging with different types of approaches. Some of them more formal. Software Engineers will try and find formal methods, but within that some would argue we still
need to work with people. Back to this if you like the fight between. Creativity and control. Various methods are created and throughout the Seventies, 80s and 90s there are all kinds of structured approaches. To obtaining requirement and structured approaches to design. So the structured requirements, structured design and our communities. Now those methods are becoming more and more structured and more and more controlled in response to the failures. We need to control the effort.
We need to know what we're doing. We need to start here and finish over there. And suffering engineering becomes quite controlled and at a time I'm teaching systems and software engineering, so I'm doing, I'm doing a lot of that. I've shifted from doing to to teaching, researching, making sense of things. And we we are particularly focused on some of the methods that are emerging. So I had a strong interest in
methods. I had a strong interest in what could be achieved and in this shift between control and creativity and they have various efforts that are going on. So in Japan, Takeuchi and Nonaka are saying, well to do, you know, when you look at innovation, when you look at creativity and how people work, it's not really a relay race where you hand over to someone else. So this is I think 1996, they're this, they're saying it's more for. It's more like a rugby match.
When you pass the ball backwards and forwards that they come up with the notion of it's a rugby match and this is the idea of scrum being born. It goes in all kinds of directions and at the same time other people are coming up with other lighter methodologies. They're calling them, but they're really methods. Lighter approaches to development, lighter life cycles. So what happens in 2001 is 17 people involved in the creation and the development of of flight and methods.
Get together. What they have in common is that they all believe that there's too much, if you like, structure and control. And we need to be we need to liberate ourselves a little bit. And we need to recognise these are the people that come into the meeting, they've all created their own methods or selling particular approaches. That emphasise working with users. Little bits of prototyping. Some of the methods can be traced back to prototyping. Some of them are.
Simply lighter approaches. They're not project managers, they're software developers. They're interested in software and what they do is they they get together for a weekend. There were a number of similar get togethers and this is this was one particular group and they do want a lot of groups to get together. Do they? They come up with a manifesto, of course. Wouldn't it be great if we could shape the world in the way we like it and everyone used like methods?
So they come up with a statement, and a statement reflects the kind of thinking at a time. It comes a little bit from manufacturing, a little bit from innovation. They're inspired by other developments that are going on at the time. And they're really saying we think we should be working in different ways. We should be using lighter methods. We should be doing things with those lighter methods that without the control, without the structure, without the imposition of of the.
Bureaucracy and they're all trying to sell their own individual methods at the time. So the challenge for them, for 17 people in that in that room is to to come up with a coherent statement that reflects. The flexibility, the freedom, the that are common across the all the methods that they're all representing. So. So how do we create a sort of a statement of independence? What is our our common cause? Did you know any of them personally? Had you met any of these people
in that room? Yes, I knew a few, quite a few of them at the time. I've met since. I've done keynotes and I've spoken at conferences with many of them, so I've so, yes, very much so. And. I knew many others liked them. It was pretty much the prevailing kind of approach. Programmers don't like structure. They don't like controls. So having come from that background, I was familiar with. I was familiar with the constraints. I was familiar with the ways of
thinking. So they basically work on the manifesto over the weekend and the manifesto is 4 statements of value. So we prefer something over something else. We prefer people over tools, essentially. The relationships and working with people of the tools and methods and approaches, we prefer delivering working products over comprehensive documentation. So so we like to do things. So the first one is saying we like to work with people. And I suppose that's quite a strong statement for
programmers. Remember, programmers are the people you keep in the basement. Now they're coming out. They're saying we want to work with people. Let me work with people. So they're working with people now. And like me, I suppose they got excited by the fact that ohh, I can shape things for you, I can shape the world for you, I can use the computer to create a better world for you. So. So we work with people we prefer.
Delivering, working things, doing rather than documenting and documentation wasn't a popular part of the software process. It's the thing that came at the end and it was you think you've done the thing. Now go and test and then go and do the documentation. We prefer customer collaboration over really contract negotiation. And we prefer responding to change over just following a plan. If something changes, let's go with it.
Let's flow with it. So it represents an attitude, it represents an age, it represents the people in the room, it represents a long weekend in the cold February holiday resort. It's a group of people saying wouldn't it be great if we could change the world? So they come up with those statements. Later on they will add the the 12 principles, but those four values really about 30, I don't know, 40 words, something like that. This is our manifesto. This is what we think we'd like
the world to look like. And one of them goes and posts it on the web because you can now and reasonably quickly there's some traction. Various communities realise that. This is going on. And they say, yeah, yeah, yeah, me too, me too. I want to be part of this movement. I believe in those principles as well. So it becomes something within the software development community, becomes a way of thinking about how we do, how we
develop software. I don't want to say how we do projects, but certainly how we develop software, because there is an implication that we're back to this creativity and control. So it becomes perhaps an alternative way of looking at how we develop. I mean, I'm understanding that it was never intended to be a management method or a project management method, but it has permeated not only project management but management in general. What's your take on this?
Do you think the people who originally created this are surprised that this has happened? What's your view on how Agile has influenced project management and what it's become now? Right. So when they post the manifesto, it's the manifesto for Agile software development. So that's quite explicit that that that's the title of it and and and it remains like that. The document hasn't changed since 2001. There was a conscious decision. It reflects a point in time when they got together.
So what essentially was the the manifesto for Agile software development became known as the Agile Manifesto. So at some point we we dropped the software development. But when I first knew it, it was this manifesto for Agile software development. And within the software development community there was a mixed response. Some people got excited about what Agile? And deliver. About the fact that it could liberate you from some bureaucracy, it enabled you to
do things differently. It was quite an exciting development certainly for programmers and a lot of the agile literature from that time is saying we don't need project managers, It's the end of the project manager. Project manager can can disappear. One of them is saying the role of the project manager is to deliver pizza. So, so there are different perspectives around that, but somewhere along the line. People get excited by the concept.
Now. The software development community, to a large extent, has moved on from Agile. This suffering the software development community, if I reflect back over the years, roughly every 10 years there's a new trend and a new trend coming comes in. And it could be structured systems analysis or it could be structured design or software engineer. Something will come along and it will grasp everyone and they'll get excited by it. And then there'll be a new toy.
And we are looking at roughly every 10 years being there's a new exciting development and to a large extent software development and software engineering have moved on to thinking about the delivery of services rather than products and looking at continuous delivery, continuous testing, continuous delivery, continuous deployment, So delivery becomes more constant. The agile community has in software development has certainly moved on.
But a lot of others, as you said, have have picked up on on Djali. Do we need to be agile? And that's without fully understanding what our journey is. Our journey is this mythical thing that we can do so we can be agile. That that sounds good. We can be. I want to be able to do magic. I want to do all kinds of exciting things. So, so we can be agile, we can do things, and we can pivot. We can respond. And a lot of what our child gives you is the ability to.
Ready to to respond to change, Going back to where I started and those experiments. Agile allows you to do little experiments and on the basis of those experiments to amend to some extent what you're doing. So you're being more responsive, you're being more adaptive. And in terms of management, management goes back to the bureaucracy to control the. So having this magic ability to, yeah, do you want to be agile? Yes, of course I do agile. Agile is exciting.
It's not boring old structure, it's I can be agile, I can do what I like. It allows me to do all sorts of amazing things. So there's a lot of traction, There's a lot of people are impressed by it. Some of the origins, as I pointed out, also go back to production and manufacturing. And innovation management. And there's a challenge around finding the balance between product development and project management. And is it a technical development that we are doing?
Can we get, can we do away with the project manager? Can we? So the project management community has found agile. Now they're very it seems like an exciting development, and there are all kinds of conversations around what replaces what does all this waterfall V agile discussion everywhere, and which is relatively meaningless because neither of them is a project management approach. We're APM, the only chartered membership organisation for the project profession.
When you become an AP member, you'll receive the resources and support you need to make an impact, delivering better projects with better outcomes. Plus you'll access exclusive training and benefits to support your ongoing career development. Find out how we can help you reach your potential by visiting apm.org.uk, Because when projects succeed, society benefits. Is the project management profession using Agile? However, they define it as an effective way to have fewer project failures.
Is it a useful tool? Interesting question. We have fewer of the old failures, but our job will introduce inevitably introduce its own kinds of failures. I've been collecting failures for literally all of my career, not just my own. I'm I'm less fussy nowadays. I'm I'm happy to welcome anyone to the club. 10s of thousands of failures have made it in and there are some major failures
and they're very different. What what Agile allows you to do is you've got small, small teams doing things and this is what the essence going back to 2001. This is the essence of that meeting. This is what it was all about. It was a small solution to a small problem of a small group of people working together to do small things. This is what our job enables you to do and this is what they were talking about. So this is it liberates a small team to do things far more creatively.
And at the time we were doing some research around Agile which the view of determining whether it had, whether it had values. We were looking at the quality of what the teams were producing and we did a one year study where we we looked at multiple projects and we looked at the outputs of teams in the banking sector and we we measured the the output and the agile teams produced far more code which means they produced far more functionality. The obvious question would be
was that useful functionality? The answer would be to spend a lot more time with your clients. So they did a lot more. They delivered a lot more. But it introduced different challenges. The fact that the software community has moved on and is interested in dev OPS reflects an interest in continuing operations.
It reflects a need to integrate other activities all the time to to to continue to develop and to have backlogs that allow you to bring in items from different areas and integrate them into your deliverables. It means that agility. So make the chef now. Agility allows you to do things differently. Agility allows you to deal with uncertainty. Agility allows you to see what you can do. Action has been really exciting during the Pandemic actually allows you to start.
You have a small team run with it, go with it. Here's the ball run. So Agile allows you to enables you to do all kinds of things. It's up to the project manager to decide whether it is the most useful way of deploying off using their team and the most useful way of doing things. It's still the skills of the project managers. I don't think the world is just to deliver pizza.
The role is to decide what you're hoping to deliver and to say, well, we can do things together, we can do exciting things and agile will allow us to experiment going back to where we're starting, you can do your experiments. You can see whether it makes sense. During the pandemic, that was amazing. You actually said to the team, so we need to start. Rules are changing from This is a Friday afternoon announcement. Rules are changing from Monday. Go run with it.
We don't really know what it means. Let's go and explore the boundaries and it it liberates teams. It enables them to do things but it it's something that allows a small a small team to do things. Somewhere along the line we realised, wait a minute, if one team can be our gel, can we have lots of teams playing in the same field being agile? So then we started talking about scaling agile, and now we're talking about business agility.
And now we're talking about all kinds of other notions which are far more complex. The problem is, if you allow one team to, I'm not gonna say misbehave. If you allow one team to have fun, you have to allow all of them to have give everyone the ability to have fun together. And that's where we come back to that good old balancing act between creativity and structure and control. One team having fun is exciting enough and noisy enough, but can you let hundred teams, thousand teams do that?
So we are back to OK, now we need to introduce scaling methods and approaches to to to getting multiple teams work together. So we are beginning to put some certain controls around that. If you ask the original people who attended the the meeting and created the Agile Manifesto what they think about it, They hate it. Their view is that Agile was this liberating activity and they don't like the many of them, don't like the scaling
approaches. There are various methods that have been designed to do that and they feel that they are adding a lot of structure, which you would need to do when you have multiple teams playing and having fun together at the same time. You want to control the noise, you want to control the space, you want to control what's going on. You need to manage it, so the challenge for project managers is to think what it can achieve through our journey. It goes back to the skills of
the project manager. We if we don't assume that all the knowledge is there, it's really how do we deploy, how do we create, how do we become pragmatic managers use their professional judgement and decide how to deliver, how to shape, how to deploy in meaningful ways. So it's creating strategies that will allow the mix that will allow the different activities that need to take place to happen together at the same time. Darren That's a. Fantastic way to end this
podcast. Thank you so much for your time. Really, really interesting speaking to you about this. So thank you again. It's a pleasure. Thank you very much. Thanks again to. Darren for joining us and to you for listening to the APM Podcast. Don't forget to look out for more episodes or to rate and review us wherever you get your podcasts. We welcome you to get in touch with your comments, feedback and suggestions by emailing us at
apmpodcast@thinkpublishing.co.uk. This podcast has been brought to you by APM, The Chartered Body for the project profession. For more information on APM, visit to apm.org.uk.