Welcome to the APM podcast. APM is the childhood body for the project profession. My name is Emma David and I'm the editor of Project APM's quarterly journal and your host today. In this podcast, we're talking about how to measure anything in project management. Yes, you did hear me write anything. It's the title of a new book that is co-authored by Douglas Hubbard, Alexander Budzia and Andreas lead. I have the pleasure of being joined by Doug and Andreas Now.
Doug is founder and president of Hubbard Decision Research, is a management consultant and the author of a series of books including the brilliantly titled How to Measure Anything. Andreas is head of Data Science at Oxford Global Projects and is a PhD Fellow at AAHSA University in Denmark.
Be prepared for mind expanding conversation about project management and how any aspect of it can be measured, including those things you thought might be immeasurable and why we should all be engaged in the most important project of all, the meta project of understanding how to manage projects better. I'd like to begin by welcoming Andreas and Douglas to the APM Podcast. So thanks very much for finding the time to speak to us today. Thank you for having us. You bet.
Let's begin by you telling us a little bit about your area of work and interest within project management. So Andreas, tell us a bit about what you're what you're working on right now. Yes, Sir, of course. So I'm, I'm Andreas lead and the head of Data Science at Oxford Global Projects. And Oxford Global projects is a small consultancy which is really about analysing vast amount of vast amounts of project data and and using it to to have a better decision making around projects.
Another of what we do is about, you know, debiasing estimates and kind of assuring estimates, realistic, basing estimates on historical data rather than bottom up type approaches. But it's also kind of bigger than that. So as the head of data science, I did some of the more quantitative engagements, things like using AI for machine learning for forecasting project outcomes or doing kind of econometric cost driver analysis.
And the big goal with all of this is, is to better projects because after having looked at all the project data we have access to, which is now about 20 data from 20,000 completed projects, the the big conclusion is that rojects aren't erforming that well. O we'd like to understand, you know, what leads to success and how can we imrove roject erformance? Thank you. So that's not necessarily a controversial finding, is it, unfortunately. But good to know.
We're here today to talk about how we can make perhaps make projects more successful through the way they're measured or what what is in fact measured. So Douglas, tell us a bit about your area of interest and and actually tell us how the book came about as well, because that's always good to know. Yeah, sure. So I'm Doug Hubbard. I'm the president and founder of Hubbard Decision Research, and for the past 36 or 37 years or so I've been in some form of quantitative management consulting.
So all of my work, including all my books, this was my fifth book actually, They've all been on some topic related to decisions under uncertainty and difficult measurements. So that includes risk analysis, making big bets, and lots of different areas of forecasting problems. And of course, that applies to projects in general.
My previous four books were not about projects specifically, but certainly a lot of my work was about risk return analysis on major new investments, which were usually projects in regards to how we met. Actually, I'll let Andres describe this because initially 2 of you called me. Yeah, so. So I was writing an article together with a colleague about cost, benefit, cost benefit analysis. Now kind of alternatives to, to typical inclusions in cost benefit analysis.
I, I also teach cost benefit analysis at the Department of economics here in all who's where I'm based. But we found that Doug's work was really interesting. So we, we asked him for an interview and kind of some ideas around what should we include in this article. And I guess it was, it was a good match. Doug was looking for his next book project and project management was one of the topics he was considering.
And so because we had access to a lot of data and a lot of analysis that kind of fit in really well with the book. I think that's fair, isn't it, Doug? Oh yeah, absolutely. In fact, I I chose this book title based on analytics. I decided to go after How to Measure Anything in Project Management because it had previously tied as one of the top 2 with How to Measure Anything in Cybersecurity Risk, which I've already wrote now.
So I wrote that one first. And what tipped the needle in the in favour of the first book of the the 4th, my 4th book. But the previous How to Measure Anything in Cybersecurity Risk book was that I had a a well qualified, enthusiastic co-author that said, Hey, I'll write this with you. And I said, OK, well, then let's get started on that. And so I was really looking for the right people.
I actually talked to a bunch of other possible candidates for co-authors for how to measure anything in project management. And my strategy is always to find somebody who's a specialist, an expert in renowned in that field because I'm always the outsider, I'm always the outside quant analyst, applying it to different kinds of problems, etcetera. So what I wanted to do a spin off of my first book, my first book was just how to measure anything. So that and I meant it
literally. So how to measure anything. And I decided to do a series of spin off books like that. And so I waited for the right co-authors and when I learned about Oxford Global projects, I thought, well, that's ideal. They've got all this data. They, they already have these guys that are, that have been studying projects and all the problems with projects for many years now. So it seemed like the ideal co-authors. That's really interesting. So Doug, can you measure
anything? Sorry, I've got to ask you. Yeah, absolutely. Including the hardest most possible things you've ever heard of. Can you, I'll think of some ideas as I as as we progress about what might be really hard to measure, but it sounds as though the book then is a really nice mix of the quality of quantitative, someone who's very deep with people who are very deeply involved in the profession and then more of an
outsider. And I think that always adds a really valuable perspective on the same topic. I'm going to dive in there and ask you a really hard question and probably very hard to answer in not very much time. But but what's new about what you're saying in project management and how it's measured?
Because I've read in the forward something very ambitious that you wrote that was that you don't just want to challenge traditional project management practises, but replace them with something more of rigorous, transparent and effective. This sounds really interesting. So. So tell me more. I don't know who'd like to go press. Doug, would you like to share
that with us? So I approach everything as a sceptic and in every field I've looked in, whether it was cybersecurity risk or operational risk management or portfolio prioritisation methods that businesses use or just decision making in general, I tend to find that there's a lot of, a lot of assumptions they make about what works and what does it. They don't really know, they just assumed it always has worked.
They they never really measured it and they're often surprised when they do measure it. So it's the sort of thing where it's not really obvious though, like if you're, if you're golfing, it's going to be obvious whether or not you're good or bad at it, right? You get your score right at the end of things, right, other kinds of sports, etcetera. But in a lot of management decision making, you have this really inconsistent and highly delayed and ambiguous feedback
cycle. So it's not really necessarily obvious that what you're using is really one of the better methods. There's probably many other methods. People tend to be very overconfident about their own judgement. We know this from a lot of other research that's been, you know,
spearheaded by many others. We also know that they're highly inconsistent, that they're influenced by irrelevant random external factors, and they dismiss too many things as immeasurable when all the research shows that you're better off doing the math on a wide variety of problems. Even relatively simple statistical models seem to outperform human experts in a variety of different fields. You can't trust the humans. Well, you can't trust the humans
for some things. It's really important you figure out how to use the humans, what to use them for, and what not to rely on them for. Quantitative methods in mathematics are really important. Human invention, slash discovery. All right. As important as language itself. OK. And it solves a lot of problems. It's a tool that we should use more often. We we revert too frequently to just rely on our own judgement.
We're having too much trust in other human beings when in fact the data shows that simple statistical models are outperforming them. So does that pass the argument in the book? Oh yes, absolutely. We, we take a sceptical view of, of a lot of the methods that are used in project management and Andres can talk a little bit
about that. But we did, in addition to the analysis of the 20,000 or so projects that they had in their database, we did our own original survey of little over 200 project managers and projects and pretty extensive literature review, more than 100 different, you know, peer reviewed articles, standards, other sources and so forth. And we were looking for what's the evidence that some methods are measurably better than others. And that was one of the surprising things.
I'll let Andreas talk about some of that. But really, what works doesn't is will be a surprise to many project managers I think. Well, what? What were these discoveries you made, Andreas? So I, I might just, I might just take a, a step back because one of the first things we did when we approached the book was, well, can can we say something about some of the, do some projects perform better than others?
Do projects use specific types of methodologies or frameworks or different types of tools for estimating how do they perform compared to others? And so the the first thing that we actually did was to look at our large database of projects, which goes back to we have some historical projects from the 1800s. And what we see there is kind of a very consistent trend in terms of projects not delivering to expectations in terms of cost and schedule and also benefits.
And and that's despite the fact that since at least since the 90s, we've gotten more and more kind of methodologies and frameworks and all these things that supposedly help us with projects. So we, if these frameworks and these tools we've developed works, we would expect to see, you know, budget overruns have gone down over time. We, we don't see as big delays. We are better at getting the benefits that we, we expect we will from projects.
But when we look at kind of project performance over time, we don't see that a performance improvement since 1990s despite all this, all this work and improving projects in terms of methodologies and certifications and all this. So that was kind of the first clue. Afterwards, we, we launched this survey where we looked into, well, what, what do projects use? And can we see some kind of statistical differences between different types of approaches to delivering projects?
And, and I think the, the evidence, what we found was was very contrasting because project management is a field where there's just a lot of claims, There's lots of methodologies that promise, you know, if you use this approach to deliver your project, you're going to get twice the benefits in half the time. But we didn't see that for any methods. We didn't see any methods really outperform others. We, we saw very few actually effects of applying any kind of methods at all to projects.
I think that the one very consistent finding we had was that projects have spent more time in the front end development of projects. They they tend to perform better and and secondly projects that apply more quantitative and evidence based methods. They also tend to perform better in terms of adhering to their estimates compared to projects that use more qualitative and subjective estimating
techniques. One of the things I thought of as I was going through the book was that I thought listeners might be interesting. Was was I, I? What can be measured in projects? Can everything be measured and what are the things you should be measuring? So the methodologies you're saying out there, many of them don't deliver any notable results. And is part of that because they're measuring the wrong things or what do you do with the measurements once you've got them?
So big, a big couple of big questions there. So I guess if you want to take, can you measure everything in projects and then which are the ones you should be measuring? I think that would be really interesting. Well, the simple answer is yes. There's literally nothing, literally nothing that has any consequences at all for the real world. Every project that anybody has ever proposed had some expected consequences, right? And every consequence is necessarily observable.
Otherwise, it wouldn't be something that everybody cared about, right? It would have to have observable consequences. No matter what you're thinking of, it's really hard to measure if you have a project that's supposed to improve collaboration or innovation or the environment or customer satisfaction, or some sort of social welfare or children's education, it doesn't really matter. In each of those cases, there are expected observable outcomes.
Now, you might be highly uncertain about them. That's a different issue, but you can identify the observations first, then the rest is about observations and a little bit of math to reduce your uncertainty. Does that mean you need to know what you're observing before you start measuring? Do you know? Do you have to have an expectation of what there will be to observe before you can start thinking about measuring
that thing? You don't have to, but you won't know the value that measurement until you know the reason for it. You can have purely exploratory discovery driven observations. A lot of science is this way. You you don't necessarily know what you're looking for, right? If I go to a, a, a, a new city and I'm walking around downtown, I don't know what I'm looking for necessarily, right?
I might discover something. But if I'm going to spend a lot of time and effort measuring something in particular, I have to know what it's for, right? And all measurements are meant to reduce uncertainty, to make better bets. You're making better decisions. That's what all of this is about. Every project manager, what management means is all about making decisions before and during a project, sometimes even after a project.
So you can think of projects, project decisions during the project as sort of intervention decisions. So what are they measuring if they're not going to do anything with the measurements during a project? That is, they can't think of anything they might do differently at all. If it was surprisingly high or surprisingly low, well then that has a information value of 0. This is a computable value, by the way, the value of
information. So it's sort of like the cost of being wrong times the chance of being wrong. It can get a little bit more elaborate than that. But generally speaking, if there's, if you can't think of any consequence that would be different at all, or if there's no uncertainty about it already, well then there's no value to that that information. Do you think that project
professionals. Oh, know that deep down that they do question why they're measuring something or is it just we're just measuring these things? So it's there's enough thought given to or consideration given to what's being measured and why, what the outcome you want from that measurement to be Andreas? So I think there's a lot of culture and project management. We typically measure the things we've measured before or that
that other projects measure. And it's actually one of the things we go into that, you know, there's all these standard setting bodies, there's all these kind of influential organisations in, in the space of project management. And a lot of them just kind of repeat. This is how we've delivered projects before. This is how we've done project management and how we've thought about, you know, quantifying risk in some way.
But they, they typically don't really distinguish between what are the methods that are proven to actually give you better results versus the ones that in, in the best case don't do anything, but at, at worst, they might add uncertainty. We use kind of a risk matrix as one of the examples of, so a way of we, we call it analysis placebo in the book, but it's, it's the idea that you, you feel you're kind of measuring a risk by using risk matrices to kind of quantify the impact versus
the likelihood. But there's multiple problems with, with that type of more subjective scale approach where all the research from, from not just product management, but also different fields show that these types of quantifications, they, there's a lot of error with them and they don't actually contribute anything good.
But in projects, we, we apply some of these tools and because we've been applying those tools, we feel that we've actually we've done some kind of assessment of a risk even though we haven't really, we call it analysis placebo that because we've spent time applying A structured framework, now we feel better about the situation even though we haven't reduced uncertainty at all. So, so I think that's a lot of
that kind of culture. But another thing that's we found was pretty surprising in the book was that projects say spend a lot of time measuring things that have the lowest
information values. If you want to stick to that framework of the value of information in terms of reducing uncertainty for our decision making projects, they spend a long time kind of estimating or measuring the CapEx cost, but they don't spend as much time measuring the OpEx cost of the kind of the maintenance of the the infrastructure, the project that they build afterwards. Where typically the uncertainties is typically in the latter and the Opecs the
same with benefits. Often times big projects, especially big public projects have wider societal benefits and those are really uncertain. But projects they don't spend as much time trying to measure and reduce the uncertainty about the benefits side of the project as they do cost because they're the cost is the big typically it's, it's the big component that they could ask questions about. They have to deliver too.
So should project professionals be focusing more on the benefits that will come from a project and measuring those? Yeah. Actually, when we compute the value of information, and we've done this many times on many different types of investments, including lots of different categories of projects, we tend to find the high information value variables. So I mentioned that's an algorithm you can actually run. These are more sensitive
variables. They change the outcome of the decision more and they're more uncertain, right, than other variables. The high information value variables tend to be the things they haven't measured before, and that often includes benefits. They spend more time historically spending things that spending their time on, things that have lower information values, things that are statistically less likely to actually improve decisions. So we call this the measurement
conversion in our books. I think this is a pervasive issue in every industry and projects and also outside of projects. I I don't know how it couldn't affect the GDP of countries. It would kind of have to. Why have we ended up this way? What? Why are we not measuring benefits better or measuring them at all? There's a general, there are a few different reasons I think and Andreas, you can comment on
this. But First off, I think there's a habit of when you label something as intangible and immeasurable, it's never considered again. Once it's labelled intangible or immeasurable, no one ever goes back and ask, is it really immeasurable? Are there really no observable consequences at all that could reduce our uncertainty about this? Have we even figured out what it is that we're measuring right? Often things seem immeasurable because we have an ambiguous
label. We're using terms like collaboration or innovation, but there's observable consequences that you can't identify out of each of those things. So there's that. Things get labelled too quickly or at all as immeasurable. That's one problem. They're all measurable. The other problem is that people are just as Andre has pointed out, it's just a matter of culture. They measure what they know how
to measure. They don't really look at a problem and say, what should I measure given this problem and then learn how to measure those things. That's not the approach. The approach is I know how to measure these things. That's what I'm measuring. So what needs to change to make projects delivered more successfully or deliver better benefits? Is is part of that looking at benefits and realising that these can be measured in a useful way?
Yes, that's part of it. The first big project management decision is approval and prioritisation. Should you even do the project in the first place? In those cases, even the decision of doing the project in the 1st place is. And that first big decision, modelling it quantitatively, is what's needed. Because when we start looking at that problem by itself, we find out that really most decision makers probably would not have accepted most of their projects. Seriously.
Oh yes. So when you look at the data, there's a way to quantify risk aversion as well. And when and when you look at all the research on what executives say about how risk tolerant they are, how risk averse they are to, and then compare that to the actual uncertainties and risk of the big projects, they wouldn't have accepted almost any of them. So fewer projects, bigger wins.
Is it one approach, right. And also spending more time on that early selection, you know, if somebody says, hey, this project is a good deal because we make more than our cost of money, that's not nearly good enough. If your return is just a little higher than your discount rates, you say, well, it's a good public project.
You know, we're going to make the the cost of this big infrastructure project is justified or the cost of this big software project or Rd project is justified because we expect these returns. If those returns are modest, like modest improvements over your cost of money, your discount rates, that's not nearly good enough.
If a project is expected, If we look at just all the projects in their database, if a project is expected to have a lot twice the benefits as costs, and this is present value discounted to current dollars, current units of monetary units. If you expect twice the benefits as you have cost, there's still so much uncertainty on benefits and costs. There's about a 14% chance of losing money still. In fact, there's even a chance of losing more than the money you put into it. And that's.
Quite something, yeah. Yes, if you spend, if you spend, you know, £100 million on some big infrastructure project, is it possible you could lose more than 100 million? Yeah, actually there is. One of the cost that are not typically included in cost overruns is other process disruptions. So you've seen that's when there's a big infrastructure project, there's a bridge being built or a new road being repaid for something like this, and then there's a traffic congestion for a while because
traffic has to be rerouted. Well, the project takes longer than expected. So that cost is the congestion cost is longer than expected, maybe even greater than expected. That's how you actually have cost that aren't in your budget. There's a other disruption cost in software cases. There's been cases of software where a lot of money has been spent to implement something. And then they figured out that this software that was supposed to improve productivity made things worse.
So they had to unravel it. They had to undo it and take it out, go back to the old way. That's more that cost more than the cost of the project to do all of that. Now that doesn't happen every time, but it happens frequently enough that it's a risk we should consider and it's often not even acknowledges a possibility at the beginning of this approval process. So that's one big thing. Fewer projects, bigger wins, you know, a 20 or even 30 or 40% return. Not nearly good enough.
You got to go bigger, spend more time in that upfront planning and look for better options. Keep exploring. There's one of the issues we talked about in the book is exploration versus exploitation. So this is kind of a classic measurement problem. It's a operations research problem. How long should I generate solutions before I pick the best one so far and and build on that one? Because there's a chance the next solution you generate is going to be better than any of
the previous ones. On the other hand, you keep deferring the project, which has benefits itself presumably, right? So that's something that isn't typically looked at and it's something you could spend more time on in front end loading is you could spend more time generating solutions. It appears that people converge too quickly on a preferred solution. They should think of many more solutions and they should generate many more decision models and options to be used
during the project. So it's not just the solutions, it's not the just the various designs of bridges, for example, that you're looking at. If you're also designing the measurement programme during the project, you're also designing the intervention options. These are decision models that run during the project. You're designing things. It's like what method should I use? Should this be an agile project or some modified version of that? That's all part of front end
loading. And it's usually those things are they're not only not spend enough time on they're, they're just ignored as choices. They're just assumed that we're going to deal with this project with this way. We're going to follow this particular standard and we're going to measure these particular things and maybe after looking at a few solutions, we're going to pick that solution. That's a big problem.
Just to add, when we talk about solutions, we mean solutions in a much broader kind of definition of that word. So when we talk about iterations of bridges, we really should spend more time thinking about alternatives to bridges as well that fulfil the same kind of requirement for the bridge.
So thinking in fairies or tunnels or other types of solutions, not just iterations of bridges, because that's one thing that we saw too, is that we we fixate on a type of solution fairly quickly in projects and then we do iterations of that kind of design. How should that bridge look or how should that nuclear power plant be? Whereas we could consider much different types of solutions as well. I don't know where to start with everything I want to ask you,
because that was quite macro. It's quite philosophical about what? What do we do and how do we do it in other alternatives, let alone the complication of political feeling or government. When you hit government mega projects for example, where the politics behind it can make it something happen without really being questioned why or is this the right thing to do. So for people listening, what is the how can they imagine someone just in a how I get a job
working on a project? How can you begin to deliver, execute, think about your projects in a in a better way
along the lines he suggests. And are you saying, let's look at the data, that let's make data more of a priority than these humans who might be too might be prone to optimism bias or and then with the rise of AI is, is this going to nudge projects more towards the way you're thinking that that if it's not measured, it should be measured, Everything could be measured and we should listen to the data and see the evidence of what actually works and be more objective about it.
I think this is an interesting, interesting thing that comes up quite a lot. So many might say, but what about somebody who's working on a project right now? I think that's already too far down in the weeds because just organizationally that person doesn't really have the position or authority to make some of the changes we're talking about. We have to start much higher and much earlier in the organisation. We have to talk about stakeholders changing the way that they approach things,
right? So it's stakeholders that are making ultimately making choices about where they should allocate funds in the first place. And something has to be communicated there that it's to their benefit to spend more time in this analysis before you make big bets. You know, if you're buying a house, you look at a few houses, right? If you're buying a car, you look at a few cars. You don't just pick the first
one you see, right? And I think decision makers often are too quick to say we got to start coding or we got to start pouring concrete, right? I, that's a, that's a, this is a different way of thinking about it. We have to spend more time in that upfront analysis, analysing our choices. Again, fewer projects, bigger wins. That's the, the optimistic part here. I think it's the bigger wins part there. There's room for really
impactful mega projects. That's where we should be thinking those really fundamentally life changing kinds of projects, socially changing kinds of projects. Like like what? Such as? Oh well, we have a couple of examples in the book, but at least in the states here, is it possible? Ideas have been proposed and it should be analysed. Is it possible to stop hurricanes?
That's a hypothesis. And even if there's, if you work out the information value that if there were only a 5% chance of that being true, it'd be worth investigating just because
it's so impactful. The cost of hurricanes is starts to compete with the cost of our entire aircraft carrier fleet, you know, in the US, right, or the entire Interstate system of expressways in the US. All of a sudden, when you add up a few big hurricanes like that, you say, well, yeah, actually, if it were even possible to even slightly reduce of the impact of hurricanes, what are the different ways we could do that? That's a big project. That could be a very big
project. It might not even be a big project, Might be a bunch of little projects, by the way. But also, you think about AI itself. We talk about AI in the project and we tell people that you're single most important project, and this is the biggest project of all of them. By the way, the single biggest project the world is facing right now is how to do projects. That's the single biggest project.
That's the most important mega project that any country or company is dealing with is how they should do projects. And this is a key message in the entire book. There's a lot of ways to run projects and a lot of ways to measure projects, and we're just not spending enough time figuring out which ways work better. Well, some are better than others, and we're still stuck on some, you know, habits that don't work as well as other
things. Thanks Doug. Andreas, what? What are those better ways of running projects then what? What actually has does give some success. So we do have a lot of, I think one of the positive messages of the book is that there is a lot of potential, there are a lot of things you could be doing to improve the projects you're doing at the moment.
So far we've talked a lot about, well, if we, we're able to reduce uncertainty about the bets that we're making, we would be making better bets, probably fewer bets, but the portfolio projects would be entirely different to what we're investing in now. And it would probably be better because a lot of what we, a lot of the projects we invest in now are the ones that look best on paper, but aren't necessarily the the best projects in in the book. Obviously the book is about measurement.
So we talk about like what kind of measurement should you be making? Why are you doing? Why are you measuring in the 1st place? How should you be doing measurement? And one of the things when it comes to information value is, is often, if you don't know anything, almost anything will tell you something, right? So if the uncertainty is really high about something you haven't measured in your project, you don't need to collect a lot of data to reduce that uncertainty.
And that's also part of the measurement inversion we talked about. We spend a lot of time fixating on the cost of the project. And sometimes to marginally reduce the uncertainty, we have to do a whole whole bunch of engineering in a project that might be really expensive and there's still a lot of uncertainty after we've done
that detailed engineering. Whereas when it comes to some of the benefits coming out of projects, since we haven't observed it so far, we, we don't really need to make a lot of observation to be in a better place than we were before, to have reduced the uncertainty of what can we expect to get out of it and how can we manage that better.
So we, we talk about, you know, you probably need less data than you think you need and you probably have more data than than you think you do. And that's generally going to be the case for organisations and for projects, especially on the things they aren't measuring so far. What what other things we talk about that you should be measuring as well as the the outside environment of projects. You mentioned politics as as an example of how it can add
uncertainty to projects. And obviously that should be something if you're doing a big project that runs over multiple years, you should be measuring kind of the political context, political support of a project, because that's going to add uncertainty and it might mean that you're building in more decision options along the way of a project. But that's also one of the recommendations of the book that, you know, the projects, they aren't in their own bubble.
They they do engage with and they are impacted by the external environment. So that's also some other things you should be measuring that you haven't been before. Thank you. Could you give an example of a project that perhaps does take on board the things you're telling me the recommend, the recommendations that you make in the book? Is there anything out there that people could find out more about?
I've run into a couple of examples in in industries that do some of the things we're talking about. So for example, the option to cancel a project, that should be an option that's defined very well in advance. It's a part of a decision model that's run every day. Now nobody's doing quite that just yet, but in the Pharmaceutical industry and some related industries, they have a stage gate process. So only about 8% of all pharmaceutical products that are investigated ever make it to
market. So how do they make any money? Well, when they quit, they quit early. And when you look at a lot of big projects that cancelled, and a lot of them do, they cancel with nothing to show for it after spending a lot of cost and having a lot of external consequences that are costly after. If you look at those kinds of projects, you often find out that they could have and should have cancelled it quite a long
time ago. It they had all the data that it would would have justified cancellation quite a long time ago. It was just running on inertia. But also there's that feeling of, I think is it called sunk cost, sort of sunk cost of allergy where you've put all this money in already so it's too late to turn back. Right. It should always be on the
forecast of the future. Here's what we here's our uncertainty about what there is remaining to be spent given what we've learned so far about the complexities of the project, because that might update our uncertainties about what's remaining to be spent. And we're comparing it to the outside world. It might not just be the project, it might be that something in the outside world has changed that changed the benefits of this thing.
If I'm. 35% of the way into a project for some big infrastructure project and AI at the same time is improving because the and it will, because these are multi year projects, right? So there will be changes in not just AI, but materials and things like this and fabrication methods during the project. At what point during the project do we say there's enough that's changed about the outside world that I should now re engineer something about this project?
That's a intervention decision. That's an intervention option that we should define in advance. So the pharmaceutical companies are one, but also aerospace does a little bit of something that we talked about where they do anticipate changes in technology. Because if they're developing a new crew re entry vehicle from the space station or something like this, they start designing it without necessarily assuming that the only materials they
have are the current materials. Because it'll take a decade or more to design these things. And during that time there will be technological improvements. They're kind of counting on them in some ways. So we should take that into account. If we have a big infrastructure project, should we be saying asking questions like if AI could have changed this design if only I waited 2 1/2 years from now, what would I do differently? Is is there a point at which it might be worth waiting?
There's something we introduced in the book called Technology Regret. Technology regret is includes things like you invest too soon in a technology that's rapidly changing, so you fixate on a bridge design or fixate on some software features at the same time while you're pouring concrete and writing code. AI is getting better. Not only AI, but materials and fabrication methods, 3D printing, hole bridges and so forth. That's getting better during this.
Now AI is actually, I think improving faster than the other two items I mentioned. So we should at least be looking at there. There's not too many big projects that are going multiple years that shouldn't also be tracking what's going on with AI and asking questions about. Is there a point in time I should consider the possibility of right now and design an intervention option where I'm so far into the project and something the outside world has
changed? And given how far I am into the project, and given the benefits of switching gears at that point in time, does it make sense to change something during the project? And you're saying that the answer could be yes, it could be, it could be beneficial to. So, so you're, you're planning in uncertainty, you're kind of planning that. That's quite probably quite a mindset shift for lots of people working on projects like this.
Yeah, I think people are used to the concept of options, though some are. They're used to the concept of real options too. You know what's the what's the value of keeping an option open, right? Actually, even things like the option to cancel a project. This surprises people, but it de risk a project. Risk of a projects go down when you build in intervention options because in the case that you do need to cancel a project, you'll at least be able to cancel it sooner.
So your expected losses would be lower if you do cancel a project. Now I've had lots of clients that would say, yeah, I've seen projects get cancelled, but prior to us developing decision models for them, they never saw a chance of cancellation on the business case for a project. How often do you see that right, if you're if you're presenting to a government agency or someone else in the corporation. I've got this idea for a project.
Should I include the fact that there's based on historical data, there's about a 15% chance of cancellation with nothing to show for it. Some we've heard this, Andres and I were just talking to a group of the last time I was in the lending group of, you know, project managers and stuff like this. They'd say, well, you know, things wouldn't get approved if that were the case. If you're I said, well, if that's what you're looking at, then just leave out half the cost too.
Why not? So the no, it's, it's about being transparent about these issues. Also, I think there's a fundamental conflict of interest here if the person opposing the project is the only one building the decision model for it, right, and the only one presenting it to decision makers. I do think there's room for some independent body of auditors. Actually, Oxford Global Projects does some of this right now because they audit people's projects, project plans and
costs and schedules. That's part of the services that they do. And one of the values of having an outside consultant come come in, in general is that, hey, we don't have any skin in the game here. We're just going to do our analysis and give you an honest representation of the outcomes. Because usually we're working for the check writers, the people who were allocated funds, not necessarily the project manager or the person who wanted
to promote the project. We're working for the people who have have to commit to the funds in the 1st place. So they want an honest answer and I think that's part of it. We have to think about some built in conflicts of interest in the way that we approach even proposing projects to begin with, but and then everything else that we just talked about. South more rigour what? Andreas, what would you like to
add to that? I love this conversation has been about you know what, What type of projects should we be making? How can we plan projects better? But we do spend quite a long time in the book talking about different types of measurements for projects that are already in flights. What should you be start starting to measure and how can you make, how can you make kind of metrics more, more decision driven?
So we, we talked a little about, we, we have specific chapters about, you know, cost and schedule and benefits progress, typical things that you need to measure on, on projects and that it's really useful to measure. But one of the things we observed was that a lot of the metrics that are currently being used and on all sorts of dashboards, they're very kind of exploratory, as Doug would put it. So they're there because decision makers are project
managers, are risk managers. They feel that once they know that once they look at the dashboard full of metrics and they see like the right combination of different figures and charts and, and metrics, they'll have some kind of they'll know what to do in that kind of instance. They'll know how to react on the metrics on the dashboard.
But a lot of the time what we discovered was that a lot of the metrics aren't geared for decision making really, and not about, you know, what intervention decision or how should we react if we see a specific metric. So the book spends a lot of time thinking about, you know, progress metrics like cost performance indicators or scheduled performance indicators and so forth, and thinking about, well, how could we make it reframe that into something
that is easier to understand. And a lot of that is reframing it into, well, how much productivity increase would we need to have to actually catch up on our delay, for example? So we do give a lot of advice in terms of how should you be measuring, how should you be presenting metrics? And I think the other thing in the book that we do is we be sure that because this is a book about measurement, there is some maths in it. But we do show that, you know, all this maths isn't complicated.
So alongside the book, we have a website where you uploaded a whole bunch of spreadsheets to show that all these things. We talk about the technology regret example, we have an example of using machine learning for predicting outcomes. We have different types of simulation exercises and we run it all in Excel, which you can download and apply it to your projects just to show that.
It sounds pretty complex if you're used to using more subjective scores and scales and risk matrices, but it actually isn't that difficult and there's a lot of already built solutions out there that you can just apply. If there's one thing you'd like readers to take away from your book, what what might that be, Andreas?
So if, if I can positively frame it, because you've talked about issues a lot on this on this podcast so far, is that there is actually a lot of evidence, even though there's in the industry, there's a lot of claims around or if you just apply our methods or our tools or our software, you're going to better projects.
That's typically what we see. If we go out to conferences and we look at like vendor stands, it's all the same software companies with some kind of reframing of if you use our products, you're going to deliver to cost into schedule, right? That'll be the typical experience. But there's so many claims out there can be quite difficult to find out what works and what doesn't. And what we've tried to do is actually just to look at the data. The research has already been done.
We've done some of our own research and we've identified that, you know, some things are better than others in terms of measurement or estimating or delivering projects. So that there's there's an opportunity to apply the things that work and kind of toss the things that don't work. Fantastic. That's very well put. Thank you, Doug. Any any one message you'd like to get across to to listeners? Yeah, we, we refer to the the most important project, the one that is about making projects
better. We call that the meta project in the book. And everybody ought to start thinking about their meta project. You know, no matter how big your project portfolio is, no, no matter how big the individual projects are, some time is worth being spent on the meta project. Your most important decision is how to make decisions. By the way, that's in life and business and government and everything. Your most important decision is how to make decisions.
So we should apply that here. And so the meta project is, is the project about making all the other projects better. What's a great way to spend 1% of a whole project portfolio trying to optimise the other 99%? That's the best way to spend 1%, right? And it might be about that much for a lot of project portfolios. If you look at some of the infrastructure portfolios, you know, even spending 1% of those portfolios is a significant meta project where a lot of analysis could be done.
And so we recommend having a project called the Meta Project and Everybody's portfolio. Thank you, Doug. I feel as though we've just scrape the surface of what we could talk about in your book. I feel like I might have to do a follow up podcast sometime soon. But thank you very much for your time. Thank you for your thoughts.
I wish you best of luck with the book, but it's been really fascinating talking to you and I'm sorry we couldn't cover much more of what you've what you've put in the book, but thanks very much for your time today. You best text. Thank you. Thanks again to Doug and Andreas for joining us and to you for listening to the APM podcast. I hope you've been left with much food for thought, so don't forget to look out for more episodes, alterate and reviews.
Wherever you get your podcasts. We'd welcome you to get in touch with your comments, feedback and suggestions by emailing us at apmpodcast@thinkpublishing.co.uk. Spotify and YouTube users, please do also leave us your comments. This podcast has been brought to you by APM, the chartered body for the project profession. For more information on APM, visit apm.org.uk.