How AI is changing projects right now

Welcome to the APM podcast. APM is the childhood body for the project profession. My name's Emma DaVita. I'm the editor of Project APM's quarterly journal and your host. In this podcast, I'm speaking to Daniel Armanios, BT, Professor of major programme Management at Oxford University's SCII Business School, and Zach Swafford, Co founder of Dot, a

San Francisco based startup. He's a project management tool uses AI to assist with brainstorming, road map planning and report generation, among many other tasks. We thought it's the perfect time to cast Sari over the rapid developments in AI over the past year and speculate on what 2025 might hold with two guests with two different perspectives on

the world of work. Listen on to hear their thoughts on the exciting potential of AI, their worries about the technology, and their advice on what you should be doing to practically get to grips with this work revolution. What's really valuable, I think, is their insights into how AI is being used in projects right now. So do listen on. I'd like to welcome both of you to the APM Podcast. Thanks for giving us your time

today. Let's begin by finding out a bit more about your work and your area of interest in AI. So perhaps, Daniel, do you want to tell us a bit about your work and your area of interest when it comes to AI and project management? I'm Daniel Armanios. I'm the BT Professor and Chair of Major Programme Management here at the Site Business School, University of Oxford. I think my work is at the intersection of civil

engineering and sociology. So essentially I'm interested in how organisations coordinate to underpin really big scale initiatives. And so my interest particularly around the AI facets, is thinking about where do you deploy what kinds of algorithms in the organisation. And so when you have a project with several organisations, several stakeholders, where best to place what kinds of

algorithms. And so I've been spending much more time on the organisational design programme, project design kind of perspective to look at where you place these kind of algorithms, knowing that AI is not all one thing, right? It's been multiple things. And so that's where I've been spending a lot of time. And the second is where is the assistance of AI really good at processing a lot of information, but also where do we need to build guard rails and caution

around the hallucinations? The ability for AI to shape the informational landscape in ways that might not be fully revealing of the truth. And so take its benefits, but also be mindful of its places where it can go off, off the rails, if you will. Thanks, Daniel. So by what you're talking about, do you do you tend to focus on mega projects, bigger projects? Usually large scale projects, Yeah, usually large scale projects.

But I think often how it's being deployed in smaller projects can be a harbinger of what could happen at scale in a larger initiative. Is that how it tends to work out in the real world, that that organisations will will experiment with a smaller project and naturally that will scale up or not depending on how successful it was or lessons

taken from that and shared? I mean, I'd be, I'd be interested in Zach's corporate experience around this in terms of what they're doing with this, the start up at Dart. But I think what I would say is, is that ideally that's what you'd like, like some experimentation on a sandbox. But I think matching with, I think this year was, was really a year of, Oh my gosh, we need to get on this.

And so what's happened, there's a lot of organisations that feel, Oh my gosh, we just have to get everyone to do it. And no matter what, deploy it. And so you would hope it's done with strategic intense, you know, sandboxing, etcetera. But I think there is a bit of, Oh my gosh, we're behind the curve, let's just deploy it. We need to show that we're doing something with AI. And so in some cases they're just experimenting and you're seeing this with AI itself.

I mean, a lot of these products are being disseminated without necessarily full on validation whereby it's like what we're going to know what's the issues or know what is by people trying it. And so I mean, even open AI when they launch ChatGPT, other things, it would just let's just see what happens. And you know, there's different perspectives as whether that's OK or not, but I think it's a

mix of both, right? Some are sandboxing, trying this, figuring out the kinks where others are just like, let's just get it out there and see what happens. Each has their pros and cons, if you will. So a little bit messy in real life. Yeah, unfortunately. Thank you, Daniel. Zach, tell us about about your work and a bit about your experience with AI. My name is Zach Swafford and I am a Co founder at a start up called Dart.

We do project management with AI, so pretty relevant and we got started a couple years ago basically because we saw in, we saw a big sea change here basically in the project management space with AI. We saw this coming a little bit and wanted to build really from the ground up a new project management tool that would incorporate AI and really specifically when I say I, I mean LLMS on into every facet of the project management experience.

Can you spell out what an LM is just for people listening? You might not know. So an LLM is a large language model and this is something like Jet GPT or Gemini is a Google One Co pilot and Microsoft Claude is another one. Those are some of the biggest models and those the idea is that they use a lot of language, a lot of text and other information from the Internet to create a chat like experience.

But really thinking beyond chat. I think in the longer run, these are the models that will act more like a like a real AI like a real Jarvis or whatever you want to think about. You know, these these ideas of big AI models that can do almost anything. These LLMS are the technology that starts to enable that more so than traditional machine learning, which can be great at answering specific questions and is an excellent tool for lots of things.

But the LLMS are the side that I am the most excited about right now. So there's lots of interesting stuff we can talk about there, get into that. We have tonnes of really interesting features in Dart. But to answer sort of the question of how are, how are organisations building up to this? I've seen it, I've seen it all sorts of different ways. I've seen, like Dan was saying, really diving right into a big project and kind of jumping off the deep end, which can be risky. Of course.

I've also seen really successful implementations where folks are trying something on a smaller project and building up to it. And I think there are there are pros and cons and there are also different levels of risk that different organisations are able to accept or willing to accept. And depending on the industry. At DART, we work with a lot of different industries.

And I always say that some of our marketing agencies, right, can accept a higher level of risk and a higher level of incorrect answers or hallucinations or whatever than a team that might be working in healthcare or, or construction or something like that. So that's another perspective on, you know, maybe different industries can move at different speeds here. As well. And Zach is have you had a background in project management before? Yes.

Great question. Yeah, sorry, I should have, should have answered the background question. So, yeah. So I studied AI actually specifically at Stanford a few years ago and then I went into a company called Plenty and we did indoor farming. So at Plenty and I worked on the hardware and software sides. There are LED teams in AI software and hardware actually. Would you mean by Intel farming? Is that literally what? Oh yeah, what do you mean by that? Absolutely. It's what it sounds like, Yeah.

So. Farming inside. That's right. Yeah. So you can think of it like a big automated greenhouse. I think that's the simplest way to think about it. We grew leafy greens, lettuce and arugula, kale, lots of fancy herbs and the indoor farming process is. The big idea was using automation and AI to really have the perfect environment for plants and grow them really fast and well and healthily. And when you have an indoor farm, you can also place it right next to where it's going.

So you also don't have to deal with the supply chain logistics and you get a lot of fresher food. So yes, very interesting work and, and plenty in particular was a really fast growing company. So we, we grew a tonne and that meant that we had all sorts of scaling problems, all sorts of project management needs.

So I filled the gaps in engineering management, but also in implementing project management, both for software, which is really my original background, and then also even building Gantt charts and helping the construction and then the builds we were, we were another startup. So there was just a lot going on and, and I filled some of the gaps there. And that's really how I got started with project management formally. OK, well that's really interesting.

So you 2 have very different perspectives. Daniel Moore, the organisational take in the bigger picture and Zach, really nitty gritty of making this work day-to-day. I think there's the thing that I'd really like to hear about is looking over 2024 comparing where the world is now compared to a year ago. What would you say have been the the biggest developments when it comes to AI and project management specifically? Zach, maybe on the ground, what, what would you say? Jump out at you?

I think that in the past year I've seen of new tools that allow people to chat directly with their projects. So this means asking questions about are there any blockers? Are there what, what are the, what are the critical path in this timeline? Those sorts of things. Or sometimes the questions that project managers or potentially even other stakeholders might formulate and then might want to ask, but maybe would need to go through.

You need to have a bunch of messages about, about that and, and go through a whole process to figure those sorts of things out. Those are the kinds of questions that if you have your project management tool or your system in place to have all that data in good shape, then you know a random stakeholder would be able to just ask the question what's

our critical path right now? What's our timeline and get a chat based answer, which is particularly nice actually for non-technical folks or people that aren't steeped in the project management world like we are. OK. That's great. Daniel, you agree with Zach. Are there any other developments that you've noticed that have been on your radar?

Definitely one of the trends I see very much in agreements with Zach. So the ability to take large amount of corpus of text because if you think of projects, they fundamentally produce a tremendous amount of language data, text data. And So what DART is doing other organisations and plan others, they're really enhancing drastically the processing capability. So I mean, you can it's get, it's quite revolutionary project

world. I mean, the idea that you can take a request for proposals, run it through these LLMS and come up with an addition, an initial Gantt chart, a racy diagram just from the data to be able to ideate things that you know are doable, but usually took, you know, burned a lot of time. And if you think of projects, often the issues are like death by 1000 cuts. You know, if you can save 30 minutes here, an hour here doing this, the fact that you can see critical pass in your data run

last planner models. And this is where the Omni models get into the way you can ensemble different models together. Perhaps even now there's a lot of interesting trends on, you know, reference class forecasting, benchmarking, for example, right, how you come up with the reference class, There's 20,000 different ways you can come up with a reference class that can be done back end automatically with some of these generative AI large language models. So that's one dimension I think

is really interesting. The ability to take all of that data and make sense of it, that would take hours, months of really specialised time. And even if it's not perfect, just getting you started, you've already saved an infinite amount of time. So the ability to ideate on this is, is is really revolutionary. I think where another trend is happening is the visual turn. So there's now models that can translate what you're doing or what you're thinking in text

into image. So there's some organisations, some companies like Icon for example, that they, what they're planning to do is take your ideas and then translate them into initial construction documents. That's an amazing turn and then schedule what would look like that from that. So I think the visual turn, even using video to do this, be able to OK, I have this idea.

Let me generate a video of what this could look like and immerse someone in it. So the connections between large language models and visual or even video motion kind of imagery that could then be tied into immersive reality, other

things. So it's not just the text and the visual, but the modality is really quite fascinating and really exciting because especially at the early stages where if you could plan and see much more of the information around you both in Word but also image, that could save so much cost at the back end because you've already kind of stress tested. Red teamed a lot of different scenarios that would have taken a lot of costs to do.

Because if you have to do that without this kind of assistance, the issue becomes I have to then think, is this time to build this scenario, to run this chart, to run these matrices? Is that really worth the time to do so? Then you have to spend a lot of time being very careful about where we're going to search more, where we're going to plan more. And with these tools, you can run many more trials than you would otherwise.

And it has huge implications. I think it's just really exciting to see that strain both in the text base term large language models, but also now the different modalities, text to visual. And we could talk about what I think is happening more recently, which is even the ability to train on really pretty interestingly accurate

synthetic data too. So it's not just looking at your own prior project data and seeing it running synthetic data that because of how they're trained, it could be quite accurate. It may not be, but at least as a starting point could be quite fascinating.

I mean, for example, in spatial computing, which is a lot of the back end Nvidia's now training, you know, if you have a project on autonomous vehicles or other things, they're actually training on synthetic data because the lighting and stuff now is so good. So autonomous vehicles, for example, are being trained on synthetic data to some degree. We get into the final one, I would say, and then I'll move on. Is this.

So we have we talked about textual, we talked about visual, but also the ability now to artificially or with AI create kind of interactions of social interactions between people. So we can take the next step and say if these different project manager are working together, how would their interaction look like? What pitfalls can we worry about simply by embedding rules into simulated managers or agents? This would be called deep reinforcement learning.

And they're remarkably adapt. So for example, they can run scenarios like what would you know, birthday parties and who's invited? I mean, the social interactions you can do with just very basic rules. So I could see in the future, and even now it's happening, run different managers with very basic realistic assumptions what would happen when they work and interact in this way over a few months, what could be, what could get, what pitfalls could

we worry about, what not? And you could do that all at the early stage. So it's really, I mean we're only scraping the very initial surface and I think that's why Dart and plan icon. These are just going to grow because of their ability to really ride that initial wave. And so with that comes amazing power. And also, you know, we could discuss the perils later, But in

summary, I think three things. One is the massive ability to read large scale text, the text to visual and motion right, It's like video and so forth. And then the third one is even to simulate what interactions of individuals in these projects or large scale programmes or even small one could look like. Is it realistic? Not necessarily, but it is at least a really good way to ideate in very different ways at

scale. So what you're explaining there sounds very exciting and we're on the cusp of big things. I'm just, I've, I haven't heard anyone talk about the, the kind of project manager interfacing with another project manager. Do you mean that you could get AI to understand the characters of individual people and how they might work together? Is that what you're saying? Yeah, I think so. So there is, and we can talk about the integration of how

these different algorithms work. But one set of algorithms are what's known as deep reinforcement learning. So essentially what happens is you populate with some very basic rules so you can look at your prior work as a project manager, otherwise say what are kind of heuristics, rules of thumb or ways we operate that we can embed into an agent, right? And then you say, and then how you, the learning aspect is when it does really well, you say, yes, that's great.

When it kind of goes off the rails, you say, no, no, no, that's not what I'm looking for. And as you train these agents with these rules and let them kind of code another and deal with the data, you could potentially simulate interactions. There's some interesting work. It's funny, Zach mentioned Stanford. I graduated from Stanford as well.

There's a really interesting study at a basic level out of Stanford where they developed this kind of, I don't know what you call it, like a kind of, it's almost gamified, but it's an interaction called Simorca. And essentially what they do is they embed agents with kind of rules. Now in that case, it's like we're hosting parties or general

social interaction. But if you use your insight and acumen as a project manager to understand here's how I break down sets of work, here's how I interact with with suppliers, etcetera, you can simulate that interaction. You can even build avatars to practise. So for example, let's say I'm going to have a difficult conversation with a supplier, but I want to practise what that

would be like. You could interact with an avatar who has these kind of rules to help you train for this difficult conversation I'm going to have about procurement. Right. So the, the interaction ability and that's only just started maybe within the last couple of months, I would say in the project management space. It's been longer in the CS space, but you're starting to see that ability in the project management very recent.

So it's not just taking my data, rendering it, visualising the Ida, but even practising how we even do the work is possible now I think. OK, So what I think I'd like you to do is if I take you away from from the, you know, what ifs and the excitement about what could happen to actually bring us back to how AI is becoming a facet of work life and projects these days. So kind of practical examples. So there are honestly lots of stories of success that I'm

familiar with. And I think one place to start is where the newest technologies are at their best, where they really shine and thinking about how that can be applied to PM as a discipline. So one of those Dan just mentioned is acting as a smart, I like to say sometimes a smart intern almost giving you an 80% version of what you need to.

And maybe maybe it's almost like a team of interns where you can send them out and proactively get all sorts of different first stabs at a project or at A at any given task really. So one place where I've seen that really successful a lot is in initial phases is in brainstorming, ideating, trying

to come up with an initial plan. A lot of this happens in software, maybe more so than construction, but at the beginning of a software project we might be thinking about what are the different ways we can tackle this. Here's a bunch of the user feedback that we have. We have a long list of tonnes of documents about how users act and behave. And now what we need is some solutions, some some ways to tackle that problem. There are great tools really tactically, I might just put

that as a first stab. I would always say just put that into Cha Chi PT. Just put all those documents straight into Cha Chi PT if you can, if your organisation allows that sort of, you know, security wise, but put all that straight into the nearest large language model and see what happens, right. And, and see, get some of that back out. Ask, ask it, Hey, can you help me solve some of these problems? Can we ideate about how to solve

some of this? And probably what you'll find is that some of the answers are good, some of the ideas are good. A lot of them lack the context and awareness of your organisation, of your practises of kind of what what you actually need to do. You'd be surprised sometimes how that's not totally necessary. But in a lot of cases, then you might need to iterate and start to provide more of that context and start to think about more advanced tools that would allow you to include that context.

So that's one way. Another way I would say is, yeah, I mean, I guess I think an important thing that I wanted to emphasise was figuring out the ways to work with your organisation to have the right security posture to allow you to use those tools. Because it is really important that people at all levels of an organisation are able to experiment with this stuff, in my opinion. And there are ways now to allow you to not trained on any of

that data, right? You can get an agreement with open AI to originally this was not the case. And so people were worried about providing information to LLMS and there was kind of a let's let's take this slow sort of approach. Now. I would say it's a lot easier for anybody to get the correct kind of agreements in place to say, OK, our data isn't going to

be used. We can treat this more like any other service provider or cloud service provider and let's you know, let's get chat be set up in our organisation so that we can start experimenting with this. Do some of that brainstorming. Do some of that initial get. Get some of the initial takes on how we can apply this. Provide the documents, provide the context, start to get back some of the answers. So it's really about using another resource to make connections.

You might not necessarily. It's like having a whole nother team of people in the room who might come up with some really terrible ideas, but some of them might be the valuable ones that you may never have picked up on in the first place.

That's right. Yeah. And I think that that's an awesome opportunity for or folks to take away some of the, the hard work, the brainstorming work, the grunt work of just building, building a timeline, building a Gantt chart, let's say, outsource that to some of these tools. And then free you up as as a honestly, as APM contributor at any level really to do some of the higher level work. Even if you're junior in an

organisation. Now if you end up with a team of interns that can come up with great ideas for you, then you can start to build the muscle of discerning what the right course is and then escalating that as your your work essentially. OK. And what about a more sophisticated level or a more senior level cross projects where AI is taken more seriously beyond the kind of initial experimentation as you've described? Where do you see that happening right now in the real world?

Yeah. I think there it becomes more about, I see a lot of the challenges at that level as more interpersonal challenges of getting stakeholder alignment and you know, you've got a plan or you've got some candidate plans and you need to, you need to collect all the resources and get everyone focused around a common goal, a shared goal. That's the kind of thing that AI doesn't do as well right now. So you can, but there are still ways that you can really apply

those tools. Daniel mentioned earlier the ability to run scenarios right to you can you can say, hey, act as this stakeholder. Let's, let's I, I need to practise for a stakeholder meeting here where I'm going to be delivering some tough news or I'm going to be pitching this plan. Help me give me some feedback on the ways that I could improve this project proposal, but also just my demeanour and the way that I the way that I present

myself and present this project. That would be a very concrete case and and anyone could just have that chat with ChatGPT right now to to practise for a meeting. You don't need some super hyper specialised tool. Once you try ChatGPT and you see where it fails, you can move to a more advanced tool for something like this specific to

the project management domain. But I would encourage everyone to just give that a try and see if see if you can learn something from it. See if you can get a takeaway there that can help you to help even improve your your inner human interaction. That's I haven't that's not really something that we've covered much before around using AI to improve your human skills. That's you always hit a negative and how it's taken away from the world of humans.

But this is absolutely putting that on its head. Daniel, I can see you nodding your head there in agreement, but what have you seen AI used across projects, programmes over the past year that would have been successful? The way I generally characterise it, I think Zach is really spot on. Maybe maybe to take a background, I think it's important to understand how these language, large language models work so that you can understand what it's useful or not.

Essentially the best analogy I have is let's just use generative AI large language models as the example. It's kind of like a stochastic parrot. So what does it do? It sees some words in a question and it says, OK, I've seen these words Co located with each other. I'm going to go to my training data and model that I've developed on that training data and respond back with something that seems probabilistically

related, right? So if I asked certain kinds of questions, it knows I've seen that question before, I've seen these words kind of matched together. And so I can give you a response that would probably seem similar. Now, why do I say this is you have to realise that ChatGPT others have been trained on essentially the realm of large scale corpus of data like

Internet level data. So why do I say this is that it's good to start where I've seen it useful is good to start for like generalist things to start things that you know are very common in project management and are very like, I would start with the thing What I've seen is you start with a very mundane things that everybody with a project has to do, a Gantt chart, a racing matrix, you know, like that level and stuff that you have that you're willing to share

publicly, right? So you start there and you get a sense of, and I think that's where the biggest, easiest low hanging fruit is mundane, painful things that you have to do. You want to have 80% trial like Zach said. And then that way I could spend my time finessing the details as opposed to building that up over hours. I can just start there. Where it gets like the added value is now if you have something that's your specialist sauce as an organisation.

Now why did I mention all of this stuff about large language models is they're chained on generalist data. So now the question is, how do I get it to understand better my refined thing? Let's say I'm doing healthcare work and I'm looking at a specialised disease just using ChatGPT, etcetera. It's going to give me everything about medicine, but it's not going to have specialised knowledge. And sometimes you want it to

train on smaller data. So there's two ways Zach brought it up about this is that some companies, their large language models, you essentially have to ping the server of that model and that's how it works. Other ones, you can actually download their neural model, bring it into your closed kind of space, give it data that's only to you in your closed firewalled environment, and essentially take the basis of the algorithm existing and train it with more specialised data.

So now if you're taking the next step and say, well, I get a sense of how I want to do this Gantt chart. I get a sense of how I want to do the race, like general things that just are pain points that are mundane. And then I take the next step and say, hey, I want actually more targeted knowledge for which we have the secret sauce. You can train that model uniquely to your data. And so that's called transfer learning essentially.

That would be the next step. Now I can get the really specialised know how to develop it. That's just for me. Now the point Zach made. And I think there's concern that, Oh my gosh, if I use this, the human will be displaced, etcetera, because AI can hallucinate. You really need to have expert knowledge of what is sensible and what doesn't make sense.

So there's a little bit of danger if you're trying to use it to explore in some newer area you're not familiar with, you need to make sure you're triangulating it with other things and, and stuff, you know, it's your expert knowledge. In other places where you're using it to experiment, you're going to need to look at other sources just to make sure. OK, that sounds interesting, but let me Fact Check it. And there's ways to, there's settings actually you can do to

change that. There's something called temperature. So if you actually make it, I think it's really, really low. It's saying, I don't, I want you to be absolutely certain about this. I don't want you to be creative versus I want you to really think crazy. You can set the temperature really high and say, come up with the most creative, unhinged thing. And even if it hallucinates, maybe it's interesting, right? So it just depends on the nature of the problem. But very quickly, generalist,

mundane pain points start there. As you build acumen, build experimentation, you want to do something more specialised, you can take in those models and train it specifically on your data in a firewalled way. There's ways to do that now. What benefits is AI bringing right now to to projects that you've seen, Daniel? I think massive time benefits already immediately, right, in terms of saving you all the times you have to deal with things.

I think also it's saving you on cost later on because if you're doing so much, the the theory is if I'm experimenting and trialling things on several scenarios and getting a bunch of these 80 percents and looking at it, in the long run, it's helping me avoid issues that I would have that I wouldn't have been able to recognise if I didn't try that experimentation

up front. But I think the most immediate for me is the time benefits, things that used to take really long time that have to be done are getting you most of the way. And I think to Zach's point, as you use, if you do it with a variety of LLMS, each trained in different ways, like Claude, like Gemini, like Co pilot, like ChatGPT, you also can triangulate pretty quickly as well. And so I think there's there's like triangulation robustness that I think in the immediate saves you time.

That's immediate. I think the cost stuff is, is the things you would say based on comparison of what you had to do in the past of mistakes you can avoid now because you've you're able to experiment far more widely. But I'd say immediately is the scheduling and time facets.

As a leader, so as a project leader, like one level up, how, how do you lead on AI within, I know your department or you programme or an organisation And how do you, I think, Daniel, this is a point you, you made around making AI explainable in terms of what it does and what assumptions it makes about project delivery and outcomes. So, so to start asking people, your workers, your Co workers to have that discernment, not to go into this blind. I mean, it's pretty nuanced kind

of sophisticated stuff. If all you've been doing is trying to experiment a bit with ChatGPT, you might not get all of this stuff. So as a leader, as a project leader, how should you talk about this? How? Where should you be about the language you're using and what you're asking people to to take

from what AI produces? I think you're really hitting a profound nail on the head, Emma, because if you look at the surveys with CEOs, executives, more senior leadership and projects and organisations like Price Waterhouse Coopers did a very nice one. And they basically said 61% are like thinking this is going to solve a lot of issues. They're going to increase productivity rapidly. And then if you do, there's another interview by Upwork on

the day-to-day worker. Look at the, I call it the front line. It's not a great word. But let's just say the, the those who are nearer to the locus of work and they find something like, I want to say 77 percent, 75 feel it's actually making them worse off and burning them out. And what's happening is, is you'd be sad. You know, we joke about the, I mean, Zach mentioned the experimentation.

You would be surprised how many senior level people are just saying this is going to solve everything without experimenting with it themselves. And So what I think is happening, the gap is, is an over expectation of what this will achieve and how much judgement will be needed. This is why I think it's really, really important.

First and foremost, senior leaders need to experiment with it. And if you are actually, they need to experiment this so they can calibrate their expectation in judgement because they're looking at the kind of what they see or hear or perceive and using that to drive. Oh, yeah, yeah. Just do it for everything. And that's creating a lot of burnout.

So that's one aspect. So the question is, and there's different ways to think about it, Do you treat it like, there's a very nice analogy by Ethan Malik at Gorton. He argues, do you look at this as cyborgs or centaurs? So the idea is, do you look at it as I even more pragmatically, do I outsource bits and we look at it sequentially where AI does something and passes it to me, etcetera?

Or are we doing it in parallel where you can't even see where the human and AI end and there's different kinds of ways you can use AI in your teaming to do that. Do you treat it as another team member that you outsource stuff and get back and sequence it? Or are they part of the ideation where you don't even see where the the team comes and goes?

That's more like pragmatic. I think on terms of your question about at the end of the day, we're dealing with a black box where we don't know all the latent features. There's a real push right now to get a sense of what is going into the algorithms, how representative they are, how do they change the data when they go in. And there's some discussion about creating like prescription labels for AI or nutrition labels with things like how

representative is the data? What data was this trained on? Where is it vulnerable? How long is this? How long is the insights I get from this algorithm worth useful or not? Does it take? Is it months? Is it days? And I think even those simple things would really help people understand. Yes, it's a black box, but given what I know is going in and what's coming out, it gives me enough kind of insight to say, OK, maybe I should try these

things versus another. To give a really tangible example of management consulting you brought up, Emma, is that they find they did a really interesting study at BCG and all it was was literally cutting and pasting ideas into the into the algorithm and there was a massive boost in performance. I want to say like not massive, it was like 17% increase just

from copy and paste. But when they gave things to the algorithm for which it was not equipped, sometimes these algorithms have trouble triangulating data or even basic math. They actually experienced a percentage loss in performance. So part of the trick is they have to understand what to give the algorithm and what is not

good to give. And so if we have these kind of nutrition labels, prescription labels to say basically, yes, these are the kind of things we're excelling at ideation, brainstorming, recombining information. And here's where it's not good, like triangulating between spreadsheets and other things in interviews or even sometimes basic math, then we're going to suffer. So I think that and that's why it really it depends on senior

management experience. It would be great if they experiment with their team so they can work on how they would negotiate, how they would bring it into their protocols. And from that they can learn more quickly what to share and what does it get completely off so that you get the performance benefits without the performance detriments, if you will. That what are you talking about there is learning what algorithms work for which data and that's the sort of thing

that. On a team, people individually might be going off doing their own thing and not thinking necessary strategically. So have you got any advice around if there's a project team and everyone's doing their own thing and they know they should be doing and it isn't the most valuable thing there to share what you've learnt?

It is. And so as a leader, that's something that you should be building into, building time into regularly to make sure that everyone knows what everyone else is doing and what works and what doesn't. And somehow there is some kind of process around that so people aren't wasting their time and everyone's reaping the benefits.

Because it's like you're very much working in, you know, with, I imagine people who aren't that used to using this kind of stuff in their projects or they're excited about it or, you know, what advice do you give them? How do you see people use it and experiment with it? And what advice would you give? Yeah, absolutely. Yeah. Well, I, I completely agree with all of the points that we've talked about so far. I think that socialising, what's working and what isn't within an organisation.

I'm personally a big believer and my background in startups and perhaps show some of my bias, but I'm a big believer in letting teams be independent and experiment with different things and even in a bigger organisation and then have the having the the successful strategies, the successful projects rise up and then be disseminated.

And so I would encourage the way that I would think about it would be to encourage teams to use different tools and experiment and try, try out what the the new, the new breadth of technologies, because we don't know what's going to work in the long run. But we're still we're still in the middle of this transition or even at the beginning.

And so there isn't one right answer that's going to at least most organisations, all but the very luckiest probably are not going to set one course through this storm and maintain that the

whole time, right. So the key instead would be to develop systems, as you're saying I'm a really where the best practises and the best new ideas coming up can be socialised and spread more broadly and spread throughout an organisation to have everyone on all the different teams follow those best practises as they evolve. I think it's really important to ask both of you where what you think 2025 will hold. I mean, I know that it's hard to

look so too far ahead with this. And equally it can be fun to get carried away the excitement to see what, what, what are the possibilities, Zach? What, what would do you hope 2025 might bring? So what do you feel optimistic about? And equally what what might keep you awake in the middle of the night around AII? Think that there's a lot to be excited about. I, I personally believe that AI will be a paradigm shift across

industries. So in the long run, I think it's of a similar scale to the Internet or personal computing or these sorts of things where every industry, every vertical, every specialisation, including of course PM is going to be changed. And so I think that whether that's in 2025 or in the next, in a few years after that, it's

a different question. But I do think that stay, I mean, it's so exciting because there's so much possibility and there's so much value from a business perspective to be created and added and so much time to be saved and so many more projects to be done. Just straight up, we can do so much more. If we if we can work faster, we can do so many more things, which is really exciting on a

human level. I think that just more and more the ability to have AI agents are sort of these both somewhat more autonomous tools that can have some long running context. One of the biggest problems with some of our chat chat based interfaces right now is that they lose that organisational context. Today. They don't know if you start a new chat, you maybe you told it about some of your best practises and and that kind of thing, but you have to start over if you start a new chat.

So more and more I think we're going to see specific dedicated agents is sort of the term of art to have that long running memory of here's how things work, here's how we do things, here's how we did the last project even right, which is something that short chat session would not have. So an agent that has that long running context and they can also bring to bear some of the general knowledge and general skills to kind of get something done. That's something I'm really excited about.

On the downsides, I think that we've already talked about it a bit, but just there is risk with any, any paradigm shift this big that some people in certain roles and even some entire organisations that don't, all of which maybe don't adapt fast enough to the, the new changes will be left behind.

And so I think that the biggest risk, I mean, the lucky thing about this risk that I'm flagging is that we can all still fix it. It's not the, it's not too late to upskill yourself and your organisation and to adapt to some of these tools and to keep adapting over the next 5 or 10 years and, and to, to weather that storm and learn what you need to learn and build a more valuable and more powerful really organisation in the longer run.

So there are risks there, but I think that we're it's still early, so we can all we can all still weather this storm. Thanks, Zach. That's so interesting. Daniel what? What would be your take on what what we can expect in 2025 and what might keep you awake at night around AI? Yeah, it's hard to think of 2025 because, you know, if you talked McKinsey five years ago, they were thinking, oh, it's 10 to 15 years from now. We're going to have, you know, generative AI and language

learning. And then all of a sudden it came. So I'm not sure when it will happen. But my, my take away from this is we got to think things that we thought were 5-10 years from now could actually be next year. So I'm just going to say the ones I think are interesting. I think there's a couple of things. One is the powering of AI. The amount of energy and water that's going to need to upkeep

the data is massive. So I'd be looking out for what's going to happen in fusion energy, what's going to happen in small modular nuclear reactors. Because for example, the projections are one study out of Yale basically said by 2026 even that the amount of energy that's going to be needed for the servers is the same amount of annual consumption in Japan. So we're talking massive energy. The other one is cooling currently with just ChatGPT 3.

The the study I saw it uses something between 4.2 to 6.6 billion cubic metres of water. That's half the entire consumption of the UK. That's nuts. So look out of the technologies in terms of the cooling and the powering. So that's one. The other trend is how do I make AI work on small data? Everyone's assuming big data, so I think you're going to see a lot more on transfer learning. How do I use parsimonious data that doesn't need an insane amount of energy?

So that's kind of 1, and I have some others, but you know, you wanted to ask a question on this. Yeah. No, no, no, that was that. That was what I wanted to follow, follow up on. What were the other things do you think are coming our way? Not necessary, not necessarily for 2025, but on the agenda at some point? I think an interesting one is the developments in quantum. So machine learning right now runs on what they call binary logic, right?

0 ones, typical computers, Quantum is game changing in this regard because now assume instead of just having zero and one, I'm looking at the probability of being zero and one, now I'm thinking of a continuous distribution that used to be binary and there is massive power that can be developed from that. Now I think it's a ways away, but look, if LLM happened earlier than we expected, there's going to be interesting things of quantum.

I've already seen some of those methodologies used in marketing etcetera. I think one that's more immediate is AI going into the contracting or procurement space. So the notion of smart contracts. So imagine now instead of human to human interaction or project, it's machine to machine. So let's say company in my project, I'm a supplier A and I'm the PMO and we've agreed that I want, I don't know, 100

widgets. I have an A sensor on the machine building it that the minute that the widgets are built, it immediately enacts the contract and delivers it. It's entirely self governing. That's going to be interesting. So you're going to see some interesting work on the procurement space with smart contracts potentially on the downside, the ability to discern between what is fake and what is real is becoming really, really hard.

So there's going to be, imagine a world where someone has a crisis, uses audio from a influential figure in politics or corporation. You can foment crises quite quickly. So we're going to have to build rapidly our ability to better discern fake versus not. And so looking at the regulatory space is going to be interesting. I think the US, the EU and even the Chinese are thinking differently about this. I think the US is a market LED approach.

They want to see what the market does and let that kind of run. I think the EU is taking a more safety approach. They're worried about validating the data quickly. This is why you see the EUAI Act versus how it is in the States. And I think in China they're thinking about the integration. They realise that some of this is going to be difficult, but they realise they have to build their capabilities to discern

what's real or not. So I think the regulatory space is going to be interesting to see about how we handle the, you know, the proliferation of this stuff at scale, but also recognising our inability to recognise synthetic versus not. In some cases that may not be a problem. In some cases you can obviously imagine it could be a very big problem. I'm going to try and wrap this up now, but are there any final thoughts that either of you want

to share? Or maybe I can ask you how optimistical, pessimistic you feel about this right now? I'm very optimistic. I think my my final takeaways would be just to embrace, embrace this paradigm shift if you can, as an individual contributor all the way up to an executive. Anyone who's working on PM problems, I would say just embrace, Embrace the change and see how you can do your best to see how you can apply it to your

organisation over time. That's all that anyone really can do. Daniel, do do you want to give us your your final thoughts? How optimistic are you feeling? Yeah, sure. That's, that's a hard question. I would say I'm in the middle, right. I'm measured about this. I think, I think there's a real lot of potential. But we're also at the same time really as a society, very ill equipped with the capabilities to really harness that potential and where it where to understand

where it goes wrong. So my encouragement is very similar to what Zach said, which is the need to experiment. Everyone needs to experiment at a level of which to do this and not to try to outsource it. The temptation is let me outsource it to a technology unit. Let me outsource it to my lower end. You need to experiment it with yourself to calibrate your

judgement. And I think hopefully with that as a society, if we can build the right labelling, the right guardrails, I think it could really harness that potential. But it's still early days. So I would say, you know, it depends on what day you get me. Listen, it just needs to say thank you to both of you for an interesting conversation that could and should go on for another couple of hours. So thank you again so much for your time. It's been absolutely fascinating.

And I'd love to catch up with you again in the last six months and see how far it's all moved since then. So, so thank you very much for your time. Yeah, thank you. This was a fun conversation. Thanks, Ella. Thanks again to Daniel and Zach for joining us and to you for listening to the APM podcast.

I don't think I've ever recorded such a mind expanding podcast before, so I hope they've given you much food for thought and practical advice on how you can use AI in projects and how to lead on experimenting with it. But anyway, don't forget to look out for more episodes or to rate and reviews. Wherever you get your podcasts, we'd welcome you to get in touch with your comments, feedback and suggestions by emailing us at APM Podcast at syncpublishing.co.uk.

This podcast has been brought to you by APM, the chartered body for the project profession. For more information on APM, visit apm.org.uk.