The Institute of Internal Auditors presents all things internal audit AI podcast. In this episode, Sue King of KPMG and Grant Osler of Workiva discuss the potential of generative AI to speed up internal audit processes and support risk management in evolving risk areas. Hi, I'm here with Grant from Workiva and Sue from KPMG, and we're talking about generative ai. What are some best practices for, uh, entering prompts into the, uh, into the tool?
That's a great question. So I think, you know, just being as specific as you can be, right? I mean, you can, you can write a very, very simple or open plain language prompt, but the more specific you can be of like, compare it to this framework or like specific to, like you were saying, like specific to audit or like, uh, yeah. The more specificity, the better an as you can get. Yeah. The more context you provide, the better you're gonna get results back.
And one of the things we were talking about before was we things in preparing for a session, we talked this, I like to, like, I'll rewrite the prompt three or four different ways. I'll tweak it and I'll compare the results back. Say, oh, okay, these things are common. Okay, I know that's important. These things are outliers. Where am I going? And ask for references like, where did you get this from?
So you can take some of the hallucinations, it's gonna do it, but I can identify 'em, I can deal with them a lot easier. Will will it cite its references? It will if you ask it. Right. That's great. And it's, and you don't have to, you don't have to be that articulate in doing it. Yeah. It fills in the blanks pretty well, but the more you do it, just like anything else, the better you get. So practice a lot. Yeah. And the more you can be, you know, tell it to like, okay, that's great.
Summarize it or what, what are the highlights out of this? Like, to, to really get it to fine tune, fine tune answers so you don't end up with pages and pages. Right. And what, what are some of the most common uses of artificial intelligence by internal auditors? What, what are the, what are you seeing out there? Uh, how are people Adopting? I think what we're seeing a lot in neutral audit, a lot of, lot of work on the front end saying, look, what are the risks in this area?
What are the most important ones? How are they gonna be, how would, what controls would I look for in that? Like what kind of procedures would I do for that? So a lot in the planning is really big. We see a lot on the backend. I don't know about anybody else, but we used to spend a lot of time wordsmithing reports. Gen AI writes better than we do most of the time. Let it take the pass. Then you go into refine it. It's hard to start with a blank piece of paper, right?
But if I can start with something that's pretty well crafted, I can then go in and adjust it and bring it to where I need to be really quickly. It frees your time up to do more value added audit instead of spending your time wordsmithing reports. And how much data entry does it take to, to get it or, you know, to get it to, uh, write that draft report. Like how much, uh, prompting do you have to give it? I, yeah, I don't feel like it really needs that much.
It's like, you know, I've got this exception. Write me a recommendation, uh, you know, make it action oriented. Make it detailed, right. Write, you can put all of that in there. Do you want it in bullet point? You can tell it what you want and it will give you what you ask for. It's amazing.
So I think that's, that's one of the great use cases, right, is like the, the planning and you know, making you smarter about you going to a, a new area that you don't really know as much about that's kind of more technical or, you know, specialized to an industry. Like make, helps you get smarter in that planning phase. Um, and then as Grant says, like gets you to that recommendation, Does it have the ability yet to generate the work programs?
Absolutely. It does. It does. Yeah, absolutely it does. So, so I think, yeah, so doing the planning, the recommendation, but then I think, you know, actually doing the testing as well to you can, you can give it documents to ingest, um, you know, be they, you know, whatever format, Excel, PDF or whatever. And you can tell it like, you know, compare and contrast.
So you can start doing some testing procedures to say, you know, like, okay, like the example that we went through in our session yesterday is around terminated user testing, right? Do I have people who are, who've got active access that are not employees?
But then the beauty of AI is that then you can say like, okay, well once I found people that should have been terminated, then you know, you can give it another data source and say, well go and look and see did that person actually do anything after their termination date? Or like you can give it the tickets and say, was the ticket issued for that terminated user and it just didn't get executed on.
So you can start getting into a lot more of those, um, kind of what if and look back type analysis very easily. 'cause there's been been a focus for a long time of like, how do we get to the point where internal auditors or value added, right? And so I think really being able to go to a control owner and not only say like, oh, you've got an exception, which okay, they hate hearing that.
But then if you've been able to get to the next level and say, well, I've done root cause analysis, the issue is like, the tickets are getting issued, but they're not getting closed out properly. Or, um, or even saying like, Hey, I did this analysis. If you first line were to, were to take that activity, this would be your control activity and here's how we would, how we would resolve it.
So I think getting, again, by getting back to that, you know, action oriented value added to the the first line, I think that's a huge point. Being able to shift from assurance to advisory and help the first and second line do their job better. We're able to amplify the impact we have. It's huge. And so the more we can help this move upstream, the more impactful we are as auditors. What, what are some of the most creative uses that you've seen of, of AI by auditors?
You know, how, uh, have you seen anything where, you know, you're surprised at, uh, at what it could do Or? Well, I think there's, I think there's a lot of different things, right? So we're, you know, we're experimenting with using, using copilot for example, to, uh, to document, uh, from a walkthrough that we're having.
And then, you know, it can create your first draft of a walkthrough and then you can say like, okay, now compare the walkthrough of what the process owner just described, the, the process to me. Compare that to the process narrative or to the rackum or to the flowchart and tell me why there are differences.
And, and you know, the, the beauty of it is the speed that then you can go back to the control owner, you know, the next day and say, Hey, when we talked about this, you never, you didn't touch on this. Like, was that we just omitted it or has it changed? Right? As opposed to, you know, historically it might have taken, you know, a week or to get back to it. And then the, the control owner's like, oh, I don't remember what we talked about.
So, so, you know, some of it is, uh, as Grant said, you know, kind of takes, takes some of the, uh, kind of more of the grunt work kind of out of this, right? So that we are more focused on, on value add. Yeah. Um, but the speed thing is, is also terrific. I think that point about taking some of that, that busy monotonous workout, every auditor I know, every audit leader is trying to staff their team with the right people. And it's a struggle right now.
If you can get your people working on the right things. A they like their job more. Nobody likes right? Narratives, I don't, maybe there is, but I don't know. 'em, they never work for me, right? So take the stuff they don't like to do, automate as much of that as you can. Let them focus on things that adds more value, that is more fun for them. Our retention will get better, our ability to keep them to, and move those people into higher value roles in the organization gets better.
So it's, it's kinda that virtuous cycle is my opinion. And what do you see as like the next frontier for, uh, internal audits use of ai? Right? Where, where are we going next? Well, I just, you know, I think going back to what I was saying earlier, right? I feel like we've all been stuck, you know, doing SOCs or, or doing some of the kind of like the standard audits.
Um, you know, but I think this tool really enables us to, to speed up and again, like really get into that value added and, and to be able to use that knowledge to say like, okay, something that maybe I might've needed a specialist to help me. Like, you know, you've got much more of a, a of a jumpstart.
But I do think that whole like value added piece, um, and, and more insight being able to, to really dig into that, uh, that root, root cause analysis that we've all been talking about for a long time. I think one of the challenges auditors have is writing a real business case in a recommendation, right? Really make the sale. And, and we don't always do that very well. You can ask Gen I to write it in a tone where it will do a much better job that'll drive adoption. If, if things don't change.
We write up a finding we're of no value, let's just be honest. We're of value when the operation gets better and they change and they have less risk 'cause they've got better controls or other things in place. And I think gen I can help in that a lot and it's gonna keep getting better. Can Gene AI make recommendations at this point and uh, you know, like beyond just a high level generic, can they, you know, can, can it really generate some insightful recommendations? Yeah.
Yeah. We've seen it. We've seen it do that. Yeah. At, at a detailed control, a control level, right? Yeah. Yeah. For sure Isn't, But it's not on autopilot, right? You still want the professional who has experience to say, does this make sense? Is that right? Based on what we saw, right? Because it, it, Charles King who is mm-hmm. From KPMG who was on our group yesterday was ready, is like, it's a huge algorithm, right? It's, it's doing math.
You need to make sure that, that it's still on point, right? So I think that, again, I don't see this replacing auditors. I see this in letting auditors be a whole lot more important, a lot more impactful, Right? Yeah. Yeah. I, I, I totally agree. Right? It's like that first version, but we all know it's way easier to edit something than it is to be staring at a blank sheet of paper. Yeah. White tape are scary, Right? Pretty
Sure. Uh, are there ways to connect, uh, an AI platform to your enterprise systems so that the AI can be monitoring or analyzing the, the data? Yeah, so, and I can speak to Workiva solution, I can't speak to anybody else, but you know, our system is designed with APIs that connect other systems and bring the data in and, and because the gen AI is right native in our application, I can go right to that data and I think it's there.
So yeah, it's really there, um, understanding what data you need and getting the right connections in. If I don't have a connection and maybe it's not something I'm gonna do repetitively, I can bring it in in the form of a spreadsheet or something like that. So it's amazingly simple actually to bring the data in and connect up to it. Yeah. Which is really fun.
Yeah. So Ultimately, how do you see, uh, internal audit incorporating artificial intelligence into its work, you know, five years from now? How, how would, what would you envision, you know, in internal audit's use of AI to be? Well, it's interesting, we were talking about it, right? You know, like things are moving so quickly.
So five years, by five years from now, like, gosh, we might be on the moon, but, um, you know, but I, I do think like this, this, you know, there's been this push, like how do we, how do we use more technology and more automation in the way that we're auditing, whether it's internal audit or whether it's sox. And I think this has really given us that, you know, that low code, no code solution Mm-hmm. To be able to start doing analysis.
You know, I think, you know, like is it the, is it the perfect solution if you wanna do continuous monitoring or whatever, maybe not. But what it allows you to do is to do that, you know, like, okay, well I, I did it using gen AI and I proved that like, this is gonna be valuable and useful, so now let's go and figure out like, what is the right technology solution. So I, I just think, uh, um, yeah, it, it's got a myriad of possibilities.
Um, it's not gonna, as Grant says, not gonna replace auditors, but really allows us a, again, to be much more valuable for the first line. Um, which I think is terrific. Yeah. Yeah. I mean it's, it's what, 15 or 16 months old now? And we're in a whole different world, so I don't, I'm not that good. A prognosticator say five years. I'm not sure I know what a year from now looks like.
But, but what I see is that as people get in and use it, they're learning ai, like everything else comes with risks. And if you're not in it using it, your audit team, you're not understanding those risks well enough to really advise your company and you're not helping your company. I can guarantee you somebody in marketing or somewhere else is using it. And if you're not helping put the right governance in place around this, you're not doing your job as an auditor in my opinion.
Right? So you've gotta be well enough versed to have that conversation with the other people, to put things in place, have those right guardrails in place. And I think that's maybe the next step for us. I don't know how many, three or four steps, I dunno what that looks like, but I think right now it's get in, get familiar, use it practice and use it so that you can really sit down and have an intelligent conversation with your business. Say, look, these are risks.
Here's how we're addressing them. What are you doing? How do we do this? And really put things in place. By the way, I'll help you write the policy And then just, uh, one thing that, that sparks. So the other thing we haven't talked about, right? We're talking about using gen ai, but I think as auditors, exactly as Grant says, you have to be familiar with it because, you know, the rest of the business is going to be using it.
And so we need to look at it, we need to come at it from a controls perspective as well, right? And say like, okay, what's the governance we've got over this? Let's make sure that we're managing it. Are there any other, uh, you know, things that you know, you're most excited about in, in AI and like, you know, bringing it to your clients or incorporating it into your platform. What, what, what, what's the, uh, you know, the, the next target.
Nobody knows better about chasing down data to make sure it's complete and accurate than auditors. That's what we do every day. And so I think there's frontiers in partnering with the business and all these things in these new emerging risk areas, right? Where we don't know, but it can help us get speed. So we can again, have those really meaningful conversations with people who do know it and help them get there faster. This is about us helping the first
and second line be more impactful. That's our job. You know, I think when you look at the speed that regulation is coming at us, right? So whether it's, you know, the cyber rule, which we've still gotta deal with, right? The new climate rule, we've got, uh, the European AI rules, I mean, it just like coming at us, but, um, you know, being able to use Gen AI to, to summarize it, get me to the high points.
Some, somebody in our session, uh, the other day was talking about policies of like, you can tell it, read a policy and find all of the, all of the must, must dos, right? What, like, so then you can come up with a list so I can use a checklist. So like, again, just that ability of it to consume information and summarize it for you is just, is just awesome. It's life changing. I mean, it really is. If I'd known this, I might still be auditing instead of doing what I do now.
It's, it's making it fun. Well, thank you Grant and Sue for talking to us about artificial intelligence and, uh, it's been very informative. Terrific. Thanks for having us. Appreciate fun To be here. Thanks. If you like this podcast, please subscribe and rate us. You can subscribe wherever you get your podcasts. You can also catch each episode on YouTube or@theiia.org. That's THE iia.org.