The Institute of Internal Auditors presents all things internal audit tech. In this episode, Charles King talks with Debbie Liu about the transformative role of artificial intelligence In internal auditing, they discuss the integration of AI tools like co-pilots, the importance of prompt writing and how AI is enhancing audit processes. The conversation also covers training strategies, real world applications, and the impact of AI on stakeholder engagements.
Well, Debbie Liu, welcome to the All Things Internal Audit podcast. We're so happy to have you here today. Yeah, thank you Charles, for inviting me. Let's tell the audience a little bit about your team at Kaiser Permanente. Talk to us a little bit about how big is the team? What are some of the big focus areas that your team's working on right now? Mm-hmm. So I have a team of 82 people, and we're divided by portfolios of risk.
As you know, Kaiser is an integrated care model in the, in healthcare. So we have one portfolio that's dedicated towards, um, our national health plan, the payer side, finance compliance, and then we have another team that is dedicated to the provider side, the care delivery side and pharmacy. Uh, we also have a team that's dedicated to IT and enterprise shared services. And then we have a team that's dedicated things like HR community benefit because we are a nonprofit.
We do have a tech enablement team. It's a small team. Um, you know, I started that team and it's doing really well. We're just in that maturity development right now. But it's done a great job for being a small team. I love that. Well, you know, as a, as an AI leader and AI evangelist, I probably wanna hear more about that tech enablement team than anything else. We, we met through an AI conversation. So let's talk about that a little bit. At, at Kaiser.
How do you think about using AI and really all technology analytics in your internal audit team? What, what does that look like for you? And what are some of the big North Star goals that you're trying to achieve? So we are a continuous improvement organization. So technology is very key to continuous improvement. We were reviewing the use of gen AI for the past year, and we joined a early adopters program last summer for 365 co-pilot.
There was about four of us, and I was one of the members of that early adopter team. And then when we rolled it out, we successfully integrated co-pilot into all phases of our audit, plus our quality assurance and quality improvement program. So we went through the entire eco uh, system. The one thing that we've learned is prompt writing is an essential skill at being successful at Gen ai. Right? So what's a prompt?
A prompt is instructions cues for directing gen ai like copilot to produce relevant information for you that you need. So what I did first in the journey of gen AI was train the entire department on prompt writing, right? So prompt writing is, uh, not a unique skill to internal audit. It's not a unique skill to any organization or any industry. It's an essential skill that you'll need for the future.
I would say for the future generation of auditors, this is a standard skill that you need to have. As a matter of fact, I've thought about putting it into my, uh, job postings as a preferred requirement, effective prompt writing skills. Additionally, um, prompt writing is crucial for guiding gen AI to produce high quality, relevant, and accurate responses to meet your needs. So that's why I went through that journey of training the department.
We also have a prompt and co-pilot guidance manual. Oh, Wow. That's interesting. Tell me about that. So, um, it's just, um, a guidance ma manual on how to write prompts like I'm an experienced auditor, you know, it has all the pieces. And who's your audience you're writing to? Uh, we also have a library of prompts. We have a crowdsourcing platform. It's called Prompt Buddy.
Right? So we have prompts where each phase of the audit cycle, because why reinvent the prompt if it's a standard prompt for risk assessment or to develop your risk and control matrix so people don't have to work extra hard. Uh, we actually have an AI landing page in internal audit, which has all things ai. So if you want training, there are PowerPoint decks there. Uh, if you want guidance, there's the manual. Uh, we are also in the process of developing agents.
So we started to develop, we've completed developing an agent for writing an effective audit finding, you know, with the components of, uh, issue criteria cause impact recommendation. So you just run this agent, which is full of prompts, right? And it makes it easier. And each, each prompt for each category has several questions or prompts to make that issue much clearer.
Excellent. I love that. You know, one of the things that you learned really early on with prompt engineering is that it's very difficult to write a perfect prompt that's gonna get a perfect output. And some of the best prompts that I've seen in my clients are prompts that start a dialogue with the user and, and exactly what you're saying, they ask questions, the user responds, and then when the AI has everything it needs from the user, then it can produce its output.
And it's a very different approach to prompt engineering than trying to put everything in one initial prompt. It sounds like you've done a lot for communication and training. How have you seen adoption? Because it's, it can be hard to get people to change the way they work and use these tools and what have you, what have you found to be effective and, and maybe what's not effective if you tried some things that didn't work? Well, I really believe as a leader, you walk the talk.
So as I mentioned earlier, I was in that early adoption program so I can learn about it and see the capabilities and the possibilities and how I can dream using it. And then when we did the training, I was the person that was actually demoing the scenarios. So they can say, oh, if Debbie can learn it, I can learn it. And also I use it daily. So when I review audit reports, everyone knows I'm using co-pilot.
And sometimes some of my review notes is, use co-pilot for this, or Did you use copilot for that? So everyone is adopting because they see their leaders using it, and my leaders are using it. We also have every Friday office hours. So we have a half hour every Friday in which you can come to the office hours, bring your questions about, say, I tried to write this prompt, I'm having problems. Or you wrote a fantastic prompt and you wanna share the rest of the department.
So the office hours is an opportunity to not only solve problems, but to upskill you. Right? That's where we teach about, like the last office, um, session was about writing agents, right?
And, uh, when I saw you in your presentation talk about applets, it gave me an idea of, you know what, I'd love to write an applet where, you know, in the report writing cycle, I can send that applet to our business user, our business executive to guide them in writing a management response or corrective action plan. Like step by step, what does a good corrective action plan look like? And does it address all the risks in the finding? And that will make it fun and easier for that, right?
I I'm thinking of the user experience. So when you think of use cases for your prompts, don't only think about your ecosystem within internal audit, you know, what makes your audit process effective, but where are your touch points? Like when you have new people joining internal audit, I'm thinking about using it for onboarding, for education, for wherever we touch our clients. So looking at that and analyzing it, where can we use Gen I? Yeah, I think that's such a great idea.
I mean, not just, not just walking the talk. I mean that is, that is so important I think to drive adoption group, whether it's AI or anything else. But I think this idea of thinking about, hey ai, how you can not just make your team's experience better, but make the other stakeholders, your auditees, your audit clients more engaged in using ai, and that's gonna make, ultimately that's gonna make your management action plans better.
That's gonna reduce the amount of iteration and back and forth, and it makes you look so tech forward, right? You, you use internal audit as this kind of beacon of innovation and you can show others what some of the possibilities are. I absolutely love it. I've also shared with my audit committee chair the use how we're using AI in internal audit. He had me demo how we're using AI internal audit to the audit committee who shared that with the board.
And it cascaded through the executives at Kaiser. And I've been doing a road show demoing how we use gen AI in internal audit with various executives at Kaiser and encouraging them that it's not that difficult. Prompt writing is like learning to write a language and the language is English, really. Right? Yeah. There are some nuances to write effective prompts, but it's not difficult. It's just a matter of doing it every day, practicing, and you'll get better and better at it.
And it learns your voice too. So when I write emails, it doesn't sound as artificial. It sounding more and more like me. Yes. Like you. Yeah. That's so great. And you know, we talk in internal audit so often about how to get a seat at the table and we think about how to become relevant. And this is such a great example of you leading the organization from a technology standpoint and really showing everyone else at the executive level and beyond how AI can change the way you operate.
And I have to think that's an inspiration for, for others in the organization. Yeah. Yeah. Well, Debbie, this has been a great conversation. I'm so glad we got a chance to talk today. And thank you again for coming on all things internal audit. Yeah. And thank you for letting me share and hopefully I'm encouraging others to just same. Hey, audit pros ready to supercharge your skills and connect with the best in the field.
You absolutely need to check out the I'S 2025 International Conference happening July 14th through 16th in Toronto. This is your chance to learn about emerging risks, cutting edge tech, and global best practices that will elevate your internal audit game. Don't get left behind. Register now@iiaic.org. If you like this podcast, please subscribe and rate us. You can subscribe wherever you get your podcast. You can also catch other episodes on YouTube or at the iia org. That's THEI a.org.