The Institute of Internal Auditors presents all things internal audit tech. In this episode, George Barum talks with Mike Kino about how agile methodologies and artificial intelligence are transforming internal audit functions. Kino shares practical insights on implementing agile auditing, the challenges and benefits of this approach, and how AI is being used throughout the internal audit lifecycle. Mike, thanks for being with us today. Thank you guys for having me. Very excited to be here.
So, uh, but I guess before we get started, could you maybe give us what you would, uh, consider a definition of agile auditing? What I think of it, I think of, uh, efficiency and streamlining processes, but you, you tell me. How would you define it? So I would define agile as allowing you to be dynamic in how you're conducting your audit lifecycle. Um, being able to conduct procedures in more of a, more of an iterative way that allows you to be flexible as certain risks may, may come up.
Whereas typically when you think of auditing, you would have very structured, sort of gated things you need to do. You need a plan, then you need to go through your, execute your execution phase, then you need to go through your reporting phase and you need to go through your monitoring follow-up. When I think of Agile, I think of doing all of those sort of in time box components that focus on one piece of scope for a given audit or for a given body of work. Okay, Mike, next question.
Um, how are you using the agile mindset in your internal audit function? So, from an internal audit standpoint, when we think of planning and and Agile, what we do is we prepare our annual audit plan and we figure out all the work that we're gonna, that we're gonna be doing in a given year. That work is then time boxed into what we would call an epic. An epic is equivalent to an audit. So in a given year, we would have the amount of audits that would be called epics.
Each of those epics would be, would have sprints allocated to them. And within those sprints, you are covering a functional scope area. If we think about, um, doing a, um, an a procure to pay audit, for example, in a procure to pay audit in one of the sprints, we may cover the business process side of procure to pay in another, um, sprint. We may cover the TE side of procure to pay.
But in each one of those sprints, in each one of those two week increments, we're going to plan, we're gonna have our field work, we're gonna have review, we're gonna have reporting, we're gonna have feedback, we're gonna confirm observations, and then we're going to continuously iterate. And so we're going to have that continuous feedback loop with the end user until it's done. And then once we're done with one component of a sprint, we're gonna move on to the next scoping area.
So typically where an audit would take three months to, to complete, by the end of three months, your report would be fully written, all of the sprints would be completed in each sprint. You would have all components of the audit lifecycle completed, and then at the end you would have stakeholders and auditees who are in fully alignment with what the output is and, and less friction and resistance.
Mike, could you give us an example of the planning and scoping phase, um, some of the critical elements that you need to consider? I do think it's important as you embark on your agile journey to define the definition of success. And to me, we look at this in, in a few different areas. Uh, number one, we look at sprint progress. So in a two week period, how much did we get done relative to plan?
We look at sprint velocity, how much can we get done or what, what do we plan to get done in two week period? We look at the quality of work. You know, when you think of a two week time period, it feels very condensed, but we don't wanna sacrifice quality. So that's another thing we assess. We look at stakeholder satisfaction. How satisfied are our stakeholders to go through their all the audit life cycle components in a two week period?
It may feel that we're pressed for time, but what we're trying to do is get through everything in an efficient way and give them a given finding, if you will, or give them, um, a certain observation that they could help address more real time. And then the last area of continuous success is continuous improvement, making sure you're reflecting on what worked well and what didn't, and incorporating lessons learned and better ways of working into the next phase of, of your, your sprint.
Okay. Alright, thanks for that explanation. How would you say agile auditing has changed over the past few years? I know we have AI and we have different aspects from a technology standpoint, but maybe if you said, uh, looking at it maybe 2020 versus 2025, what would you highlight are, are some of the major differences? I think from 2020 to 2025, the major differences definitely relate to technology and the advancement of technology.
So if you go back to when we started our journey, um, which is in between that 2020 and 2025, yeah, it was around 2022. What Agile looked for us in 2022 was having a Kanban board using project management tools to figure out what the year is gonna look like in terms of our audit plan, and then time boxing each audit into sprints and figuring out how we're going to sort of allocate resources and that work.
And also having a Kanban board where we could sort of know what's not started, what's in progress, um, what is complete, what is in the backlog. So Agile in 2022 was very project management oriented. We had some data and some indicators in terms of how work was going and how we could reallocate. And then I think as the years progressed, the big thing that changed was technology and the data we had.
So now there's a lot more, um, data involved in how we manage our day-to-day and, and the use of agile. When we are doing continuous monitoring and we see there are certain transactions being flagged, we may have to adjust our audit plan. Working in an agile environment allows us to shift priorities on demand and allows us to really focus on the highest rated risk areas.
And so the biggest difference is, biggest difference is still having a project management discipline, plus listening to the data and then utilizing the skills and program that you've developed to help address the, the highest, uh, rated risk.
Okay. Uh, in, in terms of an organization or an internal audit function that's considering, uh, going from a traditional way of auditing, moving to Agile, what would you say are maybe some readiness type things or, or what you would do to get prepared for that? Maybe some of the challenges that, that a, uh, internal audit function may face in terms of readiness? I think it starts with education. Really talking to the team about how you traditionally would audit, right?
And going through your, your various phases to really thinking agile and talking through what the benefits are, uh, are in, are in that sense. So when we started our journey, I explained to the team, Hey, this is gonna feel a little bit unconventional, but you're gonna reap a few different benefits. One of the benefits is you're gonna own to get all the phases done of an audit in one phase, and the findings and everything that, that are gonna come out of the audit are not gonna be outdated.
So you're not gonna do an audit for six months and then give all your results at the end. In the first two months, you're gonna deliver results, you're gonna have those agreed upon, and your audit is gonna be sort of completed in increments over time. And that's more beneficial to the business because they could be agile in their remediation, they hear of a finding sort of real time, and then they could go ahead and address it.
So really educating and getting the buy-in from my team that this is a step in the right direction, right? The regulatory landscape is very dynamic. Business risk is very dynamic. We need to be able to operate in line with the business raises a software company. The company operates under Agile or Scrum methodologies. So we cannot operate in sort of an elongated audit lifecycle if we wanted to really address risk and mitigate things, um, on a day-to-day.
Yeah, it sounds like, uh, responsiveness is, is a big advantage, just being, uh, able to adapt and respond to the business, respond to management, right. So from a roles and responsibility standpoint, uh, I know there are Scrum masters and different folks. Uh, would you say that that is, uh, something to identify the folks are gonna be involved in certain roles in the beginning?
Is that just kind of evolve over time or, uh, could you just talk a little bit about, you know, how important it's to, to go ahead and identify folks who are gonna be in those, those key roles? Yeah, it, it's definitely important and it's also important to explain sort of what those roles entail, right? So your scrum master is almost your, you, your lieutenant, right? Your lead sort of project manager.
They're gonna keep everyone accountable, make sure that the work is progressing, make sure your CanBan board is up to date, getting the relevant updates from the other team members. We do daily standups. The expectation when we get to the daily standups is the scrum master would've spoken to all the team members, make sure the board is reflected in terms of what's not sorted, what needs going backlog.
So when we're having the meeting, it's very intentional and everyone understands what the priorities are. And if the priorities shift, it's very clear when we're communicating and we don't need to spend 45 minutes doing a rundown of what everyone did, we can move on to the next day and say, yep, the expectation for the next day is that we get here.
And so we'll do, um, daily standups, we'll do, um, every two weeks we'll do some sort of retrospective to see what do we plan to accomplish versus what did we accomplish? And then see where some of the, the blockers were, maybe where there are certain inefficiencies and we look, um, and lean on our agile framework to continuously improve.
So, Mike, from a, uh, success standpoint, or maybe from a key performance indicator standpoint, what are some of the key things that you would highlight on how we're doing along that journey and, uh, measuring our success with Agile auditing? So when, when we're conducting an Agile audit, it's, it's extremely important for us to work with the end user and document the user acceptance criteria. What does this mean in practical terms?
It means we're documenting the requirements that must be met for a story or, or work item to be completed in the audit. So understanding what the business risk is, making sure that that's incorporated into the audit, in addition to having your standard audit procedures that you're going through, and Agile is meant to be collaborative, and you're working with the end user to making sure that everything is taken into account as you iterate through the process.
Okay. Um, from a training standpoint, um, just to try to get your internal audit function up to speed and make sure that, um, you know, maybe on an annual basis that, that you're improving and working towards, uh, being more efficient, would you say that, uh, that you need to go to training or is it more on the job training, whereas you go through different projects and different audits that you kinda learn and, and evolve from there? So I would say you need one person that's trained.
Um, I joined Brazen 2021 April of 2021 as the first internal audit hire to really build out the function. Later that year in 2021, I attended an awesome training in Florida hosted by the I a about agile auditing. So I learned before I had my team learn, and that's something that I hold very near and dear to the heart.
So I'd say you need at least one person who understands what agile auditing is, goes through, I'd say at least three to five days of very hands-on tactical training to see what's gonna work for your, your business needs, and then be able to train other people. I don't think it requires everyone on the team, but you need one person who could then retrain and get everyone into that, into that methodology of updating a CanBan board, right?
Working on sprints, understanding what the expectations are and when you need to finish things and, and how you, um, iteratively work through a larger body of work. Okay. So let's, uh, shift gears a little bit. Let's talk about artificial intelligence. Could you maybe, uh, give a couple examples of things have, uh, maybe over the past year or so that you've seen, uh, be implemented with the Agile, uh, approach?
Uh, maybe talk about, uh, you know, some examples, maybe some challenges that folks have had implementing it. Maybe just share some stories. Yeah. So artificial intelligence has definitely expedited the way that we could carry out and conduct our agile way of auditing some real use cases and, and practical ways of thinking about it.
When you think of planning, right, and when you're under such a small time box component of two weeks for planning, the best thing you could have at your discretion is technology to help you sort of expedite the planning process. So pre agile days and pre AI days, you would spend a lot of time on scoping, having my planning memos, getting all the relevant information in there. Now we leverage AI to help us scope, help us write the planning memo.
Things that took multiple hours or multiple days, but now happening in seconds. And now we just need to do a quality level of review on it to make sure it's tailored towards what we're doing. That's planning, right? Then we think of, we move to your field work phase. Again, you're pressed on time, you have sort of two weeks to cover a large amount of scope. Well, what's the simplest thing to use AI for?
Record a meeting, transcribe notes, and have sort of a prerequisite for what your work paper's gonna look like. And again, in the past, you would have a human who would've to take very detailed notes. You then have to get into a room, did we capture all the relevant information? Does everyone agree? Well, now we have a pretty accurate, and I say pretty accurate, right? Because with, with that ca caveat with ai, we have a pretty accurate representation of the meeting output.
We could then summarize from there what your attributes that you're testing or what are, what are the key takeaways that you need for your work paper? And then you've, you've again saved a lot of time. So it's definitely helped expedite in in that sense. When you think of data, data's important in any audit that you're doing. We use AI to, um, analyze tabular data. So very structured data.
Um, if we're doing data ana analytics as part of an audit, we may write the SQL queries and scripts, but we may use AI to interpret the results. If we're looking at documentation, um, PDFs or unstructured data, we may use AI to help us expedite the review and give us some of the outputs or if we're looking for a specific consideration in a report. So we try to look at it and correlate AI to every sort of phase of the audit lifecycle and where we can get to it.
So then moving on from execution to reporting. Reporting's great, right? You, you sort of have AI help you stage your work papers, and then for a report you may take a first pass, say, Hey, can you rewrite this section? So almost looking at it as a sort of assistant that could help you expedite each part of the process, but you never wanna sacrifice quality.
So with the use of ai, we make sure we have diligent review checkpoints to make sure that what we're putting down is sound and makes sense and it's aligned with what the stakeholders have have communicated to us. And then lastly, monitoring and follow up. So we use AI to maybe draft a remediation plan. We may get a written up response in terms of where a, where an auditee is in their, um, remediation journey. And we may revise it to make it a little bit more cleaner, more concise and so forth.
So we've used it at every part of the audit lifecycle and it's definitely helped allow us to save time and operate in sort of the, the two week sort of, uh, time box. Okay, great, great examples. Would you say there are any misconceptions with, uh, with agile auditing, someone who's maybe not done it before or maybe just has, uh, limited, uh, exposure to it, what would you say is, is maybe something that, uh, that's out there? That's, uh, a misconception?
A misconception about agile auditing is that it's one size fits all and it's absolutely not. And you need to understand the principles of agile and align it with what your intended output is, right?
Whether you do daily standups or whether you do standups a few times a week, you still are going with the spirit of communicating, constantly understanding when things are changing internally with the business, external factors, um, whether you're doing your retrospectives every week versus two weeks. So the timing, I think there's a lot of flexibility and variability in what you're doing. The documentation, there are very, um, structured documents that the Agile manifesto states, right?
You have artifacts and the, and a few different documents. My team does not align with those documents one to one. We may call them totally different things, but we follow some of the principles to make sure that we are operating in an agile manner and we're not a hundred percent compliant with it. And I, I think anyone who tells you they're a hundred percent agile is kind of fooling themselves.
Okay. So last question, Mike. Uh, maybe this is putting you on the spot a little bit, but what do you anticipate the next few years hold for agile auditing? And maybe if you can, uh, you know, work in any, any angles from an AI standpoint, how do you see this evolving? What do you think are some of the, gonna be the, the key, um, the items that you, that you look out for in the future? So I think you're gonna see a lot more adoption of AI and have that integrated with agile.
I think there may be ai, scrum, masters, right? Who keep your team accountable, where you had a human who was sort of monitoring the progress of work and making sure everything was on track. You probably have more aid from an AI scrum master to do that. I think you're gonna see a lot more in the data analytics space. Um, you're gonna see a lot more agentic workflows to help sort of streamline the way we audit.
Um, so I think those are just a few examples of how you're gonna see AI be more inserted. Um, within, within Agile. Mike, you mentioned, uh, the possibility of ai, uh, replacing the Scrum master, uh, role. Could you take a little, uh, a little bit deeper dive into that? Maybe give some examples and uh, how that could happen? So, you know, I have a vision that there could be the use of ai, you know, to replace a scrum master.
What this, um, AI agent would do would be to skin or sift through your Kanban board, right? Understand where there may be, um, too much in one of the various buckets, whether it be in progress, you know, not started and automatically figure out a way to reallocate story cards to end users, to free up allocation and to get more work done in an efficient manner.
'cause I think right now when you think of Agile, it's really at the discretion of the Scrum master communicating with each member of the team on the amount of work they could take if things are moving along.
And when you think of AI and having AI sort of expedite this process, it could analyze a lot more of this information a lot faster and figure out if you are time boxing something, a story card, and you're saying it's gonna take four hours and it's been sitting on your Kanban board in progress for multiple weeks, then you as a human have looked at it and sort of mis sized it, but perhaps the AI can say, Hey, over a few days, this still wasn't done. You've time boxed this to be four hours.
Is this getting done in this sprint or should it get allocated to the next sprint? So I think in terms of like ai, scrum, masters, there's an opportunity for them to rationalize your Kanban board, give summary outputs of what we believe we can achieve in a two week sprint when we're going through planning, and then work with us as humans, help tell the story and help drive greater efficiencies as we progress through the year. Okay. Good deal. Well, it was great talking to you today.
Thanks for sharing your insights, Mike. Thank you. Thank you guys for having me. Really appreciate the conversation. Hey, audit pros ready to supercharge your skills and connect with the best in the field. You absolutely need to check out the I'S 2025 International Conference happening July 14th through the 16th in Toronto. And virtually this is your chance to dive into emerging risks, cutting edge tech and global best practices that will elevate your internal audit game.
Don't get left behind and register now@theia.org. If you like this podcast, please subscribe and rate us. You can subscribe wherever you get your podcasts. You can also catch other episodes on YouTube or@theia.org. That's THE iia.org R.