Teaching Artificial Intelligence

This is Lab Medicine Rounds, a curate podcast for physicians, laboratory professionals and students. I'm your host, Justin Kreuter, the Bow Tie, bandit of Blood a transfusion medicine pathologist at Mayo Clinic. Today we're rounding with Dr. Bradley Erickson, director of the Mayo Clinic Artificial Intelligence Lab and professor of radiology at Mayo Clinic to talk about working with artificial intelligence and also how to train on it. Thanks for joining us today Dr. Erickson.

Thank you for inviting me to talk with you today. So why don't we kick stuff off and maybe define like the importance, why are computer aided diagnoses

artificial intelligence important for healthcare? There? There are a number of advantages to AI algorithms as long as they're appropriately implemented. In a lot of areas of medicine we humans tend to be more qualitative and that can be very good, but in other venues it's important to be more quantitative and computers are particularly good at those sorts of tasks and, and AI falls into that category.

For example, measuring the size of tumors or other disease processes is a quantitative task that we humans usually don't like to do all that much but computers are very good at doing it and they can do it very efficiently. And so this is one of those cases where you get a win-win situation where the computer does it better it does it faster, and it takes gru work off the the shoulders of, of the physician.

So I think that there are some areas where it's a natural win for AI to help us do our task better. The other thing is that for tests like detection and and making sure that we don't miss certain things having AI there watching over our shoulder can be valuable particularly when it's 3:00 AM in the morning.

And, and I know I'm not at the top of my game at that time or you know, at the end of the day after looking at hundreds of thousands of images, again we humans tend to fatigue and AI tools don't. And so having that kind of extra set of eyes looking at things saying, Hey don't forget about this, or what do you think about this? Something to make sure that, that we give as good attention to the last case as we gave to the first case I think is another value of ai.

You know, as you were talking there, in my head I'm thinking about checklists, right? And like in the operating room there's checklists

and you know, in the aviation industry checklist is I guess is there one way or maybe it is it one facet of AI that's kind of like the next generation of a, of a checklist? Well, so the, the more typical way that we implement checklists is what's called structured reporting. So when I interpret an examination there will be a number of, you know what about the such and such? What about this, what about that?

And particularly then if the computer prompts me and says here are the legal answers that also then can be nice training data for training in AI. And now there are already companies starting to show AI tools that can generate structured reports. And so you have that double advantage of the computers looking at everything and it lays it out in a nice organized fashion. So you, you mentioned in your first answer explaining why is it important talking

about, I kind of caught your highlight on the if it's appropriately implemented. And so it kinda leads me to the question of what's important for physicians to understand about working with artificial intelligence? So when I give talks about AI, I try to emphasize the point that despite its name, AI is not intelligent. The more correct term in the field is to call it machine learning or deep learning. And it's learning a pattern.

And so you could feed it whatever you want and it would figure out the pattern. And while for humans who are really good at memorizing patterns which is what a lot of medicine is about we think of that as being intelligent. And so that, that's kind of the, the origin of the term. But the, the computer is ultimately just doing a pattern matching thing. And the danger then is this.

Somebody I know actually took an x-ray of a pickle and fed that x-ray into a cancer detection algorithm and the algorithm said there's cancer there. The problem is that there's no common sense that we humans would think of when the AI runs. It is just saying this most looks like this and that's a big problem. And I think that that then kind of gets me to the next point which is we need to think about confidence levels of AI.

The current generation that we have basically says it's probably this, but it it doesn't give a lot about the actual probability value. It just tells you cancer or no cancer. And the ability to have it convey a calibrated probability as well as a confidence value I think is critical. If you think about your interactions with your physician and you walk in and they say lung cancer. How much confidence would that give you? You know, sometimes it is just about that clear.

But other times and probably most times it's more

of a differential diagnosis. And, and that's kind of where we need to get with AI is that we get that list of possibilities with some sort of indication of the confidence level and those technologies are being developed, but we're not there today. And, as they're getting developed, is there kind of a standard way that the community is thinking about talking about this confidence and probability? Is that going to be kind of universal for the different tools that are kind of being developed?

Or is it each, each to their own the way they kind of try to convey that? So we're still early enough on the development that each is kind of doing it their own way, you know and until we have a more clear winner I'm not sure that people are gonna put too much effort into standardizing that.

In some of the structured reporting technologies there are fields for putting in a confidence value but the precise way to interpret that is still not to defined, you know, that's actually a big problem when you think about our language today. If I read out a chest x-ray and I say that's probably pneumonia does that mean I'm 99% sure 90% sure 56% sure? Right? What does probability mean in a quantitative sense? And that's a big challenge then in in terms of creating training data, right?

How do we train the algorithm that this is what a 56% probability means but also then how do you map a number back to language that that we would understand as humans? So, that's a big challenge that that we have today is that language and humans are not quantitative the way that algorithms are and and thinking about what probabilities and confidence terms mean is a challenge. Yeah, I, as I hear you say that, I think in pathology, you mentioned the radiology challenge

when you say this is probably. You know

in pathology there's certain aspects of our practice where we're talking about something is suspicious for something is atypical something you know cannot rule out. And and I guess I'm sort of reflecting now that we try to convey that probability exactly like you said actually as a, as more of a subjective rather than you know, qualitative rather than a quantitative way. So is the thought then that there's that'll help us get away from some of the, the biases in in our clinical practices?

Yeah, you know, so bias has several different components. You know, I have a, an electronics kind of background and you know, we always tend to think of bias as a bad thing, but of course for those of you who know electronics bias is what makes transistors work, right? So bias if properly used, can be a good thing. How does that apply to AI? Well, in terms of bias and particularly, you know

underrepresented populations and so on we know that some races genders and so on have different risk profiles. And so to say I'm going to be completely blind to race or sex is probably not the right approach. You just need to make sure that you use that information to provide the best care for patients.

And so as we then start to again produce these probability estimates you know, that information is hopefully going to improve the confidence intervals be because we have that additional information about the the sex and race of the individual. So in your role as a director of our artificial intelligence lab, how do you go about or how are you thinking about how we train you know, our trainees, our residents and fellows how to use artificial intelligence well?

I imagine that's starting to kind of enter into your life and what can you share with our audience? So I try to make the points that that we've already discussed about the fact that it's not intelligent, it's just doing patterning and that as long as you give it an input, today's generation of AI tools always produce an output, even if it's nonsense. And so, you know, I think it's critical that our trainees need to get at least some exposure to AI technology to understand how it works.

And it of course, more importantly how it fails. And you know, I draw a lot of parallels with statistics that, you know even back in the dark ages when I went to medical school we had to take statistics and epidemiology. And I think that that's a valuable thing, right? You have to understand how to read the literature

but also when you're looking at, you know a BMI that's, you know, at this value, well how far off of the population norm is that and what does it mean? And I think there needs to be at least as much time spent on training about AI tools in medical school and residency and and so on, so that they understand again, the principles of machine learning, how it works, how it fails be because it's probably going to have even more application than medical care than than statistics and epidemiology.

Is, is there a good, yeah this is a bit of an ignorant question in that I, I'm not sure, like, you know, if if I wanted to get my residents and fellows exposed to AI now, like I'm not sure if there's a going into old school things thinking is there is there kind of the the recommended textbook on it or is there something, you know in our current practice now where I could have somebody go deliberately kind of practice with or are is there some online tools that, or a place

that somebody can go a digital playground and get exposure to and, and come to appreciate these points you're highlighting for us? Yeah, so as, as you kind of suspected, you know textbooks are pretty much useless. They get out of date so fast. Things like chat GBT, you know, didn't didn't exist at least in the knowledge of the population three, four months ago, right? So unfortunately textbooks probably don't cut it.

So to address this problem there's a guy named Jeremy Howard who has built a number of what are called Jupiter Notebooks. It's a way that you can execute code but it gets the name Notebook because it's like a scientific notebook where you also see the output and you can put in hypertext markup like a webpage. And so he actually wrote a textbook that is all code and and these Jupiter Notebooks. So stealing his idea, my lab and I have created a website focused on medical image deep learning.

So if people are interested, that's at midel.org. And that's something that, you know because it's web content it's a lot easier to keep up to date. We can add a new page when some new technology comes along if there is a bug, you know unlike a textbook where you have to publish errata,

we can, you know, update the code pretty easily. But I think the ability to actually see the code run and people say, gee, I wonder what happens if I do this? And they change a bit of the code and they can see the impact I think is extremely valuable for, you know, early to mid-level learning. There are courses and in fact Mayo offers a master's in AI for medical people and that gives you a more in depth learning experience but obviously requires a, a bigger commitment.

So there are a number of options, but you know I think web resources probably is the way to go. YouTube is fantastic, the challenges that most of YouTube content is not specific to medical but in terms of learning the general concepts of AI YouTube is kind of my go-to. It's, it's wonderful to hear all these resources. I'm just kind of curious have you started to have program directors approach you in your laboratory to facilitate education around AI in their department division?

Yes, and I have, you know, gone and done the typical visiting professor thing to do that. But in addition, I'm part of an informatics society that has created what's called NIIC-RAD the National Imaging Informatics Curriculum for Radiology. And that covers a lot more than just AI.

It talks about, you know, how do you move images around, how do you do structured reports and whatnot, but we've added AI content to that. And so that is a week long webinar that is available to all radiology programs. It's actually now across the world not just the US and so because it's really not feasible for many of the smaller programs to have an expert on AI. And so this is a way that we can educate, you know essentially radiology programs around the world on AI as it applies to radiology.

And there are discussions with other societies like pathology about doing a similar thing. Yeah, that's wonderful. So my, I've heard other colleagues kind of talk a about the future of medicine is is being handed over to the robots and what's, you know the role of the physician is really to still have maybe that healing touch or comfort. But as I hear you talk and and really talk about how best to use ai I would gather that's not the future vision that that you see.

So what do you think the future of of AI in medical practice looks like? So I think that, for instance AI doing more of the quantitative tasks and doing some of the grunt work that we physicians don't like to do is the sweet spot. We focus too much on doing the sexy, it can make a diagnosis that a human can't do and it's cool when that works but I think the payback for that is relatively small compared to the investment.

But I think, you know, those sorts of tools are coming we and others have published on the ability to protect molecular markers from standard CT's and MR's. That there's no way a human can see what the AI is seeing. I think that the routine quantitative measurements of things like body composition, the amount of visceral fat, subcutaneous fat, and muscle is valuable to many clinicians today. And having a human trace that out is simply not practical.

But we've already deployed an algorithm so actually every abdomen CT done at Mayo has a body composition available to it. They don't routinely report it, but it's available if if people want to see it. I think that the generative technologies is kind of led by Chat GPT is also going to change medicine. Now Chat GPT we all know about the hallucinations where it will make up really plausible sounding things that is complete garbage.

But there are variants that don't do that where that there's what's called the temperature which is how much you weight the probability and how much you want to right weight randomness. But also it can say and this is the document where I got this idea from.

And so I think for summarization, you know, going through the 30,000 pages of outside records, great task for some of these generative technologies where you say give me a one page summary of all the hematologic disease of this patient in their life. I think it's very feasible in the not too distant future. And if we can do that with text, there are also some great generative technologies for images.

And, and this is one where my lab has done some work where we can take for instance a large collection of hip x-rays or chest x-rays and if we also know the sex and the age and the race and the BMI we can train a model where you can then say generate 10,000 x-rays of the pelvis with this many from this age range, this many, this age range this many, this sex, this many the other sex, you know this many with a certain BMI, this many a certain race.

And so we can completely sample the population and none of the x-rays will be from any one individual which then gets around privacy concerns. So I think those sorts of generative techniques to improve the training of other AI algorithms is really cool thing that that we're starting to understand better. Finally, I think there are components of AI that will go back to the roots.

And so for those of you who are historians the first real AI application in medicine was called MYCIN and was a set of about 500 rules for making the diagnosis of blood infections.

And you could ask it, well what's the best antibiotic to use for this type of infection? And, and that ability to control things as opposed to being susceptible to hallucinations is a real problem. And so if you can define a set of rules and say, when you see this do this and then do this and then do this. We have a lot of workflow challenges in healthcare where handoffs are dropped or certain steps aren't done in time or they're not done according to the care process that we all agree on.

And I think that that form of workflow something called process automation that's used in manufacturing of cars it's used in the financial industry but it's not used in healthcare for some reason. And I think that that's actually a form of AI that probably is going to start to be applied in the next five years or so. Wow. I think all of our audience right now really keyed into your, your predictions for the future.

Because many of us are doing chart reviews in preparation for frozen sections the next day. And, and also there's lots of times where I have to comb through a lot of data. I hadn't even thought about that. And then the, the medical educator in me is just thrilled at the, you know what might be possible with almost essentially, you know I know what, what I would love to have and expose my learners to, but it's going and grabbing that materials that that takes so much time.

We've been rounding with Dr. Erickson on the importance of working with artificial intelligence and how to teach it. And thank you so much Dr. Erickson for taking the time with us today. It's been my pleasure. It's been great to talk with you all and I hope that this was valuable to your audience. And to our listeners, thank you for joining us today. We invite you to share your thoughts and suggestions via email. Please direct any suggestions to [email protected].

If you have enjoyed this podcast please follow or subscribe. And until our next rounds together we encourage you to continue to connect lab medicine and the clinical practice through insightful conversations.