So, really a lot of my neural network is right place, right time, and many, many sponsors who have lit the light, you know, ahead of me. And some of those tended to be these physicians who were very interested in informatics. I didn't even know that that was what it was called then. And so, I ended up learning how to work on an open-source medical record system, ended up deploying it in many places around the world, virtually, some in person.
And so that's how I got into sort of this world of computers in medicine. Hi, welcome to a new episode of NEJM AI Grand Rounds. I'm Raj Munrai, and I'm here with my co-host, Andy Beam. We're really excited to bring you our conversation with Professor Judy Gichoya. Judy is an interventional radiologist and AI researcher at Emory University, where she's also an associate professor in the Department of Radiology.
Andy, I think this conversation was special in how it repeatedly reflected Judy's unique approach to solving problems that truly comes both from her deep clinical expertise and her computational skills. I completely agree, Raj. Judy is really one of a kind. In addition to being a world class machine learning researcher, she's also a practicing interventional radiologist, a truly rare combination. She said something during our conversation that really stuck with me.
One benefit of AI that folks often point to is that AI will take over the quote unquote easy cases, and doctors can spend more time on the difficult ones that might trip up the AI system. However, Judy astutely pointed out that this would actually be kind of a terrible job for the doctors. It would probably be pretty stressful to only look at edge cases that have a lot of ambiguity. It would, in effect, amplify the most stressful parts of the jobs for doctors.
This really challenged a core assumption I had about the integration of AI in the near term, and it really speaks to how deeply thoughtful Judy is and how grateful I am that there are MDs like her at the forefront of AI in medicine. The NEJM AI Grand Rounds podcast is sponsored by Microsoft, Viz AI, and Lyric. We thank them for their support. And with that, we bring you our conversation with Dr. Judy Gichoya on AI Grand Rounds. Judy, welcome.
So, this is a question we love to get started with: could you tell us about the training procedure for your own neural network? How did you get interested in AI and what data and experiences led you to where you are today? So, I actually did not set out to do or work in AI, just the same way I didn't end up, even if right now I work a lot in BIOS, I did not set out to do that. And my journey really around computers is this passion of understanding how technology is used for health care delivery.
And so, during my medical school, which was in the western part of Kenya, I was very excited to get to my clinical years. And this is like the fourth year of medical school to the fifth-year transition. And You know, it's almost like having your child move from writing with a pencil to a pen and you're looking forward to it. And I got there and, wow, I was really disillusioned.
It felt to me like busy work and, well, whatever you were trying to look for in terms of results, uh, just was impossible to find. So, really a lot of my neural network is right place, right time. And many, many sponsors who've lit the light, you know, ahead of me. And some of those tended to be these physicians who are very interested in informatics. I didn't even know that that was what it was called then.
And so, I ended up learning how to work on an open-source medical record system, ended up deploying it, uh, in many places around the world, virtually, some in-person. And so, that's how I got into sort of this world of computer, computers in medicine. And so, during my residency, when Geoff Hinton said, stop training radiologists, we saw a drop in the number of medical students who were getting into the radiology residency.
And so, I started this journal club through the American College of Radiology, just to encourage people to at least have dialogue around this. So, it's really an accident, and I ended up in AI, but there's no P.h.D., no formal training in this space. Could I just follow, so we like to get beyond the biosketch a little bit on the show. So, how did you get interested in medicine in the first place? Like, what was it about that that, uh, led you to a career in medicine to get started with?
Actually, another accident. I was not interested in medicine. I was very interested in computer science.
After my parents told me that I could not be a pilot and so, I didn't have anyone who would support my dream to be a pilot and so, in Kenya when you do well in school you're sort of expected to go to specific professions and I didn't make up my mind you don't really go to university immediately you have like this 18-month period and I didn't make up my mind till one last day in the hospital near my village where my mother worked then, there's this doctor called Dr.
Surmat, who had this very sick patient and was driving all over the country looking for blood. And he told me that I would have a good time in more university because of one thing, that the system of learning was problem based. It would be very self-directed. And so, you'd have agency to do a lot of things. As you know, Andrew, I like a lot of that, that type of freedom. And so, I said, okay, I'll try it. And it worked out okay. I did well.
And now I'm an interventional radiologist, and I truly do enjoy living in this world of computers and medicine. Judy, what drew you to radiology specifically? Because it's very techie, when I graduated from Moi, I went to work in two hospitals in Kenya and, but I still kept doing my informatics work, really training, doing capacity building and implementing medical records system to just start what would be the first data collection.
And so, at that point, I got an opportunity to come to Indiana University at the Regenstrief Institute to train in health informatics. And then I went back, and I'll do a lot of clinical decision support. And so, in that process, two things happened: one, during my training at Indiana, what I realized, there was one day we discussed a paper that said the decision
maker in a ward round is the attending. I felt really disillusioned because I thought that you should never write such a paper and if you are already in the space where you understood what medicine was that that's like a common sense type of question. So, I figured that I would like to be the type of practitioner who lives in medicine and not be too far removed.
Now, I do know that one day I'm going to walk away from medicine, but I felt that my ideas were better for just having, being in the health care space, seeing patients and just being there. So, the second thing was the CT scanner that's changed actually, but in near my village was pretty old and you'd have to drive to Nairobi. I had a relative that we had to drive to Nairobi, which is two hours away, come back home, and then wait for two weeks to get a result.
So, you can imagine in the context of head trauma or just head injury, that's really difficult. And so, what I thought was that with my superpower of informatics that I could probably get to a point where I would help with teleradiology, just, so those were my ambitious goals. I haven't done all those things, but I thought that radiology would be a good fit because I saw the gap, and I was very drawn to the technology side of things of radiology. Could I ask a follow-up question?
So, I think I've heard you say twice, twice now in your career, you've been disillusioned with medicine. Could you talk a little bit about the source of that disillusionment? And it's a recurring theme that we've heard from other folks like Ziad Obermeyer, who I think you know well, how did you overcome that? And why were you disillusioned?
Yeah, so this was mainly the transition that I first of all mentioned was, you know, you're waiting and waiting and waiting to be now in the inpatient settings. And that was just the structure of the curriculum. But when I got there, I was like, wow, you know, it's almost like, this is not what I thought it would be. But remember, you may not know this, but, uh, during that time, it was also, Kenya was struggling with the HIV pandemic.
And so, it was very difficult to take care of patients without resources. I think, that type of disillusionment is very different. It's more of that forces you to fix the problem that you see is affecting you. Really, the electronic medical records systems I was deploying then were to support for us to take care of HIV patients, to figure out when were they dying? Who was hungry? I mean, it's not real. Like, are they taking the ARVs?
And that type of data was very instrumental in setting up one of the largest patient care cohorts in Kenya. And so, you know, to be able to live in this world where you can see how your technology is used, I think is an absolute pleasure. And it continues to today where, you know, it's an absolute still honor that, to take care of patients.
I think that everyone is tired because The busy work and the paperwork is, is a lot, you know, and, as someone who's practiced medicine in Africa, in many places, and also here in the U.S. I can tell you, it was amazing to be in Malawi a few years ago, where the doctor would come and say, oh, this is the clinical things that I'm facing. And you would come up with a plan, and they would come back in the afternoon, and you discuss what you found.
But here with technology, you may not even know the radiologist who interpreted your study. So, there's quite a big gap and I feel that unfortunately the hospital administrators keep growing and growing, but they have no love for what we do. Yeah, so I think unfortunately a lot of that reflects what I've seen in medicine. I also think it's a great transition to the next thing that I want to ask you about, which is some of your own work.
So, I think what's really unique about you is that you are a leading expert in AI, but you still are a full-time clinician. You still are a practicing radiologist. 00:10:25,095 --> 00:10:29,024 Unlike a lot of other folks in this space who have moved purely into research or administrative roles. So I'd like to frame some of this conversation around one of your papers. It's a general-purpose AI assistant embedded in an open-sourced radiology information system.
Cause I think it gets to the core of a lot of what you were just talking about. So, could you tell us a little bit about that paper? And then we can jump off from there. Yeah. So, this actually, is sort of like the planting seed of three years or four years of work.
When we've thought about how AI is going to be deployed, so if you think about the evolution of AI deployment, at least in radiology initially, and let's talk about the RSNA, which is the Radiology Society of North America, where all vendors would come, and they opened a new exhibit for startups. You would see that most people thought, well, I'm going to lock you into a platform, right?
If you're in this platform, then it's a marketplace like the Apple store, the Google store that you can get, go and cherry pick the applications you want, you run them. And that is still in use. The second, and is that this sort of the narrow use case of AI. And this is saying, okay, I'm going to look at your wrist or your chest x-ray or your brain CT. And I'm just going to tell you, you have bleeding, but I won't tell you if you have tumor or anything else.
So, you find one image can have almost 10 algorithms running on it to get you the information. Now imagine if you're a radiologist and you have to consume all this information. So, it's really work that I do with a really close friend of mine called Dr. Saptashi from Indiana University, where he always forces me to do this qualitative work research, which is very difficult to do, but it really brings you to the heart of the doctors and the patients who we are caring for.
So, this, the first instance here that you see was for us to integrate a way that would allow us to see how, so we used OHEAP, which is like an open-source viewer for medical images. And we were putting an engine and trying to say, could you do this, uh, I don't want to see anything for pneumonia, you know, like, could you even get to a point where you can personalize?
But more importantly, we were treating, it's this work that was funded by the NSF, that we were going to treat your agent that is deployed as your own assistant. So, we are for, you know, for us right now. And, you know, Andrew would have used, Raj would have used. And this assistance, what they would do is learn what Judy wanted. And then what I would say is maybe that Judy's very good at maybe reading pneumothorax, right?
So, with time, that agent would not have to keep like giving me more alerts and more alerts and more alerts. And so, this was the original work that would allow us to do the reader study, which we just finished, like three weeks ago, where we now had the radiologists using this. We learned a lot from this process. In fact, the final-end product was not on an open-source platform. And that's because we had to learn about standards.
We had to use multiple models and we just wanted something that was more pleasant in terms of appearance. And this work is not out of reach because the work that we're seeing right now is that the human machine collaboration is an area that requires a lot of work, right? Because AI outputs, what you give can make your best expert worse. And so, where we're seeing at least work in radiology, showing you don't really gain productivity gains.
What you get more is, and I can explain that second part a little bit later, but what you observe is that your experts, their performance decreases, but your novices, their performance increases. So, how do you deploy AI in this type of real-world scenario? Yeah. So, it's really interesting. So, I think there was a lot there. It sounds like the AI assistant actually isn't reading the images themselves.
It's actually triaging it to the member of the health care team who is an expert in that particular kind of thing. So maybe I think that this is a pneumothorax read, so I'm going to send it to Judy. You know, I think that this is a pneumonia read, so I'm going to send it to Raj. Yeah. And that's more of like a downstream use. It wasn't really meant to be that way. And, people have used, those types of systems in a different way.
I actually have very strong sentiment around them, not very good sentiment. I think it's good when you figure out who's your best pneumothorax reader. But you can imagine, no one wants to read very complex studies every day. If you think about cognitive burden, right, or when you've seen like autonomous AI being postulated that, well, let it read all the normal chest x-rays, then you read all the diseased ones. Imagine where we work, right? It's academic hospitals.
The cancer you see is for a 36-year-old. Those people tend to be the same age as you with metastatic disease. I personally would not have good job satisfaction if those are the only cases I read without even just the normal chest x-ray interspersed in between it. And so, what this one was doing is learning almost like your personal preferences.
And yes, you can use that as a downstream task, but you can also use it to figure out to label using few shot learning, which is to label, like if you see 10 of your radiologists are always moving something, so you could go back and relabel. The other thing was that you could also teach these agents things that it hadn't seen, right?
So, these tail-end cases, for example, if you have a rare disease that not all agents based on whoever was there was able to see, then they could say, hey, do you know, you can almost think you're almost giving personas to these agents and trying to see, how to collaborate. So, it was really like a proof of concept for us to figure out. How do you deploy AI in this type of, of network and then trying to see what do the radiologists feel about it?
So, that's a really interesting point that I had never like really appreciated until you just said it. So, I always hear this analogy to like, AIs can land planes, but when something goes wrong, they can't land them on the Hudson, you know? And if I'm hearing you correctly, if landing a plane on the Hudson is now becomes your full job, that's a very stressful job to have and not one that most people, would enjoy and that's not something I'd really appreciate it until you just said it.
Oh yeah, yeah. I mean, even just our types of work, for example, if you work in an academic, I'm an interventional radiologist, so you're always doing procedures, right? And I put more test spots than remove them. You only remove them when patients are sick or when they've cured their disease. You see patients at the beginning of their journey, of a long journey, and so, if you are looking at only diseased only abnormal mammograms. I personally would not enjoy that type of job.
This ties in perfectly to a bunch of other questions that I want to ask you. So, you mentioned the, you know, the famous Jeff Hinton quotes about we should stop training radiologists. He's since kind of recanted that I think a little bit. Andrew Ng also had a quote that was like very similar to that. So, I'm not gonna ask you the replacement question. I'm gonna ask you over the next five to ten years, how do you see radiology changing as a result of AI?
I think the most popular code there is now that if you use AI, you're gonna replace people who don't. Right? I believe that we are going to see AI deployed in new ways, not because of the downstream effect. So, it's been a couple of months. We're seeing the foundation models. If you're really using them, you can see what they're not going to be able to do. But I think we're going through a lot of hype again around AI. So, it's a little bit difficult to see between sort of like the weeds there.
But I think that AI is going to change in terms of the backend, the back-office work. If you think about preprocessing. There are a lot of algorithms, not necessarily AI, that are used to preprocess images. And I believe that that is something that is going to experience a lot of uptake. If you think about something like cardiac disease, which, has moved away from invasive cardiology to CT scans, right?
So, the CT scans needs to be very precise about your breathing, your heart rate, and then reconstruct and measure calcium. That workflow is not sustainable to humans, but this is a test if you're coming with chest pain, it's more frequently ordered, especially as the radiation doses have become more optimized and reduced. And so those type of things, I believe we're going to see more AI deployed then. So not necessarily like, more in an assistive role, you know? Sorry.
I, so you're just, I think, dropping insights on me here, left and right, because again, like where my brain goes is the sort of diagnostic head-to-head comparison.
And I was trying to reconcile that with what you said about there are M.D.s who use AI are actually sometimes worse, especially if you're an expert M.D. And so, I was trying to figure out how that was all going to shake out, but you pointed out that there's this whole workflow before you're even getting to a decision point, where AI could come in and actually make a big difference in the near term. I don't want to answer my last question, which will be the answer that I'll hold off on.
Okay. So, you mentioned something else very interesting, just now, which is that if you've worked with foundation models, you know, some of the limitations of them. So, could you tell us about that from the perspective of a radiologist? So, um, I think most of this is going to be, so radiologists have always, always dictated their reports, right? So, using voice-to-text is not new.
And we're going to an era where a lot of the medical content is because of this ambient listening is going to be generated using, you know, a large language model of sorts. And so, I think that, I mean, I have many opinions, but one of them is the making of an expert, or the distinguishment of an expert in the era of large language models is very difficult. So, Judy does not need to be an expert programmer. I probably just need a few things.
I need to hire people who can be expert programmers, but I just need a few things. And, but when you look at, for example, these subtle differences, and I think those are the new papers that we are seeing now come through like advice for oncology, right? I may just know broad drugs, but someone else is going to have something different. I think that's why we are seeing the large models film. I do not want to preempt something that I have no exposure to.
We know that the foundation models can consume images pretty well. When we think about the diffusion models and using that same approach for generating synthetic data, I think we've seen quite a lot of gains around that. And I believe that you will not necessarily need to work in an academic center to do amazing AI research.
I believe in the next two to three years, you're going to see new types of data sets that we can build models from, you know, because this was really a privilege of the past. And so, the same technology that is used to dictate what we know as radiologists is that it's very difficult to get accurate dictations every time you're always typing or doing something different. And if we are going to have a fill-in-the-blanks, right? So, the ambient listening is like, well, maybe Judy meant this.
I think we're going to see like this voice. Yes, we do know that the EMR contains a lot of duplicated reports, but we're going to even have situations where we have just junk that is not really useful, you'd rather have a short report that is more precise about what you're looking for.
And so, this challenge is again, the same opinion that we, one of the questions that we kicked off with, which is, should we try to be always deploying technology or should we really use the deployment of technology to understand the underlying problems that we're trying to fix. For example, maybe we don't need 10 notes from 10 medical students on one patient on one day, and we just need a few good notes and figure out the billing issue, right?
Then we don't have this bloated EMR that is full of work, things that we cannot use. And so, today we see the inbox, that's not a big problem for radiologists who don't do procedures at the message box, which is being taken over by AI systems. But some of the things that are being said, well, you can use order refills. Just think about the logistic and regulatory hurdles that need to be bypassed to even get something like that. I think that's what people are not realizing.
So, in my opinion, I think that the companies that build these technologies usually don't disclose what they have worked on. What we're seeing now, my guess is what they knew maybe one year or two years ago, the knowledge that we have, unless you're in the core team that is building this work. But the main things that are not going to be very straightforward is the regulatory pitfalls. And then when you have one disaster case and there's no thinking about risk.
So, the technology, even if may not be there today, is gonna get to a point where it does a lot more work and maybe the radiologist of the future is going to be also a pathologist because they're now empowered to read the RADPATH correlation, but the essence of risk, trust, and the technology being developed by people who don't see patients may be what limits the technology from ever being used.
So, maybe this is a good place to transition and ask this question, because I heard a lot about overflowing inboxes and radiologists also being pathologists in the future, and I wonder if as a practicing clinician, if you think that AI is going to make doctors happier? I think we're at a moment, so I saw your reaction, but I think we're at a moment now where, doctors are historically unhappy. I think that there's a lot of evidence in the literature to support that.
And one, I think, hope is that AI will take away a lot of the grunt work that physicians have to do. But the flip side of that, I guess, is the theory of conservation of RVUs, that you're going to have to keep producing. And if you can produce more than you will be asked to produce more. So, I think I have a sense of, of where you fall on that, but I'd love to hear. It's not going to make doctors happy. We're just going to be transitioned to another form of data entry workforce.
Yes. Yeah. People have looked at this. People have looked at scribes. Did they make life easier? Right? We're just replacing the scribes with the AI systems. The mundane work that makes people, that would make a big difference, really sometimes does not require technology. It's culture, you know, forms, figuring out who's the right person to call. I know that's very difficult for people to understand, but that still remains a mystery in a hospital.
And you waste a lot of time calling and calling and calling. That does not require AI, and even if AI has amazing results, who is it going to communicate those results to? So, I think that's part of the hype in my opinion. So, I want to believe, X files, like I want to believe, but I think that probably you're right in that a lot of the malaise is not due to pajama time, but due to deeper structural issues. And it seems unlikely that AI will fix those deeper structural issues.
All right, Raj, you want to take over now? Sure. So, Judy, we want to transition to some of your work on algorithmic bias and shortcut features in radiology. So, we had Marzyah on the podcast a few months ago, and we spoke about your paper together. Uh, that was published, I believe, in Lancet Digital Health, and it was titled AI Recognition of Patient Race in Medical Imaging, a Modeling Study.
So, Marzyah talked about the backstory of this paper, why it was surprising, and also, how you pressure tested some of your findings in different ways. So, for our listeners today, I was hoping maybe you could just briefly summarize the main findings of the study, but then jump off into what the clinical implications are of this finding. I know you wrote also in another piece, a recent article that was published in Science about some of those implications.
So, maybe you could weave those two articles together and tell us about both what you found and what this might mean for practice. So, this is an area that is actually a core focus of my research. It's not predicting a self-reported race only, but it's this concept that there are hidden signals on medical images. And the hidden signals as of today between our groups and other researchers, I can show you Judy's x-ray and you'd say, "Judy's Black. She's female.
This is her chest x-ray age, maybe 70." And you can calculate my actual age, which is 50 years. I live in a deprived area because the SD, the social deprivation index is encoded on that. And I am going to spend $15,000 in the next three years. These are all papers that have been published trying to show you can start to predict some of this, what we call, I mean, there's just no way that I would tell you from a chest x-ray that the insurance of the patient is Medicaid or Medicare.
Those would be sort of my unconscious biases, but this is not a task that radiologists do. Then on the other hand, we see different work.
One is done by Emma Pearson and Seyad, which is the work on the algorithmic knee predictions which is better than the Kellgren Lawrence, which is just an older system, not an older, really the most, the current system used to determine what's your severity of your osteoarthritis and has implications on who gets surgery and who, whose surgery is covered by insurance. We've seen work from Adam Nyala that looked at breast risk prediction using the Mirai model.
Showing that you can perform better on these image-based models for risk estimation. You can imagine everything, prostate cancer, lung cancer, all of them could follow the same way. And you can show, hey, that maybe you can get better risk prediction than the existing clinical systems, and it's even better for minorities, you know.
And then the next other examples you see, for example, there's work that got published looking at COPD, that if you don't use the radiologist labels, you just use the pulmonary function test, you get better performance than even using the radiology text. So, one hand, I say that it's not just race, self-reported race, which is just a social and legal construct, no biology around it. And in same like insurance and other things that are biological, for example, age.
And now, on the other hand, is that you have this good performing image-based only models. What we should be reading, and including more of our recent work where we've shown that from test x-rays, you can tell who has diabetes, just the ambulatory test x-rays. So, you should read this type of work and see. How is this? Right? Is it a confounder? The work that has subsequently come out is that models are lazy. They pick the easiest thing that they can work on to make the prediction.
If the easiest thing is where the data was collected and the disease probability, then they learn that. And then when you remove that shortcut, they learn, they pick another shortcut and another shortcut. And so that's how you can have models predict, caption an image without ever looking at the image itself, it just decides, okay, I'm going to say, you know, this is sheep grazing or something like that, right, with and without the sheep, just when it lands on the simple easy confounder.
So, the importance of this work is, yes, it's great if you can show me that the algorithmic knee osteoarthritis prediction is better. But if I show you that the same algorithm can tell that the race of the patient is Black or white. Do you know if it's really looking at the disease? Or it's looking at the shortest thing to tell you that this is the osteoarthritis severity. We cannot do this work in this type of shortcuts.
Others are easy to identify, for example, ICU markers or something like that, but we cannot do this work when we know, we cannot, there's no way that I'll tell you the insurance or the risk of a patient from a medical image. And so, this concept of these hidden signals and their downstream impact on algorithmic prediction is why we need to study it in something that we may never be able to see, but we need to understand when it fails. And what if, not all Black patients are the same?
Maybe in Africa it's very different. And what are these models looking at? And the current explanatory tools are not sufficient for us to do that. The last thing is that these shortcuts are not taken away with external validation. So, just because I bring my model to you. They don't go away because there is how we collect it. And so, putting a big area of work. So, I'm excited to be working on this though. No, that's great.
So, I think there are many different paths that have been outlined by researchers recognizing that race and other variables are encoded in ways that are often hidden to physicians in these images and in other types of clinical data.
And depending on who you ask, they'll say that, that the best path forward is to, for example, learn de-biased representations if you can, right, that don't encode those particular variables so that you can have some guarantee or some sort of assurance rather, that the model is not just using, you know, race to predict X, Y, Z, or to make some type of allocation decision.
Um, how do you see, maybe for your own group's work, given that finding in Lancet Digital Health, and then given the Science article where you outline some of those implications that you just talked about, do you think this is more of a set of technical challenges to, you know, to develop better machine learning tools, to learn new representations that don't encode variables that we don't want them to encode
for particular models? Or do you think this is, we're still in the descriptive stage, if you will, where we need to understand what is encoded in what type of clinical data and just make that clear and apparent to the whole community, even at a scale that is maybe appreciated by us as machine learning and health researchers, but is not appreciated by, you know, many manufacturers of these models, algorithms, or users, physicians at the end, who are using these models in practice?
You know, what do you see as the path, forward for the next few years of research? Yeah, I think we're still in the discovery phase. And here's why. If you read all these papers, most of them, they, I didn't set out to go study what self AI could predict for us. Really, actually, the what set out that project was, um, around when George Floyd was murdered, a lot of the journals started to say, we want to focus on social justice and we want to look at that in medicine.
And also remember a lot of patients who are Black and Latinx were dying disproportionately from COVID. And so, when we set out to do this, what happened was we were like, oh, we, JSCR is coming. They're looking for a paper. I attended a data thon before the previous year in Singapore, and I realized people are not using the MIMIC test x-ray data set. And this was just a little bit new-ish and they said, well, let's look at this. And when I start my research, I always say, what's the story?
Why am I doing this? Right. Maybe that's the hypothesis, but I always say. Maybe not in a very scientific way, but what am I going to teach people about this? Or what's the question that I want to answer? And here I was going to say, the takeaway is going to be, we need more diverse datasets, boom, easy paper. And then we started and yes, we had looked at the work from Laleh, which and Marzyah, which showed the under diagnosis, right?
So, you're most likely to have a normal chest x-ray when you have a true disease, a true finding when you're Black or Hispanic. But at the same time, when we brought the same models to Emory, the amplitude dropped, but the patterns remained the same. So, it was this looking at the patterns that put us into this rabbit hole for two years. Then one of the team members came back and said, well, it's because the models are learning the rest of the patient.
And we said, no, and we said, you're wrong and shamed the person. And we went back to redo the experiments and it was really the same. So, we started now structuring the research and know that it's always a shame that we never get to know what's in the kitchen, how these ideas come up to put that context. But it's not that. Today, when I understand these issues around like even systemic racism, remember I am an immigrant. I would do that research very different.
In fact, I may be too ashamed to do the research if I hadn't done it then, you know? And so it's, I think we don't know enough and especially things that we cannot see and using the current saliency, visual features for explanations, we really cannot get a hold of what these confounders are doing.
So, I agree that there is a role for the model developer, but I think we need more disclosures and transparency in terms of the data sets, how they're composed and how they end up, especially those results that are never published. That's what we need more to figure out. Oh, maybe there was something, a signal here. And the way to do, if you're listening to this and you're a researcher, is to have more people in your team who are looking at your work.
Because I do feel that it gets better that way. That's great. So you, your paper was on chest x-rays, and then you mentioned Ziad and Emma's paper on knee scans. Have you done a similar set of experiments around those modalities, so around knee scans for example? Yes. Yeah. Yeah, yeah, so the paper actually was more than chest x-rays. We looked at three chest x-ray data sets, three CT chest data sets, a digital hand atlas, that's a public data set.
We had a, an internal data set for a different project for cervical spine radiographs and mammograms. And we found the same thing across all of them. Have you done the knee scans as yet or no? Mm hmm. We did. And it's, the performance there is like 0.99. If that's the case, if that's the case, what does that mean about that Emma and Ziad's paper? We don't know. We need to validate it in the real-world setting.
Is it because the algorithm performs better for Black patients or it learns that the knee x-rays from a Black patient? Got it. Okay. All right. Lots of future work to be done. So that's great, Judy. So, I think we are going to jump into the lightning round now. Andy, do you want to, do you want to kick us off?
Yeah. Um, so this is always a fun part of the show, Judy, where we, we ask you a bunch of completely unrelated questions, and you don't have to answer them in one or two sentences, but I think the goal is to move through as, cover as much of Judy as possible. All right, so the first one is, what core principle informs your life? Lifelong learning, and togetherness, community. I truly do enjoy just learning.
I'm just a really curious person, and I enjoy reading books, and then, and challenging myself if I agree with the authors. So that means that if I pick a book, I will always read it to the end, even if it's not, it doesn't align with my principles, but that forces me to listen to other different opinions that I would not listen to. And I truly am a child of the village, and it would not be possible without just having other people to work with. And work is, research is much better this way.
And in terms of medicine, it's really to do things that spark joy and because we don't have too much time in the world. I really love that. That challenging yourself, even with people with whom you disagree, being a core principle, because I think that fewer and fewer of us are able to maintain the cognitive dissonance that it takes to do that. And more and more of us retreat into echo chambers. So, I love that. That's a core part of you. Yeah, me too.
There's this idea of adversarial collaboration, which we should all engage in probably a little bit more. So that's, that's great to hear. So, Judy, I have the next question. I think I know the answer to this. Uh, although it might be completely different from what I'm thinking. Um, and I think you give us a hint earlier. But here's the question. If you weren't a physician, what job would you be doing? You can dream big. Today, maybe a veterinary doctor in the zoo.
Not in the zoo, in the National Park. Okay, great. That's because I would be able to go in, specifically in Maasai Mara. I think I would love to be a radiologist. In Maasai Mara, which is one of the largest parks in Kenya, and I have spent time taking care of patients in Narok, and every evening would go for a game drive, and it's probably one of my best times in my life. That sounds amazing. I was going to say pilot where you started off. Apparently, they're going to be replaced soon.
Yes, fair enough. Can't land on the Hudson. Can't land on the Hudson. No, but, uh, I feel that maybe then it was it was this thing of just this curiosity of just flying. But I, maybe what I wanted to do was travel and I've gotten to do that quite a bit. So, I don't feel like I'm missing out on that part of my dream. Although I, I did plan to get my private pilot license a few years ago. Nice. I wonder if this question is now answered. I don't think it is.
But if you could instantly acquire any new skill, what would it be? Um, learning piano. So, I have and I'm not so sure about instant. But it turns out that it's actually pretty interesting. And the reason why I'm doing it, first of all, it's a gift from my spouse for my birthday. And so that, so I've been doing that.
But it's very mathematical like very logical, and so I've enjoyed doing that, and I think the reality is this concept of deliberate practice that I take from it, and I've always said that to my trainees, especially in the clinical area, that these are words that I steal from my friend, is perfect practice makes perfect.
So, it's not just practice, and so this being intentional, and so that's been like just something new, and I do, it's, it's also Just something to, that I do for me and I have to make time for it despite the busiest week. And so that's also been like really nice because it's building some other character in me. That's awesome. I've always heard the, the maxium be practice makes permanent. And so I, yeah, I love the idea of it.
Yeah. But you know, in medicine, you can do like bad things, you have bad outcomes and you think you did well. So, they're permanent. That's true. Yeah. Continuing with the music theme, what is your favorite music album of all time? Of all time? I, I cannot stay with all time. I, my music changes across. Across times and across weeks, I do really enjoy the African music and, um, because I feel like there are these giants who sing a lot about some of the struggles.
So, it's not just music for music. Um, right, I would say that the person I wish I'd seen her alive present would be, uh, this lady called Cesária Évora, who sings this music which moves everyone in the crowd and she would be singing barefoot and barely moving. And I wish I'd had a chance to see her alive in a live performance. Today I enjoy a lot of Swahili music. I don't get to speak Swahili as much. And so, anyone from East Africa with music that, a genre called Bongo, I listen to them a lot.
And I, I, that's what I jam to most of the time now when I'm driving to work. Awesome. Awesome. Yeah. Um, so if you could have dinner with one person alive or dead, who would it be? Ooh. It would be Shonda Rhimes. I hope that it's never going to happen, but that would be the person. And the reason is just looking at her work about making Grey's Anatomy, this is the producer who's all this. Some of these shows as a Black woman I feel like she breaks barriers and she brings gay people.
She brings Black doctors. She brings, you know, they're not just nurses. And in, in this world where what we see really reinforces what we think about stereotypes, I think she's really broken the barrier for so many people. And I don't even know what I would ask her, but I thought like she'd, I think she'd be a pretty cool person to have lunch with. That's an awesome answer. It's never going to happen, but...
Well, if she's listening, you know, maybe we can... I don't know that she's listening to the New England Journal of Medicine. I've heard, yeah, I've heard AI Grand Rounds is her, is her go to podcast. Yeah. All right. If you could eat only one food for the rest of your life, what would it be? Chapati. Oh, nice. Yeah, very nice. Not the Indian chapati, the Kenyan chapati. Very nice. All right. So, uh, one more question. Nice. One more question for you here, Judy.
Um, what is the most interesting thing you've either read or watched recently? Oh, it's definitely reading. And maybe I should do the last year theme, let me see it was, I would say that the book that I gave people last year was, um, I have to, clearly it was not as notable, if I cannot pull up the title, and it's The Psychology of Money. Who wrote that? Morgan Housel. What's it about?
So, it's, it's, it was a great book and it talks about, I didn't really know that this person was an, was like a financial planner till the end of the book. And so, it talks that, about that money, your relationship of money is dependent on the relationship that you have around you. For example, if you are born in depression or you're born in inflation, you tend to be very cautious, or let's say. When you were growing up, your family or someone you knew, everything was taken by the bank.
You're very cautious about how you deal with money. So, it's that this behavior and psychology of money. And so, to think about how to invest your money is that you have to think about the emotions that are around your behavior. For example, for him, if you talk to many, of the financial planners that would say never sit on cash in the bank. But it turns out that all of us will, will experience a bad, a bad thing in our lives, or someone close to us will experience that.
And so, if you invest faithfully, then, and you don't interrupt compounding, it means that you minimize the, sort of like, your reactions, your knee jerk reactions. So, what blew my mind was that he paid for his house wholly in cash. None of us would, especially now that the interest rates are pretty high, but most of us would get into a mortgage. But it turns out that what kept him in bed at night is knowing that his family would not be homeless.
And so, this was the biggest, that once that was taken care of, he never wanted to pay anything per month, that it gave him peace of mind. And so, it's this thing that your relationship of money really determines about how you and the people you are close to you deal with it. And once you understand that, then and the second thing is that you don't try to beat the market and get on the hype. Then you realize what enough is for you.
And once you realize that it gives you absolute peace of mind and you can spend the money in ways that are more meaningful. And so, I thought that that was such a great lesson to learn. And I really enjoyed the thought process about thinking about money in from a behavioral and emotional point of view. Yeah, that's great. I, I think I'm just gonna move on quickly and not dwell too long on what my crypto, uh, currency purchase, uh, says about me psychologically.
So, congrats on surviving the lightning realm, Judy. There we go. Thanks. Thanks. Thanks. Alright, so I think we want to pull back a little bit and ask you some big picture questions here at the end. So, you know, we've talked about it a little bit so far, but the thing that's currently sucking up all of the oxygen in AI are large language models. I think that a lot of diagnostic medicine is trying to reckon with what the implications for their field will be.
It's maybe perhaps less obvious to me what LLMs specifically will portend for radiology. So, I'd love to hear your thoughts on how LLMs will or will not impact radiology. I don't think directly for sure there are some tasks around processing the radiology report that are important. And this is, I think we're going to really understand what the potential is once we get the image part of the model, right? So, the radiology report is written for the doctor.
It's not really written for the patient, right? It's a, you're the doctor's doctor. And so, in this case, when let's say my report is so wordy, maybe the ER doctor just says, it's the appendicitis, and we could hypothetically say that maybe these models are going to be as good that they don't need your reports to just ask the clinical question that they want, assuming that they know the question that they want to do. I think right now, as radiologists, my biggest area is around education, right?
It's still the same. There's no radiology education. All our medical students are trained in the same way. And they're learning in this era of large language models. And we do need if you think about how quickly the curriculum changes or doesn't change, we need to really train our workforce to be able to work with these technologies. And I think we are failing in that area. And the second area, to me, is around the interface of reports and patients.
And so, again, not directly to the radiologist, but in this area where you can get access to your signed report immediately, even before your referring doctor gets it. If it's not written for you, but you can copy that report into a large language model and you can start to interact with the report. And so, it's the downstream uses of our outputs, I think is the most immediate
thing that is going to happen as of today. Before the, or I'm working together with a chatbot or something like that, more than like a medicine specialty. Yeah. That's so interesting. And again, I think it speaks to your experience as an actual physician that you have that able to have that kind of insight. I think of an example from my own family where a family member sent -- my wife's a physician -- sent a mammography report that she got.
And the family member had very little ability to parse that themselves and were asking, you know, my wife questions about what does this mean. But that seems to me to be, that type of translation between a note written for a doctor into plain English seems like a dead obvious thing that would have high yield. Yeah, and actually my students have worked on this. It's pretty interesting. And even medicine today, even the radiology report, we take care of diverse patients, right?
They, they should access their reports in the language that they understand most. It could be English, but it could be a certain kind of English. It could be a dialect. It could be anything. So, talking to the patient in their own voice, I think is, you know, these large language models are so good at changing voice and so good at pretending or emulating a certain style that that's a, another really good example.
Judy, do you think within five years, discharge notes will primarily be written by large language models? Probably, but the, the doctor will still have to edit them, you know, they're just gonna be wordy, wordy, wordy, wordy, wordy. I do know that the, I mean, I think almost 20 hospitals, I mean, it's in their tents for sure, who are already using, you know, the products now being integrated with Epic.
My concern about this really in the understanding the true ability is also some of the changes that have happened in the interfaces that you have. So, it's very, it's not very different, but It's still different when you put your query directly to the API versus around the product that let's say, for example, a premium ChatGPT. And we've seen this work that has also shown that the type of response varies, right?
So, I believe that what we're struggling with is this believability and the flowery language that is wrapped around this large language models to really understand, like, the discharge summary does not need to be flowery. It just needs to be factual, you know, it just needs to be factual. It's just busy work, but it just needs to be factual. So, how can you evaluate as a researcher the quality of something like that?
If you're wrapped around, oh, I'm so sorry now, ChatGPT cannot say I'm so sorry. That's the human intervention that we put in between the API and what we see. All right, our next question. Extrapolating from current AI and medicine trends, what worries you the most? I'm excited about these technologies.
And I think what my concern is, is for patients who are historically underserved and now they have, we are bringing technology to the doctor's rooms and not even putting transparency and understanding. Maybe it's going to be better, but there are papers that have shown, even just focus groups, that have shown that people are modifying their behaviors when they go to doctors. I have been labeled crazy when I've been a patient, and it's because of these perceptions in the hospital.
You know, this was one disagreement when I was requiring OB services and I can see now we are bringing technology in and so it's about whose voice is going to be the loudest. So, I know that in my own family and friends, I tell them the key words to say so that they trigger like action, right? When you're not heard or even my students, I say, yeah, this is the high, most, worst headache of my life.
It really has meaning about those type of things that trigger, especially when you cannot be heard. And then the second area is that we, I feel that these changes are traditionally going to be sold to administrators. They'll be told, oh yeah, we can do your billing now, we can do all these things, you can get more money. But they're not really truly impacting patient care or making a difference.
And so, we will end up with these technologies that we are now bringing in and I agree, I personally, by the way, dictate my report. It doesn't rewrite it, it just transcribes and then I decide what that is. But these next steps and just the, the LLM influencers around, I think are going to cause more harm in an area that is very difficult to study. Hmm. You may have partially answered our next question, but let me just try and ask it directly.
Given the sort of like dual use cases for AI we've talked about with, maybe it can explain some pain disparities in some cases. What do you think about, um, machine learning and AI and its potential to exacerbate health care disparities? Do you think that that's going to happen or will it reduce it? So, I think that can happen and it's not just because of health care. I mean, we see it in other society things, right? Deciding who gets Amazon Prime, deciding who gets hired, deciding policing.
So, we do see that and it's this assumption that technology is neutral. We never know how it's going to be deployed downstream. Now, as someone who's seen like police brutality, I wish that it was a machine, a robot that stopped you and traffic stops. I think that more, less Black men would die that way, in my opinion, or maybe that we never needed to stop anyone and do get a ticket.
Yesterday I was in, uh, Costco and after my receipt was printed out, there's another receipt that came out and the second receipt, I didn't understand it wasn't for me. And, you know, I was puzzled and, and the teller told me, oh no, this receipt just tracks how fast I am at counting. You know, like at the counter. And I thought, why would you ever build such a technology? You know, and I said, okay, what are the consequences if you're slow?
Then their renewal to be a teller, right, is dependent on the speed. What if I wanted to ask about something that I didn't find in the store? Just think about this simple thing and how it's deployed. You know, so we have these ideas that always sound good, but we never frame them in the societies that we work. And so that's why AI, not just in health care, can access a bit disparities.
It's terrifying to think about the analog of that Costco, second Costco receipt in the context of health care too. Uh, but it's terrifying but not impossible for me to imagine it. Alright Judy, this is our final question. That's what you said like 30 minutes ago, man. Yeah, the last one for real, I promise. So other than Indian chapatis not being the best type of chapati, what is your most controversial opinion? I don't know. I don't know because maybe I believe in it too much.
I think that maybe the thing that maybe my friends don't like me saying is like, do things that spark joy, because they say that comes from privilege, that, that to have those choices. But this is something that I truly believe in. And that means that I can, what I call cancel my order, like I can just walk away from something really easy. And even, even when I should pay a little more attention to it, but thankfully my spouse is there to help me with that and my friends.
And I think that, um, in terms of opinions of life, maybe I would say that I don't believe it's maybe my subscription to this model that, for example, some of the global health initiatives are not helpful. And, and maybe that, I don't know, I don't have a good answer for this one. I have many opinions, but I actually do, as I speak loudly, I feel that I can do believe in those, those opinions. So, all right, well, Judy, thank you so much for being on AI Grand Rounds. This was great. Awesome.
Thanks for the invitation.