AI in Medicine Part 2

Welcome to this episode of The most important thing. Podcast on the planet. According to Grandma. I'm joined by my co-host, Dr. Tyler Gibbs. Who if you weren't here recording right now And I'm joined by my co-host New York Times. Alright, good morning Tyler. Good morning. It's uh it's slowly warming up around uh the United States, so hopefully uh we get some better weather for the next couple of weeks.

I know there's nothing podcast listeners love more than when uh the hosts talk about the weather. It's like super relatable, very exciting. Yeah, it's about all I think about these days is how much longer is winter gonna last and when can I go golfing? Did the groundhog see its shadow? I think the groundhog did not sh see its shadow, so Spring's coming soon? Yeah, that's what it means, right? I always forget which way it goes. Yeah. All right. So

Question for you, Devin. Uh-huh. What was your nickname in high school? Ooh, in high school? Yeah. Yikes. I don't think I've ever had a nickname. Devin is a pretty short name to begin with. Uhhuh. Some people might have called me dev, which I don't love, but I don't hate. Um but yeah, I don't know that I ever had I was like terrible at sports, but I played all of them and so There was a lot of like laughing at me, but I don't know that a nickname ever stuck to that. Oh

How about you? Names are fun. I I had a bunch of different ones that are more and less appropriate, but um I I love asking especially s new people that I meet what their nickname was in high school because there's almost always a story behind it. Right. Um I I got I mean, with a sh last name like Gibb I got I got Gib, Gibber, Gibster, Gibbonator, the Gib the Gibbosaurus.

It's just so clever. Um, yeah, very clever. The O D M, which I won't won't explain to anybody. Okay. So keep it a mystery, love it. Yeah. Yeah, lots of different nicknames. Mostly sports and uh last name related. But Yeah, I feel like that's an honor to be called by your last name when you play sports. Again, very bad at sports, but somehow always made the team I think maybe as fodder for people to laugh at me.

It's'cause you have a positive mental attitude and you're your coach's dream, Devin. Oh, I tried harder than anybody for sure. Uh-huh. But trying hard and being good are not the same.

Yeah. That's fair. That's good. All right. So today we're gonna talk about uh some more artificial intelligence and healthcare. Mm-hmm. Part two of um our kickoff to our AI series. Okay. So I Like last week, um, I started cataloging just kind of the major ethical challenges I see arising in the literature right now. There's probably

So much more that we're not even thinking about. Um but these are the things I made a kind of list that I'll go through with you of what I'm seeing talked about over and over again. In this realm of clinical care. Probably a million things we could talk about with the ethics of AI, but I'm gonna try to narrow in on what could be the possible ethical challenges of using AI in that clinical space.

Sounds good. Yeah,'cause I AI is permeating everything else in the world from stock market trading to real estate to booking tea times at local golf courses. So Yes, I'm something that I don't know anything about, but I'll take your word for it.

Okay. Okay. So first up, we mentioned this a little bit last week, but I want to delve more into it. is um threats to the provider patient relationship. Yeah, okay. So of all the concerns about AI and healthcare that have arisen, healthcare consumers in a Pew study done last year, say they are most concerned about their provider's reliance on AI and that it will lead to a deterioration of their relationship with their healthcare provider.

Does that surprise you? Yeah, ki I I mean on its face it su it doesn't surprise me because I think when anybody is uncomfortable with a new technology in healthcare, whether it's something simple like the stethoscope, we we s there there are reports of when The stethoscope was developed and uh became widely used, that there was concerns that that physical distance between the provider and

the the patient, like not the the doctor not actually putting their ear against the chest of their patient to actually listen to them, that it that that would cause a a a distance and a a breakdown in the relationship.

I it's it's a it's a concern that from some sometimes some some of the earliest medical technologies uh had the same concern. So it doesn't surprise me, but I think that when people are uncomfortable with new innovations that that's often the the first thing that they are concerned about is that the relationship between the doctor and the patient is gonna be different and worse.

Yeah. So I've never heard that about the stethoscope. That's super interesting because you think of I think of the stethoscope as like this savior device that you've probably heard about these tales of uh before the stethoscope it was really hard to get a heartbeat just from listening.

So there were situations in which a a heartbeat could be really faint and that was hard to hear. And so there were all sorts of medical devices invented to kind of poke and prod people to create painful experiences to make sure that they were actually dead. And the stethoscope stopped all that. Um so that's really in I've never heard that before. Yeah. In retrospect a and this I've never had this experience of a doctor putting

his or her ear literally on my chest sounds um like something I would not enjoy actually. Well if you're used to it, right? If it's like this I i it's this level of intimacy and you know closeness that i uh they would use like little cones, like little metallic like brass cones, like you would hear like you would see pictures of like people with hearing impairments like use those cones yeah. Um to hear better.

Yeah, so the stethoscope was kind of billed as the same way as being a problem. But um there's also the there's also this um the the distance that's created, you know, the electronic medical record and and all of these other inventions that have come out and ch re revolutionized the way in which clinical practice happens, but so is AI more than just an extension of that, or is there something specific about it? Yeah. So I

it's kinda hard to um with these kind of big national surveys to know exactly what people were thinking when they said that this was their greatest worry. But there is this sense that they think that if their provider and I know some physicians hate that word, so if their practitioner relies on AI instead of relying on intuition or medical training or the other things that you might rely on to make a diagnosis, that that will somehow distance the patient from their healthcare.

And it's interesting because people generally say they think that AI will improve things like diagnosis and prognosis. So they're acknowledging that that is a benefit and still they're worried about what that will mean for their relationship with their healthcare provider. So super interesting. But I think you're right to say this like over reliance on technology

does create less human to human contact, right? We've all seen this with the electronic health record where your physician might be talking to you but really looking at his or her computer as they do it so they can put in their notes. We mentioned last week that physicians tend to spend more time in the electronic health record than they do in patient interactions, so

you can see why they'd be tempted to try to do both of those things at once, but it stinks when your doctor's not looking at you when they're having a conversation with you, right? Yeah. I I think so. But I don't do you remember a time in which and this is uh you know, we are of a particular age now, um, when that didn't happen? I don't have a good strong memory of

Before doctors were just looking at their computers while asking me questions? Well, I mean, I do remember working in a hospital when I did my first internship. Uh where they did only have um paper charts.

But I wasn't a patient. Th I mean, this was in my twenties, so I didn't have a lot of like patient experiences yet. So you're right that I don't remember a time and the older you get and the more, you know, chronically ill you get in my case, the more time I spend with physicians. But Yeah, I don't personally remember a time when that when they weren't on their computers, although I do remember times working in hospitals.

when they relied on paper charts. So all that to say, I I guess we don't have maybe the best perspective on but I doubt the physicians were like sitting there with their pen and paper as they talk to you. That would seem fairly strange. Yeah. Yeah. Yeah. So there there is all these th studies that show that clinicians spend thirty-five percent of their time with patients looking at their computer screen rather than directly at their patient. So not great. It's not great.

I I remember when I was in the doing my training that there was one physician whose bedside manner was um particularly questionable and thought that maybe uh a robot intervention would actually like one of those like uh droids almost that goes around and interacts with people was actually an improvement. So I don't know, two edged sword maybe?

Yeah, I mean I guess if your bedside manner's terrible, then maybe uh that physician not looking at you is the best possible scenario. But that's not what we hope for. That's not the ideal. Ideally our physicians have great bedside manner and they're able to have direct conversations while looking their patients in the eye. So will AI help this or not?

That's kind of questionable, but at the moment people have already have these worries about technology distancing themselves from others and hurting personal connection. Um and so the prospect of AI brings up I think all of that for people.

Mm-hmm. Yeah. Yeah, I can see that for sure. Yeah, particularly with the change in the technology for sure. Mm-hmm. There's also this idea that it um relying s a lot on AI diagnostics or prognosis may de skill physicians. So if you're relying on a machine to do that work for you consistently, then you might not be in the practice of doing it for yourself.

And so there's that kind of de-skilling worry. Um I don't know if we need to worry about that quite yet. That might be e even more in the future. Um but something people are already thinking about. Yeah. And we hear this sometimes when we there are concerns or complaints about kind of the younger generation of

healthcare provider, whether it's a physician or a nurse or a social worker, like relying upon, you know, just Googling something rather than actually learning the material. Right. And this is part of the concern in medical education is that w w what What's the value of people going through and learning material if they can just Google what they need to know? And I think that that is also a concern

with using AI in all other types of educational settings as well. So having Chat GPT write your essays in undergraduate, for example. You probably see this much more than I do because I don't have a lot of my students writing essays, but um

Oh Do you hear that? I mean my students would never use AI, but I can imagine Yeah, like we're trying to as um professors think of ways to both positively use AI, but also make sure that our students aren't totally relying on it because there's something about writing that we think might still be important. So it's huge upheaval in education but so I think people are sort of transferring that idea, which is maybe a little bit more relatable than being a physician who uses AI.

For diagnosis, that's maybe it's less clear like what that even looks like for most people. I I mean it's less clear for me. You can see how if you transfer the this thinking into other areas into medicine, people would have the same kinds of work. Yeah. Yeah. So I don't think it's an unreasonable worry, right? We want to make sure our physicians aren't relying so much on AI that they just forget how to make a diagnosis.

Yeah, absolutely. Yeah, it can definitely be a a tool used by folks to really enhance the the clinical practice, I think, especially with diagnostics using massive amounts of data and being able to look at patterns and predictions based upon specific information. But again, I push really hard that the practice of medicine is not a

it's not a vending machine type of relationship. You don't go in and and put in your your concerns and your your clinical information and then out pops a a diagnosis. I think there's something more um I don't know, magical, intimate human about it that maybe is completely lost in AI it's an AI intermediary.

I think absolutely. Like w we always say there's a art to medicine and then it's a humanistic practice and if a robot could do it, then there's something definitely that's lost there in the human human connection.

Yeah. Um yeah, good. Okay. So that's consumers like if we say they're consumers of healthcare, that's their number one worry. Their second worry, um, is around data security and privacy. Okay. So what could possibly go wrong when we provide AI with all of our patient records? Yeah, oh my goodness I and I think there's like low lit low hanging fruit concerns. So like what happens if it gets hacked or what happens if it gets

Um, you know, there's some sort of data breach. And we see that all the time in healthcare. We see m big systems that are hacked and uh have to pay ransoms or have to you know pay tons of money to get their information um unlocked. But uh but so that's kind of a a big picture question, um, about like s the security of it. But the the other question I think that comes up is what

what happens with the data that you're supplying? Does it just get anonymized and put into this big vat of other data or is it extractable or is it identifiable? Um and I think these questions Probably uh I mean the answers to these questions are so technical and I don't know that any b any individual or any group really know the answer to them. Mm-hmm. Maybe.

Yeah, well, so there has been a little bit of work already in this area. So as we talked about last time, in order for AI to be effective, it has to draw on huge sets of data, right? So it is gathering big data th h tens of thousands of patient profiles in order to make good diagnoses and predictions if that's what we want it to do. Um so it it has huge data sets. So just to say like your data probably if you're listening to this has already been used in some sort of AI algorithm.

Um and you didn't have to consent to that. That doesn't seem great. Yeah. For all sorts of reasons that we may or may not agree with, but you've probably already had your data taken up if you've used the healthcare system in the last couple of years to go into these data sets. So there is this question of um security.

And the big tech companies now that are gathering this data, which are the big tech companies you'd think about, Google for example, has admitted that they cannot totally secure this data. So the a big report came out in twenty twenty three where Google was like, listen, about three percent of all of this data is recoverable.

meaning somebody could go in and find and re identify patients with their medical record. And there's just like nothing they can do about that at this point. So about three percent of all this data would be identifiable and that's just kind of the name of the game. Mm-hmm. So you hope you're not in the three percent. Yeah. And that's so curious be that it would just be like a shrug almost. Like, yeah, sorry. Like your data's identifiable and out in the world.

Yeah. And so there already have been like lawsuits and federal investigations because a lot of these companies put stuff in the cloud and like how secure is that? And some of their own AI programmers can see patient names with sensitive health information. And so there have been some lawsuits around this, but they have not yet resulted in any additional regulation.

Hm. Yeah. I can only imagine that that's that the answer to that is there haven't been more regulations yet. Mm-hmm. But maybe I'm wrong. Like maybe it's the technology is such that um the regulations aren't able to keep up or won't be able to keep up. And then we get into this situation where maybe more people are just gonna shrug about it and not care about it and just view all data as not private to begin with.

Um which is kind of a cynical way of approaching the the question. Yeah, it seems like we've all kind of gotten not all of us, but most of us have gotten to that place with, you know, Amazon and Facebook and these tech companies that are gleaning all this purchasing habits from us.

And um tracking our locations and now there's sort of a process of saying no to that, but for a really long time there wasn't and people kinda did just shrug their shoulders and say, All right, well if they wanna market more stuff at me that's I'm more likely to buy

fine, if it's tracking me, fine. Most people have relented with that with like twenty three and me, that is gathering your DNA and then selling it. Like if if people aren't horrified by that, I'm not sure what they're gonna be horrified by. So I Yeah. I don't know though, there's something about your health information as it is like in your

health record that maybe people would feel more uncomfortable with. And that's probably why this is the second biggest concern by people in this Pew study, um, is that there might be re identification of them with their health record. And while we have um some laws that protect people from discrimination, you can just imagine that if you have a stigmatized health condition and your employer finds out about it or

um somebody you have a high profile job and people find out about this, that that puts you at risk for certain kinds of discrimination. Yeah. Or backlash or embarrassment or something like that. Yeah. Yeah. And there's something about It's not just that these big tech companies are gathering all this data, but that it's become a business.

Maybe I'm particularly sensitive about this, but I just think it's one thing for these algorithms to say, you know, we're gathering this data so that we can help future populations. We can start curing diseases, we can detect things more quickly. That all sounds really good. But then these companies are making money off of my data.

And I didn't give permission for them to have that data. So we're enriching tech companies for commercial gain off of our data that we never even agreed to give for free. There's something about that that feels really icky to me.

Yeah. I agree. I think that if there was a way to f like allow my data, even like vast amounts of even very private data, information about myself for like research purposes or to train like you said, s to train the AI in order to improve diagnostic or prognostic um abilities.

Very different than big tech bros making tons of money off of the the backs of my data. The back of my data, right? Right. And both those things can be true. They can be making tons of money because they're helping with diagnostic prediction.

But still there's something about you know it's Amazon, Google, and Microsoft are the big companies doing this work right now. There are smaller tech companies doing it as well, but those are already the biggest tech companies and we're giving most of our data to them in these cloud storage programs. And they have access to like our mental health conditions, I just it makes me uncomfortable for sure. So I I get why people are uncomfortable.

Do you think that it would make you more comfortable if there are more people like more oversight or if it just didn't exist or like wha what what would help with your uncomfortableness? Yeah, I I think there just being more regulations on um you know, like it should never be the case that some Tech Bro can see my name attached to my health record. That just seems wild to me that there's not more regulation around that.

I don't think that's the norm, but I think it's apparently it's possible and it's happening. So I don't know. I I think maybe just some regulations around um how identifiable stuff is would make me a little bit more comfortable. I probably will never be a hundred percent comfortable with it.

But I also recognize that if this produces a bunch of cures for horrible diseases that might at the end of the day be worth it. It's just kinda hard to know that from the get go. Yeah. So are there all these calls now to make AI more answerable and transparent to the public? And and that makes sense to me. So have you heard of like um responsible AI? There's a lot of terms that um get bandied about.

for making AI more sort of transparent to the public. So what exactly what what is the data that these companies are getting? What are they doing with it and how are they using it to make these kinds of predictions or to help improve diagnoses. That stuff is obviously complex, but I think most people would say they want there to be transparency in exactly how this data is being used.

Yeah, yeah, for sure. Especially if they're like guiding principles or certain companies use the data in more or less secure or more and less sensitive ways. I think that that would give the individual who quote unquote owns their own data to be able to decide which algorithms align more with kind of their maybe ethical intuitions or values. Yeah.

I mean that that sounds really nice. Nothing like that is happening. Yeah, and and what makes this even more complicated is that I mean obviously this AI stuff is super complicated and I don't Yeah, please don't ask me to describe what AI and machine learning are actually doing. Like it's it's complicated and

tricky thing is that these companies are proprietary. So they have no interest in being transparent with you about what they're doing because this is their like secret knowledge that if they shared, other companies could use and replicate what they're doing. And so because these are private tech firms and not federal agencies that are responsible to the public in a different way, you know, capitalism suggests to me that they're they have no interest in being transparent unless consumers demand

But at this point there hasn't been like a big you know, people don't even know this is happening, right? So a lot of hospital systems are using AI to make all sorts of predictions about patients and combing all of the healthcare records and we just have no idea it's happening because they don't have to tell.

And it's not just for clinical, uh, like diagnosis, prognosis, interaction with patients, right? I mean AI is being used to predict where resources are being allocated or maybe there's certain times of uh s the the monthly um, you know, buying, purchasing you know, saline or certain uh resources or certain medications are being used more and less and they can try to be more sensitive with the buying and and usage of those. Which I think is is great if we can if AI can help us be more

environmentally conscious about our waste and and overuse of products. I mean th I think that's a great idea, but I don't know that that is really the focus as much as it is to look at the ways in which w A I is a clinical tool to make clinical decisions. Yeah. Well, I mean, I'm sure you've heard the incredible computing power that goes into AI is just going to wreak havoc on our environment. So

There's that too. Like it oh yeah, it might be able to help in these ways. But I encourage everyone to look it up. It's like every time you y go into Chat GPT and ask it to do something, it's like you traveling to France on an airplane, it is not great. Like the computing power required for AI is so massive and these servers take up so much energy that the carbon footprint of AI is Like we probably shouldn't just be using it for funsies because it is

not great for the environment if that's something you care about. Yeah. Like the the m the size and the amount of energy and electricity and water and everything that goes into uh these massive server farms. Exactly. Big problem. So the American Medical Association has encouraged this won't help with all of what we just talked about, but it is encouraging physicians to be transparent when they are personal using AI in patient care.

So you know, there's no mandate right now that your that your physician says Hey, so I came to this diagnosis and I was really helped by I AI. I didn't totally rely on AI, but I did put all your symptoms in and compared you to other patients and AI told me that you could have this kind of cancer. There's no requirement that physicians do that right now, but the AMA is saying that's probably a good practice.

Um because if patients are this worried about transparency, then if they find out later you did that, they might be pretty angry and it might create some distrust. So better on the front end to describe what you're doing with AI with your patients.

Have you ever had that experience? No. Has anybody ever done that? Absolutely not. Me either. Uh I did have a first experience with a student who submitted something to me and they said, um, Joe, just so you know I dictated this into Chat GPT and then had them had it like edit it for clarity or something. And then they provided like their original input, like their their dialogue as like a proof of this is actually what what really happened.

it was like twice as much stuff to read and get through, but it was really an interesting so I can totally see how a physician doing that um with a patient would be I I think it for a lot of people it would be really helpful, encouraging um like a good a good step, but other people would be like, Why are you telling me this? Like just

Do the thing. Yeah. Yeah. I can totally see that. For sure. I would be super interested in my physician saying, Hey, just so you know, we're using AI to read your MRI this month and here's you know what it told us. I'd be fascinated by that and really encouraged. about that kind of transparency, I can imagine other people being like, don't care. But then you don't have to keep bringing it up, right? If you're if somebody says to you, Oh, that that's interesting, but I'm not

you know, you don't have to say more about it. And I think that should be your prerogative, of course. But in my mind, better to be transparent on the front end and see kind of what people want m what more they might want to know. Mhm. Yeah. I wonder if it's something that you could like opt out of as well. Be like, All right, that's interesting, Doc. Don't ever do that again. I only want what's in your organic brain.

Right. I don't know how you could at this point because if your physician, you know I I guess the physician could just like not ask AI to try to read anything for you, but your data is already in the AI algorithm. So you can't opt out of that at this point. So you're basically then putting your data into the record, into this algorithm and not getting any of the benefits of that, which seems like kinda only hurting you. Yeah.

But but also I can see physicians or you know, any type of healthcare provider saying, uh you know, uh we used AI or this tool to do the diagnosis or to help read it and the p patient saying, Oh, well how does that work? Tell me more about that and then you're dear in the headlights, right? I mean you can't go into machine learning and like all the the details of it in the clinical encounter. That wouldn't be helpful. No, and actually I mean if if you people really want to start digging in

there's this black box problem, right? Our physicians are just not going to be able to understand, both because it's really complicated and it's not what they're trained to know, like exactly how these algorithms are working. They're also proprietary. So they won't know exactly how they're working. They might have kind of bare outlines and that might be enough for most people. Like here's generally what AI is doing.

But at some point it's gonna spit out a diagnosis and somebody's gonna wanna know exactly how it came to that, and the physician's not going to be able to say, Will that make people even more upset? or m more distrustful of that. I don't know. It's hard to say. Maybe we'll all get, like you said, so used to it that it won't matter anymore. But right now, in this transition period, I think it will upset people. Yeah. So the next

area of this, which is related, is um this area of accountability. When inevitably AI outstrips physician understandings of AI. How are you gonna run audits of like what the AI is doing? We know now from Chat GBT that AI tends to hallucinate certain things, right? Like it'll start saying stuff that's not true. Like we've all experienced that. So what if that happens here and how can we make sure the AI is accountable to like what it's doing?

Mm-hmm. Yeah. Whether it's able to explain exactly what it's doing. But also if there's a mistake or an error or a hallucination or an artifact that comes out, like who bears that responsibility of there's harm caused by Yeah. How can you really elicit informed consent, you know, to treat somebody and that treatment plan is based on what AI told you and you can't explain the AI?

Right, like what does that look like in terms of informed consent? How do you engage in that conversation? Again, this stuff is really complicated and it could be privately held information and so It's not that physicians are too stupid to understand AI. I mean they they might be, we all might be, but it's also that this stuff is privately held algorithms and so nobody can explain it.

perhaps not even the person who made the algorithm, because at some point this stuff is supposed to start making itself. improving itself. That's the whole point of AI, right? It's it's not just data and an algorithm. So how who's accountable then if something goes wrong? Yeah, and from like kind of a legalistic perspective, like whom do you sue, right?

I mean if there's a mistake and the mistake leads to harm, I mean somebody should in our system somebody ought to be held responsible for that. And if it's everybody shrugs and looks at the the computer computer screen and says, Well the the AI s the I A I did it even though it the result is a harm in somebody's life or death or Some types of other people.

injury based upon the the misinformation coming from from AI. I mean that happens all the time. I think the the error rate or something is, you know Probably not fifty percent, but the amount of incorrect or unverifiable information that comes back from public chat GPT profile that you can log into is

surprisingly high. And it's I mean, the re the reliability and the trust that we have in the AI I think is something that is only going to get more complicated. Mm-hmm. And as it gets better, it's gonna be harder to know that it is telling untruths, right?

So right now, that's maybe in my mind one of the benefits as a teacher that ChatGPT is not really great at pulling accurate information. Like that's not really what it was designed for. It's more about, you know, having an interesting conversation. So from my perspective, it's like great. So students can't rely on it and that's good for me for right now. But the better it gets, the less true that's gonna be. And so it's gonna be harder and harder to check.

Um but you're right about this legal stuff. So this is already happening. So there's these huge questions around if a generated diagnosis is incorrect, whose responsibility Is it the physician who relayed the information? Is it the programmer or the AI vendor that produced the AI algorithm? So it right now in the US.

Huge questions around this. And so far it looks like it might depend on and this is wild to me. The answers to these questions depend on who developed the algorithm and their pathway to FDA approval. What a terrible I Yeah, I'm sure s very smart people were involved in that decision but Tell me more. Yeah. So so there's been a couple of these lawsuits, um and they've gone a couple of different ways, but just to say like

the FDA is trying to figure this out right now, like trying to figure out how to have oversight over this stuff. Really, none of these lawsuits yet have produced more regulation. This is the theme. of this right now is like there have been lawsuits and it hasn't changed anything. Um I think eventually it'll have to, but this AI software is adaptive and self designing.

So like who could possibly be held responsible for it? I I just I don't know. And I don't think there's been any resolution to But presumably this will happen more and more in the future is like people'cause we have a like a lawsuit based society in America is like the only way you can get compensated for

something going wrong in the medical encounter is by suing. And so the more lawsuits that happen, the more we'll kind of get clarity on this. But right now the F DA has just sort of ruled as like, okay, well if you you took this pathway to your um approval, then either the physician or the programming vendor can be responsible. It's just a the wild

Mm-hmm. Yeah, and that's very uncomfortable to a lot of people, not just the the patients, but also the healthcare providers as well. I mean they're a very litigation shy group of professionals. Right. To begin with. Right. So it might uh quell some use of this if providers think that if they use it and it's wrong that they could be personally suited.

Because they're the ones who delivered the diagnosis. Which is kind of how it works right now, right? So the at the we usually say the buck stops with the physician. So they're the ones who are ultimately responsible for carrying out treatment plans. So they are responsible.

But man, the AI stuff makes that more complicated. Yeah. Yeah. It especially because it's it's unpredictable and unexplainable on the physician's part, right? I mean if they're using, you know, textbooks or they're using other types of online database. based material to inform their decisions, that's it's a different ball of wax.

Yeah. Okay, so the last kind of big bucket is concerns about bias. Okay. How can robots be biased? Well, because they're not. They're not biased. Unless they're created by humans. Oh. I guess. Yeah. So machine learning when it draws upon these large data sets to make predictions about diagnoses, risk management.

prognoses, all sorts of stuff. It's drawing on pre existing data, right? Mm-hmm. And in case anyone was unaware, this should not be a shocker, our healthcare system already has a ton of bias built in. So this is both kind of implicit biases on the parts of clinicians and healthcare providers. It's also like systematic bias against certain populations in the healthcare system as a whole. So if you have biased data going in, you should expect biased data coming out.

That makes sense. Tell me more about what are some examples of bias information going in that results in bias information coming out. Yeah, so like right now we know that racial and ethnic minorities are much less likely to enroll in human subjects research. Right. So because of a long legacy of um American researchers and you know globally too, but in our context, there's been a lot of really terrible human subjects abuse.

And so many minority populations do not want to enter into human subjects research. And so if you don't have their data and then you're generating outcomes, it might overlook this population. Or if you have um folks who are less engaged in the healthcare system, which is also true of certain populations in our country.

So some people are much more reluctant to go to the doctor, which means that we don't have as much data about them. Or it might look like certain populations are much more likely to die of certain diseases, but that's because they didn't engage the healthcare system until way further down in their disease progression.

And so you're getting all this data that's not giving a full picture of certain populations. And so we shouldn't then be surprised when the algorithms give us information that doesn't well represent certain populations. Mm-hmm. Yeah, that makes sense. So the data coming out of the the AI tool um is only and and this is like so obvious I think that it maybe

goes without saying, but it's not a good so AI is not a good predictor of future information. It's a kind of a l a retrospective. It's a look back at what we've we've put into it. And if we are baking into picking into the cake all of the the the problems and issues and systemic and historical bad things that have happened. Uh the the AI algorithm doesn't know what input is valuable or or or not valuable or harmful or not harmful. It just looks at data as being data. Mm-hmm.

Yeah, it's in I mean, somebody described this to me as like, it's because it's so entirely backward looking and medicine is built off this idea of progress, there's a misalignment here. So it can only sort of replicate stuff that's happened in the past and like predict things based on the past.

But we're always wanting to like innovate and and make things better in the future. And so AI and innovation don't necessarily go together really well if what we're doing is mostly combing data from patient profiles. Mm-hmm. Um but there are already some studies that show that AI does a bad job at assigning risk level to certain racial populations. So just if it again is using that data is already sort of like racism is baked into the system, it's not good at predicting risk for future.

So there there is an issue here for sure. Yeah. Yeah, that's interesting the the misalignment of kind of the of the ideals of the two kind of separate systems. I haven't thought about that for. That's really interesting.

Because I think that if we as a healthcare field, healthcare industry um were to only say we're just going to do what we've already done really, really well and not progress, then I mean that seems opposite of the inclination that um people go into medicine and research in order to accomplish.

Yeah, we'll see what ends up happening, but there are some worries already that like this AI might actually stifle innovation rather than produce it. Okay. So Giant buckets were damage to the relationship between the provider and patient, privacy, transparency and informed consent in medicine, accountability, like who is responsible for the AI, and then bias. Is there anything that you feel like I've kind of missed here or that you've heard around AI that we should

also be worried about. I think the only thing that I hear in addition to that list is the the generational divide within the practice the field of healthcare. Uh there there are folks and it tends to be people who are newer in their career or just coming out of training who are all for it, you know, full full steam ahead. And I think that's part of being kind of digital native. natives and and being more comfortable with technology. And then

people who are towards the end of their career. I've also noticed them being more engaged as well, uh, with AI. People who are maybe in the sunset of their career thinking, Well, this is a new fun tool, let's let let's really kind of embrace it and try to

use it as much as possible. But then you have like the people who are in the the meat of their career and, you know, maybe been out of training for ten or fifteen years, but have another ten or fifteen or twenty years before they're looking at retiring and really resistant to the disruption that it's that is happening within that AI is causing within the healthcare system and and with their patients and and

kind of also these bigger questions about research. So lots of generational differences. Mm-hmm. Yeah, that's super interesting. But it would be like the young folks and then the older folks who are more like, Let's try it and then the folks in the middle who are like, Ugh, I don't know. My job's hard enough Yeah.

But we'll see. I mean the only thing that we can can f you know accurately predict is that w number one, we're gonna be wrong about whatever we predict and number two, it's gonna be it's gonna change and it's gonna be uh interesting to watch. Yeah, I mean I bet in a year we'll re-listen to this podcast and be like, what naive little doves we were. Oh sweet.

Sweet child of summer. Um th that would presume that we go back and listen to any of these podcasts though, which I uh our avid listeners are going back and listening to the best of the best. Oh man. Yeah. I can't even listen to myself so once so. All right. Well, so in the next few weeks we'll have other people talking about AI, other aspects. People who are far more experts than me.

Uh and so I think we'll learn a lot more. Yeah. The one good thing that we could bring to the table, Devin, is we've got really smart friends and we're gonna bring them on and help us talk about it. Absolutely. All right, until then. All right, bye. Bye. Thanks for tuning in to this episode of Bioethics for the If you're into what we're doing,