Welcome back to the Stay On My Operating Table podcast with Dr. Philip Ovadia. I'm your host, Jack Heald. Dr. O., before we go into the subject of today's meeting, I would like to say congratulations. I got notification that this podcast has hit 5000 downloads. And that's pretty impressive in the short amount of time that you've been. You've been doing this.
Thank you, I was very excited to see that. And we're recording this a few days before Thanksgiving. And I just want to say how thankful I am for everyone that has been tuning in everyone that has been listening, all the great feedback that we've been getting, and just glad that we're able to get useful information in the hands, so to speak, I guess the minds of the audience. And I think today's topic is going to be quite useful information for people to understand.
Useful, you know, that's one of the very best things about the things that you've been teaching me is, is this is not all theory there. It's very practical things that I can do. And my wife, as she's listened to these things, we do them. She's actually in the office with me right now. She's changed how she eats. So you're making a difference out there. Alright, so let's talk about this article you sent me. This is I think a preprint that you
sent Me, I'm not sure. But the title of the article is how to survive the medical misinformation mess. And it's a, I guess, a research report by the main main researcher was John Irenaeus, which is, I guess, fairly well known in your world. So Edie Stewart Brownlee and strike, how to survive the medical misinformation mess. I read the whole thing. And I was, I was horrified. I'm a layman. I would like you to set this thing up for our audience. And then let's go through it.
Sure thing. And so the copy I had sent you was the preprint. But it's important to point out that this was published in peer reviewed this article, as you said, in titled How to Survive the medical misinformation mess was published in the European Journal of Clinical Investigation in November of 2017. And I think as we go through the discussion, people should keep this in mind that
this was published in 2017. You know, before the current medical misinformation mess that we find ourselves in, but professor in netus, you know, has been on the forefront of this topic. And I think, you know, as I said, this paper is very timely for us to be discussing today/
Let me just give the headlines and then - you're the expert. So let's do that first. The gist of the article is that there is a tremendous amount of medical misinformation floating around, not just amongst the lay people like myself, but also in the medical community itself. And the authors list four key problems of medical misinformation. Those four are and I'm just going to read them.
Much published medical research is not reliable, or is uncertain reliability, offers no benefits to patients, or is not useful to decision makers. I mean, that all by itself, I just my head, it made my head explode as we used to say, and that's just number one. Number two, most healthcare professionals are not aware of this problem. Number three, even if they are aware of this problem, most healthcare professionals lack the skills necessary to evaluate the reliability and usefulness of
medical evidence. That was an eye opener, and number four, patients and families frequently lack relevant accurate medical evidence and skilled guidance at the time of medical decision making. Wow. This feels like it should be like a 12 part Document. entry, but we're just going to do at least one podcast here. Open it up for us Doctor Oh, number one, much medical research is not reliable, or is uncertain reliability offers no benefit to patients or is not useful to decision makers.
Yeah, so, you know, to fill in a little bit of the context here, you know, that I want people to understand the challenge that faces physicians these days, you know, every physician goes through their education go through medical school, in their training. And, and one of the common quips, you know, that you hear oftentimes as you're going through medical school, is that half of the information that you learn in medical school is going to be proven wrong, by the time you
retire. We just don't know which half and you know, every doctor wants to stay up to date on the latest, you know, developments on you know, the best way to treat their patients. But they're faced with a kind of
overwhelming problem. And, you know, in the beginning part of this paper, they kind of outline that problem, that when we go through PubMed, which is the kind of database of record of the published medical literature, there are currently, and this, again, was four years ago that this paper was written, but at the time that paper was written, they stated that there were 17 million articles in PubMed, that were tagged as
human. And there were over 700,000 articles that are defined as clinical trials, as well as over 1.8 million review articles. So, you know, obviously, no physician can keep up with, you know, that volume of literature. And the problem that, you know, problem number one that they outlined in this paper is that much of that medical literature is not
reliable. And, you know, I have seen statistics that show that anywhere between, you know, 20, and 50%, of medical recommendations, is inappropriate, or medical services, I should say, that are delivered in the United States can be inappropriate at times,
and may even harm patients. And, you know, they talk about in this paper, how, when researchers have gone through this sort of grade studies, and determine, you know, what studies actually meet the criteria for being, you know, good medical evidence, as few as 5% of them. So, as few as 5% of the published studies that doctors are relying on to, you know, help guide their treatment, as few as 5% of them
are actually good studies. And, Lord, that is a big problem for, you know, us as practicing physicians, because, honestly, most of most physicians don't have the time, or, you know, as we'll talk about later, as we go through this paper, don't have the skills to properly determine what studies are useful and what studies are not.
So, this, this is essentially a study of studies. And the first the first conclusion they came up with came to at least the person when they highlight here is that 95 out of 100 of these studies are deeply flawed. Is a is that did I read that? Right? That's yeah, what I took away from it.
Yes, that's exactly right. You know, so less than one out of every 20 papers that a physician, you know, ends up looking at is going to be actually a, you know, good quality study that should impact how do they take care of their patients?
So, I mean, this honestly, this problem is, the way they describe this problem is so it was distressing to me.
Yeah, it's
I I'm, I'm, I really been looking forward to talking to you about this since you sent it to me, but I I'm just, I'm, frankly, I'm lost. I'm so I mean, I knew things were i i knew things weren't great, but this is this makes a case that I don't want to overstate it You're on the inside. Tell me what what the view from the inside is? As it relates to this 5% of studies, if they said 5%? Are what's the word that they used, purer than fewer than 5%? past a validity screening for an
evidence based journal? What does that mean? Help me out.
Yeah, so what they did in the study that, you know, this paper is referencing is, basically they had eight criteria that they felt were important to judge whether a, a piece of clinical research was, you know, useful was going to be
useful. And real quickly, you know, those eight problems, they say, were that, first of all, it requires the existence of a real problem to address that is then placed in the proper context, that there is sufficient information from the research, the research should be patient centered, it should be pragmatic, it should provide reasonable value for money, it should not be futile, and that the research should be
transparent. So, you know, someone listening to that, I think, certainly, you know, myself, as a physician, would say, those seem like, pretty good criteria for, you know, what I should then translate into the care of my patients. But again, when they looked at over 60,000 studies that had been published, fewer than 5% of those, basically, could, could meet at least six of the eight criteria that they that they
just outlined. So basically, you know, again, one out of 20 studies is actually going to be useful. But these all balled 20 ended up getting published. And, you know, as, as a physician who's trying to stay up to date, and he's flipping through his, you know, he's reading his journals every month, which, you know, is a very time consuming
effort. And, you know, basically, you can imagine that, I would say, the average journal, you know, the ones that I read, that say, you know, contain somewhere between 30 to 40. Public, you know, 30 to 40 studies are in each issue of
that journal. And not all of them are going to be, you know, clinical research, but, you know, there, the chances are in each journal that I go through, there may be one actual useful, you know, high quality study, and oftentimes, there isn't going to be any useful high quality studies, you know, in that entire issue of a journal. So, this is a big problem for practicing physicians. And as we're going to talk about in the next problem, most physicians
aren't even aware of that. They assume that, because that study made it into the journal, it's been properly vetted by the author by the editors of the journal, right, so that it is a high quality study, and we should pay attention to it. But as we're going to discuss as we move along, that oftentimes is not the case.
Well, you know, this may be a little bit of inside baseball here, but - maybe not as an outsider, as just a lay person who's a consumer of medical services - I think I'm not unusual. I assume, by and large, my physicians are using the best information available. And I understand that it might
not be right. But I've always assumed that they're using the best information available and that there is a system in place a series of processes that have been refined over - I just would have assumed - hundreds of
years. So that the the beliefs, the techniques, the methods, the treatments that the doctors are using and and recommending have been vetted for these six or eight things that you just listed, that somebody has gone through and said "oh yeah, these things are 1. placed in the proper context 2. solves a real problem as sufficient information gained 3. is centered on the patient's well being 4. is pragmatic 5. has a reasonable value for money 6. isn't futile and oh, by the way,
and 7. our methodology for evaluating this is transparent. We're not hiding anything. I've always assumed that that's going on. And if it wasn't, I would have assumed a reasonably competent doctor would know that. But it sounds like I'm wrong. And I don't want to sound Pollyanna here, but this is - it's a little alarming. It's very, very alarming.
Yeah, I agree with you. I think it is alarming. And, you know, the first part of your statement was that physicians are using the best available tools, you know, resources that they have. And I think that's probably true. These are the best available resources, you know, the, the medical journals, especially, you know, what are considered the sort of high impact, you know, medical journals with the best, you know, reputations behind them, probably are the best tools that we have
available to us. In keeping up with new developments in medicine, unfortunately, I think what this paper is showing is that those tools, despite being the best ones available to us are not very good tools. And, you know, this, I think, is another important piece of what we've been discussing in this podcast. about the ways that the system is broken, the health care system is broken.
This particular episode is the least optimistic one I think we've done. And I don't like that, because I am by nature, an optimist. So I see a problem. And the first thing I want to do is, is say, Okay, well, how do we solve this? But maybe it's too early, maybe we need to really settle into what the full extent of the problem
is. But if you've got ideas here, for how an average Joe like me can navigate inside a medical system that we're suddenly finding out is deeply unreliable, I would sure like to hear.
Yeah, I think, you know, as we go through this discussion, hopefully, we'll come up with some, you know, possible solutions. And towards the end of this paper, they kind of talk about, you know, maybe some possibilities, I think the most important thing is something that we, you know, we've talked about on this podcast before, and that is that people do need to have a healthy skepticism, as they, you know, approach the medical system. Yeah. And as they, you know, discuss things
with their physicians. They need to, as I've advocated, you know, they need to be proactive about their health. Yeah, they need to be proactive about, you know, talking to their physicians, about the kind of data that's behind the, you know, whatever treatment it might be, that their physician is, is advocating for them.
And that reminds me of all these, these ads on TV from the pharmaceutical companies, where they will usually say talk to your doctor, bla, bla, bla, bla bla. And with the implication being that your doctor is going to be an expert. And, and what I'm seeing reported here is, not only is your doctor not an expert, which is bad enough, your doctor probably doesn't know he's not an expert. He thinks he's got
good information. And in fact, the information he's got is probably at best, of very low quality and quite possibly wrong. Ah, okay. Problem. Yeah, go ahead. Lead.
I was gonna say I think that leads us into problem two very well.
Problem two, most healthcare professionals are not aware of this problem of low value publications, low quality publications. Why? I mean, obviously time, but that's not new. You know, For as long as we've had human beings, we've had 168 hours a week. So what's the deal?
Yeah, I think, you know, this echoes a problem that's, you know, common throughout society today. So, you know, many of us, sort of grew up and still exist today, I would say, you know, hearing that, for instance, the New York Times is the, you know, the newspaper of record. And, you know, oftentimes, we see that that's not the case, and that, you know, their reporting can be
inaccurate. And in the same way, in the healthcare industry, you know, as physicians, we are taught that, you know, the medical journals are sort of the, you know, the record, and they need to be trusted. And, you know, certain journals, you know, we do have sort of ranking systems for journals, I guess, I would say, and I think many physicians, you know, sort of know which journals are the most reputable ones, and which journals might not be as
reputable, right. But even within that hierarchy, we see problems. There was a very, you know, a recent example, where one of the probably, I would say, the world's most preeminent Medical Journal, The Lancet, published a study that was shown to be completely falsified, you know, shortly after it was published. And, you know, you have to really start to question when you see something like that, as a physician, hopefully, you start to question, you know,
what is going on here? You know, how did a study such as this, that was, you know, felt to be a very important contribution to the, you know, medical literature on a hot topic? How did that make it through the whole review process that these journals have in place? When it was, you know, not a little bit wrong, it was found to be a completely fabricated study, the study, yes, the data behind it was found to be fabricated. And somehow this made it through, you know, the review process.
And, you know, some very astute clinicians, you know, looked at the data that was published in that study, and something just didn't seem right. And they started poking around. And ultimately, as I said, it was determined that the data behind the study was was fabricated. But, you know, that made it into the top medical journal in the world. And I would say, there's a pretty good chance that if I surveyed 100, random physicians, most of them would not even be aware that this had happened.
So, you know, this is the problem that we're running into. They talk about, you know, again, in the paper, they talk about journal reading habits of physicians. And, importantly, you know, the study that they reference looked at physicians who had kind of gone through an extra training program, it's called the Robert Wood Johnson,
Clinical Scholars Program. So these are physicians who, you know, had gone above and beyond just their usual medical school and training to really, you know, understand how to interpret the medical literature. And in that study, most of the physicians admitted that they usually didn't read, you know, the full article, that they oftentimes will rely on just the abstract of the articles as they're going through things, and that they relied on the editors of the journals to assure, you know,
the study quality. And as we talked about, you know, in the last section, that trust in the editors of the journal is probably, you know, misplaced. Wow.
Okay, so, I remember back when I was a programmer, I subscribed to a journal that was specifically for folks who who use the particular tool that I used, and I eagerly look forward to that because it would typically almost every issue would have a problem that I had run into, and a solution and you know, the cool thing about about programming about working with a computer if somebody says I here's a problem, and here's a solution, you can do it
yourself. And you can see yourself, yes, this solution resolved that particular problem. So, so getting those those journals was, it was a big deal to me. And I generally read every word of every article that had any interest to me whatsoever. But these were not huge journals, they would have six, maybe eight articles per journal per month. I've just always assumed that doctors do
the same thing. As a programmer, I couldn't just read the summary, what would what would be the abstract, because that wouldn't give me enough information. I couldn't actually go and prove it myself. And yet this study says, that's not how most of these well trained. Physicians read. Okay. It's too early in the morning for me to
be this this wound up. I was I was looking at something that you highlighted in this, in this article you sent to me here under problem two, it said nearly half of the abstracts of randomized controlled trials contained biased reporting of study results, implying benefit when there was no significant significant, no statistically significant difference in the primary endpoint between study arms. Would you would you explain that in English?
Yeah, basically, what this is saying, and I will admit, I find this to be true is that, you know, when you look at the abstract, the published, you know, kind of summary of an article, it oftentimes will word things in such a way that imply, you know, that there might be some finding some effect of whatever they were studying, that isn't actually supported by the data
within that study. And, you know, this, again, is something I oftentimes, you'll see, as I'm going through the medical literature, around nutrition, or around heart disease, and some of the, you know, interventions around heart disease, where, you know, the paper just won't show and, you know, whatever, study, whatever the author's, you know, of the, of the study, we're trying to show, the data might not support that, but in an effort to basically, you know, make the paper look more
important, get it published, they will kind of word things in such a way that imply that there is an effect when there isn't one. And, you know, this becomes a problem when, if you're only reading the abstract of a paper, and you might not realize that, and then what happens is, and they go into this as well, this problem can then get magnified.
Because one of the tools that's oftentimes used in medicine is his, what's called a review paper, a systemic systematic review paper, okay, where you know, someone is interested in a topic. And so they'll go back through all the studies that have been published on this topic, and they'll try and combine them and summarize them. And, you know, see what the
effect is. Because sometimes, you know, one of the issues we run into in medicine, is running large trials, you know, that are necessary to show some of these benefits may not be possible. So sometimes we only have a bunch of small studies. And, you know, you're trying to combine those that try and figure out the effects of certain things. But oftentimes, you know, the authors of those review papers will only be looking at the abstract of the papers that they
are reviewing. And they'll say, this paper showed this effect based on what the abstract says, and it may not show that effect. So then you start to combine these things and these problems get magnified even more. Wow. And again, it's a it's a huge, it's a huge problem in healthcare, in the healthcare
literature. And I think it just goes back to the fact that you know, physicians oftentimes don't have the time necessary to properly go through literature, you know, physicians don't get paid for reading medical literature, they get paid for
taking care of patients. And, you know, reading the medical literature to be able to do that effectively, is something that oftentimes they then have to, you know, do on their own, after hours, and all that, so, and then there's just such a high volume of medical literature published these days, that, you know, most physicians only have time to skim through some abstracts. And, and, you know, maybe there'll be one or two articles that they're particularly interested in that
they'll read in more detail. But for the most part, they don't.
Ah, you know, our conversations have been... disruptive. They've delivered information that has allowed me to make decisions primarily for me about what I put in my mouth, but also about exercise as well. You know, it's given me concrete things I can do. This paper, the effect of reading this paper on me, has been to become deeply distrustful of all medical information. Now, that may be an
overreaction. But if if only five out of 100 studies pass these, these high quality standards, then it's not an I don't think it's unreasonable of me to, to assume that 95% of what I'm hearing is based on bad bad studies, poorly, poorly structured, poorly written or worse, falsified results. So
You know, honestly, I think that was the what the authors of this paper were trying to do, you know, I think that they are trying to get physicians to have that same attitude. Because honestly, that is the attitude that we need to have, as physicians, we should be largely mistrustful of the information that's being put out
in front of us. And by doing so, it's going to allow us to figure out, you know, what is the information that we should be trusting, because there is that one in 20 paper that is actually, you know, useful information that we should then put into practice. But the only way we're going to figure that out, is if we are mistrustful going into this process. And we are skeptical about every piece of, you know, research that we
are reading. And we go through this exercise to figure out, you know, what are the useful pieces of information. And ultimately, I think the hope of these authors is that if more and more physicians are doing that, then the journals are going to kind of respond to that. And you know, they are also going to become more responsible about the, the literature that they allow to be published.
Well, I'll tell you what we've we've, we're 30 minutes into here and only halfway through this, why don't we? Why don't we just say part one, and part two is coming up in the next episode. So y'all tune in for the next one. Is that okay with you? That sounds great. All right. Well, for Dr. Philip oveja on Jack Heald. This is the stay off my operating table podcast. We're gonna finish this conversation on the next episode.