External Data in Clinical Trials

Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in.

All right, welcome. We are back. Uh, in the interim, this is Barry Consultants podcast of all things statistical, all things scientific with clinical trials, uh, medical decision making, uh, and. We have an interesting podcast today in that I don't know the topic, so my, my co-host on in the interim, uh, Kurt Veley is here and he has a surprise topic to talk about today. Kurt.

Hey Scott. Um, alright, so I wanna start with a story. I went to a conference not that long ago, and standard statistical conference. People are talking about the data they had, all the methods that they've been developing to. Understand it, what they've learned from the data, and essentially all of these. Talks.

well, well let's, let's, let's sort of figure out the topic and the interesting part of this. So what, when you say is data good or bad? Do you mean that When I design an experiment, I'm saying I want to collect the following data and I can analyze just the data in that experiment, which we've been doing for a long time. Uh, frequentist, uh, rarely Bayesian, but we, we do that. You're saying by data, is that outside the experiment? Is there any room in that experiment? For external,

Yes. So what? What do we do with essentially the totality of human knowledge? Prior to the experiment, should that enter into the experiment in any way and wanna avoid this? I mean, there's lots of rhetoric we could do with this. If you want to, if you want to say bad things about this, you talk about biases and confounding and everything else. If you wanna say good things, you talk about totality, the evidence, but really let's get at, you know, what is gonna lead us in good and bad directions?

Yeah. So let's, let's, let's lay out why not. Why? I mean, why, what are the reasons people give that we shouldn't use other data in our experiment?

Obvious answer would be. It, it could lead us in the wrong direction. So the, uh, experiment might have been done with a different set of patients at a different time. There are key differences between that old data and what my current experiment is. If I use the new data or use the old data in my new experiment. I can get, uh, answers that are biased in certain ways. I can draw wrong conclusions.

So is this, is this a frequentist issue? Is this that we calculate the operating characteristics of the new experiment? Type one error is only the new experiment. And if I use any data outside of my experiment, you can inflate type one error. You can get bias, all these bad terms, uh, uh, in it. Is it a frequentist problem that data outside the experiment all of a sudden type one error means something very different?

So I don't know if it's a, I think there is a frequent DYS Bayesian divide. I think Bayesians are trained with the idea of collect some data, update your beliefs, collect more data, update your beliefs, and it's a natural thing to do. I. To bring things in, but I don't know if, if I were to say that the problem is not being a frequentist in terms of our, do you care about long-term error rates? I would lay the problem at the feet of, you have to have this 2.5 type one error.

Yeah. Uh, so every time, so let, let's, let's talk about a case where we would use external data, and we have, uh, we've designed trials where we've used external data and what that looks like and what could be the potential problem. So, um, suppose. I've got results from another trial, a phase two trial about the relative efficacy of a treatment, and I bring it into my next experiment. And I use that as prior knowledge. And when I'm done with my new experiment, I combine them together.

are certainly, I mean, there are cases where you are gonna run into those problems. I mean, we've talked in examples and antibiotics. The development of resistance over time, we expect therapies to change. There are situations where we think that problem exists and we need to address it or not use the data. There are also cases where diseases are very stable over time, and those issues may not be as big a concern.

Yeah, so I, I mean, I'm, I, I, I'm very much on the side that we should be using external data. I, I, uh, I think we go about this in a way that's just so strikingly conservative that it slows us down, uh, in it, that, that. That, you know, something may happen.

So I, it's an interesting question 'cause it certainly can be wrong on occasion. So I, I wonder if as a society and you, you ask about frequentist and whether the error rates are driving this, I think you can be a, a perfectly good and wonderful frequentist and use historical data, but what you're gonna have to accept is most of the time prior data is leading you in the right direction. If it's not, we should just see science altogether. Because we have worse problems.

Yeah, it's a, and I'm struck by, I, I'm struck by the idea of using that data and in and in Bayesian we do

I.

borrowing of that information. We do things that are dynamic. I'm struck by, there are tons of trials that we use, objective performance criteria. That it's a single arm trial and the drug has to jump a particular hurdle. That's a number. That number's always based on prior data, and we do that all the time, and presumably that's okay. Or you run a trial where your only, your data comes from the new experiment, and that's your control.

so just, just to interrupt you there, I mean, we go to the FDAA hundred times a year and have three major disagreements. That's much better than with our wives, so.

Yeah. Yeah. So we provide an example that we wanted to go in with a new therapy. It was oncology and we were, we, the, the new trial. We didn't want to enroll one to one, to a standard of care. Uh, it's easier to get patients in if they're more likely to get experimental treatment. Uh, we wanted to enroll three to one, three to experimental, one to control. There's a good bit of information about the behavior of the control.

It's interesting, it's almost you have to cross the Rubicon of what are you willing to completely make this assumption or not make this assumption at all? And you can't test the assumption.

Right. Right. And the other extreme of this, and I know, I know, I know this one, uh, gets at you a little

Don't do it. Don't do it, Scott.

Yeah. That we talk about using real world evidence all the time. Digital twins or real world evidence. So for, for control, we bring in external data. And people seem to be almost comfortable that your control is entirely external data, that's okay. Um, but if you were to do something like use non concurrent controls from your own experiment, that seems to be unacceptable despite the fact that these are unbelievably phenomenal, uh, historical controls.

Yep. The, um, I mean, it, it's, there are different legal reasons that real world evidence is being handled in certain ways right now compared to other. Um, but, you know, FDA is not a, um, it's not a homogeneous organization. There are lots of people, the academic community is a, is many people, industry is many people. Um, but I think we're gonna, if we're talking about where we're going, one of the key aspects of this is a hierarchy of what's more or less dangerous.

Yeah, and, and you, you mentioned a key point to that if you're enrolling some new controls, you actually get a comparison. You get the, you get the controls in your experiment against the previous controls, and you can judge some level of, of similarity between those and, and what's become okay.

Why don't, why don't you talk about that a little more? About just the details? 'cause I think there's often a sense of. the fact that there is a hierarchy between a single arm trial, dynamic borrowing, static borrowing, get an idea about what those methods are and how they mitigate those risks

Yeah, yeah, yeah. So, so the extremes on this, and I can run an experiment where I only include data in my new experiment, uh, one-to-one randomized control to treatment. And I'm only gonna compare those two arms. Uh, in that I could do the exact ex, uh, very much extreme from that, where I enroll only my treatment arm. And I, I want to know, is the treatment arm behaving, you know. Better than, or is the treatment benefiting these patients?

And I think, you know, that's really what's going on in the academic literature right now in terms of trying to figure out the best ways to make that assessment, minimize the risks, amplify the benefits. Um, we're talking now more about covariate matching between studies. I. So not just looking at all the historical data, but which patients best match our current enrollment.

Yeah, and, and part of this is it. Part of it is statisticians are really smart and they can figure out how something can go wrong. That this, the, the, the historical data's not the same as the new controls we've got, uh, beha, we've got problems with the analysis there. We've got potential type one error inflation. We've got potential bias in the estimate of the treatment effect. And statisticians can figure this out and say that, you know, this is bad.

Well, we basically, it's like data. We design an experiment, data comes into existence. There is this bright shining moment of about five minutes where we can do something with it. And then we're meant to put it aside, at least with regard to future experiments. Obviously people are gonna reference it in deciding treatments and so forth in the medium, but it, it's interesting, it worries me as a statistician, I feel deep-seated guilt actually putting data aside.

Yeah. Yeah. Uh, and, and, and again, it's hard, uh, but there's, there's huge ramifications to not doing it, uh, in that ways. And I think there are ways of doing it very well. You can be prospective in your new experiment. The analysis plan is completely written down, written down. It's explicit. When you present the results you, you show the results. Just in the experiment, you show what happens using the modeling of external data. I think we can do this really well.

Well, it also, it, it has tremendous implications towards design because if you can't borrow, suddenly sponsors have an incredible incentive to claim that their patient population is homogeneous so that they can pull everything together and maximize their power.

Yeah.

you say, you know, as soon as you say there's two different groups of people, you have to run twice as many patients, no one's gonna wanna explore those kind of questions.

yeah, so it's the same bizarre thing that happens, that we run trials that are single arm trials where we use a historical control a hundred percent, or we don't use it at all. And you're describing exactly what happens in trials. We run a trial and you do a common estimate across the entire population, a single estimate, which is this, which is full pooling, it's exact same mathematically of pooling populations altogether to get a single answer.

So let me ask you about one other thing, which I think is a real oddball in this conversation. Um, there is a situation where we routinely combine data across different studies, and it's often viewed as the pinnacle of evidence for clinical trials, which is a

I, uh, I have no idea. I have no, oh, meta-analysis. Okay. Yep.

where do you think meta-analysis fits in, in terms of the, the assumptions and how it fits into this debate?

Yeah, I, I, I think meta-analyses are incredibly valuable. Um, the modeling makes a ton of sense and incredibly important. Um, I, I think people overweight single trials and the, the totality of evidence. Now, not every meta-analysis is every meta-analysis. Uh, there's, and, and, and I think people, I sort of think meta-analyses are bad because there are some that are clearly biased. There's publication bias, there's, there's availability of data bias.

is two is two 0.49s as good as a 0.051 and a 0.001 on the P-values. Clearly you'd want the latter.

Yeah. Yeah. And, and, and so I think of a meta-analysis the same way I, I, I would do a meta-analysis of my two phase three trials for what, what does it tell me to the answer? So it, you know, those techniques, they tend to be Bayesian. But, uh, largely the availability of data. And there are some where we know there's missing data, we know there's issues with it, and we would not take those at. Uh, you know, at, at a high scientific degree to that.

We would certainly say that on any kind of borrowing, bringing together data that doesn't belong is bad. You can borrow in a way that does generate horrible biases and will generate bad conclusions. Uh, but we also are talking about what can be done well with experience. Um, we're running outta time. Why don't you close us out?

Uh, so, uh, I, I think this by the way, is the future. I think as we learn more and more about diseases, we get more and more subsets of them. Um, we are getting more and more quality data available from other clinical trials. The sharing of data from clinical trials, the sharing of other types of resources, we're gonna have explosion with medical data. The idea that we don't use that in our new experiments, if we don't use that in valuable ways, we're running worse experiments.