Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in.
All right. Welcome everybody. Welcome back to, in the Interim. I am joined today by somebody you're all familiar with, Dr. Kurt Veley. And he helps me, ho co-host some of, in the interim broad, broadcast. Today we are gonna talk about, uh, time travel. maybe not quite time travel, but statistical time travel. We are gonna talk about the time machine and its relevance to platform trials. we'll sneak in a little sports, uh, all in this. So welcome back to in the interim.
Thank you very much. Good to see you, Scott.
Ready to see you. Alright, so the topic of the time machine, and we'll call it the time machine. we, we have called it that and we'll sort of describe why, we'll describe the relevance of this topic and why it amazingly has lots of people. That get very flustered or frustrated by it. And, um, we'll, we'll talk about that. So let's introduce the time machine.
Well, so let's, um, yeah. let's let's go back in time speaking of that too, cause you've been working on, you've been working on this since the nineties, something like that. When all said and done.
Yeah. So let's go back in time. And it's interesting how this, I remember, and many of you have heard some of the episodes we do about sports and drug developers in sports. And as a kid, my father Don, we were talking about the old and the old, pub fodder is the best term I have for this. Sports, sports enthusiasts will argue endlessly about the different talent level of players at different eras in sports.
And if you watch the MLB All-Star game, which was very recent here, they did a really nice, Look back at Hank Aaron hitting his 715th home run and he beat Babe Ruth in the total nu the, the, the, the career home run record. And it was a big deal at the time. Of course, they played at very di very different times. Even though Hank Aaron played for an incredibly long time, their careers never overlapped. And so questions about who's a better home run hit.
Her, and you could have those questions about today, who's a better home run hitter? And Don said something to me that stuck a little bit. He said, if you take two players like Babe Ruth and Hank Aaron, they never played together, but they played with players who played together. So back in the nineties, the, this I wanted to. To put this into actual algorithms. And so this fantastic thing in sports where Babe Ruth never played with the great home run hitters of today.
Aaron Judge to New York Yankee, wonderful home run hitters. They never played together, but Babe Ruth's career. Overlapped, Lou Gehrig's career who overlapped. Jimmy Fox's career overlapped Ted Williams' career. Ted Williams' career overlaps Mickey Mantle overlaps. Probably Hank Aaron at that point who, who played for quite a long time. So they, there's this bridging, if you will, of Babe Ruth to Hank Aaron, and even to Aaron Judge today. So they never played at the same time.
But this overlap, you could estimate the relative home run ability of Babe Ruth to Jimmy Fox. And Jimmy Fox. To Ted Williams and Ted Williams. To Mickey Mantle and Mickey Mantle. To Hank Aaron. You could estimate all these players by simultaneously estimating. The effects of age, I'm sorry, the effects of era age is a different thing will come to, but the effect of the era in which they played and era matters in home runs.
There was a dead ball era that different effect, pitchers varied over time. ERA matters in which you played, but you can always estimate the relative skill of players that played at different times. So I was involved in an effort with Pat Larkey and Shane Reese, where we wrote a paper that we did exactly this, and we did it in baseball. We did it in scoring in the NHL and golf. And in many ways, golf is probably the simplest of those because there's only one goal.
In golf, there's really only one stat that matters. How many strokes did it take to play 18 holes? And you have exactly the same setup. If you wanted to compare Tiger Woods to Jack Nicholas or Scotty Scheffler, to Jack Nicholas or Ben Hogan, Tom Watson, you have this incredible overlap where they played in the same tournament at the same time. In the NHL, we have the same thing where they played in the game at the same time.
And you can estimate relative performance of players bridging across this. And then we estimated this relative efficacy and we estimated the effects of error, putting everybody by, adjusting by earth, putting everybody on exactly the same scale and estimating. Comparing these apples to apples on exactly the same scale. So you could compare Hank Aaron to Babe Ruth. You could move Hank Aaron from 1973 to 1927 and say, what would he have done Now?
We called it a time machine because we weren't interested in. Suppose Hank Aaron would've grown up in 1920. Now he wouldn't have played in the major leagues because he was black and he couldn't have played. suppose it was the nutrition of the time and grew up at that. We weren't interested in that. We were interested in a time machine moving Jack Nicholas to 2025 as a 25-year-old. Superstar, what would he do compared to Scotty Scheffler now or Tiger Woods in, in 2001, moving them in time.
So that was the paper and that was the effort by the way, that was published in. Journal of the American Statistical Association. It was actually picked as the, applications and case studies paper to present at the joint statistical meetings. That paper, that effort. So that was the introduction to a statistical time machine, Kurt.
so who did you say was the best?
Yeah. Yeah. which everybody wants to argue about all of that and so Fantastic. interestingly, in baseball we did home runs and batting average, within that, and it, we did this in the mid 1990s and, if I remember, it's been a while. I think Mark McGuire was the best home run hitter, of all time in that, the best golfer of all time at that point, I believe was still Jack Nicholas, but that was pre Tiger Woods. Recent updates of that, tiger Woods is the best.
within that, the NHL was fantastic in terms of scoring, because there's amazing things when gr, when Wayne Gretzky played. He played in an era where the goal scoring was much higher. He was still the greatest scorer of all time, but it's not nearly what I expected. His numbers dwarf everybody's, but he played in an era and so these are all the fantastic things to talk about.
By the way, as we start to move this to clinical trials and we'll, what we'll figure out, why would anybody care about a time machine in clinical trials? Seems like such an odd thing. It's much more challenging in sports because players over that time are at different ages an age. The age of the participants certainly matters. within that, Jack Nicholas at 55 was not Jack Nicholas at 30, so we had to model.
How players age within the different sports, and it's different in golf and baseball and hockey. So there's a component of that which, which adds to challenges of that. and go to that paper and you can certainly read about all of that. but that's different than, clinical trials.
Alright, so let's move to clinical trials. I think the first time you see a time machine is probably I spy ish, where you're starting a platform. It's 2010. why don't you talk about the start of I Spy and how a time machine got into that.
Yeah, I SPY two is a, platform trial in neoadjuvant breast cancer. And, just very briefly, it was doing personalized medicine in largely I'll refer to it as four, but it's actually eight, four types of. Of breast cancer stratified by hormone receptor status and HER two status. So you could be positive and negative, positive, negative. Additionally, there's MammaPrint positive negative, so there's really eight. let's talk about eight.
And what happened in the middle of that trial is, and that different therapies work differently in those eight subtypes, and the standard of care had changed in one of the subtypes. And as that platform trial with different arms at different times in that trial. So one experimental arm might be there in the first two years, and then another one is there in year two and three. Sorry, one and two. And then the other one's there in two and three. Another one, three and four. Four and five.
So multiple investigational arms are in I Spy, and the standard of care was changing. And the beautiful thing about it was the standard of care being introduced was an arm that was in as an experimental arm in I SPY two. And so now we wanted to make comparisons to a new standard of care in say, year six of this platform trial with, at this point maybe 15 arms. And this is like players in.
We, we'll call it golf players in golf that had been playing in this common arena at the same time with overlapping eras in the trial, and now we have a new standard of care. And Don and the design team said, how are we going to do this and make these estimates also to the old standard of care? It became infor important to also do relative comparisons to the old standard of care, but the new standard of care. And this problem lays out exactly like.
Bridging eras in sports where different experimental arms in a platform trial. In this platform trial were there and we were interested in making comparison to previous arms and current arms, actually all the arms in an apples to Apples way. But they were in the trial at different times, and we needed to be able to make those comparisons.
So you got a couple differences here. You mentioned age, you don't have patient or you don't have players getting older or younger, per se. maybe they change a little bit 'cause of standard of care evolving or whatever that may be, but it's a lot less. But you also have a lot fewer arms in the trial. So you know, you have thousands upon thousands of players in these databases. what do you think is the what? What was the magic number that said to do this?
Yeah, that's right there. The, at any time in I SPY two, there may have been two or three arms over there, and the nice thing is in many of these subtypes of women, there was a constant control arm. It's almost call it Hank Aaron. Hank Aaron played for almost 25 years, so he had this really long career. The control arm in that trial overlapped a lot and there was always a connection. So there was always e, every particular point. You could always connect it from a particular arm.
So you need at least this connection from one era to the next. If you ever all of a sudden. Stopped every arm before and brought all new ones in after it, you'd lose that connection. So you need at least one connection in them, and the more connection you have, the more overlap of these bridging, certainly your better ability to understand each era in the trial.
So this is, I've seen a lot of papers consider this. they bring it back to meta-analysis where, you know, ConnectEd's actually a phrase in meta-analysis. How many arms. Overlap between trials. This is essentially the same idea. It's just instead of individual trials, you've got what happened in this trial in 20 17, 20 16, 20 15? Those what are what need to be connected.
Yeah, that's right. And so I spy too, just to set this up, so we in, we instituted the time machine and I think it was the first use of the time machine in a platform trial. And that trial had run over 10 years, 25 ish arms. it's had a new evolution where actually some endpoints have changed and all of that, where we can talk about ways in which we wouldn't want to do this, but over the time, maybe 25 arms.
And it estimated the relative comparison of these arms and the effects of ERA incredibly well over those 10 years. And it brought huge value to that trial because we didn't have to over enroll the control. Because of this overlap, we could enroll one out of five patients on the control, and we had it, it was estimated incredibly well. So with less resources, we had great information on the control. And by the way.
If you gave over the course of ice py two, what was the estimate of the era effects? That's actually fascinating in the paper of how does, how do the eras change home run rates, batting average rates, scoring rates going back and look at, in that trial, it was very similar over the 10 years if you gave the same treatment. Seven years apart to the same woman with the same subtype of cancer, you had very similar outcomes. So, there wasn't much time, there wasn't much era effect in that trial.
so Hank Aaron's always 25 and 1930. Looks like 1970.
yep. And in part it was the incredible aspect that in. In breast cancer, they had identified these four really important, subtypes that affected outcome and affected treatment effects. So they had it so well characterized that there wasn't much effect. You could imagine other diseases where we don't really understand that very well and it may affect some and others, and you may get more era effects.
And I think this brings out, one of the key assumptions in the time machine. and it compares it to, using historical borrowing, for example, from other trials. If you are bringing in data that's completely, there's none of these bridges you need. That external data to look, in absolute magnitude equal to what's currently enrolling. The assumption here is really that the relative differences are stable over time, which would really be in terms of the treatment effect.
The differences between arms are stable, but generally, you talk about a dead ball era and a live ball era and so on. That's adjusted. And here it would be essentially the same. Drugs work well in 2013 and 2016. It's not that drugs turn on and off, so to speak.
Yeah, so I think let's get into that. so I think people can see now we've instituted. The time machine in multiple platform trials Now that we're using this, and it's, by the way, it's a new thing in clinical trial science with the advent of platform trials, that we actually have this incredible opportunity in the same trial to have this ongoing set of data where we have overlapping, treatment arms where we can build this and provide better estimates of treatment effect.
So in sports and in this setting, it makes this assumption that we're estimating the relative treatment effect between arms. And we can call this in many of the models we have, we can call this an additive effect. Is that fair to refer to it that way? Now, this could be additive on a logistic regression scale or on a time to event scale, a quantitative scale. but that the relative effect of two treatments is constant in any era.
Now if you have that assumption, this model works incredibly well and there are multiple papers on this. The Roig paper, I don't know if I'm saying her name, Marta, I know Marta, but Roig, that paper that you were involved in that shows you get unbiased estimates as long as you have this. additive effect of treatment over there. So while we're using controls from different times, using this adjustment, we get unbiased estimates mean squared Error is tremendously better.
really nice paper showing this. We have a paper, the Bayesian time machines, Seville et all, that, that walks through this. So that, that's awesome. Now what, how might this breakdown, and in sports, I'll let you talk about it in clinical trials, but in sports, this works really well. What might be an interaction, and when people saw this paper, everybody wants to argue that you don't incorporate this or that.
But suppose Babe Ruth. Yeah, in the 1920s is really good when pitchers throw a lot of off speed pitches. historically, pitchers then threw nine full innings. There weren't much in the way of relievers, and so he's facing guys that are throwing more off speed. And if he faced guys that threw a lot of fastballs, he might not be very good. And we have an advent now of a lot of fastballs, relievers coming in every inning, throwing a lot of fastballs, and maybe he wasn't a good fastball hitter.
So the relative difference from him to Aaron Judge might be different in the twenties, the 1920s than the twenties. Twenties, a hundred year time lapse here. Now you can create interactions like that where the additive effect doesn't hold. You can do that in golf with equipment and all that, but largely, for example, in golf, I don't believe there's any interactions. You can dream 'em up, but they're not real and a great player relative.
if somebody was better in the 1950s, they would be better now if you move them and they had to play with that equipment and all of that. same largely in baseball. I don't believe those interactions. Now, what does it mean in a platform trial? Kurt?
So for a clinical trial, generally speaking, your treatment effect, it depends on who you're studying and what the disease looks like. So the prototypical worry is COVID, where you've got multiple different, there are variants of the disease and in addition, if you talk about 2021. 20, 2020. You've got lots of severe people in the hospital, maybe 20, 22. There are more beds, more people making the hospital that otherwise would've been sent home.
you're treating different people and the idea is that you could have a drug that because you're treating different people, it has a different treatment effect in 2020. It works well on severe disease, but in 2022, you're not treating severe disease anymore. And the treatment effect now, just like there's lots of stuff on observational data, causal inference. When we talk about that assumption on the treatment effect, you can adjust for covar.
It's after those adjustments so you can mitigate this risk. But that's the concern and it certainly would break this down.
Yeah, so infectious disease is one scenario where. it might even be a different disease. So an antiviral that attacks COVID in 2020 might be different than an antiviral in 2023, but let's think about a LS Duchenne Muscular Dystrophy, breast cancer, lung cancer, Alzheimer's Disease O obesity, all the, almost all of these other diseases.
This would be very rare that it, treatment A is better in 2015 than B, but it's worse in 2025 than B. you almost have to try to be creative to even create scenarios where that is. Maybe there's a different background therapy with an interaction to it, but by the way, we, we have that information, we can look at that so that could be the potential. A couple parts to this is the important thing. You and I are both shocked by the reaction to this and we'll come back to that.
But you brought up external data. So there's lots of talk in the world now of real world evidence use of historical information in trials where we might run a clinical trial in a in breast cancer, and we might want to bring in. Information on previous controls or in a trial in a LS in pre previous controls, and this is somewhat all the rage that people are trying to do that external data, and you brought that up, it's from a different.
Protocol patients meet different inclusion exclusion criteria. The data fidelity is different. The data collection is different to this. it was a different time. All of those things are different using external data. In a platform trial. These patients are being collected in the same trial, the same protocol, the same data. They meet the same inclusion, exclusion criteria. They're in the same trial. They were all randomized. So we have this phenomenal, we can call it historical data.
I think it demeans it. In a way that is inappropriate. It's so much, it's the greatest historical data ever to have in the same clinical trial controls. The only difference is time in which they're enrolled and we can model that.
And it's a test. It's a testable assumption because we've got all of this over time. We can look, do the treatment effects look different era to era We also, you mentioned that I-SPY changed the control arm, but a lot of platforms don't. You've got this very stable arm that's in there as an anchor to the model the whole time. certainly these things can happen. But they're, you're at far less risk doing a time machine than any other kind of borrowing or real world evidence.
Now, we should be upfront here, this is not a randomized comparison, so there's a reason that people have this reaction. the magic of a randomized clinical trial is you have at any given time a control group that's comparable to a treatment group. We're talking about bringing in other information, so that is gonna break that assumption. The question is just like anything else we borrow data for, do we get a better answer?
Do we treat patients better by using the information as opposed to ignoring it?
Yeah, I, I, we, we haven't really explained one thing, by the way. we should I, and maybe I'll throw this to you. We, how do you model time? So this question now in a trial, suppose you've got a platform trial that runs for 15 years and you've got all multiple overlapping arms. And let's just say we model a year effect. you could model what you could model each year effect. Independently, or you could smooth over time. So our Bayesian, our Bayesian model is a little bit different.
So describe those two approaches.
So e essentially, as you were saying, we. Bin time. So we could do every year, every six months, every three months. statistically the right thing to do is every time you change allocation a new drug coming in or not, that should be an error. But a lot of times we approximate this by a number of months and just bin, set time intervals. We often call a time categorical model where we just do a fixed, it's almost like we blocked on time and we're setting a separate effect.
the time machine, it's got exactly the same structure, but we smooth those estimates and I think you and I tend to sit on the on. Slightly different extremes of this continuum. I tend to do more of the categorical approach. you tend to do more of the smoothing. I think this is like dose response modeling. If you have lots of doses, it makes sense to smooth. If you have a few intervals, maybe it doesn't help as much, but that's the basic difference between it.
But basically you're blocking over time and then the question is smooth or not.
Yeah. And so that particular problem is largely a time series problem. We're modeling this covariate over time and how do you model that individually? within that and the Bayesian time machine is generally that, that it's smooths, it tends to use a Bayesian smoothing spline to do that smoothing, over time. And I, and. A lot of the trials I'm involved in maybe has small intervals and it's, and there's not huge sample size in each interval.
And hence there's benefits to the smoothing where if you have big chunks with lots of patients there, time categorical is largely the same thing.
Yeah.
Okay. So, um, there, it's interesting in this setting, We can even talk about frequentist things like unbiased estimate of the relative comparison of any particular arms and all of this as long as we don't have this interaction effect kind of thing. Now. people can present examples where if there's an interaction, you run into trouble with this estimate, which is a fantastic thing for us to think about because nobody's ever talked about this before.
maybe an infectious disease, that there's an interaction, and if there's an interaction. All of the stuff we've been doing previously is hugely problematic.
I actually, I tend to be a troll when I'm reviewing papers like this, and I, for what it's worth, I'm often reviewer too, but I'm usually a nice reviewer too, at least. but I, I usually end up, there's a notion that, oh, this doesn't work and this could happen, and you have these examples. But I often go back to, the protocols that have been written and they repeatedly refer to the treatment effect. The treatment effect. The treatment effect.
So it's something that, it, as you said, it's not considered and. The breakdown is actually pretty bad because if you don't think a drug does the same thing in 2022, as in 2023 as in 2024, how do I approve it in 2025 and use it in 2026 and feel confident? Obviously, there's a leap of faith there. I think we all know that's problematic, this isn't a new issue that people have suddenly gone, oh, it's terrible here, and it doesn't exist in the other place.
Yeah, and I, if you believe that this interaction exists, it means that it's hard to ever approve a drug going forward. We need to constantly be testing it in the year. To say, does it, does this drug work? Which the cases of this would be frighteningly rare, but it's almost as though a statisticians are really good at pointing out this can happen and all of a sudden, because we're now doing this as maybe a new problem and pointing out.
So you and I have both been really surprised by the pushback of this and I'll, for last week we got comments from the FDA. By the way, I'm a huge fan of the FDA, but they were questioning the use of this in an oncology setting in a phase two trial of the potential use of the time machine for this. and it's so weird to me that this is a concern in that setting, of course, within a therapeutic area where historically we have this, where we create.
Objective performance number based on historical data of a 15% response rate that you're trying to beat. And now we're trying to use this time machine that there's this, almost this reticence to use it, which means you ignore a ton of data from the same trial. And it just that, that has surprised me greatly that there's been a pushback to this.
And I'm gonna, maybe I'll push back a little bit on one thing you said. I do think, some of these interactions could happen. You could imagine a treatment effect. It's varying from 20 to 25% over the course of the trial. Some unmeasured covariate. usually the magnitude, people worry about type one error inflation in this. if you're talking about small differences, you get small type one error inflation, it's 2.6, 2.7, 2.8. A log rank test will do that.
doing a, basically most asymptotic results do that. We do this all the time, so I accept the possibility, but I don't accept that it's a catastrophe.
Yeah. Yeah. Yep, yep. and ignoring really valuable data, it has huge impacts, much more than the 2.6.
Type two era. Type two eras matter two.
Yeah. And the other implication to it is you collect a whole lot of new control data that you probably don't need to collect by using the data you have in the trial and really smart particular ways
And certainly I wouldn't recommend ever eliminating a control arm completely. That's not what we're talking about doing.
Yeah. and that's the huge value of being able to use time is having this overlap and a control arm is a huge value to that. Now, I don't use the time machine. In, in, in some study there are some diseases, like I was involved in a number of COD efforts and the remap cap trial, for example, does not use. This non concurrent controls, and it was the disease setting. It was thought not to do it. Now, we used time adjustment.
Even though we don't use non concurrent controls, which actually confuse some people thinking we were using non concurrent controls, in this setting, but disease specific and there may be other diseases where there's massive changes in the disease and in the way it's treated that we worry about exactly those things. So this can very much be disease specific. And then there are other diseases we know largely there have been very little differences over time.
there are some cases where you never know what happened. I remember when we were, involved in Los Angeles modeling of COVID, and there were about three times during those. 18 months, whatever it was exactly where clearly the amount of time people spend in the hospital, it changed. We don't know why we had to address it in our modeling. These are the kind of things you can't anticipate.
yeah. Now, as it, it's also one of the huge values of these platform trials is actually this estimate of time. it becomes a scientific quantity of interest. And I brought this up in the sports example that looking at. Major league baseball from 1920 to 2020 and the relative, what a player would've done if they played at the different eras. Those estimate of that is fantastic stuff. for that.
And looking at the NHL for example, in the mid 1980s, a player would've scored many more points by playing there than they would play. Now it's just a different sort of game and that fantastic thing. The same disease learning happens in these platform trials with these estimates of time, and we have a number of these trials showing this, and as you say, we can explore. Is there any evidence of interaction within there, if that's the thing that people are concerned about in these trials?
I think one thing we ought to say here is I think people get reasonably nervous about the thought of, if you're talking about I spy borrowing data from 2010 to, a month. A drug that was enrolled in 2020. So the notion of, what does that really mean? Are you pooling that so on? And certainly this isn't what that model does. it depends on the overlap. So when you're doing sports, you've got thousands of. Players going back in time. in a clinical trial we don't have that.
We maybe have a couple dozen arms. If you are looking at a drug in 2020, you give some weight to the patients that were enrolled in 2019. You give less weight to 2018, less weight to 2017. And it depends on this overlap at how many patients, we're not. Pooling patients from 2010, they're largely discounted because of the lack of overlap in a clinical trial.
Yeah, that's a beautiful thing, and I think you did this in the paper at all, where you look at. In a time categorical, you can actually calculate the contribution of a patient depending on the overlap from earlier, and it's this beautiful variance. Covance Matrix gives you this beautiful weighting of the patients and you get this natural down weighting. It's a beautiful thing.
and the Healy trial uses this to some extent because there's a window, it doesn't borrow back in time in infinitely. It borrows back to a certain degree.
Yeah. And you're a fan of potentially even just, algorithmically cutting a, cutting a line and saying, we're gonna do this for the last five years or something. And that might be disease specific as well, depending on what we think about how the disease, how the treatment has moved, within these trials.
I think you could just basically look and say, this is only contributing 1% of the effective sample size. This isn't worth it. You avoid problems with, people having to publish and there's live data that's still being used. A lot of problems go away. If you eventually realize this really isn't helping us much, let's put a line in the sand and cut it.
Yep. Yep, yep. Alright, Kurt, so if you could use this time machine. And go back and tell Kurt Veley, Carnegie Mellon as a graduate student, you could go talk to yourself. What would you tell yourself?
Oh, that's a terrible question to ask. let's see. So I remember, I remember when I was in grad school, probably the least likely thing that you would've ever thought I would do was end up as a biostatistician. 'cause I was incredibly computational, incredibly theoretical. you'll remember we had this bet on who could use the, you wanted to use the fewest theorems in your dissertation as possible. So whereas I had a, I had pretty much ruined that already.
So I think that, this notion of the applied stuff is where you can actually really affect the world. Would be on that list.
Yeah. Awesome. Awesome. I did have a theorem in my dissertation, by the way.
Did you, you had one.
Yeah, I think I had a couple yeah, by, we need to do one of these on my dissertation at some point. which, which for those interested, it was how to optimally play, hide and seek, which you may find interesting. I, by the way, if I could go back in time, I might tell myself to do a different dissertation topic, but I enjoyed it, I loved it, and, we're in a good place.
I, I have never revisited my dissertation,
Yeah, we're in a good place. You know where we are. Kurt, we are in the interim. And till next time, thank you all for joining us here in the interim. Cheers.