Welcome to Berry's In the Interim podcast, where we explore the cutting edge of innovative clinical trial design for the pharmaceutical and medical industries, and so much more. Let's dive in.
All right. Welcome everybody. Back to in the interim today I have a really, uh, cool topic. Uh, I was really excited about this and the, the topic is a new kind of trial design and we'll get into what's new about it, but the old version of. Powering the trial at 90% running it and finding out was the trial successful or not, doesn't work here. So we'll introduce that in a minute, but first I want to introduce my guests. So, uh, two guests from Barry consultants today.
Uh, Liz Lorenzi, Dr. Liz Lorenzi, a senior statistical scientists at Berry Consultants, and Dr. Amy Crawford, a statistical scientist at Berry Consultants. Welcome both.
Thanks. Thanks for having us.
So let's start, let's start with introductions. Um, I was told that we started one of these podcasts and I didn't do a very good job of introducing our, our guests. So let's, let's do a little bit of introduction, Liz. Um, you've been, tell us how long have you been at Barry consultants?
I've been at Barry, uh, I think six years. Actually. I started on July 1st, so I just hit my six year anniversary. Uh, before then I was at Duke doing my PhD in statistic. Um, and since being, being at Barry, I've been really enjoying participating in lots of different clinical trial designs, all of which are adaptive, innovative, exciting.
I have been a part of a lot of design teams for platform trials, so I have found a particular interest in working on the design of platform trials, which I think we'll be discussing an example of one of those today.
Yeah. Yeah. Very nice. Very nice. And, uh, so, uh, Liz, do you consider yourself a Bayesian?
Oh, I do think I'd consider myself a Bayesian. Yeah. I think I have to say that going to Duke for my PhD, but, but I do think, uh, of myself as a Bayesian. Mm-hmm.
and you'll get a flavor of that, I think today as well. And Amy, um, uh, tell us a little bit about yourself.
Sure. So, um, I've been with Barry for just over five years. I started almost exactly a year after Liz. Um, I've also been working on some platform trial designs. Um. But I've also, I've also been working in stroke quite a bit, so, um, which I think we're, sorry, Scott teasing the topic. Um, getting, gonna talk a little bit about a stroke trial today. So it's a particular clinical interest that I've had since starting at Barry.
Um, before, before I was at Barry, I was doing my PhD at Iowa State University working on something very different. Um, not as Bayesian of a, of a group there, but I, I think I also consider myself a Bayesian. Um, yeah.
Yeah, and, and, uh, we'll, we'll see some of that flavor today as well. Okay, so the, the, the topic for today is part of the step platform and a, a, a shameless plug for everybody to go back and listen to the podcast with Eva Mystery and Jordan Elm. It, we go through the, the big picture of the step platform. It's a, it's a fabulous effort and it's a, it's a really nice. Potential of what platforms can be. And I think the NIH is actually a place that platform draws can be incredibly powerful.
So the, let me, let me not introduce this and, and Liz, do you want to tell us just overall the step platform, the goal of the step platform?
Sure. So Step is a platform trial in acute stroke, and this is funded by the NIH and kind of organized by a big group of stroke researchers under the umbrella of stroke net. And the goal of this platform trial started as in one of the areas of answering the question of who do, who, uh, what population does EVT benefit. So about 10 years ago, they learned that this therapy called endovascular therapy, it's a, a clot clearing device, um, was very effective in a particular patient population.
And since then there's been numerous questions of whether that we can expand that patient population and try to learn whom, uh, EBT can benefit. And so one of the major questions under the step, uh, platform is to learn which patient population can EBT be beneficial for. Uh, there are other questions that step answers, um, more on the medical, uh, therapeutic side. So thinking of different.
Uh, other treatments that are a little different in the question of maybe more traditionally trying to answer, is this therapy better than a control? Um, but I think the one that we're going to focus on today is this question of EBT. Um, and it's a very cool structure to have a platform trial where gonna have this master protocol, uh, guiding the enrollment of patients, guiding the inclusion, exclusion, guiding kind of, uh, organizing sites in the infrastructure of the trial.
And adding a lot of efficiency so that we can answer this question.
So the, the in the step platform, it's, it's structured so. An individual, a team can propose a question to the platform. The platform has this larger master protocol aspect and questions get asked. And a huge goal of this as, as Liz described, is finding out who does and doesn't benefit from endovascular therapy. And we're gonna say EVT over and over again. So this is the, the, the clot clearing device.
Within the structure of this, when this question comes into the platform, there's a design team. So the design team is part of the master protocol, and it involves, uh, scientists, stroke, neurologists, statisticians. So the three of us are part of this design team. There are, there are many others within this design team. Uh, several others at Barry consultants, several at Kansas. Kansas. University Medical Center, uh, several individuals from there, MUSC at Jordan Elm with that.
And so this is, this is the design of that part within the master protocol. So in the setting where. All trials historically were run where, and and they really optimized. If you're running a trial and you wanna show endovascular therapy is beneficial. Historically, they ran trials in those patients that were really, from their perspective, the most likely to benefit from endovascular therapy. Relative to the control here is just a, a background medical management without the, the device.
And so they, they ran trials in that population that they thought were optimal. They powered it for that population and demonstrated endovascular therapy was good and that's within a certain population and it was incredibly effective. In that. So now I think it, what, what neurologists in the medical community are stuck with is, okay, here within this optimized population, this, this therapy's tremendously beneficial in, in the neurological status of patients.
90 days afterwards, they, they incredibly beneficial. So let's figure out in those we don't know if it's beneficial. Um, whether it's beneficial. So now the first, first question came in one we're really gonna, uh, focus on is we'll focus on the first question, which large vessel occlusions were, were generally thought that EVT was beneficial, but how about smaller size strokes? So can you set up a little bit of the.
Patient population, and I know we're all statisticians here, Amy, but a little bit of the patient population that this question now is, does endovascular therapy work?
Sure. Yeah. So like you were saying, Scott, there's, there's this, um, set of patients where we know the answer and I like to think about it as you can sort of draw a box. Around, around that set of patients, right? You, you present with a stroke at the hospital? Um. And based on how you present, you know, the, the doctors figure out whether you should get EBT or, or should not, and, and that, that indication is relatively narrow at the moment.
Um, and so what we're looking at here is for smaller vessel occlusions. Um, I think, uh, we're, we're thinking about what we call medium vessel occlusions. What about patients that are just outside the box? What if you present to the hospital and you have a baseline characteristic of your stroke that doesn't quite fall inside the box, a median vessel occlusion, uh, you fall just outside the box. And the question is, you know, should you get, should you get, um, EVT or not?
And that's kind of the, the, the question we're trying to answer. So these patients, um, where we're, where we're trying to answer this question for, um, they, they have a little bit, um, uh. Sorry. Their, their strokes are a little bit different than those that are part of the indication. And I'm, I'm gonna fumble the, the clinical piece here.
And that's okay. Where you get, you get a lot of leeway, uh, in the clinical piece. Yep.
So, um, they, there are a couple baseline characteristics that we look at, um, at for strokes when we're trying to answer this question. Uh, one of them is time last known well. So patients who present earlier, um, after they were last known well with their stroke. Um, we think that maybe they, they benefit more from EVT, um, in this population than patients who present later. And so the question is, you know, what should we do for patients that present later?
And there's another baseline indicator called, um, the NIH Stroke Scale. Um, and so on this NIH stroke scale, um, you, you present with a stroke, you get a score. So certain patients with certain scores are indicated for EBT in this population with median vessel occlusions and, and certain patients aren't. So the question that we're trying to answer for these patients is, um, you know, what, what kinds of baseline scores on this scale should we give, give the therapy to.
Okay, so let's, let's set The problem up here is we, we're now going into medium vessel occlusions. We're enrolling people. Let, let. I know papers have come out that have changed this, and so we're updating the design. But let's go back to the first version of the design for, for simplicity, where it was medium vessel occlusions and the, the thought was that. It, it's the, the relative efficacy of EVT compared to medical management is very likely to depend on their, the clinical.
Symptoms, which is the NIH stroke scale. So this is a stroke scale from one to 30, say, uh, I think it actually goes a little bit higher than that, but those are very rare. Was it 28 maybe? Uh, that the scale, so this, this is the, their, their range of symptoms and really the size. This is a, a, a proxy for. The, for a medium vessel occlusion stroke, the clinical symptoms are maybe telling you how much brain there is to save through endovascular therapy.
Now, all of these patients are within 24 hours. Last known well, so the, so the statistics problem here that trial design is we wanna find out who in this range is a function of their stroke scale from one to 28 benefits from EVT or not. Now what would be wrong, Liz? With us running a traditional trial and saying, let's enroll everybody in there and do a test, whether EVT is beneficial in that population.
Yeah, so you know, we believe, we know part of this population, you know, from these previous trials benefit. If we just say, let's expand, run a giant trial, randomized 1:1 and at the end test we might, we would likely get a null result because likely there is heterogeneous treatment effects within this population where some are going to benefit, some will likely not benefit and medical management might be better.
And so what we really wanna do is try to learn, um, based on patient characteristics. How patient or who, which patients are benefiting from EVT. So if we think about this, N-I-H-S-S, as one of the covariates we could use to try to stratify the patient population, we might wanna learn within each bin of N-I-H-S-S what the treatment effect of EVT might look like. Um, but the question isn't necessarily what is the treatment effect in the bin.
But it's really is EVT better than medical management and it's more of a generic question of, you know, which of these bins should benefit, which of these bin bins may not benefit? Um, so it's really kind of learning where that change might occur in the patient population.
Yep. So you, you said the word change, which is gonna be a key word, let's not, maybe not jump to change quite yet. Um, within this, so if we enroll this entire population and we run a test and find null result. We might say, oh, in this population there's no benefit where there may be people in that population that benefit a great deal. So the mathematical problem here is we have a scale from one to 28, and we're trying to find out for those patients who on that scale, benefits and who doesn't.
what somebody could do and to attack this problem if we're stuck in the old way of running a clinical trial where we run a trial and we test a hypothesis and it's only upon rejecting the null that we say, oh, EVT is beneficial, which was, which were done in the first ones. We might go at this and say, let's just take the big ones. And this was largely what, when people wanted to originally show it was beneficial. They did that, but they took the big strokes.
And so let's take everybody that's 20 above and test to see if EV t's better in that scenario, and then if that's successful, maybe then we go down to 15. You can imagine how slow science would move if we're restricted to run these Yes, no questions in the trial. So we're gonna do something different in this trial, and we are going to enroll everybody in this patient population, and we're gonna try to estimate who's better and where does EVT work or not.
Now we're gonna use the scientific information that on the bottom part of the scale. Um, that medical management is likely to be better on the top part of the scale. EVT is likely to be better, and somewhere there's a change point here. So tell us about doing this with a change point model in trying to estimate that.
the idea here is that, um, we want to figure out where to draw the line. Who, who should get EBT, who should not? And so the I the idea is if we enroll everybody, um, across this scale where we don't know the answer. We can let the data inform where the line should be drawn. And we do this in a structured way because as Scott said, we know that on the higher end of the scale, somewhere out there, EVT is beneficial.
And the question is, you know, how far down can we drag that line down the scale to where EVT is still beneficial for patients that lie above the boundary. And, and we let the data tell us what that looks like. Um, and, and the way that we do this, like Scott said, is with a change point model.
And what the change point model does is it, it looks for that single point of where we draw the line, where we, where we draw the boundary to expand the indication of EVT along this baseline covariate scale. Um, and, and the data informed that. And so we, the, the really beautiful thing about a change point model.
Is, um, it, it allows us to make these informed decisions in a structured way so we don't have to test, you know, you could, you could think, um, you know, maybe if you're coming at this, um, for the first time and you wanna say, oh, does it work? And. Bin, you know, 28, does it work in bin 27? Does it work in bin 26? You know, you can think about, well, let's test it in bin 28. Let's test it in bin 27. Let's bet test it in bin 26.
It's going to be another extremely inefficient way of doing this, right? And so using all of the data and, and the neighboring bins inform each other, all, all of that data to figure out where the single point, the line should be drawn, um, is just a really nice framework for how we answer the question.
So within the model, if we learn and we get a lot of patients near 1516 and EVT is doing better. It tells us that change point is to the left of that, and 20 is also better. 25 is also better. So in some way, we're trying to figure out this one place where it shifts from medical management to EVT in the setting. Now, um, we, we use a Bayesian model for that. Um, not surprisingly, I'm talking to two Bayesians here. Um, they wouldn't let me get away with a frequentist model here.
So we're using a Bayesian model, uh, in this setting. Now, Liz, it didn't really work when we had a single change point. And you know, we thought of that, that break point of where it goes. Why didn't a single change point work?
Yeah, the, the single change point assumes that to the right there will be a benefit of EVT, and to the left there will be a benefit of medical management. And what we were learning was that it might be a little bit more nuanced than that, where there might be a subset of patients where. There's essentially no difference of the two therapies. It's essentially like a, a coin flip.
And so by ch changing it from one change point to two, we're allowing ourselves to kind of find this period or this, this area of somewhat equality or a similar effectiveness, and then we're allowing to say to the right of that, that's where the EVT would be. So to the right of the right change point, that's where EVT is better. Between the right and the left change point, they're equal. And then to the left of the left change point, we think medical management is better.
so it allows us a little bit more flexibility to add that kind of piece where we don't actually think there's a big difference between the two.
So we were simulating and we discovered a bit of the issue with one, and in part it was the, uh, the, the old bugaboo for us, the null hypothesis. So if you simulated a scenario where it's flat in a, in across the entire stroke scale.
The model doesn't really work well, and you can imagine it's, it's almost non-identifiable that the change point could end up any one of these places and maybe it ends up right in the middle of that and it, it, it somewhat is misinformed by saying it has to be positive on the left and negative on the right. I think I said that backwards, but, uh, that it's, it's beneficial on one side or the other when there is this range. So when we simulated that, we found it really didn't work very well.
So we created two change points. Amy. Yeah.
Yes. Yep. So we created two change points and, and what happens then in the null, um, scenario that you were just describing, Scott, is it allows those two change points to, to scoot away from each other if that's what the data are, are saying should be the right answer and, and allows for that region of equivalence or equality that Liz was mentioning. That we would estimate between the two change points to cover the entire scale, um, of that covariate that we're modeling over.
And so it, it allows us to handle this null scenario, um, much more intuitively than if the model were to have, to just try to figure out where a single change point would need to be. And the data look the same across the whole thing.
And then we're, we really care about the place where Ev v t's better. If they're the same, you wouldn't do this invasive device. So it's, it's somewhat of a statistical and clinical. Better fitting of the data and it functions much better. Another value of clinical trial simulation to carry out a range of scenarios, even if we don't think it's very likely. And, and, and we've created a better model in that. Okay. Tell, tell me a little bit about the code.
Um, and, and, uh, I think, uh, Amy has carried the brunt of the, the, the code building here. Um, now can we just create our code to run this, to run simulations? That didn't really work.
Yeah, that didn't really work. Um, so. When we, when we write code in R or um, one of these other languages, what we do when we run Bayesian models is we tend to use some, some built in packages. Um, statisticians listening have probably heard of Stan or Jags. Um, and And these built in packages don't necessarily facilitate. Uh, learning the type of model that we're describing here. Um, and so we've had to create custom code.
We, we wrote our own, um, our own sampler, our own, our own code that fits this model, um, to run custom for this trial. Mm-hmm.
So, and we wrote that in a, a lower le you wrote it, we, we, uh, you wrote it in a lower level language. You wrote it in c to then be called for simulations and, um, um, within that circumstance, and it's, it's a Bayesian. Model using Markov chain model where the unknown is the, the location of the, the change points within that beautiful, beautiful code. Uh, and, and there's, I'm ignoring a good bit of work to get to the right place and all of that, which was, which was fun. Um, uh, in that.
So now we're running the, the trial design. In this circumstances were. We, we may, as the data start to come in, we don't want to enroll 2000 patients and then analyze this. So that's the model that will be fit. But what do we do with the design? So what's the design that goes with that model, Liz?
Yeah, so this is in an adaptive. Trial framework. So we, through the step platform, we'll have interim analysis that occur. At that time we would gather the current data, run the analysis model, and then evaluate against some pre-specified decision rules to just see if there's any conclusions that could be made. Um, and what's nice about this is, you know, we have a lot of information on the edges, like we've talked about.
So. You know, we may not need that many patients to inform those bins that are, you know, right outside that box that Amy described. And so by doing these adaptations, we may be able to get some, uh, results and conclusions out more frequently, change, uh, practice, um, facilitate, you know, better learning of this question through these adaptations. So, um, I think we have them set to be done quarterly. So as we're enrolling the trial every quarter we would gather the data. Run the model.
Um, and that would be done by a blinded team, so, or an unblinded team. So we will have a separate team from the three of us that are the Baker Berry design team that will be, um, unblinded to the data and firewalled from us on the design part. And they would be the ones that get the data, run the model. And if there are decisions, they could then make a public announcement through the DSMB, et cetera. So I think that's currently the, the way that we have this set up.
So this is, I mean, this is really cool in the way of, we, we've got this model and as patients come in, it's a, it's a wider frame of, we don't know who EVT benefits and who doesn't, but as soon as we learn that, we stop enrolling that group. We announce that, uh, for patients, medium vessel occlusion with NIH stroke scale above 18 EVT is beneficial.
We publish on that and meanwhile, those patients aren't enrolled and we're narrowing in across this scale on who should get each particular treatment. And one of the things that was wonderful when we were doing simulations of this and we're characterizing how well this worked, is it's not, it's not just about power. But what fraction of the patients, when the trial's over are we going to treat? Well, and how are we going to treat patients when this trial's over across this scale?
Now we, we might give them medical management. We might give them endovascular therapy. We might give them the wrong therapy that they would've benefited or they're harmed by it. And we can create scenarios of truth, run this trial, and then evaluate how are we caring for patients when the trial's over, which is such a cool operating characteristics. And so. Targeted to the trial.
Uh, so that, that was, there were many of these, but that was one of the cool things we evaluated in the simulations. So science pulled a, pulled a bit of a, a switcheroo on us and, uh, this is a very active area of research, not surprisingly. And so new trials have come out where. Uh, medium vessel occlusion trials have come out.
We're really questioning whether EV t's beneficial, but it has given us new information in this, in this population that a key parameter is very likely time last known well, how acute is the injury is probably related to endovascular. So now this has become. Not one dimension of NIH stroke scale, but two dimensions in it.
And we have a new submission to step, and I don't know how public that is, so I won't describe the clinical context, but that one is also endovascular therapy versus medical management. And it's within the realm of the patient population we don't know, which is also two dimensional. And it's the same structure that on one side of the scale. Uh, we know EVT works and in the other side we know, uh, uh, at the very extremes that medical management is better off of the, the patients we're enrolling.
So we know that direction in two directions. So this is, you know, we know up into the right is one treatment and down into the left is the other one. We just have no idea where the break is in this. So now the model's gotten a little bit more complex, Amy.
It has. Yeah. So, um, our model now actually has four change points. Uh, like you kind of described Scott. We're, we're working on a grid. Um, and, and so across NIH stroke Scale, we're trying to figure out, um, within patients who present. Early, you know, it's less time, less known. Well, um, you know, maybe they should be treated differently than patients, uh, that present later.
Um, they, you know, they, it's been a longer duration since their time last known, well since they've had their stroke. And so what we're doing now is we're trying to draw lines on. On a two dimensional grid instead of a one dimensional, um, baseline covariate. And the way we do that is with four change points, and they're restricted and, and, and they, they move around with each other. But, but I think the big kicker with, with this, uh, more complex.
Framework is that it follows clinical belief and knowledge. And we've been able to adapt and update what we're doing based on what is being presented in, in changing practice. And so now we're, you know, we're, we're shifting our, our approach and it still is going to follow this structured data-driven decision-making framework and the model. Matches the decision framework that the clinicians and the medical world, you know, would expect to see based on, based on these recent findings.
Yeah, that is, that's so cool. Um, and now Liz, you are going to jump into one where it's not necessarily two change points, but it's a. Change curve perhaps in two dimensions. And so this, this idea was presented just this week that we have a two dimensional space. Think of, uh, quantitative variable on the x axis, quantitative variable on the Y axis. And we, we.
We know what's in the upper left and we know what's in the bottom right, and there's a curve that we now have to fit, and there's mono tonicity across this. So you're jumping into a pretty cool problem with a bit of uncertainty how this is all going to behave.
Yeah, and I, I think what we learned from Amy's, you know, development of the original change point model is that this really needs to be simulated and it needs to be discussed and, you know, constantly put in front of the clinicians to understand what they're thinking and what they're expecting, and how we can make sure the model aligns with that question.
I, yeah, I'm not sure if I know exactly what that model's going to look like, but I think, you know, as we continue discussions, we'll have a, a clearer picture and simulation will be our, our friend in these, in these conversations to make sure that we get everything right.
Yeah, we would be incredibly fearful if we just kind of put a model together and the first time we use that is in the actual trial. So we simulate millions of data sets. We see what it looks like under a range of scenarios and make sure it functions well. Because we don't want the first time we use it to be the real trial. We want that to be the 10000000th and third time that it's used. It's just been used on the simulated data sets.
So we will, we will, we will drive you crazy Liz, by test driving this over and over again. But it's sort of the fun of science here. All right. So we, we, while they are enrolling, actually, I, I, I believe there's a bit of a pause in enrollment, um, in this setting. So we are actively working on this. The step platform is a, is a very cool platform. Uh, by the way, if that kind of modeling interests people. There's another NIH trial ice cap that just stopped enrolling that has.
Analogous models. It's a different model, but as an analogous model. And that we'll be reading out sometime in the next few months if that's kind of interesting. And then hopefully we'll get step results out. So, uh, Liz and Amy, thank you for joining us in this weird place in the interim and sharing this really cool modeling.
Yeah. Thank you so much for having us.
Yeah.
Thanks Scott.
I appreciate it. So everybody out there, thanks for joining us here in the interim. Until next time.