Spending Alpha

I. Designs which use interim analyses or adaptive analyses during the course of a trial, and hence the name in the interim. We are spending some time in the interim here. So I was, uh, I was working on a project last week. I. And this is something that was, was said to me, which, uh, has said to me, has been said to me hundreds of times in the 25 years that I've been doing this, where they were interested in potential adaptations in a clinical trial.

And when I mentioned potential adaptations, their first co comment was, will this cost us alpha? Okay, so the title of this podcast, spending Alpha, we're gonna talk about Alpha. Uh, I will try to explain what Alpha is. I will try to set it up. Many of you know what Alpha is, but that comment, uh, it, it, it, it's such a challenging comment because in almost all circumstances you don't lose alpha. So what is Alpha? Let's back up.

Typically, in a a clinical trial, you're testing, and we're gonna talk about a superiority trial. There are other kinds of trials, comparative effectiveness, non-inferiority trials. But a trial, a phase three trial that's trying to demonstrate a particular investigational therapy is better than the control, let's call it the placebo. At, at the center of this, when the trial is over, the primary analysis of most of these trials is trying to demonstrate that the treatment is superior to placebo.

We do that through hypothesis testing, and many of you are familiar with the structure of a hypothesis test. We refer to the null hypothesis as, as the thing we're trying to disprove, and typically that's the, the treatment and the placebo are equal. That the treatment is not better than placebo. You can phrase it that the treatments, uh, know better, potentially worse, and the alternative is that the treatment is superior to placebo.

If we're able to collect data in the trial that demonstrates that data is unlikely under the null, under equality of treatment and placebo. We can, we, we would claim with that hypothesis test that the treatment is superior to placebo. That's a standard hypothesis test and a standard phase three trial. We, we typically draw a line and say, how unlikely does that data have to be under the null hypothesis, assuming the null in order to reject the null and claim that treatment is superior.

We, uh, you'll, you'll frequently hear 5% alpha, uh, in it. And it's such a, that this is not the topic of this conversation, but it's something else that, um, I, I've always struggled with is that in those trials, it's a, it's a, it's a, uh, it's a one directional null hypo, uh, alternative hypothesis that the treatment is superior. In that scenario, you really get 2.5%. You get half of that alpha to demonstrate superiority.

It, it's interesting because I taught at Texas a and m for five years before I got into clinical trials, and in a basic stat course, if you phrase that problem of hypothesis testing, and you said set up the hypothesis test and tell me about the type one error. They would refer to it as a greater than, the treatment is greater than the placebo, and there's 2.5% alpha. If you read most protocols of a superiority trial, they still talk about two-sided alpha, that you get 5%.

You don't get 5%, you get 2.5. Most of the protocols would be marked wrong in a basic stat course, and there's this weird tradition about still talking about two-sided 5% alpha, which which doesn't really make sense in that setting. So. Throughout. Throughout our discussion, I'm gonna talk about 2.5%, and that's the standard for a superiority trial.

That if you can demonstrate with a 2.5% type 1 error that the treatment is superior to the placebo, that's typically referring to it as a successful trial, and you've demonstrated in that trial that treatment is superior to placebo. Now that quantity, that 2.5% is usually labeled it with the Greek letter alpha. α So that's the typical standard of a phase three trial that you get this 2.5% type 1 error. Now, what does that mean?

Why, how does that 2.5%, just to back up a little bit more, is that if the treatment and placebo are exactly equal. And you run the experiment. I'll talk about a 400 patient trial. You run the, the trial, the probability, assuming they're exactly equal, that you would get data as extreme. As what we need to see, assuming they're equal, is 2.5% or less, and they refer to it as a type one error.

If they're equal, the probability we make a mistake and say the treatment is superior is 2.5% in that trial. And that's a, that's standard for a phase three trial. Now before we go on, many of you will recognize me as a a, a Bayesian. I, I am a practicing Bayesian statistician. By the way, for those of you out there, you don't actually have to declare your allegiance to one side or the other, but I think like a Bayesian, I was trained as a Bayesian. Uh, it's the way I set up models.

I, I much prefer to do a Bayesian approach, which is really quite. Opposite of the frequentist approach and type one error, and it uses posterior probabilities that the treatment is superior. Uh, but I live in this world of clinical trials where the, the standard is frequentist. Hypothesis testing. So let's talk about that and let's, let's live in that space for now.

Even though I spend a good bit of time doing Bayesian trials, even doing Bayesian analysis, taking advantage of frequentist approaches. But let's stick to, to standard frequentist, uh, phase three trials, 2.5% Type one error Now. Historically, the f the the most trials were fixed sample size. Let's enroll to the end and do a single analysis.

When you do that single analysis, if the P value, which is the probability of seeing data as extreme, that the treatment would be, would, would beat the placebo by as much as it does in the trial if they're equal, is the P value. And when that's less than 2.5, we say that we demonstrated the treatment is superior. Now we, in a traditional trial, you do a single time point. You do that analysis and you get 2.5% alpha straightforward.

Traditionally, the first types of adaptive designs that were done. We're looking at potential early analyses. So before that final time point, you do an analysis and maybe we've demonstrated superior before, superiority before that. So let's take a trial that originally a fixed sample size of 400. Now we might do interim analysis at 200 patients before the 400 and at 300 and then at 400. This is sort of the first type of adaptive trial. It's now called a group sequential trial.

It's probably always been called a group sequential trial now. Now, because you do an analysis for superiority at 200. Then you do another one at 300, and you do another one at 400. You're really taking three shots on goal for showing potential superiority. If you were to do that, using the same alpha at each test.

We reject if the P value's less than alpha at 200, at 300 or 400, the experiment itself would have a higher type one error than 2.5% because you took three shots at getting superiority so that that would not satisfy the experiment having 2.5% type one error. It. So what we do in those trials is we adjust the alpha at each of the three time points. We lower it from 2.5 when we create three different alphas referred to as the nominal alpha.

At that test, we have an alpha at 200 and alpha at 300, an alpha at 400, and their beautiful theory. Uh, O'Brien, Fleming and group sequential theory, many ways to allocate different Alpha at the three interims so that the experiment has 2.5% type one error. For example, if you did O'Brien Fleming's spending boundaries at 200, 300 and 400, the Alpha 200 would be 0.0031. At 300 would be 0.0092, and then at 400 would be 0.0213. So all of those are less than 0.025.

And you can show that if the treatment doesn't work, that there's a 2.5% chance that you would be successful. At least one of those three analyses. And that's group sequential testing. And this is historically the first type of adaptive trials that were done. Now what notice at the final analysis at 400, the, the, the alpha value, the nominal alphas 0.0213. If you didn't do the two interims, it would be 0.0 2 5, 0. It would be bigger. It because you do those two interims.

The final nominal alpha is less than 0.025. That historically has been called a penalty. That's the penalty for the interim analysis now. This is where I start to struggle with that. I hate the term penalty. I, I think it's actually caused a lot of misunderstanding, and I'll describe the sort of first misunderstanding to that is you get 2.5% type 1 error. You've just allocated it over the three analyses. You haven't lost anything. That experiment gives you that same 2.5.

You've just allocated it at three points. Yes, you've lowered the number at 400, but you had a bigger number than zero, which you had in the fixed trial at 200 and 300. So you've just spread it out. You've distributed across three analyses. And there can be benefits to that in the experiment. The trial, the, the treatment could be highly effective and you learn it at 200, you save time and patients, you get the treatment out to patients earlier. So there can be large benefits to that.

But because this, these were really the first time adaptive trials were done, this notion went to. When you do those interims, you have to adjust alpha, that there's a penalty for it. And it became that there's a penalty for looking at data and it, it's still pervasive today that the notion is looking at data is bad and you pay a penalty for it. And neither one of those is true. Looking at data is not bad and you don't pay a penalty for it.

You just spread your alpha at three different time points. So hence in introducing it in adaptive design and a a clinician or a trial design sponsor says. Does it cost me alpha? It's such a hard thing to respond to because the answer's no. But yes, you have to distribute it. So it's one of those things you have to be careful. I, I have to be careful how I answer that. Now let me, let me walk through a couple of these. What I mean by it's not bad and you haven't lost anything.

So, for example, if we are running a fixed trial of 400 patients. And we have, uh, continuous outcomes. So we're doing a t-test at the end of the day. We're testing a, um, we're testing a, a functional rating scale or a clinical outcome, or an exercise test or a biological measurement of something within a patient. And most tests. Are essentially normal. This could be a test of, of of, of dichotomous outcomes.

The rate of mortality that if that 400 patient, that 400 patient trial, if we have what's called an effect size of 0.3, so the, the, the treatment has a benefit relative to placebo, that's 30% of the standard deviation. So if we're doing a six minute walk test and the standard deviation in the six minute walk is 50 meters and our treatment has an advantage, uh, of that of 15 meters, that's 0.3 15 divided by 30 is an effect size of 0.3.

That would be in the trial of 400 patients, it would be 85% power. And you could run a fixed trial and you'd have an 85% chance of demonstrating superiority that your p value at the end is less than 0.025. Now that's, and, and you get, you, you, you do 0.025 at the end because you're doing a single interim. Now, suppose we do interims at 200, 304, and we do the final at 400 using exactly the alpha values that I just gave you. That trial has overall less power. The power goes from 85 to 84.

It's essentially exactly a one percentage reduction in power. Now that happens, you haven't lost alpha, you don't lose power because you. You, you, you pay a penalty in Alpha. It's because you distributed some of that alpha to smaller sample sizes. So you took a shot on goal at 200, but it's smaller than 400, and hence, your power goes down a little bit. From that, it goes down a very small amount. 1%. For that 1% you save on average in that same 0.3 effect size, you save.

87 patients on average, on a 400 patient trial, the average is 313 patients. That can be a huge advantage. The time to enroll those patients, uh, the, the, the number of patients enrolled to demonstrate superiority. So you're dropping power by 1%. You're saving 87 patients in a 400 patient trial now. That's a different trial design, and you get to decide as the sponsor, as the constructor of that trial, would you prefer that? Now there's no penalty to that.

You've distributed it just like in that 400 patient trial. You didn't save any alpha for 500 patients. You've spent it all at 400, but we don't talk about the penalty. For a sample size of 500, you left zero to that. So partly what happens is this, this notion and this, this, this penalty and it, there really is this thought that looking at data is bad and it's not bad. It actually restricts us from constructing better trial designs. So for example.

Let's do interims at 200, 300 and at 400 and allow the trial to potentially go to 500. So let's not spend all our alpha at 400. Let's leave some leftover for 500. So again, I do O'Brien Fleming boundaries at 200, 300, 400, 500. That same effect size that had 85% power. With 400 patients, every trial is 400 and it's 84 per 85% powered. This new trial that allows it to go to 500 is now 91% powered. I've increased the power from 85 to 91.

I've reduced the number of failed trials when my treatment works. It was 15% of the trials times the trial fails. Now it's now it's 9% of the time the trial fails. I, I've gotten rid, rid of 40% of my failed trials with that, and my average sample size is 368. So by allowing flexibility, my average sample size is smaller than 400. My power is greater, goes from 85 to 91.

In many ways, this is a much better trial design, and I'd much rather see that trial design getting over 90% power with a smaller, average sample size. If we were to do this trial over and over and over again, instead of the 400, we take less time, we spend less patients, and we get the right answer more often, and our type one error is no bigger.

But we don't do that a lot of times because there's this thought that it's bad to look at data and I'm spending alpha and I don't wanna spend my alpha, I only have 2.5. Again, you're not spending alpha, you're distributing over different time points. The name matters. And let me give you another example of, uh, uh, of the name meaning something and it, it actually hurting it. So the.

The, when you're driving down an interstate in the United States and you drive in the left hand lane that lane, many people in this country will just sort of refer to that lane as the fast lane. I, that's a harmful name for the left hand lane on that, because the notion is, oh, that's the fast lane. If the speed limit is 70. And I'm driving 73. It's okay to be in that lane. It's the fast lane. The name, the, the lane is not a fast lane. It's a passing lane.

I. The name of that left hand lane is a passing lane, and there are road signs all throughout this country that say, left hand lane is a passing lane. That means it doesn't matter how fast you're driving, if you're not passing somebody, you shouldn't be in that left hand lane. If you're driving 90 miles an hour in that lane and you're not passing anybody, you're, you're violating street signs what they tell you to do. But everybody refers to it as the fast lane, so it's okay to drive.

75 or 80 if I'm not passing somebody. It's the same thing here. By calling it a penalty, you get people not wanting to do it. Who wants to, to take a penalty in this? So looking at data's not bad. Uh, my father, Don Berry, of course, is a, a statistician, as many of you know, and during a particular election, he made the reference that he turned his TV on and off many, many times, hoping to affect the outcome of the election.

Spending his alpha over and over again hoping to change the particular outcome of the trial, which of course it didn't change it. And looking at the data doesn't change it. You can do this incredibly well. Uh, we won't get into operational bias, we won't get into all of that. Implementing trials. We did an episode with Anna McLaughlin and Michelle Dery talking about implementing adaptive trials, and this is something we worry about there, but this general notion is. Spending Alpha is bad.

Distributing alpha allows you to create better trial designs, and that's something that, uh, this, this weird name in this thought process prevents us from doing. Okay, so let's, let's, let me describe a few examples of this. So. What, how does it happen then? The, and and it re there this wide belief that when you look at the data you've spent alpha and it's absolutely untrue. Just like Don turning the TV on and off. He didn't change the election. Looking at data doesn't cost alpha.

It do, it doesn't cause you to distribute alpha actions you take at the interim. Are what are critical for understanding their potential to type one error and the potential to have to adjust the, uh, the most straightforward example of this is a futility analysis. Suppose you plan afu futility analysis in that trial at 400 I, I'm sorry, the 400 patient trial. You do a futility at 200. And you write in the protocol, no success boundary will be done.

No success stopping will be done, but we'll do a futility analysis. So if the treatment is doing worse than the, the, the placebo, at that point you might, might wanna stop the trial because it's very unlikely to be successful. Now you can, you can figure out the right way to do that. You can simulate the trial, figure out good futility, stopping boundaries. You look at the data. A a, an independent stat group looks at the data. There's no alpha spend for that.

You don't have to adjust your final alpha for that. Doing futility actually decreases the type one error of the experiment because the only thing you do is stop the trial for futility, and hence, you don't claim superiority. That can't increase the probability that the trial demonstrates superiority. It's the ap, it's the opposite action. You could do a hundred interims for futility in your trial, and your final analysis is 0.025. Now a whole separate topic is by doing futility.

Should I get a higher alpha at the end for that and binding futility and I'm buying back alpha. I actually am not a big believer in that violates the likelihood principle and other things. So we're not gonna go there. So we could do a hundred futility analysis, retain our 0.025 at the end, and we don't have to adjust it in any way because no action we took during the trial. Change the probability of of, of making a type one error. That's critical.

Now that we write down exactly what we're gonna do at interims. Because the action you could take might affect type one error, and it's clear to lay that out prospectively. So the experiment is completely prospectively set up. If the predictive probability of trial success at 400 drops below 5% at 200, we're gonna stop for futility. Otherwise, we're gonna go to 300 and do a success interim. Then we'll go to 400.

If that's not successful, we would have to adjust at 300 because we're taking a shot on goal. We might wave a flag and say the trial that the treatment was successful, sure we would adjust alpha for that. Again, not bad, you haven't lost any alpha in the trial. It doesn't squeeze away, um, that, that you pay a tax to it. Now there are pharmaceutical companies out there, big ones that when they do a futility analysis, they write in that they're going to, they're gonna reduce their alpha by 0.001.

It, Dr. It it? Yes. It drives me nuts. It's almost disingenuous a little bit. Because if I were a DSMB member and I saw that, uh, okay, are you taking a success? Look, if I see data at that time point that you're doing futility and it meets 0.001, should I stop the trial? Well, no. Then you, then you write in that you could stop the trial for superiority, but by throwing 0.001 alpha.

In the wastebasket when your action does not in increase type 1 error, it's really sort of disingenuous and it, it's sort of lacking understanding of what, what alpha is in the trial. Now, there are other things that are done at interim analysis. What, what could those be? You could expand the sample size. There's the promising zone design that may be in that design of 400. When I get to 200, I could say I want to expand it to 500.

The expansion of the sample size in some cases means you have to adjust alpha. In other cases, not, um, by the way, it's, it's not a good design. I'd rather go to 400 in design. Should I go to 500 than making that decision at 200? Yeah. Other things you can do at interims, you can do response adaptive randomization on your experimental arms.

Maybe we have two arms in the trial and two doses, and we're doing response adaptive randomization On those doses, I do an interim, I update the randomization of those. The, the, the extreme of that is I might drop a dose at that. There's no alpha spend to response. Adaptive randomization. No. That, that, that's a topic for a different day. But if you write down, here's the action that's gonna happen. There's no success stopping, we're gonna do response.

Adaptive randomization, you don't adjust alpha for that. It's actually, uh, it's kind of a neat little, uh, thing that by doing response, adaptive randomization to the arm, you increase its sample size. So when the data are good, you increase it sample size. It's the exact opposite of the group. Sequential stopping for superiority, where when the data are good, you stop. You quickly stop and say, we won here. You actually increase the effective sample size.

So we have, uh, the Sepsis Act trial was, uh, it conducted several years ago. Faring was the sponsor. You can go on to clinical trials.gov and look this up. It was a, it was a treatment cell oppressant for the treatment of sepsis, septic shock. The trial was a phase two three seamless trial, meaning it started in a phase two setting where it had three doses in a placebo.

And from 200 to 800 patients, we did interims Every month in the trial, it adjusted the randomization to the three doses, and the trial could shift to phase three and pick a single dose. And then in phase three, it would test only the selected dose and it would go to 1800 patients. Trial design's, public, you can look it up. More than 20 interim analyses were planned in that trial. It all depended on enrollment rate in the phase two part, the expansion into phase three.

The final alpha at the end of phase three was 0.025. And that was reviewed by the FDA. It was given a special protocol assessment, an SPA, meaning that they dug into all the details of this. So here's a trial that did more than 20 interim analysis, looking at the data, and yet not a single action in that trial would increase the probability of a type one error and 0.025 was given at the end. That's a demonstration that it's the action you take.

It's not the fact that somebody looked at the data in the adjustment of the alpha. Now this question of do I lose alpha if I have to distribute it? I showed you an example of a group sequential trial. We can build a hot, more highly powered trial with a smaller, average sample size by distributing alpha. Let's talk about a seamless two three trial.

So we, we worked on a project where in phase two there were two doses and a placebo, relatively rare condition, uh, a relatively, uh, uh, bad condition to have a severe condition. Rare condition. The phase two part was gonna enroll 30, 30, and 30. And at that point, the trial could stop for futility if neither dose was doing very well, or a single one of the two doses could be selected. And at that point, it would move to a one-to-one randomized trial of 90 versus 90.

So the entire trial is 30 30, 30. That's 90 total patients. And then the second part of the trial, one to one randomizes 180 patients. So with that design, you could run a separate phase two and a separate phase three. By doing that, you use all your alpha at 0.025 on the 90 versus 90 part. And in the expression we, we get full alpha at that.

An alternative, you could do an inferentially seamless trial where the 30 patients that were on the dose from in phase two could be included at the end of the second part as is, as are the placebo. So rather than the 90 versus 90.025, we could do 120 versus 120. Now that action of picking a dose and bringing its data forward on that dose and not bringing forward the data on the other dose that does.

Increase the probability of a type one error because you can pick the better of two arms, and so that act of keeping those 30 patients mean you have to adjust your final alpha at the end. In that example, by keeping the 30 and 30 on the two arms, keeping those 60 patients in the primary analysis at the end, in addition to the 180, our final alpha is 0.01693. So it's lower than the 0.025 in, in the, in the term I hate, but I'm gonna use it. We've paid this penalty 0.001693.

We've dropped by almost 0.009 that we dropped by almost a third of that. You have higher power. You, you pay, you, you distribute alpha in that way. Because you're including those 30, those 60 patients, you have higher power by including those 60 patients and doing 0.01693 than you do by analyzing the 90 versus 90 at 0.025. So when I'm asked, and I, I'll be asked that question this week, does it cost us alpha? The, you know, the technical answer is you have to adjust your alpha.

But your power's higher. And so the question I don't even think the question really is, I think they want to know, do I lose power? And they ask it, do I do, do I adjust my alpha? Do I, do I pay a penalty? It's an example where, uh, you want to, you want to pay that tax. It increases your power. It increases your power to an extent that you could make the second part of that trial, 60 versus 60. So that the total sample size on the dose you carry forward is 90 and the placebo is 90.

That's what the original second part was. In that case, you pay a bigger penalty 'cause it's only 60 versus 60. It's 0.0 1 6, 1 6. Again, you adjust your alpha at the end. That trial has higher power than the 90 versus 90 by itself. You, your trial ends up 30 on the dose. You don't pick 90 versus 90. You've saved 60 patients versus running a separate 90, a separate 180, and you have higher power. Now, now that I, I should back up.

I should be, I should be clear about that you have higher power as long as your two doses are both effective. If you're taking a shot on one of those doses in that, and it's ineffective, you might actually have slightly lower power. But if your, if your second dose has at least 50% of the effect of the other dose, you actually have higher power. So adjusting alpha can be good. Looking at data can be good. It can make a more efficient trial, a better trial design.

Uh, it can accomplish the goals easier in those circumstances. So let, let, let's sort of summarize this. Looking at data doesn't mean you have to adjust your alpha. It doesn't mean that at all. The actions you take at that interim can affect type one error. That hence the incredibly important role of a prospective design laying out exactly what's done at every interim. And following that design. So looking at data doesn't cause alpha adjustment, but alpha adjustment is not bad.

It's an incredibly powerful tool to make an adaptive design more efficient than a one look single shot. Look at 400 patients or look at whatever. You can be much more efficient in your trial designs by distributing alpha across the different interims. So. Can we get rid of referring to a superiority trial as a two-sided 5% alpha? Let's get rid of that and let's stop using the term penalty.

Let's use something much more positive because it's a good thing to do as distributing alpha, uh, uh, uh, within it. Um, it's a much more positive term than paying a penalty, which nobody wants to do. All right. Well, I appreciate everybody joining us here in the interim. Until next time, uh, try to keep all your alpha and I'll see you next time in the interim.