Revisiting Seamless 2/3 Trial for GLP-1 Agonist

Things you'd like to hear about. We had requests to go back to some trials and talk about, uh, issues that came up, barriers, how you overcame them. A little bit of maybe, um, the sausage making, if you will. So Don and I are going to revisit. Um, in older trial, uh, talk about the design, talk about the building of it, the different aspects of the trial design, a number of barriers, uh, in it. So this trial is. Uh, a fascinating story.

It's, it's, the name of the trial is the Award five trial, but the, the trial was for a drug on Eli Lilly. All of this is public. Uh, dulaglutide was the name of the drug while we were working on it. It's a Glip one inhibitor

It had an active comparator sitagliptin and it had placebo. So it was a nine arm trial. The trial had a phase two component, with adaptive randomization favoring the doses of Dulaglutide that were doing better, and we'll say a lot more about what better means.

At when it enrolled 200 patients in between 200 and 400, it could shift seamlessly to phase three and if it met certain criteria on the efficacy of, HbA1c but also additional parameters we'll talk about, it could move to. The phase three components, which was fixed, randomized, which could be one or two doses, uh, with sitagliptin, the active comparator and placebo, that size of phase. The the, the phase three component.

The second stage of this trial was adaptively selected, and that represented a phase three trial. The patients from both com, both parts of the trial would go into the primary analysis at the end of phase three. We controlled type one error as part of that. That was part of the trial design. The trial was in, was run entirely by Bayesian algorithms. The final analysis at the end of the trial would be a frequentist analysis, an ANCOVA analysis of HbA1c So what happened?

The trial at the first time, it could possibly go to phase three. It did. It selected two doses, the 0.75 milligram and the 1.5 milligram doses moved to phase three, enrolled phase three. At the same time, it spawned additional phase three trials with the doses that were selected. The trial, the trial and the additional phase three trials ran out. the, it ended up, it was a non-inferiority trial to the active comparator. It ended up showing superiority.

It was very effective in weight loss, not supply, not, not surprisingly. Um, the drug ended up getting approved by the FDA in 2014 and in 2024 it was a $6 billion selling drug. and it's been a multi-billion dollar selling drug since the trial. Eli Lilly talked about saving 12 to 18 months of development time with the seamless phase 2-3 trial. So that I'm gonna pause and throw it to Don to add to any compartments. So that's the trial design.

This really, quite, elegant, reasonably complex phase 2-3 trial with interim analysis every two weeks in the trial resulting in a successful trial. And we thought we would go back and revisit the trial in different, aspects of the building of this trial and what happened. So Don, anything to add to the overall structure of the trial?

You know, in complex innovative designs, uh, within the setting, we, the original trial design was, uh, meant to be a short term, four months, four doses, 50 patients on each dose. Phase two trial, let that trial read out and then make development decisions.

But in order to do that, we couldn't necessarily do fixed sample size on all the doses because then it grows linearly and now it's, you know, seven fourths is big. If we're gonna go to more doses. So the discussion of the number of doses in that tied to response, adaptive randomization, the trial ended up we started to explore, uh, using response adaptive randomization. The ent, the eventual trial did fixed randomization for only the first 50 patients.

But then did response adaptive randomization at that point. The one of the problems, or one of the issues with that is you don't just want to do one uh, of those analyses because you want to continually learn as the datas was getting more and more rich, and so now we had to do multiple. Uh, analyses of this, which eventually grew to doing analyses every two weeks during the course of the trial,

Had to be not elevated too high for risk of cardiovascular events. So this became a big question in the design is how do you do response Adaptive randomization? How do you do dose selection when you're worried about those endpoints? And later on down the story, weight loss became a really important part to that. We maybe we can come to that. So that became sort of barrier number one, is how to incorporate these multiple endpoints in the design parameters.

I don't know if you remember how this started. But one of the, the really interesting things that happened is we came into this and they were very against a clinical utility index initially. That they had tried to do this in some other settings, and they, they, they were against it. So we started creating a number of rules that we'll take hba one C, but if the posterior probability of, uh, heart rate increase was above some level, we would stop randomization to that.

But if this was true, we would do this. And if this was true, we would do that. And we ended up creating, um, something. And I, I, I learned the terminology at the time, a Rube Goldberg machine, and I think you referred to the design at one point as that. Do you remember that?

They would bring a new scenario and it wouldn't necessarily do what they wanted it to do because of the complexity of all these rules that were kind of custom to every scenario we got. We'd create a new rule, but every time we got a new one, it wasn't doing the right thing. So you and I were reviewing this. We were talking with the statisticians. By the way, Brenda Gatos said Eli Lilly was a huge hero in this. In her work and the, the work and the design.

So you and I may talk about our, our discussions, but, uh, the, the team at Lilly was fantastic in, in all of this.

So we were looking at simulations of a number of these designs, and we saw this as a Rube Goldberg machine and we were really concerned, and you and I thought the only way forward was gonna be to create a clinical utility index to create a weight function of these different values that if heart rate went above a particular value, that dose had to be. You know, pushed down the value of that dose was lower.

So we took the scenarios that they had created for us and what dose they wanted to pick, and we, at this point, we were probably simulating 30 different scenarios over the different endpoints, and we knew what they wanted to pick. And so we started to create mathematical functions that represented their decisions. And we thought this was the only way forward.

So we went and presented them simulations of a new approach to pick the dose, and we showed 'em how it, how it behaved, and how often it picked different doses before we kind of jammed the clinical utility, uh uh, and the function on them. And they loved it. They love the choices of this, and they became somewhat advocates for the utility function.

Uh, we had a multiplicative, uh, CUI, um, with the exception of one joint distribution or one joint function of two of the end points that were, uh, sort of the, if then, uh, that were related. So, except for that, it was multiplicative.

It had zero value, so it wiped out that dose as a value. The fourth endpoint that came into it wasn't really safety and it wasn't the primary efficacy endpoint, but was weight loss, which of course we understand now, uh, plays a critically important role for Glip one agonists, um, in this. But they knew that if the drug increased weight, it was actually one of the, the, the early parts that they worried if it increased weight, nobody would take the drug.

And if it actually decreased weight, which they had hopes for, but you know, in, in many ways, they were wanting to prevent weight gain that had additional value as a, as a therapeutically good dose. So we showed them the utility function and we simulated a bunch of trials and showed 'em the selection and they ended up.

Looking at these and, largely agreeing and then they tinkered with a little bit when they disagreed with the rule, with the outcome of the trial, they changed the utility function as they molded it around what they thought was the the risk benefit. Uh, of these endpoints, and it got to the point where we were simulating these trials and when they, when we showed them results and said, this is what happens in this example trial, they said, oh, I don't know if I'd do that.

I think they'd pick the other dose. They actually got to the point where they, thought they were wrong, and the algorithm was right to the point that they believed this utility function was absolutely the right way to go forward. And they instructed the DSMB eventually on that. So the utility function was then used to do response, adaptive randomization over the one that had the best profile. It drove whether or not a drug would, would graduate and go to the second stage.

Whether it would bring a second dose along into the trial. And the algorithm drove all of this, which was a Bayesian model on those four endpoints, a Bayesian model that calculated the posterior distribution of the utilities. Uh, and it drove, uh, all of those decisions.

So the, the, the, the one other, uh, uh, another really interesting part to this was the longitudinal modeling within this, um, which we described the original trial was maybe four months they were gonna look at, but now the primary endpoint in phase three was gonna be 12 months. Within it.

And so we wanted to make our inferences based on the 12 month results for hba one C. It was six months for, uh, heart rate, blood pressure and weight loss because that was a comparison to placebo, which they could only give for six months. Um, and so we built longitudinal models of the outcomes over time.

That predicted the six month for three of them in the 12 month for hba one c to drive the decision making in the trial became a huge aspect of, of being able to make these decisions continually during the course of the trial.

They had done quite a bit of work in diabetes, and so they were able to provide us data on fasting blood glucose, and hba one C, and so we fit the longitudinal models to the previous data and hba one C greatly outperformed fasting blood glucose as a predictor of hba one c. And so we were able to abandon fasting blood glucose and use only the hba one C values during the course of the trial. So, um, we now, this is in the age of si, we were simulating control of type one error with the regulators.

With, um, uh, uh, in the scenario where we're carrying forward, the doses selected from the first part to the second part, we ended up sending them over 300 null scenarios to simulate to go to regulators and working with them to, to approve the seamless phase two, three part of it. Now we made some concessions to the regulators on this where at least 70% of patients had to come from the second part. They felt comfortable in that.

Do you remember what Rob Hemmings said when we came back to them with the updated simulations? He was at EMA at the time.

Has the, the reason you said, you know, the 300, uh, null scenarios is because they of course, were, uh, anal about controlling type one error. But type one error depends on, for example, what the underlying control rate is. It depends on accrual rate. Um. And so if you know, if, as Scott indicated, if the accrual rate is fast, you may not be able to take too much advantage and the type one error is affected by that or if it's too slow. So we had to do lots of accrual rate, uh, simulations.

They became in the next version of the adaptive guidance, which was nine years later, uh, they had learned. Uh, to be more comfortable with simulations and they had learned, uh, how to handle this sort of a scenario. So, for example, we've designed phase three trials, uh, uh, much, you know, since then. Uh, and we don't have to run nearly that number of simulations. We have to make them comfortable and, uh, promise them that we, you know, we'll do post, uh, analyses.

To make them even more comfortable, should that be necessary. Uh, and so that's actually written in the updated adaptive guidance. Just while we're on the FDA, uh, this was at the time the critical path initiative of CER, uh, uh, dates back to 2004. Uh, but you know, this was, uh, uh, Janet Woodcock's baby and, um. 2006, it got somewhat red revised and uh, but this was regarded as part of that critical path initiative.

Uh, even though as Scott indicated it predated the CID the complex innovative design, uh, initiative. And it was probably, I mean, I don't think there's been a CID that is more complicated than the award five trial.

You send this out to operations and they enroll gangbusters, which you can do in diabetes. So we simulated a range of scenarios and we looked at the likelihood of making good decisions, picking the right dose, uh, before it went to phase three. And Lily, they, they created boundaries and they said, we have to stay in these boundaries. And they did not want to enroll more than eight per month, for example, because they knew that the, it was going to make worse decisions.

And so they told operations, you can't go above this number, which I think was a huge part of the success of the trial.

They could stop the whole trial for safety, so that was still a huge part of it. But they knew simulation control of type one error was a big part of the design, and the utility index was a big part of it. And so the DSMB was fully on board with this. So you and I were, uh, advisors to the DSMB and we were unblinded, and if you remember these, these would happen every two weeks on Wednesday, uh, every fortnight on Wednesdays, these reports would come out.

So, uh, a number of really interesting aspects to the DSMB, which have been made public. Uh uh, so yes, jump into that.

Um, and it, it, it toward the end of that period when they could make, they could drop a dose for, uh, safety, uh, they were worried about the highest dose and they understood they were actually tracking things and, you know, uh, how things were going and how close to making a decision to pick this dose over that dose. And so they were, um, worried about the highest dose, and I said, you can drop it now if you don't drop it now.

And next week when we get, you know, the permission to pick the two doses, if you dropped, um, a, a dose that would've been picked, we still go with two doses according to protocol. If you wait until next week and the highest dose is one of the ones that's picked and you decide to drop it, then we continue on only one dose. So they understood that and so they hastily drop the, uh, the highest dose, uh, for safety. And it was, you know, the combination of, of, um, blood pressure and heart rate.

Uh, considerations that, by the way, um, these doses are now part of the marketed. The, the, the marketed doses were the ones selected in these phase three trials, that the other phase three trials were the same ones. And they were happy with that and, you know, we got the right dose. Um, but they, uh, learned over time that heart rate, even though it was increased. It was not that important and didn't have clinical ramifications.

And so the, the marketed doses now include higher doses than, you know, one dose that's higher than the ones even that we considered and the award five trial.

The tho, neither one of those doses were part of the original four dose phase two trial. So at the time, this 1.5 milligram and the 0.75 milligram were, uh, you know, really good doses in terms of this and they may never actually have been used. With the other, uh, uh, with the other phase two trial. So it had a number of aspects sped up development, and by the way, then it jumped to phase three. Those doses moved on and in into the other trials.

And hence going back to the start of the story, uh, it ended up showing superiority to Sitagliptin, won all of its phase three trials, showed benefit in its cardiovascular risk trial, and is now, now Trulicity.

What, what ha what, what we would've predicted from the award five trial. Uh, but it trulicity is not approved. Uh, Lily hasn't looked for it to be approved for weight loss because they have other Glip one agonists that are, you know, next generation that are even better.

Mary Jane Geiger, who Don mentioned Andy Anderson and Brett Brenda Gatos, who are on those papers as well, doing, uh, uh, huge work and, and uh, uh, working, worked with them on the team. And this was about nine months, uh, that we built this trial and got it ready to execute.

Um, all the way back to, uh, Charlie Sampson who set up, uh, biostatistics at, uh, Eli Lilly, and also was a co-founder of the. Muncie meetings for those of you that might have been a a around back then, uh, Muncie, Indiana where, uh, the ball State University is. And, uh, I presented their adaptive designs because of, uh, Charlie and others at Lilly. But we've, we've, and, and they designed, this was pro Prozac days. They designed adaptive trials in Prozac.

Uh, so they, you know, shout out to, to all of them. If you pick a random pharmaceutical company and drop us, um, by parachute on their, onto their campus and see if we can persuade them to do this kind of thing. I'm not sure of the results, but it's not a certainty that we would be able to do it because we don't have the, the trust build up and. And, and the, the great working conditions.

So it was all a great story for us, and it was a great story for Eli Lilly who decided that we gotta simulate every trial and every trial that we run, we're gonna simulate. And I don't know whether they're still doing that, but it, it's not a bad thing. I'll tell you.