Thank everyone. My name is say It Heather. I will be moderating session. Welcome to the panel. How forarma is adopting jen AI across medical development? I think previous speaker has set us well. Those are the things that hopefully we will be addressing answering for you, and some additional questions may come up as well. So with that, I
just wanted to quickly introduce myself. I work at MERC Pharmaceutical and I lead AI in clinical trial operations, help support AIU spass strategy and so on, and in my leisure time, I'm also an adjunct faculty at Columbia University, where I am affiliated with Data Science Institute. So okay, with that, I can I can start with our panelist here. So on my right is Henry Wee is from Regional s JATA, and then we have Shamir Kadir from Sonfi
Jatasha from Moderna and Dave Appel from JANJA. And I ask you to introduce yourselves when you answer the questions.
Thank you.
So, like I said, we're well said to address these topics that we will be I will be asking questions of our testing panelists and we will learn more about it. So The first question I asked is for Sujata, share your organization's experience regarding citizen JENNYI for each user or employee.
Thank you sired everyone. Hi, my name is Sujatasha. I am from Maderna and I am kind of not in.
The digital team. I come from medical writing.
But we've been exploring this idea of AI for quite some time at Maderna. If you kind of take the journey of evolution with me for just a moment. Remember when we had these big supercomputers that just a few nerds, as David said, used.
And then they become.
Smaller and smaller and smaller.
Until we now have supercomputers in our hands and we use them all the time for everything.
That's kind of the evolution that.
AI has been taking. So, as El mentioned earlier today, AI is not new.
We've been using it, but it's only been.
Used by a select few people that were experts digital people. But now it's getting to be like the smartphone in our hands. We all have access to it. And at Maderna, the leadership team realized the evolution that was taking place in the world of technology pretty early, and so a few years ago we launched what was called the AI Academy, and everyone in the company was required to attend AI Academy, where we learned what was happening in the world of.
AI, how things were evolving. Pretty soon we were all going to have access to AI.
So it started creating this buzz around AI at Maderna. From that, from the AI Academy, we launched a hackathon and that kind of helped us identify this.
Lead group of people that were not.
Necessarily digital, but we're really enjoying AI, and so then we made it really attractive. We created this really exclusive club called the AI Champions Team, and these were the people that had adopted AI. They had participated in the hackathon, and they were from all different parts of the company, so they were talking within their teams about how AI
was going to help us do our jobs better. And because it was an exclusive club, as you know with any exclusive clubs, everyone wanted to join it because these were the people that were getting early access to the newer technologies as they were becoming available. And I know one of my colleagues was on that team, and every day I would ask him, how do I get on this?
I want to be part of this, I want to get I want to get access to the technology, and so that just created that buzz of everyone wanting to use AI. And from there we started a partnership with open Ai, which allowed us to get enterprise level licenses for ched GPT for the entire company. And at this point, what we've done is we've opened up the creation of agents or GPTs to everyone, and it's very transparent to everyone.
So I can see the GPT that David has built and say, oh, I can take parts of your GPT and adapt it to something I want to do in my job function point. We built about seven hundred and fifty was the count last week. It's probably eight hundred by now or so. Agents within Maderna that everyone is
allowed to use. It ranges from everything like make this email a friendlier or what are the benefits that are available to me through Maderna, or all the way to data analysis and being able to identify the dose in a study. All of this of course is with engagement with human beings. So the human judgment is of course a part of all of this, and that that's where I'll stop, and that someone else.
Thank you, go ahead, David.
You share with day A bell I lead External Innovation, so partnerships for RNI data Science at Jane Jay Innovative Medicine. What you said totally totally resonated with me, and it was great to see Mary Jill and Bardie stalk right before this, which actually has data on a lot.
Of these topics.
Not to put us down button contrast with the opinions we will and maybe a little bit experience that will offer here today.
So I mean we're.
Now not quite two years into the chat GPT era, which is you know, that's really the moment where leaders at companies and pharma were said, oh aha, there's this huge opportunity here and even if we can't fight quite grasp it yet, we see the We see that data science and AIII is not going to be a niche technology. This has the opportunity to improve a lot of what
we do. And so I mean we're working across all of the different vectors there, meaning deploying AI tools for everyone's you know, regular workflows like writing emails, which unfortunately takes a lot of time for probably everyone in this room, and then also specific use cases like medical writing, which a lot of companies, including ourselves are looking at as areas of opportunity.
One last thing you said rested.
To do with me, which is getting people excited about this right, and in particular you highlighted the uh, the power of the fear of missing out. FOMO is an excellent an excellent driver of human behavior and it is for better and worse.
Uh.
It creates a lot of a lot of a lot of buzz. And so in contrast with two years ago where you know AI and generative AI, we're seen as this niche, niche tool, you know, living in honestly in like discovery use cases for protein folding and structural biology.
Now everyone wants to be a part of it that creates its own problems, but it's also really exciting for those of us who are excited about the opportunity and want to realize that opportunity for the benefit of the company and the benefit of patients.
Most importantly, absolutely, thank you both, and I just can just re emphasize that you know AI education, AI engagement is very important for driving and building their trust confidence. We saw from Mary Joe's presentation building that you know, trust deficit. So my next question is for Dave, what is strategy by a form should adopt for you know, deploying these solutions at a scale, so as we.
Saw earlier from Mary Joe and I think reflecting a lot of probably reflecting the experience of the people there. There are many many use cases in development, however, there are few that have been truly scaled and that that makes good sense, right like pharma is uh risk averse and and and.
That's appropriate.
Given the you know, the the importance and risk of what we're we're working on.
The advice I would give is to take a.
Testimonlear an approach to run pilots with with partners who run pilots internally in use cases.
That are.
Either uh you know, lower risk for for patients or the company, or in in areas where it's kind of contained and isolated. And in doing that, you can actually address a bunch of the other questions. You can get the buy and you can need, you can illustrate the r o I, and then after doing that you can you can scale up. It's the same as any new technology. It's something that I think a lot of companies are
struggling with. We are actively building that muscle and kind of at the edge of moving things from you know, pilot stage to.
Scaling up a bunch, So that would be my advice. You got to you gotta get started.
Thank you.
I could not agree more.
And you know, thinking about value proposition, thinking about feasibility, and we are all at nissant stage. We saw that from Mary Joe's sort way that the industry overall is at a nis into stage and adopting AI, so we're all in it for other. My next question is for Henry, what approaches have you seen to separate the high chrome reality? For GENNYI use cases.
Thanks. Hi, I'm Henry.
I'm a doctor by background, and I had an innovation for global development at Regeneron and we try and make trials faster and cheaper and better, so to separate the hype and reality.
I used to be a lot meaner about this.
So if any of has heerd about like for Matt's last theorem, sorry apologies and salespeople in the room, But I would basically make a word version of that and say like, oh, can your solution basically solve for integers that you know were the exponent is? And they'd be like oh yeah, yeah, big time, And that's like mathematically difficult to prove like from a historical perspective. So that's one way of flushing it out to just kind of pose questions that we know to be actually hard problems
for humanity. I just not say there impossible. The more practical actually follows what Maderna did, which is that we actually have an open innovation model that we recent implement. I'd invite folks to go to regeneron dot depthpost dot com.
There's kind of a.
Dumb video of me there, but there are about fifty some entries where we actually asked folks without actually implying use of AI or not, solve useful problems to solve, and many of them actually tried to implement AI, and we got to see whether or not it was feasible or not. We also got to see patterns of how folks were obfuscating and trying to cover up the lack of utility of their AI, because we were able to pop up in the code and see it was basically an empty shell.
So that was very instructive for us of having a diverse external group. Hammer at that and other companies have done similar exercise. A non to the literature and our peers doing great work with CSR writing I've seen in the literature, so I'm a big fan of that kind of inclusive, all hands on deck type model to really figure out what works early on.
Absolutely, thank you, and I think I saw one of them talk yesterday. We're on the gardener hive cycle. We are now on the other side of the curve, and now reality is hitting the door, so it cannot be more true.
All right.
My next question is for smir how do data types influence the performance and design of algorithms and why is it often such that the quality and the structure of data are more critical than algorithms themselves.
Thank you saying thanks for having me. And this is actually my third d farm and last two times i've given talks around advanced algorithms and graph EMIL and approaches, and I never seen a crowd lighting like this. So first of all, let me acknowledge that you know, for sure GPT is the iPhone moment that we're all looking for, right, and then at the same time that that core piece
of technology is only six years old. It's built on top of a lot of other technologies, you know, whether it's transformers and bert and others that's built, but it's really geept's a six year old technology. Just so just imagine defarming another six years and in a way that but from a clinical development perspective, we should think about it. Like you know, I'm sure many of you use algorithms. What drives an algorithm sixty percents your data? How good
is that data? How well informed, well connected is that data? So this is something that you all have to think about for that matter, whenever you're designing and investing in an initiative like this, because right now you know, all the way from your tech team to the c suit, everybody is interested in going GENAA. But I know there's also session on GENAI. But I want to tell you that many times you might not need a GENAI strategy. There, you might need an AI strategy. You might need a
digital transformation strategy. By the way of introduction, I come from the Christian Medicine and Computational Biology function and safe we support our end their immunoscience and other therapeutic areas. I lead the computational commutational team within Santa FE and in my personal opinion is that you know, the data matters a lot A lot of time we invest in the latest trend latest technology, but we often forget the importance of the data. So you know, down the lane.
Over the next six years, I can tell you that all of our companies will have their own foundation models built using our own internal data. So if you don't have data centric efforts in the company, maybe this's a good time to start. I may just also add that from a from an AI perspective, Santa Fe is a company that's on a mission, on a transformation to be AI first in its approach and then also immuinoscience centric.
And then we are investing heavily in both what we call as our CEO or the piece, and I think Folbes as a snackable AI that's open to almost every employee in the company and an expert AI where you need advanced AI you know all code, low code or you know, no code skills to do that interpretation. So we're seeing that already.
You know.
Then they are a large organization with you know, fifty one hundred thousand people are already embracing AI, and almost every SIGNI is now one a chat interface to the scientific databases or to have an interface with the patient instead of having a normal challenge in people. One generative AI challenge and so this is here. But remember you know your data matters a lot because a lot of the other pieces, like the whether it's a GPT or others,
they are now commodity assets. Now you know, you know, you can, you can plug and play with different algorithms. They're all getting better over time. So let's make sure that you know the data. Data is the king in my opinion, Thank you, Shamir.
And with data, I recall that people usually ask me when this model was trained and we have to tell like it knows the world as of August twenty twenty three or September twenty twenty one. You know, so these models are trained at a certain time, so they don't know the data that has occurred since then. So a great feedback. Thank you. My next question is again for Henry. How are you thinking of assessing the performance of GENI solutions.
Yeah, I'm very.
Excited about this, and I would nod to a colleague, John Curry, International Man of Mystery, one of our closest working colleagues, having a really solid data foundation, not in training data, not in kind of data for in France, but actually instrumenting your operations and having great discipline around measuring cycle times and performance and other attributes of human activity is extraordinarily helpful at helping you identify where the true pain points are and whether or not these solutions
are actually helping things along. So a lot of this is predicated on just great operating discipline and strategy being put in place well in advance of folks actually even considering JENII, and it pays div ends right there and then so I can't say enough for just yeah, how are we thinking of assessing the performance of Jeni? It's the question is actually how are we thinking of assessing performance?
Very nice, thank you. Yeah, this is a key for AI success implementation that we are able to compute the metrics, collect the metrics, and then demonstrate that actually helps leadership and the business to understand and adopt we are making value, offering value and where things still or may not be as feasible at the moment. So thank your gain and just.
Quick yeah, very quickly.
So from a medical writing perspective, we're using and testing AI to summarize tables or do the simple things that medical writers do. And here we really realize how important the quality check is because hallucinations are common with AI, and a lot of our medical writers will say, well, I'll use AI when the hallucinations are all gone. I so, well, if the hallucinations are all gone, then what are you doing? So the importance of that human judgment in every step of the ways is really important.
This is one of the most critical and trickiest topics because all of these models are proliferating, all these use cases are proliferating.
So many companies are approaching.
BARMA customers with solutions and trying to figure out which of the good ones is. You know, a considerable effort, and it's sort of a like by definition, a multidisciplinary effort. You need the data science tist in the room, who as assessing the architecture, who's assessing benchmarks to the extent that those are available.
But then you also need.
The domain experts, right so in medical writing you need the medical writing.
Expert to look at this and saint does this work with workflows? Are they solving the right problem?
And this is challenging, But I think what's interesting about it is jen AI has forced a lot of people to work together and come together that never work together before.
So Jane Jay, the data science team which a month.
We're now working super closely with the medical writing team, but I think like two years ago that was not the case and there have been no collaboration. So you know, you need to bring both parties to the table and kind of work through the kings there. But you need a whole bunch of people to really do this, do
this effectively. And so the data science side, I would just say, we, you know, it's critical that we value the expertise of our of our functional business partners who really own the problem and understand it at a level maybe the data scientist down, So take some humility.
To absolutely and I think that, Oh, go ahead, tru.
I just want to, you know, just with respect to the performance of Genia solutions, maybe I just want to ask a question, with your permission, how many of you you think you currently use an AI solution as part of your clinical development? Fifty thirty percentage maybe in the room? How many of you have taken a decision based on a recommendation from an AI solution? Maybe thirty percentage of that?
So you see that, right, like you know, when you when you you know the performance, you can come up with any type of KPI and OKR and metrics like that, but it's really up to them, those who actually have taken an action based on that feedback. So that human element, it's very difficult to capture, but it's very they actually brought that AA solution into their decision making. It augmented
the team. It make the team move faster. That's not the case with all the use cases, but at least in some of the use cases, we see that here. So think about that, and it's an important that you should, every one of us to teach our colleagues, our executives about this. AA is not a magic wand it needs training, it needs help, and it's just another tool and you know, and it's moment right now, but think about its limitations.
The real magic comes when your subject same is use this to I don't know, ten X or you know whatever improvement in their productivity. That's what you always think about. Metrics are great, but don't forget the human factor.
Yeah. In fact, you know this term has been around for some time, maybe more than ten years. Citizen data scientists. I think in all practical purposes, it is coming to life that GENEI empowers each one of us as a data scientist. Think from that perspective when you're crafting your prompt you know, when you're asking a question, it just empowers so as as. And then the multi is disciplinary.
I was attending FDA workshop last month in DC, and multidisciplinary was one of the key them that you know, you need teams which are multi disciplinary. You need data scientists or medical writers or medical writers who want to be data scientists.
The lines are blurring.
AI has just democratized these skills, these opportunities that the lines are blurring. So thank you again.
With that, I.
Will we'll move to call for action. Dave should be start with you the call for action.
Sure, I would encourage everyone in this room to embrace the ambiguity and get started in trying to use AI and jen Ai in your in your data workflows. It is coming for all of us, and we have a responsibility to patients to make our trials and get therapies to patients as quickly and as effectively as impossible. So get started in using this in your workflows, bring ideas forward, test it out.
Thank you to Jota.
Please, my call for action is really simple, just start using it. I hear all the time, Oh I need a reference manual I need this to start using DPT. Did you have a reference manual to use your iPhone? No, it's just like that.
Just start using it.
I would say, you know behind you know, we're all in this business of finding medicines, you know, life saving medicines, and behind every data point that we handle the is a patient. So you know, be responsible about that. A I can make mistakes, you know. That's why we need semes. That's why we need I like this term thoughtful AI. You know you we need to have that filter. These
are machines trained using massive amount of data. They can make mistakes, but it's an amazing tool that if you can tackle it to solve appropriate problems, and it's it's probably the most exciting time to be in our field as well, because we're all building it together. So enjoy the ride. I would say, for for sure, I would.
Say, make sure you're finding that path is a very deep empathy.
This is scary.
This is unfamiliar terrain for folks who don't have a quantitative background. And if you're like a sociopath and you're incapable of empathy, just fake it. But somehow, you know, really understand where other folks are coming from first.
Thank you very insightful. I would I would suggest that you know, I recall a little bit of my tenure at Behavioral Science that there is this theory called theory of planned behavior, which is about motivation, opportunity, and skills when you want a desired behavior, so I think AI education and sharing success stories uplifts the motivation provides the skills and just unlocks that, you know, adoption that we are looking for. Thank you everyone for being here and providing your perspective.