We are going to now move on to our next session for accelerating patient enrollment by deploying a product model approach, and I'd like to welcome our two speakers to the stage.
Hello.
I am Tara Schultz. I am an Associate director in Drug Development IT at Bristonal Meyer Squib. I am the IT product manager for patient Recruitment him.
My name is Chen Deep. I'm the product designer at Bristo my squip.
Thank you.
So there's been a lot of talk at this conference and in the industry about AI. This talk will not be about AI, will be other things in it. We're going to discuss how we implemented a product model and product thinking which helps us with a build a strong foundation so that you can take advantage of all the technology has to offer, including AI.
Right. And it's also not about diversity, but it's about talking to your patients about problems that you're going to solve for them.
So we're going to talk about our new website, which is BMS Clinicaltrials dot com. It is a global website.
Yep, so we have a global reach. We had available globally in multiple languages, attracting a significant number of visitors comprehensive search in terms of accessibility for our patients, caregivers and healthcare professionals for BMS active recruiting trials. On our website, patients can pre screen and for any recruiting trial and then they can self refer themselves for the trial, find out the sites, and register themselves for the trial for those sites.
So our current our prior website which is BMS studyconnect dot com, we got a lot of feedback. We talked to a lot of people. There was a poor website user experience. Our patients gave us feedback that they were having difficulty doing the primary things that they wanted to do on the site with inefficient SEM and virtually non existent SEO. So our search engine marketing efforts were not as effective as they should have been. We were getting a low referral rate and no one was seeing us
on the seer piece. Patients were struggling to find the trials and find what they needed.
And.
We also didn't really appear on any search results, so we couldn't leverage what we were doing right.
So to overcome these obstacles, these problems, we adopted a product thinking approach dive into what that means product thinking combines product model and design thinking principles, and what it allowed us to do is to really focus on user center design while making iterative improvements. We started with defining our product vision, what problem our product is going to solve? Is it going to solve and how is it different
from other products that are existing in the market. So we looked at other competitive websites and saw what value they were offering to the patients, what kind of problem they were solving, And we also did our own user research, so we reached out to patients. We interviewed the patients and understood and mapped their journey throughout the trial and different steps about from diagnosis to finding about trials, searching
for trials, and then referring themselves for the trials. At a very early stage, we showed them the new website markups, compared them with the older version, and try to find out what really works for the patient because we wanted to listen to them, We wanted to understand their pain points and we wanted to see how we can solve
their problems by listening to them. And we spent most of the time defining the problem and then itating prototyping building it with our business stakeholders and with of course our patients, testing and validating getting the feedback on the improved designs. So on the right hand you see typical design thinking steps and on the left you see the product model approach. How it really couples to call it
a product thinking approach. And this is an example of the patient journey that we mapped out from diagnosis to finding about trials, lanning on the website, checking their eligibility of pre screening, and then selecting a site and registering them for the trial. And for each step along the way, we identified their actions, activities, who they were talking to, what kind of interactions they were having, what might be
influencing their decisions, their sentiments, thoughts and challenges. Now this is the simplified version of the patient journey. We, of course, you did it very exhaustively with our patients, and this was fundamental for our transition from BMS studyc connect dot com to BMS clinical trial.
So how did what did we do differently this time that we hadn't done before. So the first thing we did was we focused on MVP. That's our minimally viable product, that is the least amount of functionality that you need that will have the greatest impact. So in order to get to the MVP, you have to ruthlessly prioritize. That means saying no to most just about everything. You really got an arrow it down. We said no a lot.
We deprioritized a lot, but it did help us with the first goal, which is focusing, and everything that we did had to correlate to an actual business, a measurable business outcome. If we couldn't do it, it wasn't going to get prioritized in our release.
Right. And the outcomes were impressive. We achieved three times faster delivery and significantly reduced time to market. We achieved seven times more refles than we had compared to last year. Our solution was cost effective as compared to the vendor solution at that time it was proposed, so we saved ninety percent or within that budget, and then also conducted user research within the same budget, so we were able to define value and focused on key metrics rather than
website metrics. We were focusing more on patient metrics, what how many referrals we were getting. We were showcasing active recruiting trials on the website. We reduced the number of pages external or exit links, and that really significantly helped us reduce the bounds rate and increased engagement and search engine rankings, which in a way means that patients could find us higher on the search results when they would Google for a clinical trial, for example Looper's Clinical Trials.
At the same time, this also made our campaigns more effective, so more ROI on each taller spent and we were continuously improving delivering iteratively within two week sprint so business could see value more quickly, so business was happy as well. And all of this didn't come without any challenges. Our approval process was the most difficult, as you would know working in FHARMA. Navigating approvals from business stakeholders to legal to IRB stakeholders and then aligning them with our product
roadmap was the most challenging task. And then maintaining redirects, ensuring seamless flow of analytics for the campaigns that were already running and making sure when people are searching for
those campaigns they were hitting the right URLs. When we switched to the new domain and those analytics or metrix also flow to the dashboard for our business users and of course, resource allocations is a lot of work was UI development and development on aem Adobe Experience Manager on which our website is built, so that required a small investment as well. And now Tara will sum up what it meant actually for our different stakeholders.
So we had three groups of stakeholders. We have our operations, counterparts, our patients, and obviously we're in it, so we're stakeholder as well. So for the business we're we did accelerate our delivery significantly prior to the year's past. Uh, it's much. We've provided a very cost efficient solution and we increase the visibility on our search engine result pages. Now for our patients, you know, which is what this is really all about. We gave them a much much easier use
to use interface, so the user experience was enhanced. We've received seven times the amount of self referrals this year versus last year, and the trend is accelerating. So this shows us that people are really coming to our site to find a trial that they can qualify and enroll in. And we had to reduce bounce rate. A bounce rate is how long someone will spend on your website, so the lower it is the better. It means that people are actually coming to your site, want to be on
your site, and are engaging with your site. And from the IT perspective, we came up with a templated design, so it makes it much easier to implement, so we can keep that delivery pace of every two weeks and making sure that we're giving the business what they need when they need it. The reduction and the amount of pages, as well as the size of our site map makes
it much easier to maintain. There's a lot of focus right now on IT maintenance costs, so any solution that we put in, we really had to focus on how much the maintenance was going to be around that product. So a few things that we did so we did strategic collaboration, so we worked with the business and the
operations team and they give us the strategy. They left it to it to do the technical or the tactical implementation with our technical tools using the patient insights that we get aathered, you can really accelerate your product goal if you have clear high level goals. If your goal is to increase patient recruitment, that's the goal. It's one goal, it's not seven goals. It makes it much easier keeping our scope tight. This is always difficult because everybody likes everything,
but we weren't able to do that. We dropped a lot of functionalities and in each priority, in each product cycle, we prioritize what could be dropped rather than added. So you really need to look at your backlog and see what it is that is really going to move the
needle for your patients and your organization. And since we had so many stakeholders, we had a prioritization tech list so we could balance the goals of the different groups and assess the impact accordingly, because it doesn't always align correctly. So in this instance we have a few backlog items. We give it a number from one to five for business value, same scale for patient impact. Effort is the inverse, so we want it to be a low effort item,
urgency and then the maintenance burden. It gives us a total score and that helps us prioritize our items so we can deliver more effective solution quicker.
Awesome. Now it's time to showcase our product, though the links might not work, so we have put together screenshots at the last moment, so it is I see after all, Come on, yeah, I know technical issue, Bree, could you show the screenshot for the older version of the website. What I really wanted to do here is to show you the old version and how it looks like. This is our older website, and since our website is also available, we get in the transition transitioning phase, so it's available
for some other countries as well. This is the Canadian version. If you see the hero image, it doesn't really connect with all the diverse patient profiles that we would like to capture, and of course it doesn't focus on the patient much. And then there's no menu on this website on the older version, so that's also not an SEOS practice.
You have to have a menu so that when the search engine bought really crowls your website, it finds out the most important pages and gives the best experience to the user who are searching and finding that relevant search in Google. We can't, of course scroll it down, but this website had a long scroll and then also a lot of fasted real estate and a lot of exit links right from the homepage, which we rectified in the new version. So can we go to the new version now?
So this is how the new version looks like it's modern, but it's also very minimal and it focuses on the patient more. You cannot see the carousel of pictures that we have implemented, so we have put together screenshots capture all the diverse patient profiles. These are not all the diverse patient profile spot although that we could get approved in the minimum time frame that we could. It does put our patient and focus so they are able to connect.
And of course the name of the website, it's no longer called BMS Study Connect because we realized Study Connect could not connect with our patients, so BMS Clinical Trials was a more obvious choice. And please go ahead check it out on your mobile desktops and let us know your feedback. Thank you so much for your time. If there any question answered, we will be happy to take it. Don't hesitate, this is not a plant question.
Full disclosure. I talked to Terra yesterday this and it's fantastic because the multiple of seven referrals is very interesting when you're in the clinical operation space. Obviously, the one of the key pieces a lot of us are interesting when you're advertising studies is the content itself. I'm just very interested in because of the basis of the functionality, it sounds like it's pretty much a keep it simple,
stupid type mentality. We think in that with regard to the content thing and then presenting, because obviously the protocols themselves are extremely complex, and a lot of this conference is about how we best engage patients by making it as simple as possible. So I guess how you're generating that content, and then how you guess being able to that process of bringing it to making sure it's as understandable for patients to engage with right.
So I think as part of our prioritization we focused on as part of our MVP, we started with the protocol page three design and from our patient interviews, we understood what's more important to the patient. For a patient, what's important is their disease. For sometimes when you go to any website, any clinical trial webs you see various categories, but they don't make sense to the patient. For the patient,
it makes sense as their disease. So when you see the new version of the website, you would see there are two filters there, disease and location. We also understood from the patient interviews that location is the most important factor when they go ahead and select a site after they are matched to a trial. So it's important for them to know if you know, the site is closer to them, and it's also helpful later on reducing the
drop rate. Right, So, considering these two important factors, considering the patient experience and what we understood from the interviews, we kept that as the most important item on our backlog and move that and push that for the MVP. And that's how our protocol pages also look like I can't show you right now, but if you go ahead, you know when you will check it out. You'll see on the top right we have the nearest recruiting site section. As soon as you've you entered the location, you'll see
if there's a site closer to you or not. So yeah, really, patients driving that experience for themselves is what is the approach that we are following.
Thanks,