We're going to talk about artificial intelligence and we have Carlos Colonga from where again? Hospital Regional Materno Infantil de Especialidades de Monterrey. He's from Mexico. We have Em Gootee, who came as my medical media fellow from Turkey after graduating medical school, and I was fortunate to recruit her to stay on as one of my colleagues now at Cincinnati Children's, helping us in innovation.
And Rami Shahaban, who I was introduced to years ago, who is an ENT surgeon who is now a Ph .D. and a professor in instructional technology at Utah State University. Rock on. All right. This is the team. They're going to update us on artificial intelligence. Take it away. So we are going to talk about AI, specifically regenerative AI use cases in medicine and medical education. and we have three categories of use cases research productivity and medical education So let's start
with that poll question here. So which of the following is false? Is false. Okay. So that's tricky a little bit. ChatGBT can help to organize literature review. ChatGBT can analyze data within an Excel sheet. ChatGBT can run statistical analyses. ChatGBT can help in the research design. Or none of the above. So what do you think? Any thoughts here in the room? Do you think any of these are false or all of them are true? What about the unpaid version? It still doesn't. Yeah. Mike.
Yeah. Oh, right. Yeah. We're talking about both of them, yes. Let's keep going. All of them are true. We are researching how ChatGPT can help us in the research process. I personally use ChatGPT in systematic review and meta -analysis. When you do systematic review, you screen thousands of articles, and then you have a title and abstract screening, and then the full text screen. And this process is intimidating. You have to have
many people involved in that process. So we tried ChatGPT to do the title and abstract screening, as well as the full text screening, and it does a great job. Do you want to explain how to do it? Yeah, sure. I think we have the next slide here. Talking about that so we have punch of articles to go into title abstract screen And
then we asked chat GVT this prompt. I'm conducting a systematic review of the use of AI in organ transplantation I need you to act as a data analyst Please analyze the attached excel sheet and screen the included articles for title abstract by the way that The AI in organ transplantation paper was published already so we used chatgbt in that use the following inclusion criteria we have some inclusion criteria here and then after screening the title and abstract so now we want chatgbt
also to manipulate the document itself so it's not only just giving me text in a answer through chatgbt but also go inside the excel sheet and then read the data from excel sheet add columns to the Excel sheet and say whether this article is relevant or not relevant or maybe relevant. Because we don't trust ChatGPT a lot, I asked ChatGPT also to add a column to explain why ChatGPT
decided that decision. So you have a better understanding about this data and if you don't agree with ChatGPT, you can manipulate the prompt, engineer the prompt to make it more accurate. Okay, and we get an excellent result with that. We had to do the human title and abstract screening also because we want to have a reliable answer, but we did inter -rater reliability between ChagVT and the human reliability, and the inter -rater reliability was excellent. So ChagVT could be used in that.
part yeah so what is what is the first cycle and second so yeah okay so uh first we wanted to use chat gbt uh to give us yes or no question yes or no relevant or not relevant and then we just redesigned the prompt a little bit to make chat gbt say whether it's relevant not relevant and maybe relevant so we decided to consider all of these versions of the prompt and then make the inter -rater reliability between all
of that columns. Question? I've done this, not for the exact same purpose, but I don't get the same answer each time. So my question is about reliability. I know humans are not reliable either, but it sounds like every time you offer this question, at least to ChatGPT, I haven't tried other ones, you get a slightly different answer. It approaches the problem like... for the first
time every time? I think so. Yeah, so if you use ChatGPT in the general search, you get different answers, but we created custom GPT for that. So you can walk ChatGPT in serial of prompts until you get the most accurate results, which is just not that difference between if you run it multiple times, you get a slight difference. And we had to have human reliability here also to just make sure. that it is reliable. But that's a good potential for the future iterations for
. I think also it might be one of the questions that, what is the AI limitations when you use it for an actual paper? and go for publishing, what is the disclosures that you do for the journals, do they have any limitations, do you have any extra information on that? Yeah, so we use ChatGPT here as a part of the methodology and part of the reliability process. We don't use it to, for example, as an act of plagiarism or writing, yes. So, yeah, our journals are very, yeah, of
course, yeah. And we frankly say that this cycle of screening is done by . The second cycle of screening done by . Third cycle of screening is done by humans. So, yeah. You always have to have a human in the equations. So we have to take in mind that these AI models, they help us increase our efficiency. And sometimes we equate wrongly efficiency to productivity, but sometimes you get productivity out of the efficiency. So it helps us be more efficient. by accelerating
how we do the work. But we always, at least in the medical and the science part, we have to have a human in the equation. Because we have to remember that even if we use the prompts that help you get the same results, because ChatGPT is a black box model that we don't know what it is trained about, and they're always changing the weights, it'll change. So it needs us to tell it what to do and to understand and read the answers of ChatGPT. Yeah, you want to move
on to the next one? Yeah. So let's go to talk about Jenny AI, whose motto is use AI to supercharge your research paper. So what does it do? You open the Jenny AI, which is one of different academic models that has been fine -tuned with research papers. And you literally put your prompt. I want to write about this, this, and that. In this example, let's talk about pediatric interception. You put the prompt. And the first thing that you'll get is a suggestion. You can start writing
this. You, because you're the surgeon that has the hand on the wheel, can say, yeah, I like it. I want to modify it. I want to change it in this and that way. And then, the next slide, please. You can also talk with its in -house chatbot. and say, hey, I really want to give it this pin. I want it to research specifically in this. And the chatbot will give you the answers
that you're searching for. It will cite papers to you, and it can help you cite on your preferred type of citation specifications into your paper. Next, please. So here. You've decided that, yes, I like what it is saying. I also want you to add this, this, and that on my academic paper. And it gives you a little suggestion of the summary of the papers that you want to introduce. So we can see how this accelerates your paper writing, but you're still at the command of it. You're
still in the helm. Next, please. So let's do an audience poll to see how you're feeling about this. So how recent are the articles suggested from the use of these AI models? So they have a cut -off date of 2023. They're older than five years, older than 10 years, and they're on par with the latest published research. What do you think? Any ideas? Can we see the answers from the international audience, please? I don't know if the poll is up. Oh. Can we see the poll results?
Okay. How do people know what's up? Yeah. So, mostly, nowadays, they're mostly on par. We just have to research into what model we're using and what library they're using, but it is on par. We just have to remember that they're searching. for those latest articles in this specific type of library, which we'll talk about it at a later moment. Well, that was the moment, yeah. So... Also, there's another tool with the Gen -E AI.
It's called Open Evidence, and I think you said this is the difference that this is not necessarily helping you write it, but it helps you do literature search. Exactly, which is amazing. This has been... This was part of the program accelerator over by Harvard team and the Magic Clinic. And they especially focus on the Elsevier Library. So you can ask any type of question. It will reply in a conversational manner, trying to answer your question while summarizing the papers, the
latest papers that it's citing. So it gives you a conversational answer. And then at the end of it, It gives you the papers that you can personally check out to confirm what it's saying. If you want to press the details, it gives you a bigger summary. And then it suggests, hey, so you're talking about these things. It's like you just cited Dr. Russell's paper in the second part. Yeah, yeah. And if you wanted to go... Accurate. Yeah. And it also gives you suggestions. So I
know you're talking about the appendectomy. Hey, you could also talk about this and that, which is up in the next slide. And you can ask follow -up questions. Yep, yep, yeah. So it's pretty great. So what would you say the personal practice changes for your part? For my part? Okay. So we are going to talk about the medical education part. Is that right? Medical education is coming
next. Okay. We're going to first talk about the productivity part, but I think it kind of summarizes that large language models can enhance the efficiency and productivity when you do research by analyzing the large data sets for you, or you can use those tools to help you write. This is especially very important for the international audience that have, once you have something on hand and work on it, it's way easier. and writing something
by yourself from scratch. So these large language models also can be an editor and help you write these papers. And we're gonna move on. Yeah, Jose. Dr. Holcomb, what's the JPS policy about these AI tools? I think we wanted to submit something. They asked if we used it. Yeah, so you can use it for background information, but you can't use generative AI, that is, to have AI write you the paper. And are you making sure it's not being used, or you have to disclose it somewhere?
Well, first of all, if you used it and you disclosed it, it's probably not going to get accepted. But we have, we being Elsevier, has tools to... to figure out if it's been used or if it's being used. So that's sort of the policy is you can't use generative AI to create your paper, to write your paper. I think it's okay to use it. You could, for language, it's probably appropriate. And for literature, you know, background information,
it's certainly appropriate. I think it all started a couple years ago when ChatGPT started getting used in different fields. The first couple papers were written and ChatGPT was among the authors. And I think it's not been permitted anymore. You cannot do that. But I don't know. Probably it's the same for all the journals. You cannot ask ChatGPT to write a section of it and show
it in the authors. Yeah, most of the journals I see is just asking you to, if you want to use ChatGPT, you use it to just enhance the writing style, not create the ideas. So, yeah. And our next question, because we're going to talk about ChatGPT and other AI tools and how you use to enhance your daily productivity, not necessarily for research. And we asked the audience, how often do you use ChatGPT or similar AI tools? It could be Cloud, Gemini, anything. It's a spectrum
from never to always. And one third is never. Whenever Todd sees those numbers. He kind of has a heart attack every time he sees people that never use chat, GPT, or AI. So if you first start with productivity part with MeetGeek, this is one of the tools that helps you. You basically synchronize this tool with your calendar. It
jumps in. Meeting that you attach to your calendar and it takes notes for you and at the end of it It sends you a mail of the transcription notes and the summary And it basically tells like who was in the Hoover in the meeting what was the summary and what you need to do it goes with the recording and it assigns action items per person who are in the meaning and Fireflies is a similar one with the Miki if you go look in the internet to see meeting summarizer AI tools.
They're like a bunch of them. Most of them does the same job. They're on different price points. Most of the time, it's just personal preference. They mostly do the same thing. I know you started using MeetGeek and then you transferred to Fireflies because we... Influenced you? Yeah, I think Firefly, yeah. So I use mid -gig, I use the paid version of mid -gig, and then Firefly is the free version, and then the paid version. And when you invite
Rami to a meeting, shows up as four people. Rami, AI note -taker one, Rami, AI note -taker two. The human version is not there, is that right? So I think Firefly is... My personal preference is because it has also action items that So after the meeting it it gives you the action items for what's next so it's just to keep you Informed it is great. I use fireflies because of you guys.
I started using Carlos also comes as a two people every meeting The issue that we've run into, because I was addicted to it, I needed it because it gives me takeaways and all the key points. Here's what you need to do. But then our hospital, people got worried, especially with some of the meetings I was having, that it was auto -recording everything. So you just have to figure out what's
the safe way. At Cincinnati Children's, they now have... co -pilot approved by co -pilot that's approved by the hospital but even then our lawyers asked me to stop using it on half of my meetings because they're legally sensitive well you talk about confidential things yeah confidential things so um it's it's We're going to make progress with that because this is such a valuable tool, and we have to get past the security part of
it, but it is game -changing. Especially if you're using an institution level, it needs to be a lot of preventative, like preventions be taking place for security purposes, especially like you and Dr. Winallman's level executive meetings, but like you and me in a meeting. Fireflies can record that. There's not going to be that one.
We've actually. Added it to our platform. So we use a co -pilot and that's the only thing that's basically approved You know for Carl stores to use in regards to note -taking or getting information Recording stuff and then you know analyzing it afterwards I think it's the same at Cincinnati Children's and they keeps that they Generally send us like weekly AI guidelines of how should we use it? And what kind of information should not be used when you use co -pilot or
ask for it? meeting summaries and everything. And I think that's a good step to endorse people to use with not giving away the secrets of the company, basically. Yeah, we have to have these talks. We have to learn how to use it because it's here to stay. We can't close our eyes and don't admit that everyone's using it to increase their efficiency and improve the way they manage their day -to -day lives. Zoom AI basically does the same thing because some companies have Zoom
instead of Microsoft. But the plus I saw in the Zoom, if you're late to a meeting and you have Zoom AI companion, you can ask if your name is mentioned, what people ask you to do, and recap the meeting just before you join what happened. So I think that's a very cool thing. And if you're generally late to the meetings, I would definitely recommend. One minute? Okay, sorry. Okay, this is Todd's favorite. Read aloud GPT. We created for it. You can use this QR code and you can
use it by yourself. It basically helps you listen to your long emails or any text that you need to listen to instead of you read. We know Todd talks, like have to not talk, sorry. He has to read a lot of things, the emails and everything. And sometimes, what do you call walk and talk? T 'walk. T 'walk. So it's hard to read when you walk. So you listen to it and you use the ChatGPT to have conversations. Instead of an audio book, I will drive. I'm about to drive to Cincinnati
after this. And I will talk to ChatGPT for an hour about a book. So I'll say, summarize this book. And I will go back like a book club. I'll say, tell me now, what does it say about this? Tell me, let's go into chapter two. I want to know this. I don't get this. So I will have a conversation while I work out, while I am driving. And I think this is one of the most important ones for this audience here. Todd said he's getting a lot of requests for letter of recommendations.
And we created this GPT for him. And you can use the QR code if you can't put the slides up here. But they keep saying we created GPTs. It's very easy to do. So if you're using the same prompts over and over and it's a series of prompts, you can create it and just keep reusing it. So when I have a letter that I have to write, it goes through M -Media for me. So when you use this QR code, you're going to go to this letter of recommendation GPT. All you need to do is
upload the CV cover letter. Why are you writing this letter of recommendation? It's going to give you a draft. You can start working on it and personalize it for the candidate. We do not recommend the first thing you get from Chachapit to use it, but it's a great start for... It will sometimes be over... This person is from heaven. I actually almost always come back and say, I really don't know this person. I'm giving a reference, so please tone it down. And then it's like, this
person is very good, blah, blah, blah. So you have to just tell it what you want. Like with Halpern, I'm going to say, please really tone this down. He's not that good. Do you want to do the medical education real fast? Yeah. Sure. Just one minute. We'll just talk quickly about that. The medical education part, if you want to have a personal trainer, we use also ChatGPT
and other generative AI in that. You can, for example, ask ChatGPT to upload a guideline to ChatGPT and then ask ChatGPT to create a training session, an interactive training session between you and ChatGPT. Can we put the slides up? Thank you. Yeah, so you can ask ChatGPT to create a training session out of a guideline so it doesn't have to go into hallucinations or fabricating
information. It takes the guideline and then it creates a series of interactive questions where you can interact with ChatGPT and get trained on specific things. So, for example, that's an appendicitis case that we ask ChatGPT to create problem -solving activities. And also we ask ChatGPT... to score these activities and give us feedback at the end. So that's the feedback
that it gets at the end. On a similar note, you could also make it do any type of case, a step one, step two type of case, any type of case. You can ask for it. You do the correct prompt, and with that, it'll help you do the medical case in a step -by -step manner, and at the end, it will give you feedback. All right, this was awesome. I let us go way over. That's my fault. I'm sorry. AI is cool. Keep using it. This is
phenomenal. These guys are great. We're going to keep doing series of courses throughout the year for these kind of questions. But guys, thank you so much. That was awesome.