[Music]
Hey, welcome back to the Papers Podcast. We review the Help Professions Education Literature, laugh about it, critique it, make you smarter, and we do it all with you in our community. So, the usual hosts, trouble makers, and crew are all here. Welcome, Lara. I'm going to be the host, not the trouble maker, over here. Host, okay, and Linda? Hi there, everyone. And John, now, no, a bunch of us are jet lags still. This is the third episode of the jet lag series.
But John's off to work, and he's popping caffeine pills. He's drinking red bowls. He's got to stay up all night and work. So, good on you, John. It's your turn to have a paper. You know what the real tragedy is? I have a very, very fancy coffee maker. If you're a coffee shop like me, I have a juro micro. It broke today. Oh, no. I legitimately cried. I then put in the front seat of my car, but the seat belt on it so it didn't get hurt.
I drove it to the service center, where they say they might have to send it to Switzerland to get it fixed. Oh, my God. And then I broke down. Like, I really had a moment. Did they give you like a, like, you know how? A loner. People will give you a loner card. Yeah, they gave you a loner thing. The terrible Starbucks instant, which is terrible. Oh. I was, I'm snorting it because I didn't have any. But anyway, I thought you could give me it intracically. If I could, I would.
I don't know what that means, but it sounds nasty. Moving on. It's not exactly what you think it means, but it's pretty, it's pretty fainful. All right. So, I have a paper for you. Let me just, let me, let me warm you up. This article discuss the utility of chat GBT for academic writing. The author recommends using chat GPD to sport laborious writing tasks and refining its output with one's unique voice and style for creative writing. And guess who wrote that kids? Chat GBT.
I decided, why am I using my brain and my fingers like a sucker? Let's get that JNAI out here and do all the work I need in prepping for this podcast. In fact, I best or a producer Samuel to see if he can deep fake me and just put me in. I mean, I gotta do better than Jason. The deep fake version of me has gotta do better than Jason for sure. Actually, I need to interrupt at this point, ladies and gentlemen, or listening. We've actually been deep faking John for several weeks.
He's not aware that he's a robot driven with the chat GBT drive. But if you wouldn't mention it to him, he's a bit sensitive about it. My patients are quite pleased. I've seen too many of you. Even though you've had no idea. You know why I had to do it. Well, yes, I've been known to do that too. So you all know I had to do it. Generative AI is at the forefront of all of the things I'm thinking about in my academic world right now. I was part of a university task force on AI and education.
And I gotta say it's a bit of a face bomb because we forgot to ask a computer scientist to be a part of that task force. No. Yep. Well, anyway, here we go. I've been looking at the HP literature for a bit for any paper that allows us to jump into that conversation. You'll remember back in the late winter, we covered one of the first past the post. Is that a Canadian idiom Laura? Is that a worldwide kind of thing for a person? I don't know. I'm Canadian too. I feel it works for us.
Okay. Anyway, in that paper, we talked about chat GBT 3.0 successfully passing the USMLE. And I gotta tell you, if you haven't looked at Gen AI recently, it's gotten better and better and better. It's starting to get really impressive. And we're talking based on changes month to month week to week. But the literature hasn't really gotten much better around it. Everything seems basically a bunch of copycat manuscripts that show new tech is better and faster at conventional educational tasks.
You should look at some kind of issue around assessment. And so I haven't seen much about how we can use Gen AI in other ways. For example, what's the role for it as a teacher as an educator? So I'm curious to know Jason Linda Lara. What's your thought about Gen AI in HPE? Is it ethical? Is the sky falling? Are you excited? Were you landing on this? Are you as preoccupied with this phenomenon as I am? That's easy. No. Because John's pretty preoccupied with this.
Because you love all new things, John. Everything new. Listen, I am really excited about the potential for Gen AI to be a supplement I support in practice. And to be a helpful adjunct to teaching and learning, I am not so excited about it to help with writing manuscripts where I'm concerned that there may be some ethical issues there.
I'm less concerned. I really do think that this is a tool. And the more savvy you are with the tools, like all the other tools we have in our disposal, then these are resources we can use to make our work better and make it easier to do some tasks. It doesn't make the harder tasks any easier. And it'll certainly create other tasks that are hard.
But I'm in. The only thing I need is I need somebody to sit down and teach me how to use it. That's like literally how to find it and how to log in and stuff. And then I'll do the prompting and all that. I have no problem with that. We're still working on your microphone technique. We're going to get to chat GPT next week. Wait till I can figure this thing out. And then the VCR.
So I've gone from where Jason is to our lures. I was concerned I was thinking about plagiarism and how would we tell whether something was chat GPT, although I gather there now is a way of doing that, you know, for student papers, et cetera, to hearing that, yeah, you know, it actually makes a lot of mistakes.
I love the term, hello, sonating, which I gather is the right term for when chat GPT makes things up. And what it tells me is you can't let it do the work, but you can let it help you do the work. So I've titled this episode just kidding. I didn't title this episode at all. Guess what I did. I made chat GPT title this episode. That's awesome.
So chat GPT title this episode about itself as chat GPT unplugged the academic writer's secret weapon. The actual title, which I don't think is good is what chat GPT came up with. That's better than anything you ever done, John. That's really good. I'm not. Seriously, that's like a good one. Yeah, I know. Dang. So the actual title is writing with chat GPT an illustration of its capacity limitations and implications for academic writers.
It's published in perspectives on medical education. It came out just to the start of July this year and the author is Laura Lingard. Now, just a bit of a background about choosing a commentary. I never choose a commentary because it doesn't kind of match with the vibe that I think we should be going with for this podcast.
We should dissect the methodology and the empiric evidence before we all jump onto our bandwagens and our high horses and argue with each other and have some kind of evidence to at least inform our position or opinion. Unlike a research study, the commentary is all opinion, but I'm going to make an exception today because I think this commentary provides an excellent worked example of the potential benefits of generative AI.
In showcases where I think academic writing is going to go in 18 months. Lingard provides a counterpoint to those you think, you know, sky net is sentient and the sky is literally falling or is it burning? I'm not quite sure what's going on. So this is the purpose from the author. Drawing on chats I had with chat GBT in March and April of 2023 to illustrate its capacity and its limitations.
I extract a series of cautions and insights and guide writers on how to use incremental prompting to train the software. How to use it for brainstorming and generating content like outlines and summaries and how to employ it as an editor. So essentially we don't have a method section in I foreshadow that someone. Oh, I see some head shaking. All right, Linda, you think that there's a method section we should talk about. Go ahead.
Yeah, I think there is a method section with because Laura lie describes her interactions with chat GBT and I'm sure you're going to tell us a little bit more about that. But then she actually draws some inferences from us. So she's basically doing a qualitative analysis of chat GBT's responses to her prompts. So I think there is a method section that reminds me of of clinical studies that are in of one right you you try something and see what happened.
And I actually I think this bit of method to this, but that's okay. I thought you're going to make a Lara's head explode and someone's going to say auto ethnography with computer. And I'm glad we didn't do that too. Oh, that's a good idea. I actually say it. That's great idea. Do you know chat chat GBT helped me predict the future I asked it. Is there a methodology this paper in it said it said no, but Jason Alinda are going to try to make an argument.
But it's a study. But it disagreed that there's no case study methodology. It actually can predicts the future. It's it's pretty shocking to me. So I don't think there is a true methodology in terms of the rigor and the insight and any type of reflection or meta thinking around here.
But I do think that this is not all opinion base. This is a worked example and it's as such. I think it kind of performs above the the typical level of a commentary of I think or I have an opinion of you can actually see it happen. Lara, you want to jump in here? Do you want to go back to the auto ethnography? No, I don't want to do that. I want to ask you to.
It's kind of a definition of terms because I just want to make sure I'm really I really do think it's a tool, but I want to be sure I'm clear about what it is because it's a language prediction tool. Right? Correct. Now I'm not a computer scientist, but essentially what chat GPT has done. It's been exposed. You play one on this podcast. Compare to you. I'm a computer scientist. Hey, there it is. And I'm also an education rocket surgery specialist.
So I signed up for that weekend. Of course, if you are computer scientist, many apologies. What chat GPT is is it's been exposed to basically the entire internet up to 2021. Speaking, we should state stamp that right now. As of October 2023, Microsoft was financing open AI is going to let it start to do to searches and start to learn moving forward.
It's one of a number of large language models and what it does is using machine learning. It finds correlations between words and tries to predict what is the next correct word that makes the best sense when a query is posed to it. So it is not it does not possess knowledge. It does not reason it tries to use probability to predict what is the next word that can come along. So for example, it can hallucinate by making up what it thinks is a really good solution and presenting it to you.
Factually, when there's no factual basis for that, it's not reasoning from first principles. It's simply trying to use language to say what's the next best word that comes along.
And as it is learned from trillions and whatever the bigger word for trillions is, trillions of trillions of dialogue, it is able to predict what should come next with a high degree of of alacrity using these machine learning non-uclidean kind of mathematical models that tries to find correlation where there might not be a rationality to it. I think John just threw a bunch of big words in there. Like after a while it was like, sounds smart. But wait a second, those words don't put together.
I mean, I feel like I'm going to say non-linear. So it's a non-linear analysis that tries to say correlation is not just a equals b or a equals half b. It can find ways to see relationships that you would never see in a linear kind of way. I have no idea what that's what I'm just saying. We're going to go back on that channel. Alright, anyway, let's get to the results because that's really this is a work to examine this commentary. So let's hear what is it a commentary?
Let's and moving on. Okay. Alright. And I'll just signal for you if you don't know Laurel Lynn, Laurel Island guard. She is someone who has kind of led what's called the writers craft series with a bunch of other collaborators.
And this is a series where they try to help scholars and academics right with parsimony with clarity with accuracy with engagement. So they try to give you over the course of the last I think it's about seven years now that the series has been writing regular subscriptions in PME around here's best practices in writing.
And so this is not a novice writer. This is somebody who would probably regarded someone who writes a lot in and this is her commentary about how you can use this emerging technology to improve writing even if you're kind of at the top of your game. So first is you need to train chat GPT via incremental prompts. The first thing that you put in will give you an answer that's generic and superficial.
And so what I was articulating is these large language models. They don't have any ability to search information and to clarify. They're just trying to find correlations and predict what comes next. The more precise the more narrow the more nuanced your prompt which you can build bit by bit by bit and keep saying no refine this or do that you get a more accurate outcome in what you want.
And so for us when you're writing from a academic or a scholarly point of view it is entirely dependent that you have subject matter expertise. Otherwise you'll easily be fooled by a hallucination from this model that's trying to give you an answer. And so Lingerd can articulate there's a couple times when chat GPT starts quoting her previous work incorrectly in only because she's an expert in what she's actually written.
She identifies that as a big error. And you need to train it every time you start a new query. You can't work with chat GPT and then come back and start open a new chat box and imagine it remembers or even cares who you were where you were. It starts back at zero. And so this is always a start from scratch process, but you can go back to saved queries and build from there.
So it's not a you're always going back to ground zero, but don't imagine after you've been working on something and you come to a new project. It's learned or no or knows who you are or the lens that you're pursuing information around. Second, chat GPT is great for the LP brainstorm. It can help create an outline for a topic or paper you want to put. It can give you a great title. So I asked it for a title and said, no, give me 20. And then I looked through those 20 and said, oh, I like that.
It took maybe a second for it to me, 20 titles. But because it's because when you ask it, things for which you need true outcomes that are not just general ideas, you will be fooled. So you can say, give me 10 references that I need to think about as informing my research question because it is not keeping current and it will only predict what the next word is. It's not actually looking for those references. It's not actually trying to adjudicate which reference is good here. I've tried this.
It makes a great reference. It makes a lot. It does association. Yeah. And so the authors are authors I sometimes recognize, but the titles are wrong. And I when I started first starting to play you're out of this, I was fooled. And there was like, oh my gosh, this is really dangerous in the hands of someone who's not a subject matter expert as I was like, I didn't know that person wrote that paper. I haven't read that in blah blah blah. Oh, they haven't because it doesn't exist.
There's a demo by Eric Warren from Cincinnati live demo. He said, hey, chat GPT, please update my CV and display it. And it was like the first one was right first item. Then oh, a Nobel Prize something about Catholic history and then like just like hallucinated the rest of the document. It was awesome. Wow. I like keep that CV and submit it for the most. Nobel Prize. There's a little bit of impelianology. Oh, as one does, you know, on the weekend.
Next, chat GPT can review your writing and generate counter arguments saying, I've argued this. What would be account argument to it that I found compelling. Chat GP can review your writing and generate a summary. So sometimes putting together that first draft of the abstract, which can be the most tedious, the very last step that I do. I make and read your manuscript and generate that as a first draft for you.
And then finally, I'm chat GPT can provide copy editing, improve writing. It can help you with those transitions between paragraphs and improve that. And particularly if you're publishing in English and English is an additional language for you as an author, it can serve as a very inexpensive and very accessible resource as you make those. Translations from your first language into English. So Linda Lara Jason, I'd love to hear from you.
Of all those ways that chat GPT can help you something resonate with you or are you still concerned with the ethics of this? So I'm not concerned with the ethics and I think the thing that resonated with me the most was. Generating a summary or an abstract and improving clarity. The other thing which I think was mentioned in here somewhere was the idea of training chat GPT to actually come close to your voice.
I think the functional word there is training. I got to say I actually love this paper as with many lower life papers. It's written well. It's very readable. I could give it to a non professional audience and have them read it would be fine. My only complaint is that they use this screen capture to capture the chats. And I needed a bloody microscope to read it. The font is so small other than that I think it's a fascinating paper and I learned a lot from it.
So I'm going to echo much of what Linda has said. I think the two things one is the one thing that I hope all of our listeners do is take the time to go out and find the writer's craft in perspectives on medical education. That column is worth all of the time of actually sitting down and thinking about it and using the skills that are in there. It is legitimately one of the best writing resources.
Never mind in our own field like scholars from other domains can use these lessons. They're really excellent. So first, please take the time use the use the resource sets that's provided there. The other thing I just want to touch on in terms of the ethics. That's the only place where all of this gets for me a little bit tricky.
Not because I'm again I don't think there's a den of evil people somewhere going. Oh, I'm going to chat GPT for the rest of my life and never work a day and get paid for it. I just I don't see a reality where that's happening anytime. John like if he could he would. Right John. Is that true. He's not talking to you. He's just smiling. He's just saying. He's not worth it. Yeah.
No, so the ethics of it is something that I think at some point when it becomes more than language prediction. Maybe at that point we have to do think and do some thinking about the ethic. But the work of being creative of having created ideas of doing rigorous research all of that that's that's the bread and butter of the scholarship of the work that we do.
The chat GPT is not going to do that if it helps you write it better. Honestly, if it is if it's a field leveler for those individuals who don't have English as their native language as their first language. I'm really okay with it. Like then it actually it is incredibly ethical then because it's taking out that it's a it's a leveler and so for that I've got yeah I like it.
Okay, I'll just pile on very briefly. I love everything that Laura Linger writes. She has a magical way with words and this just flows nicely and it keeps you moving through it. This is a very basic introduction to the properties of chat GPT. So if you have never seen a presentation or haven't read anything else about it, this is a great start. It gives you just some principles to as guidance as to the current state of it.
But there's a lot more to the potential for chat GPT beyond this article. For example, at one point in the article, Laura Ler, Laura, sort of says, like you can't put any more input into this. It gives you an error. Well, there are ways around that. There are ways of programming and folding your questions. So there's a lot more beyond this. This is a great great intro.
And it'll help some people sort of get their heads around it. The thing that was most interesting to me, I reported John's question is looking at thinking about through counter arguments. It's like having a little thinking partner. It's kind of cool. The last little point that I think is comes from this article is you need to be cautious about what are the journal criteria and guidelines around the use of J.A.I.
And you should be thoughtful in terms of doing that. I will tell you that I have recently produced a book chapter for the good Dr. Varrio. And I used chat GPT 4 and had it acknowledged and you guys scratch your head. It's like, well, we didn't see that in common, but I'll let you sort out how you want to do that. There was no guidelines for you asking to write this. There's no guidelines for this. I don't know what we're doing.
I want to get the big meta narrative, which is embedded in view throughout the whole commentary, which is chat GPT is a tool in the same way that SPSS is a tool and Google scholars a tool and so tarot is a tool. But it's not an author and that's where the ethical stuff breaks down. And so if you imagine, oh, I don't have to do anything more. I'm just going to throw some data into chat GPT and say, make me a paper or write me an introduction.
It's a tool to help you be better at you coming up with your own argument, own ideas, own insights and own analysis. And that is a tool making more efficient, but it's not an author. And that's really where, you know, at the end of the day, the adjudication about the ethical thing breaks down for me. I think we're going to have more challenges for people who are coming up and don't see the nuance of that argument.
So I think we need to be being front and center about how and how these generative AI tools are being incorporated and a fundamental disclosure about where and in what process of the scholarly production where they incorporated. You know, I just to add to that, I think it's interesting that was mentioned in here. There are journals that won't allow chat GPT to be an author because it cannot take responsibility for what it's writing.
Okay, let me quote from the author herself from the in terms of conclusion, I'm going to move us to a random votes. Distinguish the laborious from the creative writing test. Use chat GPT to support the former and keep the latter for yourself. And always do what is generated as a first draft, which you will refine and rework, in using it with your own particular emphasis, unique voice and style.
In terms of a round of votes for methods, I'm going to say NA despite the pleas of Linda and Jason agreed and let's go to our educational impact. How many how's a scale of one to five or any give it. You don't get the reference to you. I thought you're in big enough nearer to get the sci fi reference there. I have no idea what we're howling about how not howling. Oh, how? Oh, I'm not I understand the reference now. I'm just deaf. Okay.
Jason's for Jason's have making smart comments, but he's on mute. So we've missed them all. All right. Jason Linda Lara educational impact. I was saying it was John's accent. I agree with you. I couldn't tell. Maybe it was howls or howls. I think this gets three house. It's a really good intro to some of the most basic principles of interacting with JNI. Is he some suggestions on how to use it to supplement as a tool.
I'm going to give the methods of four. No, I'm going to give the methods of three, but you didn't ask me that. But I'm going to give it a four for the paper a because I can show it to people and say if you're writing looks like this. This is the way it should look. It just flows so nicely. Second, it's not just an opinion piece is an opinion piece with enough examples that it makes a wonderful story.
For all those reasons, I'm giving it a five. I couldn't have asked for more out of a paper like this. That's got these purposes in mind. It's a five. As a commentary, I'm going to give it a five. In my sense, a few people read a commentary. They might be cute for an idea. And that idea will lead to some further work. But commentaries are rarely referenced. I think this might be referenced a number of times.
I think it has a very short half life and it's not a function of the writing, but it's a function of where this space is going. In 18 months from now, all of us will be doing all of this and none of us will need to be reminded or or bootstrapped up to efficiency. This will be the norm. And we'll like really think in 18 months, we're all going to be chat GPT. I thought I was being generous. I think it might actually be shorter than that. Yes. I got to look at this thing.
I think it's also a generational piece. Lara, you have your process for writing. I think new generation of scholars are going to see this as a very helpful and functional tool. Particularly when they're writing at large volumes in grad school. So I think no one's going to need this to get bootstrapped and get it get up to speed because it will be inculcated and everything we're doing. So that's my take.
Let's time stamp this. We'll come back next year at the same time and see where we're at. Yeah. No. Okay. That's good. With that, we love for you. I mean, somebody needs to babysit me on this. Like somebody needs to actually come and help me figure this out. John, if you were in the same city as me, I'd be like, so at your door.
Is someone who has been an engineer or something? It's a literal rocket scientist. He writes all the code. No, he writes the code when you bring him to like a completed something. Any like user interface. He's got no skill. None. He's looking for zeros and ones. I'm like, dude, you're not open. He's got to be helpful. But that's why I have friends like you. I think this is why I have my other brothers and sister. I was just saying. Linda, Jason, do you have a partner relationship work?
What kind of near top hack happens at the Virpio dinner table? One's in zeros and no graphic analyses and philosophy. I can just see ethnography. The answer between the zeros ones on the ethnographic on the other end of the table. Poor kids. I feel we should we should rescue them and take them to a baseball game. Oh, I take them to all the sports stuff. It's super fun. But I actually think my youngest is more prof than we are. I'll just start and eat a topic and just go right.
The rest of us are waiting for him to breathe. Where Tweed was. Nobody has a really nice like very sophisticated little house code. He came in and gave me a hug. Anyway, dear, we got to sign off. Dear listener. Thank you for staying with us to the bitter end. If you're still here, I feel we're just talking to empty space. But if you are still here, we'd love to hear from you. You can find the abstracts for all of our episodes at thepaperspodcast.com.
We'd love to hear from you if you want to communicate with us. You can reach us at Gmail. So the papers podcast at gmail.com. And if you're subscribing, we love for you to give us a thumbs up or a five star rating. And if you really, really like us, tell a friend. We're doing a lot of this to help advance and grow a community and we just want to be able to, you know, have that conversation with the growing audience. So thanks for listening. Take care everybody. Talk to you later. Bye bye.
You've been listening to the papers podcast. We hope you made you just slightly smarter. The podcast is a production of the Unit for Teaching and Learning at the CareLinska Institute. The executive producer today was my friend Teresa Sado. The technical producer today was Samuel Lundberg. You can learn more about the papers podcast and contact us at www.thepaperspodcast.com. Thank you for listening everybody and thank you for all you do. Take care. [Music] [Music]