What Should Colleges Do about ChatGPT?

And this was not well predicted at prediction markets. He had had around a one percent chance of becoming the pope. If you look at various shortlists and long lists published by various userganizations. You know, the New York Times mentions him in like the twentieth paragraph. The Washington Post doesn't mention him at all, for example, and yet he won, I believe after just the second day of the conclave. Do you know how many ballots there.

the United States Congresses for example. There weren't a lot of leaks, and I think it reflects the kind of changing global climate toward the United States, right. You know. One of the few outlets that did it in an article about him was Axios, and they're like, well, this would never happen because the Catholic Church doesn't want to give the US government more influence, or the United States

and not the government, I guess, more influence. It's like, well, what do they want to influence the United States, right, and have some counterweight to President Trump. You know what's a good counterweight, you know, probably not the Speaker of the House. Maybe the pope is like the only person who can be like on a stage as a counterweight. And you know, Greemost had had an active Twitter account too, right, He had criticized mostly through retweets, not things he wrote himself,

criticized JD. Vance criticized the deportation of Kilmar Abrego Garcia Marilyn Man who was accidentally sent to an l Salador in prison. And so it's making a.

markets work. So the first part of the psychology part, I think people had kind of bad priors, right, and let kind of their biases in an historical precedent influence this a little bit too much, saying oh, you know, well, an American could never be pope and you can't have that sort of blanket statement rights, that's a bias where you are you're actually kind of anchoring yourself to a really incorrect starting point, which gets into my second point

that you've written about, which is that this is not an event where we have tons of data and statistics. Right, this isn't the super Bowl that happens every single year. Like the last time that we had a papal election

was what twelve years ago? And you know, there have been decades where there haven't where there hasn't been one, and the world changes and different things change, and people don't know much about what happens inside the Vatican and all of that, and yet we think, right, we're a little bit overconfident in thinking that we know more than we do and that we have more data than we do. It's always incredibly incredibly risky to make predictions based on incomplete, faulty data.

of Cardinals. Right. You know, technically speaking, any baptized Catholic male is eligible to become pope. Right. I'm not saying you just want to spin a wheel one in two hundred and fifty one or whatever. But but you know, especially after the white smoke came out and people like, oh, now we know for sure, right, this is a case where the conditions were not in place for prediction markets to be particularly wise.

And I think from a psychological standpoint, you also get into these groups think echo chambers, where you start amplifying things that are already kind of front runners, et cetera, et cetera. Not for any good reason, right, but oh well, because he said so, I guess I think so. And then on and on it goes. It's like those psych studies right where there's a number of statistic that's been cited thousands of times, and then you actually try to unravel it and figure out, way, where did we get this?

And you find the study from nineteen sixty four that was based on twelve people, and that's where the number came from, and you're like, whoa, whoa, whoa. This number that has now become gospel is actually incredibly suspect. That has happened to me so many times where I've actually tried to kind of retrace the steps of how it happens. But especially with social media and kind of the way information spreads these days, it's very easy to create those effects.

the market applug, you just have a twenty percent chance. So maybe the twenty percent actually of the good bet here, and if you're making our sports bets, you learn that it's not predictable when the market knows something and when it doesn't. Obviously, markets are tough to beat in general, but sometimes it is full of shit, right and many times it's not, And you're the one feeling embarrassed after all. Instead you say, boy, you know Vegas is so smart,

but like it's not always smart? Right if it was always smart? Yeah, piggyback off it and you kind of can't.

are where they're not right. Is there is there a way that you think about that, like how to figure out you know what, I shouldn't be listening to the experts in this particular case, I shouldn't be listening to the markets particular case, because I think that's actually a really interesting question.

crypto adjacent prediction market startup betting type person Right. Next is is there are there institutional investors professional class of investors who are who are sucking up the week money to make the market balance?

repeated thing where people can calibrate their models? Is it even mathematically modelable by anything beyond like very naive methods?

You and I, by the way, didn't even mention his name. Right, he wasn't on our radar because he wasn't on the radar of any of the sources that we actually well.

interesting that people are betting on the pope. We're not sure how we feel about that as Catholics, but they're way over confident because nobody knows anything right and that proved to be prescient.

out at three out and four outed. For our non poker playing fans, that means, you know, improbable combinations of cards kept getting there against me, and so I was just very discouraged. And I ended up winning the final event, which was a really really nice way to end the trip. And the final that was actually a turbo structure, which

is very different from kind of your standard tournament. It means that the blind levels aren't increasing quite quickly, and so your strategy actually has to change a little bit, because, yes, you might start with two hundred big blinds, and whereas you know, in a normal tournament you might have that for forty minutes, half an hour, an hour, whatever it is. Here like suddenly, if you don't do stuff in twenty minutes, you look and you have thirty big blinds left, right,

Because the blind structure is moving so quickly. It's such a dynamic tournament, and so in situations like this, you need to be first of all, more aggressive, right. You need to play. You can't just sit around waiting for good hands, because otherwise you will blind out. This is one of those cases where doing nothing is doing something, and it's not a good thing, and you have to actually be a little bit different with the hand combinations that you choose to play because very quickly, this is

not deep stacked poker, this is short stacked poker. So the value of hands changes. So you need to play high card hands much more, low suited connectors much less, because you'll get into situations where people are going to get very short, especially people who didn't play correctly right and who now find themselves with very few chips, and they're going to shove on you, which means they're going

to put all their chips in the middle. And you have to know that in advance and know how you're going to respond and be in a situation where you can have a hand that can call those shoves, because otherwise you're also going to bleed chips right if you open too much and then keep folding when people are aggressive against you, and so it's a slightly different mindset, and it does require I think a lot of preparation ahead of time because you don't have much time to think.

You just need to know your range as well. You need to know your cards well, you need to know your response as well. We had a ten second shot clock in this tournament, so you had ten seconds to make every single decision, no matter what street you were on. And we had a few time banks, and each of those time banks and it was ten seconds. So it's

actually kind of beautiful. It's a very it's fun because it's so dynamic and so much happens, but there's also an element of luck when you get into a fast structure like that.

go home. So how do you balance that? I mean, again, I'm not saying you have tilt. You can tell us about that if you do right. I'm saying the fact that like people are maybe calling down lighter and bluffing a little bit more. At the same time, with a turbo structure, you do need to widen your rain, which is quite a bit, right, So how do you how did you manage that?

there was one guy who was clearly very tilted. Because this for older French gentleman who just wanted chips right, he clearly just was frustrated. This was very early on in the tournament, so we still had a lot of chips and no one had busted out yet, right, like this is how early we are. I think we're level one,

maybe level two. And you could tell that like he just like he had been playing every hand, like he just really wanted chips, and so he bet, I raised, he called, and anyway, he ended up losing right and busting and was the first player to bust out of the tournament to me. And he had like I don't even remember, it was like ace three off suit, right, he didn't even have a good ace. It was a shitty ace, but he just didn't believe me, and I think he was tired of people picking on him. And

he was definitely tilting. And I've played with him before, like he's not that bad of a player, but it was just an awful play and he just gave me all his chips, which was a great start obviously to the to the tournament. But I think that you need to look for spots like that and be willing to kind of take full advantage of them. So like, for instance, some people would say, oh, you know, don't get greedy on the river, don't go all in because he's not

gonna call I was like, no, actually he will. You know, do get greedy because in those in those spots, you're going to get lucky against people who are just done right and they don't want you to bully them. And so especially because I'm female, that dynamic is very strong. People don't want me to bully them.

you know, be a little bit more aggressive. You can bluff sometimes, but like the multi street desperate call downs are like the most easy way to like literally burn money.

right in the right spots. So like be willing. I did this as well to kind of throw your cards away in a heartbeat. So I had a few very successful bluffs that got through, and then I got very lucky at the final table. When we got to the final table, I had a good amount of chips, but the chip leader, so the person with the most chips was to my immediate left, and we got into a big hand. So this was it's not good ICM. So ICM is we've talked about this on the show. It's

the independent chip model. So basically, you know when they're big pay jumps and when you're down to the final table, you don't really want to tangle with the chip leader when there are lots of people who are shorter than you,

because you could be committing ICM suicide. But in this particular case, someone had opened, someone else had flatted, and I had pocket jacks and like thirty big blinds something like that, and so I jammed right because there was dead money in the pot and I had you know, I had a very strong hand, and I didn't want that many people in the in the pot because pocket jacks are a strong hand. But you don't want it five way pot with pocket jacks because you know, Ace

King queen. There are lots of things that can happen. And then the chip leader rejammed over me and I was like, oh no, right, that's the one person that you want to fold, and he had Ace King and I ended up holding and winning that race, and so he was still in, but that put me into the chip lead and that's huge, right that hand.

Right there are you know, ancient and modern versions of cliffs notes and things like that, if you uh, you know, the line between working with a friend on a problem set and copying your friend's answers has always been blurry, I think, but like but now chat chipetem makes it very very tempting to cheat, and especially for like written essays, but also you know problem sets for economic classes and things like that, and and so yeah, I tried to

actually I imagine that I had given myself an assignment. You know, the philosopher Michelle Fuco. Of course, French guy weird into s and m into lots of stuff ball you know, but like very famous philosopher. My whole mo in college was to always mention, always named up Fuco right, And every time I named up put then I would get at least an a minus on the paper. Right, But like I gave myself an assignment to compare the concept of the panopticon, which is this all seeing prison

where there's like a guard tower in the center. Right, it was Bentham's idea who co wrote a lot about this is a metaphor for modern surveillance states. And to say, how does this apply to like modern day China? Right, it seems like the kind of prompt that a student might be asked in a class about China or a

class about pukou or whatever else. Right, And like, you know, first of all, if you ask chat ChiPT to cheat for you, it doesn't push back at all, Right, and you can say things, can you like, please remove any traces of this is GPT, you know, please make the essay a little reugh around the margin, but have one or two good points of insight.

point and then remove further traces of it. But anyway, it's it's very hard to have an assignment now that you can confidently predict was not written by chat GPD, any type of written homework assignment that's not kind of done with active supervision, and the article is very gloomy. A lot of students are pretty open about their cheating and say, hey, this is great, right now, I can save a lot of time. Right. A lot of professors

are very despairing of it. Right, you know, you were in an era where because it's hard to prove, like you might be ninety percent sure that a student cheated, but ninety percent is probably not good enough to flunk similar to kick them out of college or things like that. But like I will say to me, it seems like there are some methods that you could use to prevent this,

which require more work. So you know, you know, when I went, I took my junior year abroad at one school economics in the UKSES, and you just take one big exam at the end of the year and that's your whole grand and that's in person. Back then, we use notebooks. I'm sure you can use you know, iPads or something like that now, right, but that's quite hard

to cheat. If you want. Students have the internet. You could say you can have the Internet, but we're monitoring you're internet access, right and so, like if you use chatchipt, we will know, right. Or you can say, okay, well we're going to have you submit a paper and I'm going to have a fifteen minute conversation with you about

the paper. I imagine it'd be hard to fake your way through even like five or ten minutes if you really didn't know the subject matter at all, if you're being quizzed about your own paper, right and so, But professors don't seem to want to do it in anyway. What are your thoughts here, Maria, Yeah.

off times. So when I was when I was teaching back in my grad school days, I had a student hand in a paper that just looked suspicious, and it was a word document that was kind of scrolling down and all of a sudden, my, you know, the cursor changes to kind of a you know, the way that

the cursor changes when there's a hyperlink. And so he had copied and pasted directly from Wikipedia, and he had removed the visual hyperlink, but he hadn't actually removed the hyperlink, so I could actually click on the text to get to the Wikipedia article. And I was like, wow, right, like that that is uh, that is like the people who these days hand in essays that start with as an ai I And I brought this, you know, to the department and said, you know this, this kid cheated.

I want to fail him. And they didn't let me because they said that, you know, there were parent issues and all these things. And I was allowed to give him a D minus, but I could not give him an F because that's not how Columbia did things. And

it was very disparaiting to me. Right. It was one of the things that one of the reasons there were lots of reasons I didn't want to go into academia, but I was like, fuck this, right, like, you need to have consequences for cheating, so that it has happened for a long time and it's obviously only getting worse. But I completely agree with you, Nate, that you should change in response to this, how you grade and how

you create assignments. When I went even kind of as an undergrad, we had so many in class exams right where you had to write essays in class. And then when I was a grad student, we had oral exams where you know, as part of your degree you have to pass orals where you are interviews. It's very stressful for a lot of people. Right. There are people who ask you questions. When you defend your dissertation. You have an entire committee, right who's pepper you with questions about

your work and about your research. And if you're not the one who knows the data and who knows what you did, then you're going to get screwed, right, You're not going to pass. And people have failed, by the way, different dissertation defenses. People have failed oral exams. People you know, fail these types of assessments all the time, and so I think what we need if we want to actually first of all, like what's the point of higher education?

I would say, you know, to create critical thinkers and people who actually are educated human beings and have well rounded who know about you know, history and all of these different things. How do you make sure that they're actually learning what? You change the You change the way that you that you have your class. You change the incentives right so that they can't cheat their way through it, because it's not going to go away. It's only going to get worse. And so you can't just pretend that

it doesn't happen. And you can't rely on cheating detection software because it doesn't matter anyway. Instead, you need to kind of create an environment where your students learn and I would say, you know, are motivated to learn, right, So make it worth their while, like make them understand why this is important. One of the students, Nay, in

that New York magpiece. He first he got into Harvard, then he said they rescinded his admission and he ended up transferring to Columbia and was kicked out within a year because he cheated on everything and he openly admitted it and created cheating software and said, well, I don't care, Like, I don't care about actually any of this. The only reason I want to go to a good school is to meet my future spouse and co founder.

Because you can obviously take a much more cynical approach, right, which is that it's a factory to produce degrees and to kind of, you know, make a lot of money running these very expensive institutions that have now very large

non academic stabs. A lot of the time that the kids are coddle, that there is a ton of great inflation, that during the pandemic, that colleges were so concerned about angering highly neurotic moms and dads and maybe professors to some extent, right, and so they're totally undermining the educational experience, right, But like, it doesn't stop the factory from like churning out degrees, even if classes are easier and easier to gain,

and so, yeah, I don't know. I also think that like you wind up with students in a kind of prisoner's dilemma type situation where it's a really pretty rational to cheat if everyone else is cheating and it can't really be detected. And by the way, like I think that, you know, when I write essays for Silver Bulletin, then I ask chech GPT question, right, I will run my copy through j just P and say do you correct

any typos or errors? Right? And things like that. Right, Sometimes it all flag errors that are fake because I can't believe how dumb things are. It's like we can't really have one hundred and third percent tariff on China, right. People might take that exaggeration seriously, Nate, So you have to remove that from your estley. I'm like, no, things are really that dumb. But like so and working with

AI is important. I mean, it's kind of intrude to I mean, I'll stay that like interpersonal skills become more important in a world where CHECHPD do is kind of like the cognitive tasks easier. It's probably more about sales, right, who are you going to hire to be the nice person to brought you the services that you could probably do yourself with chet GPT, your group assignments and things.

People feel charmed by humans. So but yeah, I think colleges are on the way to being less relevant and that they seem not to have particularly much urgency for This is kind of lame. But like I said, if you were a student, would you use chet cheapet if you Let's say you let me set this up for you. Right, Let's say you know for a fact that eighty percent of the students in your class are using chat cheapt.

I'm trying to make a sympathetic set of facts. Right, Let's say you don't think the class is well taught, and so you don't think you're really getting that much out of it. Let's even say that you kind of done half the reading and so you're kind of using cheapt chechpeeding way. It's like clearly over the line, but you're adding some original inputs and insight yourself. But it

goes against the nominal recommendations of the professor. Would you cheat in that circumstance or use chec ChiPT or not?

I want to think through you know, I wanted to be kind of a better, you know, a better person, someone who who has that knowledge, you know. I wanted to kind of have it in my head, like if you're on a desert island right with no CHATPT and you just have kind of the stuff that's in your head. I wanted to kind of have good things in there. And so I never I mean I never even like gray Zone cheated when I was in college, Like I didn't collaborate on assignments. I didn't do anything like that.

That said, like I never told on anyone either, right, Like I was not someone who would be like Nate use chat GPT, right, Like, if you want to use chat rept, fine, but even if everyone was using it, like I I think I wouldn't I wouldn't feel right about myself if the teacher actually said, do not use

chat GPT. And like I said, for most of the classes I took, a huge chunk of the grade was in class, in class, exams and assignments, and so that was something where you know, chat GPT wouldn't be helpful. But I also I wanted to be a writer, Nate, like, I wanted to learn how to write well, and I wanted one of the reasons that I wanted to go to Harvard was to study with very specific writers. I

wanted their feedback on my writing. It would just completely defeat the purpose, right if I took a fiction writing seminar and had chat GPT write my short story, Like, what am I even doing? Well?

want students to have regrets like this. One of my biggest regrets from undergrad was I took this course AX ten, which was you know, your introductory economics class my senior year, because I needed it as a requirement, and I took a pass fail because I could write and I didn't. At that point, I thought that I didn't really care about economics, want to know any of this. And it

was really early in the morning. I think it was like at nine am or something, you know, a time that was just ungodly for me even today, but as a college student like, no, no way, And so I ended up skipping a ton of lectures and almost fail

the class. I really regret that because, first of all, the class was taught by legendary professor, and the speakers we had in that class were insane, right, We had like secretaries of State, We had the things that I missed right now, when I look at our syllabus, I'm like, what an idiot?

you know, take advantage of this. There is not going to be another time where you get like an hour one on one with Larry Summers to ask him quite about the economy.

problem sets with his mission sometimes? Right, that's cheating, isn't it, I think, Solia, Yeah, it's funny because like if you're doing the homework together, it's probably not. But like I think it's cheating. And I don't regret it because the class sucked and and still most of the grades based on on the end of the year, right you still, I mean, you know, so if I fucked myself over, then that's fine, But I don't I think that was a rational decision.

on how well we colored them in. And I was like, this is bullshit. I have zero artistic ability, like I hate you know, I can't draw, stick figure, I can't color. I was like, this has nothing to do with science, right, this has nothing to do with what I'm learning. This

is insane. And my sister is a beautiful artist. So she actually, first of all, I had she had taken the class, you know, six years before I did, so she gave me her birds to like help me figure out how to how to color them in, and then when she had time, she would color them for me. And then I had another friend who was who loved drawing, and she would color my birds for me. And I hardly ever did my bird coloring in myself because I just thought it was the.

homework are not often the correct incentives. And when you have those exams so I had, I took a lot of seminars where class participation was a huge part in it, right, and if you are one of the people who did not do the reading that week, it becomes very clear pretty or pretty soon in the discussion that, like you, you are not adding much value and that like this was just this is a week that you were mentally

not there. And if the purpose of learning is to create individuals who have deeper knowledge, critical thinking abilities, kind of the ability to synthesize information, to analyze information, the types of skills that we want right in our citizens, I would hope then homework assignments are much worse at assessing those than your ability to kind of have a class discussion, your ability to kind of think through things

and reason through things. So I think that we need to figure out how do you kind of create an education system that incentivizes the sorts of thinking and behaviors that you want your students to emerge with.

actually need more actual classroom hours. Right. I had a friend who used to work at Amazon, and the culture there is that if there's a meeting where there's a briefing to read, right, they assume that you did not take the time out to read that briefing, right, So like, take twelve minutes now and read this briefing and then we'll talk about it, right, And it's like kind of a part of the meeting, you know, obviously, like I'm a big fan of reading in general, but maybe it

doesn't have to be as Zidak to like prove that you read this, Like no, I mean, look, can you hold intelligent discussion in class? And then I don't know. People always talk about the EDU cases of students who are introverted or neurodiversient or English is a second language, and yeah, it sucks, but life isn't always fairer, you

know what I mean? And if colleges become less effective at differentiating which students were more effective and when we're on it, I mean, you know, then the whole signal becomes less valuable and college degrees become worth less, particularly the most elite degrees. Right, all of a sudden's just now basically like a signal that four years ago you were considered a talented student by an emissions committee. Right, yep, I completely agree with you.

By the way, if you don't actually do any learning and instead outsource everything all the time and using.

this right. And so I mean the other irony I think this is becoming true and more and more domains evolving AI is that like, if you are pretty smart, then your productivity is enhanced by the AI, Whereas if you don't know what you're doing then and maybe you can kind of fake your way through it up to a point, but you may be more fooled by the hallucinations, You may be susceptible to misinformation, you may just not have any retention for the things that you're having it

do right. And so, like, you know, if I were using chat schipt to write a paper or I could I don't know. I think I can write better than chet chip Bet by a long shot still, but like, but I could figure out ways if I were allowed to that to make my work to summarize long reading like that seems like a potentially legitimate use for a student. Right, here's this very long reading that you have to pay way too much for the fucking textbook anyway, Right, can

you extract the important parts from this? And guess what you can do if you're Chinese, Nat, and now you can have it translate to Chinese. Right, you can up and down the world level of complexity. Right, you can say this point might seem obvious to you. Can you explain this point another way? Right? So, like, you know, there are a lot of ways that llm's II tools could be quite helpful for the educational process. But I

just think academia is moving too slow. You know, the opportunity costs of four years in college given how the world is changing. I mean, there's a cohort of students for which it might not be the right decision.