Multiple Choice Questions: A Different Approach

So this is uh , Bruce Hoskins again with a show contributor, Sean Davis , our joyful teacher in residence here at MiraCosta. And look today we're going to do something, you know, on a much more on the practical side, we're going to talk about multiple choice questions and how we use them. But more importantly I think is, is the question of why

Um, I remember you saying Sean saying in an earlier, or I shouldn't say earlier, cause we don't know the time how I'm going to release this, but a , in a different podcast where he was talking about, I was like, look, are you upholding policies that actually don't work towards your overall goals for the class and student learning and stuff like that and success. And I'm like, Yo, that is a really dope way of saying that. And so it was like, here we are with these multiple choice questions.

And my question, I mean, and to whatever degree, most if not, I don't want to say everyone's using them because of course everyone's not using them. Uh , but a lot of us use them.

The question I have is have we really thought about how we use them and are they working towards our goals and whatnot is , so I'm gonna let Sean jump off into a space and cause, cause when I think about the, the use of them, I really think about this, this one example comes up, I'm not going to say no names and again, I'm not trying to get at people specifically, right?

It's like I'm really trying to get us to think about teaching and learning and what we can do in order to, to, to address our teaching and you know, and , and to make a student equity effect , right? To affect , you know, the sports disproportionately impacted groups and things like that.

And , and so when I think about multiple choice tests and, and how someone was talking to me about it is that they used a a hundred multiple choice test or questions from all the different chapters in their textbook. And you know, right. So this is a cumulative freaking final with a hundred multiple choice questions. And it's the final, right? High stakes, high stakes , right ? And it's so super high stakes and everything. And I'm like, so, and they use the word rigorous.

That was the word that they connected to the space they said in that is rigorous. And I'm like, Yo, I hear you when you say that's rigorous , that like, that's hard. But my question is why are you doing that? Right? Why are you doing that? Is that getting you the results that you want and what exactly are those results?

And I didn't get into that conversation with the person because I didn't , honestly, I didn't even have these questions for myself back when I, when I had heard this , um , and whatnot. But it's like those are the kinds of things that I want us to, to really think about is the why. Why are we doing any of those, any of these things. Um, and so, you know, so, so what Sean is like, he teaches online and on grounding .

And um, I wanted him to talk about how he uses multiple choice in an online setting to, to set us up with this. First thing I want to say is testing is not

However, testing and frequent testing and frequent low stakes testing has been shown to be very powerful in the teaching and learning process. And so I applied this to the online setting by giving weekly quizzes that are pretty short, 12 questions. But the students have unlimited attempts on these, on these quizzes. And they have to do writing assignments, they have to do other projects and activities where they have to create things.

So there are multiple ways that I'm assessing their learning or their rather their application of concepts, theories and things like that. But with the test, they do this before anything else. They read the textbook. I have a couple chapters assigned. They do this test, which is called the readiness assurance test , and they can, they should get 100% because they can take it over and over again. And I just think of if you, and they're, they're pretty difficult questions.

So if they take it over and over again, I feel like they are getting a deeper sense of what I think is the most important, what I think are the most important concepts in terms in this chapter and how to apply them because they're all application based and some of them have multiple answers, so they can't just guess one and it'll be like a 25% chance if it's a through d that they get it right. You know , they have to probably do a and C and e right.

And if they just do a and C, they didn't get it, so they got to go back and do that again. Right. But then there'll be the feedback in there that says, this is where you need to go in the textbook with link to the open source textbook so that when they're taking the quiz, they just get shot. Right. To the area so they could review the concept and see if they can apply it on the next turn . So I do this because again, it's low stakes, lower than any of the other assignments in the class.

And I'm not looking at this quiz as necessarily an assessment tool. I'm looking at it as a teaching tool of this is how you apply these concepts, this is what I feel is important. And then when they're finished with that, when they move on to the next page in the module in our learning management system canvas, I'm s I set up my classes in modules, the first thing they see as quiz, the next page, they'll have a lecture from me.

And I know based on the statistics that are generated, the data that's generated from canvas wit from previous semesters, which concepts students struggle with most. And then that's what I provide in this 20 minute lecture is just covering those things and it's pretty consistent, right? And so then they go onto a writing assignment, supplemental readings and things like that. So that's what I do in , in my online classes.

Right. I'm not obsessed about like, are they looking at the book while they're taking the quiz? If they're looking at the book, I feel like I'm winning. Right. If they're looking at the book at any time, if they're looking at the content or they're , you know , searching it online or they're doing what they have to do to try to do well on this part of the course, I feel like they're in the material. And overall that, that's one of my goals.

So, so for me to say like, you know, you can't use the book or first off, I can't enforce that. And second, it's like not consistent with what I want and what I want is for them to look and engage, you know, engage with them.

It's like, are the students getting the hours ? Like are the students getting that deeper understanding or is this just , or is this time pressure or, or are you trying to control for cheating when you put in the clock, as I'm really asking this as a question

this dumb thing. So this is something we talk about all the time. You and I, Bruce is the question of, you know, what does it mean to prepare students for the real world? Right? As if our classrooms one at the real world, right?

And , and this is one of those examples to me that I even tell my students, I'm like, I've never been in a situation at my job where they said, you're going to sit here quietly with all of your colleagues and you can't use any resources and we're gonna put this paper in front of you and you better score this much or higher on this test right now, not using anything else, but what you remember.

Like there's a way of getting it to where you can manage that and getting people to actually interact with the material and at that and to , to come up with the goal of I really want them to just know what I value in this in the textbook or in this particular chapter and to have a overview for, you know, and , and then set your lecture around things that they may have more complications in regards to , to knowing and understanding.

And, and that to me is like the , the biggest difference between having a class based on multiple choice questions versus using multiple choice questions as a teaching tool. Right. And what not versus an assessment tool, which I get why people do this. I mean there's, I mean there's very real reasons to why multiple choice fills so much more. I'm going to use the word objective, right? It's like don't get in.

The way that I set my multiple choice up , um , is to, is to where the students can find the answer in the book type of deal. And so that they can't argue with me if you would about is it in the book? Like where is the answer? And it's like, it's in the book and, and so when we're talking about like what's in the book, right? It's, it's, it's there, it's to whatever degree it's objective.

And so we don't have to, like I said, there's , there's really no argument about the, the, the con the, the answer, right. Um, and whatnot. So the , the students, they don't come at me with that. And, but the other thing that I know, and this is, like I said, this is what the utmost love and respect is that a lot of us shoes , multiple choice questions because it's fast. It's a fast way of grading.

But is it a fast way of assessing and that it , I think it has to be something that we, that we really slowed down for ourselves. Is it a fast way of grading? Yes. As a matter of fact, there was this guy, I was doing this workshop about, you know, multiple or teaching and learning and all this and I was like, does anybody think? And I made a bad assumption in the space. I was like, you know what? Because we all know that multiple choice doesn't measure critical thinking and it was dead silent.

And I'm like, hmm , well I've experienced this kind of silence before. So let me backtrack on my thinking here real quick and let me ask you another question a different way. And that is, do you think that multiple choice measures critical thinking and then one guy in the front use the psychology psychology dude and so, you know, I forgive him already. Um , that was really funny to me.

But anyway, but he was like, Hey, what , what research has shown is that if you have a five answer multiple choice with a, I think it was a all of the above responses and none of the above response in it that that gives you the same grade. Like students tend to s test the same, the , they tend to get the same grade as if you give them a essay based test. And I was like owl , I was like, you know what, that's kind of new to me. So let me sit with that a little bit.

But then I started asking him some questions, like there's the in the, in the space, I was able to ask them some questions to get him to, to actually explain to me what he meant by that and whatnot . And so I got to, I understood that he was talking about grades, right? He wasn't talking about like skillset or learning, he was talking about grades. And so I'm like, okay , cool. Multiple choices, really good for grading fast. But is it really good for assessing?

And like I said, we need to slow that one down for all of us, like I said and think about. And so for me, the way that I used multiple choice in my class is that, so first you have to know that I wrote my own introduction to sociology textbook and then from that wrote my own, entered my own multiple choice questions, which is way harder than what I thought it was going to be. But that's a whole nother podcast.

But what I do with them is that I don't use those multiple choice questions as assessments. I don't have a quiz or a test that goes along with that. What I do is I give them the multiple choice questions in advance in a pocket , and I give it to them in advance and I have them do it for homework and they do, you know, chapter one, multiple choice first and they get a week to do it and then when they turn it in, that's when I talk about chapter one, right?

You usually write to the patient of it, and I've actually been able to solve a problem of students not reading the textbook because of the way that I do this now. Right? As that, having them do the multiple choice first means that they've gone through the textbook. That's like, I've literally eliminated that, that challenge from the classroom just because of how I'm doing this. And the second part of it is that they've been exposed to content, right?

We've talked, you know , in another podcast about content and it's like they've been exposed to all the content already. And um , the way that I wrote the multiple choice questions, I want you to know how to read a textbook and so how to pick up the important things in a textbook.

And so I'm looking because what we know, I'm here in America , so I don't know if it's anywhere else but here that one of the most common classes that a student is going to take as a first when they're a first time student, the most, the , the, the most common classes they're going to take other than an English class, it's introduction to sociology. I know that from our data, right at our program review and all of that stuff.

And so if I have all these first time students in my intro class, then let me use this as a space to introduce people to things that they may not have been exposed to, things that they may not have been as good at in the k through 12 system, which is how to get information out of a textbook and using the multiple choice as , um , scaffolding, right? It's like scaffolding their, their reading experience and going, okay, these are some of the important things to pick out of this space, right ?

Versus you're going to read this whole thing and then I'm going to pick 20 random questions. And if you didn't, you know, if you weren't interested in that section, you know, you're just, you're Sol, right. And whatnot. It's so that's so, so then when it comes to using multiple choice, it is, for me, it's like I said, it's as you said, a teaching tool, but also I want them to come out with a skillset of how do I learn how to read a textbook, right. For those and for information and what not .

Because like I said, I know that , um, that our introduction to sociology classes has a lot of time

So you're answering a lot of those questions by saying, here's your little scavenger hunt, right? These are your questions. Go look for them in the book. Look all around. I wrote the book so you know, you can ask me anything. And then when you come back, you already have this foundation. And then now we're gonna go deeper into this topic and I'm going to give you some other examples that you've yet to be exposed to, right?

And you're going to come up with examples too and you're going to apply this in different, different ways. And so that's what I do to also in the face to face classes. So I have the team based learning model , so there's six to seven students in each team and what they do is they take their, they're asked to read a chapter, they can bring a page of notes to use. I give them objectives, like these are the things to take notes on and this is where to find it in the book.

And then they take an individual readiness assurance test or what is called an IRR and then a t rat, which is a team readiness assurance test. The first one, individual readiness, individual readiness assurance says 12 questions. It's the same exact quiz when they take it as a team so they can use their notes that they took. They come to class with their notes, they do the 12 questions individually and then they take it again as a team. They get points for both.

There are more points for the individual than for the team, I guess for obvious reasons. Right. Like you want more accountability for the individual in this case. But the real reason is because they want more accountability for their individual score in the beginning of the semester. But by the end they're like, I wish this team stuff was worth more because we perform better as a team. Right? Because you have more people together.

So what I really love about this is that the amount of coverage of the material, right? And then the uncovering of the material that comes later. And we talked about the uncovered in a different uncovering material in a , in a different podcast , but they read, they take notes, they take it individually, they take it as a team in their team. They have different roles. They have the person who reads the questions out loud and the answer options.

They have the person who actually moderates and makes sure that everybody in the team has heard prior to answer being selected. And then they have the person who records, and you've seen this before, my classes, they have these really neat scratcher sheets. Yeah, it'd be like a through d and if you scratch c , right. It's like a lotto ticket. If you scratch off C and c is the answer, there's a star. But if , if not, then it's blank.

And so if there's like four different levels, if you get it right the first time, it's four points. If you scratch and it's blank and you get it right the second time it goes down by half to two and then to one. And obviously if you scratch all three and it's that last one, he gets zero points, right ? They score really high in the team scores, right? So that builds their confidence as well.

And then I lecture on the things that I know that they're struggling with, but when they take it as individuals, I can run those through the scantron machine, see the ones that they're the, the concepts and the things that they're having difficulty with and then tailor my lecture to that. Right , right. And so, and then with the midterm and final, I know you wanted me to talk about this.

I do it a little differently because again, they're in teams and I have the scratcher sheet and I allow them to take the midterm and the final in their teams and then they're there looking at red. There's a lot of energy in the room. It's not just sitting there, everybody's quiet, they're talking about the concepts right there . They're discussing it, they're debating, they're arguing and then you know, few semesters back this team was like, can we talk to other teams?

And I was like, of course you can go for it. Right? Like that's a very innovative way of looking at this situation. And so then they go, when they talk to the other teams and it becomes, the whole class is engaged in the material in a way that is so different than any midterm or final that I had been a part of. And for me, they're talking about it. They're discussing, they're engaging, they're , they're looking things up.

And that's why I make those really low stakes though, because the other things that they're being assessed on through essays and through other kinds of reflection and projects, yeah, you're going to have to show me that you can do some stuff on your own. But why can't we just explore the content together and have a discussion about it and make this a exciting atmosphere as opposed to one that is very somber, right ? No, absolutely right.

So what does the classroom look like if we're not focused on content, content delivery, and we're focused on uncovering content, what does the classroom look like if we're not focused on multiple choice as assessment versus using multiple choice as a teaching tool? I think you know what, and I'm just thinking, I don't use this, but I know that there's people who use like cahoots , like they're using the gameplay and as a teaching tool and things like that.

And so all of those things are, you know, really innovative. I don't know how innovative they are, diverse. They are different than using it as assessment. And it's like again, challenge us to, to think about these things. And you know, for our student equity take, right? For me, when I think about multiple choice questions and I think about student equity cruise right, disproportionately impacted groups, especially as it pertains to black and students.

A lot of, you know , the research actually says [inaudible] and some self fulfilling prophecy into this stuff that when you have black students don't know that they're being, you know, to whatever degree being assessed or something like that, they do just fine on tests , um, and whatnot. But when you put the, when you say this is about your intelligence or you put that high stakes on it, like a sat or act, all of a sudden score started dropping and what not .

Of course, that's not all the, you know, that doesn't explain all the , the, the, the difference in the scores and everything. But it should make us think about like, what are we doing to perpetuate, right? The inequalities when we use standardized, when we use multiple choice, like it standardized testing, right?

And, and when you create that same format, that same , uh , space in your classrooms, then you are to whatever degree privileging the rich, right, rich or you know, upper middle class and whatever, however you want to define that.

But there's definitely a group of people and everybody acknowledges that at the sat and the ACP act people and anybody that does standardized testing, they know all across the board that people in higher, higher class status definitely do better on these standardized tests for whatever reasons . Mostly because they have different resources and all this stuff. But you know, just , we'll just leave that there.

And so you're literally creating in your classroom a space that privileges people, that upper middle class and whatnot and, and rich folks , uh, you're, you're, that's what you're doing. And so then if you really want to try to affect student equity, you may want to try a different approach with that or, right. I mean, I don't know how people can , uh, you know, help people from these disproportionately impact the spaces do better on a multiple choice tests.

When I could say , yes, you're dealing with trauma. I don't think a lot of people recognize that people are dealing with trauma when they're dealing with stuff like this. They're just thinking, oh, the student doesn't know how to learn this. And so I could teach them. And it's like, well , there's trauma that goes along with this also though and whatnot . And so, so that's my student equity take.

It's like if you're, if you're really, you know, pushing the standard , like using multiple choice , like a standardized testing, well, there's a lot of trauma for especially black and brown students, but there's other groups or , and poor students, black and brown students. There's a lot of trauma that goes into that space and that , um, if we reimagine how we use multiple choice, we can definitely , uh , do better for those students. Yeah, I like that.

And my equity piece would be to flip the script on that. And what does it look like usually during a multiple

You have very low stakes or no stakes testing and you have these multiple choice tests and you say you could talk to anybody in the class, partner up with people and talk in groups, you know, and you could put some music on. Just change the environment completely so that their mindset is different about how testing happens in your classroom. Right. And then if you do have a more traditional testing setting, at least they've covered the material.

They've been through this in a different way and they're coming to it not as like this big scary thing, but a continuation of,

If you learned something today that you think would help close your student equity gaps, how long would it take to incorporate this into your classroom? A year , a semester, next month. Today. No matter the timetable , we must commit ourselves to becoming better teachers. Our students deserve it. All of them, not just the ones that are good already. [inaudible] .