00:00:00 Alex
ID stands for instructional Dr.
00:00:20 Camie
In today's episode.
00:00:21 Camie
Alex and Camie are taking a curious, honest look.
00:00:24 Camie
At course evaluation.
00:00:26 Camie
We'll talk about using student feedback, grade patterns, and your own insights, not just to measure your course, but to understand it, improve it and celebrate what's already going.
00:00:38 Camie
This isn't about perfection, it's about perspective. Let's dig in. So when you see a really controversial video on social media, what is the first thing that you do?
00:00:50 Alex
Like, comment and subscribe follow for more.
00:00:54 Camie
That might be the first thing you hear.
00:00:57 Alex
No, I can't. I follow every controversy dance that I find online. You gotta really confuse the algorithm. My first.
00:01:05 Camie
Like first inclination, after sharing it of course my.
00:01:09 Alex
Right, obviously.
00:01:09 Camie
Favorite actually is like.
00:01:09 Camie
To share it, but after that.
00:01:12 Camie
Is go to the comments section because and if I'm lucky, I'll go to the.
00:01:13 Alex
MHM.
00:01:16 Camie
Comment section before I.
00:01:17 Camie
Share it because then you can, you know, tell other people to go there.
00:01:21 Alex
Sometimes there's more gold in the comment section than there is in the actual content at the beginning.
00:01:25 Camie
I always find that just to be an interesting.
00:01:30 Camie
Support there.
00:01:32 Alex
It can be validating too, because oh, I was already thinking that. Oh, and I'm not alone. OK, I'm not crazy. Everyone sees what I'm seeing. Yeah.
00:01:37 Camie
No, they're 35,000 other people.
00:01:40 Camie
Were also thinking.
00:01:41
There you go.
00:01:43 Camie
Or you can get a different.
00:01:45 Camie
Perspective.
00:01:46 Alex
Which can be helpful in other ways if you're.
00:01:48 Alex
Open to that this.
00:01:50 Alex
Comment on the Internet just changed my mind, said no. Whatever.
00:01:56 Camie
Doesn't have to change your mind, but it can go. Ohh I didn't.
00:01:57 Alex
I know.
00:01:59 Camie
Think of it like that.
00:02:00 Alex
There. That's fair.
00:02:02 Camie
And evaluating your course or.
00:02:04 Camie
Looking at your course evaluations.
00:02:06 Camie
And kind of feel like that can kind of feel like reading the comment section on on a video.
00:02:12 Camie
And sometimes it's just as mean as comments section.
00:02:17 Alex
Sometimes more so.
00:02:18 Camie
Right.
00:02:19 Camie
It and you know, so it you could be it could be confusing because students sometimes when they leave feedback may not.
00:02:27 Camie
You know always, always be direct, so it could be vague and you could not understand what they're talking about.
00:02:34 Camie
It could be unhelpful and even a little soul crushing because.
00:02:38
Yes.
00:02:39 Camie
It's personal. Your course is very personal, but you could also get something super helpful and think, hey, I was thinking that too, but it's something, you know, in the midst of grading in that really busy week. I didn't think to change.
00:02:54 Alex
Yeah, it's always important to have a growth mindset when it comes to reading these kinds of comments, not a fixed mindset.
00:03:04 Alex
Going in and you don't have to always take everything that's said.
00:03:04
Yeah.
00:03:08 Camie
We're not neutral when it comes to valuations because.
00:03:10 Alex
Right.
00:03:11 Camie
This is like.
00:03:12 Camie
All the work and that you've poured into it, it's a personal experience.
00:03:17 Alex
At the same time, though, I think this is slightly off. The main gist of what we're talking about, but ideas are also not.
00:03:23 Alex
Our identities, yes.
00:03:24
Yeah.
00:03:24 Alex
So it's also important to not be so invested that yes, the.
00:03:32 Alex
Simplest feedback or even some.
00:03:36 Alex
Rather mildly antagonistic feedback doesn't send us spiraling.
00:03:41 Alex
Right, because we're never going to please everyone in course development, nor should we. Everyone's experience is different course development is as much.
00:03:43 Camie
MHM.
00:03:50 Alex
Of an art as it is a science.
00:03:53 Alex
And so with that in mind, some people just aren't gonna some people. Yeah, some people just aren't going to like what you develop.
00:03:54 Camie
And it's ever changing, right?
00:03:59 Alex
But that doesn't mean what you've developed is bad. We want to look at data points. We want to look at feedback we want.
00:04:05 Alex
To look at.
00:04:08 Alex
Actual metrics that we can use to inform us and and get a holistic picture when we talk about course evaluation.
00:04:15 Camie
I think when we're talking about course evaluation, it's really important to get that bird's eye view instead of zooming in on those handful of, you know.
00:04:26 Camie
Antagonistic comments from students on their course evaluation.
00:04:30 Camie
We have several other tools that we can use to kind of zoom our lens out a little bit and look at the bigger picture of what's happening in our course. But remember, when you're doing it, it's not about judgment like.
00:04:42 Camie
Even when you're looking at data points, this is not about the judgment of you or even your work. This is about curiosity and improvement and bring out what's working, what's not.
00:04:51 Camie
Thank.
00:04:54 Camie
You know shift, how do you respond to what's going on in that moment? We also know that.
00:05:01 Camie
Of each grouping of students is a little bit different, right? They have different student needs.
00:05:06 Camie
And as we have.
00:05:08 Camie
No.
00:05:09 Camie
Different generations come in.
00:05:12 Camie
That can make a big difference too.
00:05:13 Alex
Yeah, that's.
00:05:15 Alex
Maybe what I was hitting on with the idea of it being as much of an.
00:05:18 Alex
Art as it.
00:05:18 Alex
Is a science. You can deploy the most evidence based pedagogical implementations in your classes, yet there can be a million factors going on outside of a classroom that impact how students individually and then even collectively perceive of courses.
00:05:35 Alex
Experience, or even perceive you.
00:05:38 Alex
And again, a lot of those things are going to be largely out of our control. So what can we control and that's where evaluating feedback unit data points.
00:05:47 Alex
Thinking of it as a constant experimentation process, staying curious straight, almost like a scientist, know input that you receive is necessarily.
00:05:58 Alex
Condemnation. It's just new information to kind.
00:06:01 Alex
Of retool.
00:06:02 Camie
Respond.
00:06:03 Alex
And respond to and then.
00:06:03 Camie
To.
00:06:04 Camie
Yeah.
00:06:05 Alex
Develop and grow from there.
00:06:06 Camie
If that continuous improvement.
00:06:08 Alex
Yeah, growth mindset.
00:06:12 Camie
And we also, you know it supports student success and sometimes you have to do this anyway because of accreditation or institutional review and that type of thing.
00:06:22 Camie
There are a lot of reasons why we're course evaluating and so one way.
00:06:25
Yep.
00:06:28 Camie
That we sometimes evaluate our courses is by looking at.
00:06:33 Camie
The grading metric.
00:06:35 Camie
Our student scores overall.
00:06:37 Camie
And sometimes after a lot of times, people even zoom in on the D's and W's withdrawals.
00:06:45 Alex
Yeah, not Dallas, Fort Worth.
00:06:46 Camie
Right. No, no, we're not talking my airports here these S&W's.
00:06:52 Camie
In individual courses, you can track them overtime so you can look at it just for your.
00:06:57 Camie
One semester, how did that go?
00:06:59 Camie
You can look at them over if you've taught this course for several years in a row.
00:07:04 Camie
Has that percentage changed overtime?
00:07:08 Camie
Now.
00:07:09 Camie
There's a certain mindset that you have to be in for.
00:07:13 Camie
Looking at this type of metric right and that is.
00:07:17 Camie
Now the the problem with it is that the underlying assumption with looking at these S&W's any rating metric like that. If you're looking at percentages is that it's based on.
00:07:30 Camie
The Bell curve framework.
00:07:32 Camie
And the Belco framework just says that you have, you know, the most C students. You have some B&D students, and you have the fewest A&F students, just like on a natural bell curve, there are different percentages that come out with these. When people talk about it.
00:07:49 Camie
But do you just I kind of think of it in terms of you?
00:07:52 Camie
A's and F's make up about 10% each. B's and B's are about 15% each. C's are about 50%, and again, these are just approximations. It will look different if you actually research this concept.
00:08:05 Alex
Right.
00:08:07 Camie
But that's just the way that my brain.
00:08:08 Camie
Kind.
00:08:09 Camie
Of I can picture what 10% looks like right? I can picture what 15% looks like. It's easier for my brain to kind of chunk that up that way. When I'm thinking of this concept.
00:08:18 Camie
Now, that's not to say that this doesn't.
00:08:20 Camie
Tell us anything though.
00:08:21 Alex
Right. Because I I think the question I was going to ask and I think I.
00:08:25 Alex
Know your answer.
00:08:25 Alex
But I want to hear is on an initial assumption of that, would you say that is a?
00:08:31 Alex
Accurate way to look at.
00:08:33 Alex
Grade distributions in courses in general. Do you think that most students should be receiving CD?
00:08:38 Alex
'S.
00:08:39 Alex
Based on performance, the fewest should be receiving A's.
00:08:42 Alex
And F's.
00:08:44 Alex
By design or by.
00:08:44 Camie
I I actually do not follow this this approach.
00:08:49 Camie
I am more geared toward the mastery approach and honestly, when I look at, you know, we.
00:08:56 Camie
Work with a lot of instructors.
00:08:58 Camie
And when I look at the student grades in these courses.
00:09:03 Camie
That's also not how they grade. They don't grade. No one that I know that I'm working with currently is using this bell curve approach. It's a little mismatched from what is happening and what we're looking at, right?
00:09:16 Alex
They're saying, oh, if I see that I have 50 students in the class and 30 of them got to see, OK, I'm doing a good job because I'm my average amount of my students are doing an average amount of the majority of my students are doing an average level of work.
00:09:28 Camie
Right.
00:09:29 Alex
That wouldn't be.
00:09:31 Alex
I wouldn't think maybe there are some instructors that might think that way, but I don't think that would be to a good chunk of instructors that that's what they would want to say if they were bragging about their course to I set up this great course and I got. I got so many students, got a 72, it was great. I only got one guy, got a A1 gal. Got a 90%.
00:09:43 Camie
Most of my students I can see.
00:09:51 Alex
Again, you could have some some folks like that, but.
00:09:55 Camie
But I don't know any. I I have not seen any in the however many years I've been here now three years.
00:09:57 Alex
Right.
00:10:00 Alex
Well, it's it's the, it's the, it's.
00:10:02 Alex
The cinematic stereotype of an instructor like I only only three of you out of the 100 in this lecture will.
00:10:08 Alex
Pass.
00:10:08 Alex
You know it's, it's that.
00:10:09 Camie
Right, because we.
00:10:10 Camie
We create based on mastery. Most the time are you mastering this concept or the skill?
00:10:17 Camie
It's not. We're not looking at you and going OK, how many of the students have worked around here for the class? It's this average. No, we're saying. Did you meet these expectations? We'll talk a little bit more about mastery and what that looks like in a little while, but that doesn't mean still that these these F&W's can't tell us anything.
00:10:37 Alex
They still are data that we want.
00:10:39 Camie
There's still, right? There's still data that you can choose.
00:10:39 Alex
To look at.
00:10:42 Camie
To look at.
00:10:45 Alex
And we don't know what it's telling us.
00:10:45
We.
00:10:46 Camie
Right. And so that's the thing. You know that you can see that something is happening but you don't know why.
00:10:53 Camie
Do you know if it's signifying how engaged students are? Is it showing that maybe there's a really large class size, especially for those Gen. Ed courses, and that might not be working out well for student success?
00:11:07 Camie
You know, is it showing that there are challenging topics that don't have enough foundational resources to support students bridging up to those concepts?
00:11:20 Alex
Right. And do the gradable items within your course actually scaffold in a way that allows students to reflect their learning in accurate ways or?
00:11:28 Alex
They.
00:11:29 Alex
Dealing with high stakes, high test anxiety outcomes because 80% of the grade is built on to midterm like a midterm and a final. Yeah, and that's it.
00:11:32 Camie
Right.
00:11:38 Camie
Or it could even be that you have one of those upper level courses that don't have a prerequisite, but.
00:11:43 Camie
Should.
00:11:46 Camie
This can it can look like a lot of different things and so, but it's difficult unless you look at other things you.
00:11:51 Camie
Can't just look at that.
00:11:53 Alex
Have you watched the show the pit?
00:11:55 Camie
It's like I've maybe seen an episode.
00:11:58 Alex
So it's Noah Wiley from ER, it's his new show where he's the chief.
00:12:02 Alex
Your.
00:12:04 Camie
I've just seen the commercials. I haven't.
00:12:04 Alex
Doctor.
00:12:06 Camie
Watched it yet I'm excited.
00:12:06 Alex
So it's great the the the connection I'm making with this is you know when they so that's a 24 hour or it's a. So the show is centered around one shift in Pittsburgh Medical Center trauma unit, the ER there and he's got a crop of interns and Med students and his residence that he's working with and just the.
00:12:26 Alex
Layoffs that ensues and just work.
00:12:28 Alex
And in one of the busiest.
00:12:30 Alex
ER's in the in the country, and So what would the thing that it is making the?
00:12:33 Alex
Connection with me here.
00:12:34 Alex
Is they have a patient come in, you know you've got two or three.
00:12:39 Alex
Medical professionals in the room talking to the person about what's going on.
00:12:42 Alex
And they have.
00:12:43 Alex
To essentially use all these data points and but they're looking at symptoms looking at in the DFW is are the symptoms right? Like, oh, they're coming with fever, aches and pains, chills. That's not telling you it. It could tell you a handful of things. It could be, but you're going to have.
00:12:58 Alex
To do more digging.
00:12:59 Alex
They have to then do blood work or.
00:13:03 Alex
Run tests, run panels. They have to ask more questions, they do something. Differential diagnosis like what could be the other thing. This could be outside of the norm that and again, that's some medical expert could inform.
00:13:10 Camie
Yes.
00:13:16 Alex
Much more clearly that I'm informing right now.
00:13:19 Alex
But that's I'm I'm seeing the DFW.
00:13:22 Alex
Grade distribution as the similar as the symptoms when you come in when a patient.
00:13:26 Alex
Comes into the ER.
00:13:28 Alex
It's telling you something is going on. Something is wrong, right? But we, but we need to investigate. We need to look at other components to figure out what this could actually be the root cause of what's causing these symptoms.
00:13:32 Camie
You have to.
00:13:39 Camie
Yeah. And it's funny that you mentioned the differential diagnosis because that is for me one of my favorite tools in medicine.
00:13:46 Camie
Because you look at those symptoms and then you go what are not just the probabilities, but the possibilities, and then you use evidence to exclude things. It's not just going, oh, I don't think it's that it's there's not evidence to support that. It is this thing. And so I do like that a lot and because that's exactly what we do when.
00:13:52 Alex
Yeah.
00:13:57 Alex
Right.
00:14:04
The.
00:14:06 Camie
When we're looking at these and we're going OK, could it be this? Could it be that? How do we investigate that? I will.
00:14:11 Alex
We're basically doctors.
00:14:17 Camie
Oh course. Doctors, every time. You know, sometimes it does feel that way.
00:14:18 Alex
Yeah.
00:14:24 Alex
For saving lives here, folks.
00:14:28 Camie
So the DFW is a lot of times those lower scores or the withdrawals are what we look at because when people see that they see.
00:14:38 Camie
On the opposite end of that are A's, B's, and C's. Most people don't look at these when they are going through course evaluation, but I believe that they should because they do tell a story just like those D's F&W's.
00:14:53 Camie
And of course, some of that story could be that your course is set up really well and that students are succeeding in your course, right? They're mastering the things they should master.
00:14:59 Alex
Right. If if, yeah, if you set it up for the desire for mastery, it could be that.
00:15:02
Right.
00:15:04 Camie
But.
00:15:05 Camie
If.
00:15:06 Camie
Like the DS and W's assumption, we focus on that Bill Curve. A disproportionate number of A's and B's would also be problematic, right? And also, you can kind of.
00:15:19 Camie
Look at your course and know we we already established that most instructors are not grading on the bell curve.
00:15:25 Camie
Most instructors are grading based on mastery, but also their courses where it's participation. So is it showing mastery at that point? If you know we're talking more about a participatory grade?
00:15:37 Camie
Or or things like that. So we even with that, we don't know exactly what it's saying even though it, you know the assumption would be that things are going well, that's not necessarily true.
00:15:48 Alex
We can establish.
00:15:48 Alex
That the DFW's and the bell curve is a possible way to interpret it.
00:15:55 Alex
Maybe not the best, but at the same time the grade scale.
00:16:00 Alex
And great reports can be data to tell us that something's going on, but we've alluded to master. We've talked a little bit.
00:16:07 Alex
About it, but.
00:16:08 Alex
We both would probably agree that mastery teaching for mastery, evaluating for mastery is.
00:16:13 Alex
MHM.
00:16:13 Alex
A more comprehensive or reflective way to look at course outcomes.
00:16:18 Alex
Evaluation. So let's just say more about that.
00:16:19 Camie
Right evaluation, because that's what we're already using. This is a little more, it's going to be a little bit more specific. So just for those who aren't used to this concept, mastery focuses on the mastery individual students achieve on specific task or knowledge in your course.
00:16:22
Right.
00:16:37 Camie
They are largely using objectives and specific aligned assessments to evaluate the level of mastery per objective, and I would be remiss.
00:16:50 Camie
If I did not mention this next thing which we mentioned in every episode, and that is rubrics.
00:16:55 Alex
Got to mention Rupert because.
00:16:56 Camie
10 rubrics are used in this form of observation if you're directly aligning those assessments and objectives.
00:17:05 Camie
That's not always true, though. Some people without that. But, but that's one of the forms.
00:17:11 Camie
That people use.
00:17:12 Camie
This one that does give us a little more specificity because you know we're leaning in to where students are struggling. You can see where the holes are. It's not just that overall grade, but it still lacks the ability to give us a call.
00:17:27
Yeah.
00:17:28 Camie
You still don't know why students are struggling in those spaces.
00:17:31 Alex
No, but we can though.
00:17:34 Alex
Especially with rubrics and we can we can double click down deeper into evaluation of individual components. This is where you ever worked with a A staff member and ID at global campus, we hammer home backward design. We hammer home alignment of objectives, we hammer home, blooms, taxonomy, those kinds of things, measurable verbs, because we want to be able to.
00:17:56 Alex
Like you said earlier, when the doctors are doing the differential diagnosis, not just say I feel like this might be the case, you know, they have to look at evidence. If we anchor it to.
00:18:06 Alex
Backward designed.
00:18:08 Alex
The outcomes are linked to measurable assessments. Those measurable assessments are linked to measurable activities with measurable objectives. We can find. Investigate those sticking points more easily, right because it's easier to say if I if I want a student to be able to identify the differences of these different.
00:18:20 Camie
It's easier to investigate, yeah.
00:18:29 Alex
Glasses or species of of butterfly?
00:18:33 Alex
When they struggle in that particular assignment, that is evaluating that, I can see how they they missed comparison and contrasts with the way I said the segment better than if I just vaguely said I.
00:18:44 Alex
Want them to learn about this right?
00:18:46 Alex
So again, that's why.
00:18:50 Alex
It it starts really, I mean, I know we're talking about evaluation, but it really starts at the beginning of design and development from the onset, we'll get better evaluation, evaluative outcome.
00:19:00 Alex
When we set up the course.
00:19:03 Camie
When we plan it and then design it and then execute.
00:19:06 Alex
My, my, my brother-in-law puts it this way. When he he works with CEO's and coaches them and his. But he basically says like every system is designed to get the outcome it produces.
00:19:18 Camie
Yes, and and I am.
00:19:21 Camie
I'm a system thinker, and so I also kind of.
00:19:25 Camie
Lean into that a little. Of course, there are always anomalies. They're always, you know, students. So we're not saying if one student is having a hard time, but when you notice patterns in data, you know, if 80% of your students are not understanding this concept.
00:19:42 Camie
And there's something in the design that we can look at.
00:19:45
Right.
00:19:46 Camie
And so looking at grading in general.
00:19:49 Camie
Things that we can do to kind of.
00:19:52 Camie
Boost that, understand the data a little better, always to ask ourselves through whether we're looking at those DSW's and going with this or we have specific information from our mastery base.
00:20:06 Camie
Rating here it's give students have any learning gaps that might have hindered?
00:20:12 Camie
Progress and just to make a little plug here, we do have an episode on so.
00:20:17
I like it.
00:20:18 Alex
Subscribe for more.
00:20:21 Camie
Back and and take a listen to that. If you're unfamiliar with learning gaps, learning gaps aren't always about knowledge, they can also be about resources, environment, motivation, and skills. So.
00:20:29
Yep.
00:20:35 Camie
There are ways to assess for those, and sometimes that that is the problem.
00:20:40 Camie
You can also look at yourself as the instructor, right. Did you hold students accountable to mastering those materials and skills?
00:20:50 Alex
What would that what would?
00:20:51 Alex
That look like in an example.
00:20:53 Alex
Just to make it more concrete, we put.
00:20:54 Alex
You.
00:20:54 Alex
On the spot.
00:20:55 Camie
Rubrics. I mean it, it can. So I think for that for me, when I have been in a an instructional position then it is about saying, OK is what?
00:20:59 Alex
It all comes back to reprint.
00:21:12 Camie
Here.
00:21:13 Camie
Our goal is for this week.
00:21:15 Camie
This is what you need to learn and then making sure that they're producing evidence that they have learned that thing.
00:21:23 Camie
On my end as the instructor, it's my responsibility to.
00:21:28 Camie
Resource them low so they can learn that and facilitating that, learning that they're doing, and then I have an assignment that that proves it right and and they know how to get there.
00:21:42 Camie
So.
00:21:44 Camie
They know what they should be doing.
00:21:47 Camie
And then they do the thing, and then I, as the instructor go through, I give them feedback, whether they've done really well. I don't just go. Ohh, excellent job and move on. Right, I say.
00:21:59 Camie
Hey, you did an excellent job here, here and here. Here's how you can extend this next time.
00:22:04 Alex
Because mastery is still, should be seen as an evolutionary concept.
00:22:09 Camie
Always. Yeah. There's not, like a.
00:22:12 Camie
I know people want to say, oh, that's a plus. That's.
00:22:14 Camie
As good as.
00:22:15 Camie
They get no. You can always get better.
00:22:19 Camie
And because that kind of leans in more to that growth mindset and now that doesn't mean that you're not making yay, right?
00:22:25 Alex
Right. Because I could see somebody maybe in like the mathematics field say, well, I've put these 10 problems in and they answered all 10 correctly, right. But what we could do is if you have them show their work.
00:22:39 Camie
M.
00:22:39 Alex
Are you seeing any areas where maybe they arrive at the right conclusion, but?
00:22:43 Alex
Are there areas you can improve the the flow from which they work through an equation to get to?
00:22:49 Alex
The final out kind.
00:22:49 Camie
Right. Or show them a different approach to do so, or or even. What is the next step after they solve this equation? What could they do with it? How do they put it into context? Or maybe even like visualize the math process? So yeah, so there are lots of ways to kind of.
00:22:53 Alex
Right.
00:23:05 Alex
Yeah.
00:23:07 Camie
Hold students accountable.
00:23:09 Camie
In your course.
00:23:11 Camie
But a lot of it has to do with you providing feedback and them giving you feedback on that feedback, right? That may look like an opportunity for them to increase their score it, but it could also just look like an opportunity of ohh hey, what are some resources I could look over for this feedback that you gave me holding your students accountable?
00:23:31 Camie
And.
00:23:33 Camie
Make a huge difference in the course, but it does take kind of that relationship I think. Have you know with them?
00:23:42 Alex
Right. And that also having the relationship?
00:23:46 Alex
Changes then how you view the evaluative feedback, whether it's on the positive side or even on.
00:23:52 Alex
The slightly or?
00:23:53 Alex
Significantly negative side if you've built the rapport with the students, then typically when they when you do get feedback, most time it's anonymized, but you at least.
00:24:05 Alex
Hopefully it's coming from the place of mutual respect and understanding that we want to grow and we want to improve.
00:24:13 Alex
And if we're talking, I'm talking maybe specifically about, like an end of course evaluation.
00:24:17 Alex
But I think this is where instructors can be more open to iterative feedback in a more regular cadence built into the course, like even more developing courses this summer for some new ones to be launched in the fall next spring. And I was reviewing our planning document with an instructor.
00:24:27 Camie
Yes.
00:24:37 Alex
And the instructor put a mid semester evaluation of the course in the midst of the planning document, and I wrote on there. I comment on this is a great practice like keep this kind of getting feedback because ideally you'd.
00:24:51 Alex
More often, but that's no go ahead.
00:24:55 Camie
We'll get into this a.
00:24:57 Camie
Little later too, on student feedback. Sure, sure.
00:25:02 Camie
No, and I will. I will say in terms of more fixes for grading.
00:25:08 Camie
There's a tool in Ultra where you can analyze your quiz or test questions.
00:25:14 Camie
You can also use AI or your own brain to analyze.
00:25:20 Camie
Questions or assignment prompts.
00:25:22 Camie
See if they are.
00:25:24 Camie
Written in a confusing way and also make sure that you have resources available.
00:25:30 Camie
That.
00:25:32 Camie
Students know that they can learn those things or perform those things, even if it's a skill or like a certain format. You're asking them to use.
00:25:40 Camie
We talked about these things in several.
00:25:42 Camie
Episodes. So this is not a new topic.
00:25:45 Camie
But doing that gives you better data for then going oh, wait a minute, are there certain skills that the majority of students are missing?
00:25:54 Camie
In some way and is not fixed for that. Adding resources to my course or is the fix for that requiring A prerequisite to this course?
00:26:04 Camie
And that's a a conversation, obviously, to have with your department, so.
00:26:11 Camie
But just taking a look at that.
00:26:14 Camie
Is really helpful.
00:26:16 Camie
One other thing, I'm just going to mention right here is that.
00:26:21 Camie
If we can.
00:26:25 Camie
Also kind of listen to students in terms of I had a course recently where it was a number two of a part and there was like A1 and A2.
00:26:35 Camie
Yes.
00:26:37 Camie
And in the first course they had it set up a certain way. Well, they're having a lot of student feedback, but the second course.
00:26:45 Camie
Wasn't really giving anything new to the students.
00:26:48 Camie
So kind of also keeping that in mind as you go is how do you it's not just this is too difficult but also are you at the appropriate level and and?
00:26:57 Alex
Yeah, the desirable difficulty is is a real.
00:26:59 Camie
Yeah. And that for me is also part of like holding students accountable, right?
00:27:04
Yeah.
00:27:07 Camie
But as Alex mentioned, student feedback is one of the other things a lot of times we look at that year end thing, we do end of course evaluations and I even know some structures who will avoid these because.
00:27:21 Camie
They've either gotten burned in the past from looking at them.
00:27:26 Camie
And sometimes.
00:27:28 Camie
Unjustifiably so, you know, sometimes they just get some mean comments in there and not not really related to anything in the course that could be improved or on them that could be improved just because you have a disgruntled student sometimes.
00:27:41 Alex
Yeah, there's a million factors. It's the end of the semester. Students are tired. They want to move on.
00:27:43
Right.
00:27:45 Camie
Yes, but that's also kind of the crux of the problem, right is you have students at the end of semester well.
00:27:49
Yeah.
00:27:52 Camie
How are they making comments about the first of the semester? Because they are tired.
00:27:57 Camie
They have forgotten that's, you know, if this is a regular term that's 16 weeks ago. So that's a little.
00:28:02
Yeah.
00:28:05 Camie
It's a little.
00:28:06 Camie
Far fetched to ask them to remember details.
00:28:09 Alex
Yeah. And so it's it's almost another way to look at this is you could see that as one way to do it and it's not that those end of year evaluations aren't without merit or value, but they can be 1. Again, one data point and a collection of data points. But this is where you start to see in certain.
00:28:27 Alex
Project or process management development modes.
00:28:31 Alex
Something like, I mean in in our ideal world we have an evaluation model that is set up to only evaluate once something's created at the very end and then deployed. We have other models that you essentially create beta versions of something tested in active.
00:28:41 Camie
Right.
00:28:48 Alex
Ongoing scenarios and then take that information and go back to the drawing board, retool, rethink and continue to deliver till you get to that gold standard. And I like those more successive approximating models more so when possible because hitting it what you're saying there students are giving feedback when it's fresh when the experience is fresh.
00:29:10 Alex
So in a.
00:29:13 Alex
Classroom situation, whether an online or face to face, this is where at the end of the unit or the end of the lesson, you provide them a link or a form that they can just quickly fill out. Hey, what? What made sense this week? What was tough, what sticking points were there? Was there material that was exceptionally challenging or why?
00:29:33 Alex
Or, you know, asking those kind of formative questions that help you understand how they're feeling as.
00:29:38 Alex
They work through.
00:29:38 Alex
The.
00:29:39 Alex
Course, but then also reveal some of the content matter that might actually need more instruction and more assessment to help them work through it more clearly.
00:29:48 Camie
Yeah, and.
00:29:51 Camie
There's actually research out there that says students actually.
00:29:58 Camie
Are better equipped to evaluate their personal experiences in terms of like content, difficulty, or engagement than they are at evaluating your instruction.
00:30:10 Alex
Oh yeah.
00:30:10 Camie
As an instructor, right. And so when we do get student feedback then.
00:30:17 Camie
And their experience, like that's valuable information but it.
00:30:20 Alex
Ohh yeah, how they feel is in the instruction is way better than their qualification to talk about your ability to to teach, right? That's not what we should be.
00:30:26 Camie
Right.
00:30:28 Camie
Evaluating right. And so when we're talking about this, we look at, OK, well, how do I make this more engaging? It's been not about you, it's about the content. And I think in that terms.
00:30:40 Camie
Iterative evaluation after the lesson. At the end of the unit, something even.
00:30:49 Camie
More regular than mid semester right? Can be really helpful.
00:30:54 Alex
Sure.
00:30:56 Alex
What I mean by that is like even.
00:30:57 Alex
Doing something, yes.
00:30:59 Alex
Even if it's even if you feel like this is a lot to to add in, even if you can only to start off with a mid point in the semester, check in yeah is going to be better than waiting till that very end of the semester. But I I agree the more regular you can build that into a cadence if you do it from the onset, it becomes a norm.
00:31:17 Alex
Kids are used to it. They just see it as part of the.
00:31:19 Alex
Part of the participation in the class.
00:31:22 Camie
Yeah. And there are so.
00:31:25 Camie
Surveys when you build them into the course, they also don't have to be separate. You can just put one survey question at the end of your lesson quiz, right? That doesn't have to have points attached to it because an ultra you can you set points per question not per quiz.
00:31:34 Alex
Yeah.
00:31:41 Alex
Right.
00:31:42 Camie
So you can do that. You can even make that a bonus point question if you are so inclined to do so.
00:31:50 Camie
That can be really helpful because again, it just gets to be.
00:31:53 Camie
Part of your.
00:31:55 Camie
Course culture to give that feedback. This can also come as a course cafe style continuous discussion board if students are having problems or things like that, your inbox becomes flooded with discussion. It's bored and you go oh hey, something's happening. So it's a little bit easier.
00:32:14 Camie
To kind of have an open dialogue when it's when it's a norm.
00:32:17 Alex
Yeah, that's a good. It's a good like asynchronous online course standard for sure in, in, in the face to face, it's the.
00:32:25 Alex
Opening, opening that space up in class at regular intervals throughout the semester.
00:32:31 Alex
Asking those spaces or or sending out emails or having that built in, even if you're teaching in a face to face.
00:32:37 Alex
You still have usually access to an LMS where you can put that in there and it's just fine to have it right there.
00:32:41 Camie
Yes.
00:32:44 Camie
It is so I will say just some other options for evaluating your course other than grades and student feedback is to look at outside evaluation frameworks. We use quality matters for all of our online asynchronous courses.
00:33:00 Camie
That also applies to in person courses. If you are using Blackboard as a content option.
00:33:06 Camie
Because it focuses on alignment and accessibility, you can even use it for your in person courses on the alignment portion, even if you're not posting on Blackboard.
00:33:16 Camie
And saying hey.
00:33:18 Camie
I actually have objectives and I'm assessing them with these assessments. That's what I'm asking students to do.
00:33:22
Yeah, yeah.
00:33:26 Camie
So finding an outside framework that works for you and your context is really important.
00:33:31 Camie
Also self evaluation and just like the students in terms of self evaluation or in terms of valuation.
00:33:39 Camie
If you are self evaluating, do that consistently. Weekly or lesson or unit.
00:33:47 Camie
Monthly. You know what works for you? Where?
00:33:50 Camie
You can remember the most about.
00:33:54 Camie
What did students ask about the moose?
00:33:56 Alex
Right.
00:33:58 Camie
Where did you spend the most time troubleshooting? What surprised you the most in your grading?
00:34:06 Camie
Are your assessments evaluating the knowledge and skills they should be and what have you done to ensure students walk away from this course with skills they can take to the workplace?
00:34:16
Yeah.
00:34:16 Camie
So you can add of course questions to that, but think through what does it look like?
00:34:23 Alex
And one of the implements I've seen instructors do really well with this and some of the courses that I helped develop are.
00:34:30 Alex
They'll obviously, if you find a system that works best for you, that's great. But one thing is put a hidden document at the bottom of each weekly lesson and just have it, say instructor notes for the week and they ask they write out some of those questions, or they write out what were the main points that got brought up this week, or what were the main messages that I that I got, or how did this.
00:34:46 Camie
MHM.
00:34:51 Alex
Lessons seem to function and they keep that both for their own.
00:34:55 Alex
Reflection, but then also as formation for if that course is ever taught by other instructors in the future. Hey, this is a this is a historical annotation of things to keep in mind when you're teaching the course. If you encounter some of the same problems, then you can understand. OK, that's something that I was warned about and maybe that instructor was able to get ahead of.
00:35:13 Alex
Them, but then also.
00:35:15 Alex
So again, it's it benefits you in your own teaching and evaluation, but it could also be beneficial when deployed in some capacities to help other instructors too. So it it can be a win win in that regard as well. And obviously it hopefully helps the students as you then reflect on.
00:35:32
You.
00:35:32 Alex
Your own instructor self eval to improve the experience of the course.
00:35:38 Camie
Yeah, it's.
00:35:42 Camie
It's an easy way to also prompt yourself to remember later if you don't have time to make those changes or make the changes in the dev show. If you're working in a dev show. Looking back on that in addition to kind of being that.
00:36:00 Camie
Living history of the course and you know things that have gone well and whatnot, helps prompt you to make changes prior to the next semester. Moving forward, you can.
00:36:09 Camie
Also do that as.
00:36:10 Camie
A survey for yourself and you would just get the results in a in.
00:36:14 Camie
A different format.
00:36:16 Camie
But that's harder to share with other instructors, right?
00:36:18 Alex
Yeah, I guess that's fine. If there's a a workflow that works best for you, that's just something that I've seen and it's what's nice about that, especially in our instance here, our university is these courses that we help design.
00:36:33 Alex
You know are in part developed and owned in a sense by the instructor, but then also ultimately belong to the university so that we can continue to develop them and use them for other students and instructors down the road. And so having that share repository space of knowledge is really valuable. And as we continue to try and ever improve.
00:36:53 Alex
Is like we already mentioned, mastery never truly arrives. We want to.
00:36:57 Alex
Have that growth mindset.
00:36:59 Alex
I'm just going to hit all.
00:37:00 Alex
The highlights of the.
00:37:01 Camie
Talking points. Let's pretend that you do master and become expert, right. That doesn't mean your expertise cannot grow.
00:37:02 Alex
That's far.
00:37:08 Alex
That's what my degree tells me right now that I have a masters degree.
00:37:11 Alex
And I continue to learn. How about that?
00:37:14 Camie
So this week try adding a one question check in at the end of a module or quiz. What's one thing that confused you this week? It could be the smallest change that leads to the biggest insight. Thanks for joining us on the Pedagogy toolkit. Don't forget to subscribe.