[MUSIC PLAYING]
Hello, and welcome to another episode of Undercool, the materials education podcast. So today, Steve and I are hanging out in the office. And we're going to talk about the acronym that is no longer an acronym. It's just a name all by itself. That's right, ABET. Maybe once upon a time it stood for something. But we'll get into the history with Steve here.
But today, we're going to talk about ABET, what it's good for, what it's useful for, what it might not be good for, and how it can inform what we do in our materials programs and help us teach better. So Steve, to kick things off, why don't you just tell us, what is ABET and why should materials departments care about it? So ABET used to stand for the Accreditation Board of Engineering and Technology. Now it's just ABET. And it's because they do more than just engineering and technology.
They do applied stuff as well. But whatever, it's really the same organization. It's a federation of 30 to 40 professional societies that get together and they make bylaws and set up all the rules and regulations for doing accreditation for engineering schools. And ultimately, it's about making sure that the people who are designing your bridges, your airplanes, your boats, actually know something about what they're supposed to know.
So as a country, as a society, as the world, we can actually trust the engineers that our educational institutions are putting out. That's the big reason. And it's even sanctioned by the United States to do accreditation for all US programs, engineering programs. But ABET also performs many international accreditations at the request of schools in other countries because it's become kind of the gold standard for engineering accreditation.
And other countries want to have-- I mean, it's kind of funny. They want to say, we have the same accreditation as MIT does. So for whatever that's good for, but we'll get into that later on. So why do MSE programs care about this? Well, they should care because we usually have a few students who want to go into careers where they must become professional engineers. And as you know, Tim, the TMS has a committee that sets the professional engineering degree for materials.
It's really metallurgy mostly. And although I don't think more than 20, 30 people a year take that test, it's there in case they need to be. Where would they need to do this? It would be companies like Exponent. These are consulting companies that often do a lot of litigation. When something goes wrong, in some company or some person, who's another company or another person, goes to court. And in court, there are all these rules about who's allowed to testify as an expert witness.
Anything involving the government, only people with accredited-- with professional engineering are going to count. But it's also to impress the jury. So the jury is going to want to know that it's a professional engineer. And guess what society is also a member of ABET, the Society of Professional Engineers. And a long time ago, they insisted that-- well, hey, they make their own rules.
If you want to be a professional engineer, you must have graduated from an accredited engineering program or have 11 years of equivalent experience. So do we really want our students to spend 11 years before they can start a job? They might want to start right after in our department. Maybe 1% or 2% of our students every year want to do it. So we do it for them. And it makes parents and alums feel really good. So those are the two main reasons why we partake in accreditation.
OK, yeah, it definitely makes sense about wanting to give students the opportunity to earn certifications, to have access to different career paths, different opportunities down the road that ultimately rest on the program that they graduated from, having demonstrated a certain minimum quality of the education that is being provided there. So that seems reasonable.
Now, my understanding is that to get certified as an ABET accredited institution, part of the process is having ABET, essentially, inspectors come to the program and do an on-site physical visit. What do those visits look like? And what's the hardest part of preparing for a visit? What do you have to do to get ready? That might not be obvious. So that's a great question. And first of all, it's very important to realize ABET doesn't accredit institutions or departments.
They only accredit programs. So they look at the degrees that are being granted. And in our case, we have one material science degree that says a bachelor's in material science and engineering. That's a program. And that is what is accredited. And that's all ABET accredits. So what you need to do to be accredited, it's a little different the first time, but we've been accredited for so many years. I wasn't even here when we first had accredited.
I've been involved in some, but I think you need to have graduated at least one student before you can be accredited. And when you go for your first accreditation visit, you're visited by two evaluators, not one. So that's a little different. But for most of us in most programs, it's about 125 accredited materials programs in the country. And the vast majority of those have been accredited before.
So what you need to do, your institution will call up ABET and say, hey, we're ready to be accredited. We're going to start the process. Here are all the programs we'd like to do, because ABET does a visit for all the programs at an institution at the same time. The first thing you need to do once all that-- that's all Dean stuff, so don't worry about that.
First thing a program needs to do is start six years before that moment, because they really need to be preparing for this right from the day after their last visit. And I'll get back to that. So what you need to do is create what's called a self-study. And ABET gives you a template to fill in. ABET is all focused around their criteria. They have eight criteria. I'm not going to go through all of them. The most important ones are the first few. So criteria one is all about the students.
Where the students come from, how are they advised, what kind of tracking do you do, how do you handle mental health issues, how do you do admissions, all that stuff. And the second criteria is all about program educational objectives. And these are those statements that talk about what a graduate should look like a few years after they graduate. What's our aspiration for that? And those don't even need to be measured, because some of our aspirations are we want them to be creative.
We want them to solve the world's problems. And how do you actually measure those things? But it's important to have high aspirations, because that kind of drives the whole thing. I actually believe that the objectives are the most valuable part of the ABET process for a program.
It's also one of the easiest parts to be compliant with, because all you need to do is consult with your constituencies at least once every six years, and ask them, are our objectives as written, meeting your needs as a constituent group? Our program has three constituencies, the students, the faculty, and our alums. Some programs get crazy, and they say, their constituents are the universe. Well, how are you going to ask the universe what their needs are?
You know, we just can't get to some of those planets. So that would be a very unwise thing for a program to do. It's great to just have three. It works. But it's so important that you meet with them and document that thing. That's actually easy to do. The third criteria is the simplest, because ABET says, here are the student outcomes, one through seven. And you can add more, but what crazy programs can add more and add more work to their plate? So just do what ABET asks you to do.
You just list it. Those are your outcomes. I think there's a table. You show how they're related to your objectives. That's easy to do. So that's easy. It's the fourth criteria that usually stumps everybody. And that's the criteria for continuous improvement. In that-- and it's very short if you read the words the criteria. It's not much language. But it says that you have to have a outcomes assessment process that is performed regularly.
And in ABET, that means at least two cycles during the six years between when you started to when you get accredited. And it must be appropriate. And that's a big catch word. That could mean literally anything. Yes, it can. So those are the two words that usually catches most programs. And so it really all comes down to, how are you going to assess the outcomes for the students? How are you going to do it in a way that is regular? And how is it appropriate?
So after being on many ABET visits and going to many ABET symposiums and even being on the board of directors for ABET for a while, I've gotten a good sense of what they actually mean by that, even though it's not explicitly written down. So my takeaways for a lot of this are that what ABET really cares deeply about-- and you'll get this from anyone from ABET you talk to-- they care deeply about continuous improvement. This all came from ISO 9000.
And it turned into the EC2000, ABET, all of a sudden, with these words. But basically, they changed ABET dramatically around the year 2000, where they wanted to be more like what industry does for their continuous improvement. And there's a lot of good ideas in there. I have my own personal beliefs. I don't believe students in education. It's like a product that's on an assembly line.
So it's a little harder to measure students' achievement of outcomes than it is to measure quality control on the dimensions of a part, the hardness of the metal, all those things. Those are very easy to measure. Measuring learning is really hard. And actually, none of us really know how to do it in a scalable way. So that's the hardest thing. But what ABET cares about is they at least have a process that makes a very serious attempt to measure.
And what ABET cares about is you must, in the criteria, it says that you must measure the extent to which the graduates of your program have achieved the seven outcomes. Now, let's unpack that. The extent to which ABET doesn't say anything about that everyone has to have an A. They just say your program needs to know how well your graduates are doing so they can use that information if you need to to improve your program. It's a diagnostic. The other thing is the word "graduates."
They're not looking for learning about what students at the first or second year are doing, because as you know, it's hierarchical. You build on the knowledge. And the kinds of things and the outcomes are-- the outcome one is to solve science and engineering problems, complex engineering problems, which usually means open-ended problems. That's not something we expect our freshmen to do. It's something we expect our seniors to be able to do.
So it really makes-- because of that, it makes little sense to do assessment in the first or second year if all you care about is meeting the objectives of ABET. Now, there's lots of people who might want to do assessment earlier for their own purposes, and that's fine. And in fact, the mantra-- at least it used to be-- I hope it still is-- at the TMS Accreditation Committee, used to be improve your program for yourself first and worry about ABET later.
Because if you're doing a good job improving your program for yourself, you should have no problem documenting your processes and showing ABET that you did it. The converse is sort of a fool's errand, right? To just do it for compliance alone is kind of wasting your time. If you're going to do something, make it meaningful. So that's what you should really be doing.
But at any rate, this is the number one criteria that shortcomings are delivered to that there either is no good process or the program only did it once instead of at least twice, or that it's not appropriate. And what they mean by that is either doing it just running through the paces, but it's not giving them any information. The last part of the criteria says that you must use the results of the measurements, the assessment, as input to your continuous improvement process.
And you must have a continuous improvement process as well. The last line of that criteria is the most valuable. It says you can also use as input anything else to improve your program. Now, this has been debated by many people at ABET, but ABET has made it exceedingly clear that the word input is there and not output.
So if you do your assessment and you analyze your results and you show that everyone is doing great above whatever threshold you decide is important, then you may not be able to use the data you collected to actually improve your program. And that's OK. Because then you can bring in those other inputs, information from your advisory board or from industry partners to get that information instead. And so we have really good students.
They come in, they all take our intro engineering course, Engin 100, which is really a fantastic course. It's really an English course, a communications course, but it's cast in a framework of a design, build, test environment, teaching freshmen how to do engineering design right from the get-go. So our students who take that, they learn how to work in teams, they learn about ethics, they learn about design, they learn about doing experiments.
They do everything that our ABET outcomes ask us to do. So by the time they come in our department, in Michigan, they don't join our program until they've had at least one term. Wow, they're amazing. And I remember the days before Engin 100 when it wasn't that way. I'm sure they were dark times. Yes. And so things have really gotten better in the sense of at least outcomes two through seven. Outcome one is the toughest outcome.
Outcome one is using engineering, math, and science to solve complex engineering problems. That's difficult. That's the one we beat on the students hard. We have very high expectations for them. And so the scores for that outcome are always lower college-wide for all of these things. So all that's kind of cool. So at any rate, you know, that's what's the most difficult part. And so we've developed a whole new approach to doing this. We started this six years ago.
We actually started in our department earlier, but we've rolled it out and are now doing it on a college-wide basis, automated, trying to follow some basic principles to make sure our process is completely sustainable. And so this is the process for measuring the student outcomes. Yes. OK, so from the instructor side, I can see my perspective on it.
But I want to actually come back to something else you said earlier, which is the priority order of what to do for improving the program versus what to do for compliance. And when I'm thinking about making changes, hopefully improvements to my courses, I'm always saying, well, what do I really want to do with my class? And then looking at the ABET outcomes and saying, is there an outcome that this change I'd like to make happens to be well tied to?
And if so, to me, that's a good indicator that it's something worth pursuing because most of the ABET outcomes are pretty transparently things that I think we should want our students to do. We should want them to consider societal, economic, environmental implications of their engineering work. We should want them to design experiments that have scientific validity. These are just good things to do anyway.
So as I'm thinking about my courses, anytime I can point to an ABET outcome and say, by the way, I'm also achieving this. In addition to just doing what I believe is good teaching, that's always my anchor for how to make those changes. And that's great. There's kind of two viewpoints of this. There's the ABET viewpoint that they believe that they're driving all of our education by mandating these outcomes. But, you know, their outcomes are kind of motherhood. It's kind of obvious.
And I'd like to think that our program actually has many more outcomes beyond just what ABET requires. And I think any good program will, of course, do all the things ABET wants. But the real improvement of the programs, at least my experience here in Michigan, has never really come from doing the assessment of the outcomes. The real improvement comes from people just like yourself having that good attitude of trying to do what's best for our students in a broad range of areas.
So we have a very robust undergraduate committee that reviews constantly our curriculum. We do curricular reviews apart from ABET because we think it's important. We try to have meetings where we put together people who teach Thermo and Kinetics and ask, are you getting the right kind of background in your students? What's missing?
And you know this better than anyone because you're teaching a math course because we're not getting students coming into our Thermo and our Kinetics courses with enough math. And so we're going to tailor the math they need to supplement from what the math department gives them so they can perform better in our Thermo and in our Kinetics courses.
Another example of massive improvements in our system of our program didn't come out again from ABET but came by the work you did that we talked in another podcast about your alloy design module in the lab. You worked with the people who taught Thermo because while they're taking Thermo, our students are taking your lab, whether you're learning how to use tools like ThermoCalc and design their own materials, their own alloy.
Using those thermodynamic principles they're learning in Thermo actually make it pour dog bone test specimens and test it and then try to understand why it didn't work because it rarely works. But you know it will once you get really good at it but you've got to start somewhere. So how is that captured in our assessment results? You need to go way beyond just
assessment. You need to you know be creative and innovative and you need to inspire a culture in a program of the faculty caring about their undergraduate students. And I am so happy to say I think we've got an amazing culture here at Michigan doing just that. So the way I try to document that I know that every faculty member innovates in every single course they teach.
And so when it comes time to write the self-study I put a note out to the faculty asking them to write me half a page to a page of what they're most proud of in developing in the previous six years. And that's the I think the best part of our self-study because it's honest it's from the heart. It's the real actual improvements that all these individuals do. Often with the help and support of the undergraduate committee or other things.
But what a great way I think to document all of that and to really demonstrate that we're doing massive improvements to our program all the time. And it's coming from what the students tell us because we have we have town hall meetings every year with our students to hear comments. We heard a comment last term that the BioMed course is actually too biological and not enough material science. And guess what we're acting on that.
And so Brian Love who also agrees with that assessment even though he's taught the course before is actively trying to change that course now. We also get comments from our external advisory board who are people in industry who are giving us heads up about things they need. We get information from our alums with whatever careers they followed. So it's you know it's a lot of good stuff that we use to improve our program.
But I view this outcomes assessment process which we're mandated to do is actually a critical part of this process. And the way I think about it is these are the diagnostics that if we don't do we can get ourselves in serious trouble just like engineers design that little material inside your brake pads. To start squealing when the brake pads get small. You need that there even if you change your brake pads religiously and never hear it.
You want to know before something bad happens that it's about to happen. And that's how I view our outcomes assessment at the very end. I talk about the future of what we can do. I think we can even learn more from our outcomes assessment. Well let's get to nuts and bolts for a minute here because we have this big picture aspirational vision of here's what all this assessment accomplishes. Right. Here's what it helps our program get better at. But there is still that implementation layer.
The actually doing it which might not be obvious especially if a program is trying to step up their game in terms of having an easier more efficient time with checking the ABET boxes. While doing great teaching. So how do you actually do outcomes assessment. How is that implemented at a practical level. That is absolutely the critical question. We all know the biggest problem with doing outcomes assessment is getting our faculty to do it.
The only people who can really probe the students in an efficient way where we distribute the load of the work is to have every single instructor of any course. That we're using to assess our students. They have to do the work. So it comes down to some fundamental concepts to make a build a process that everyone will participate in in the easiest possible way so that it becomes sustainable. And we do it all the time.
So the first thing is although a bit you know they don't tell you how to do this. They just tell you have to do it. I've seen a lot of suggestions and I don't like most of them. Like ABET, symposia. When they say oh well you could do outcomes one and two this term and outcomes three and four the next term and then new outcomes five and six the term after that and do that and then cycle and by the time you're done six years you've done it two times.
The problem with that is everyone forgets what they were supposed to be doing because we're creatures of habit. So we need to do all of the outcomes every single term in my opinion and find a way to make it as easy as possible for our instructors to actually do the work. So that's what I first make it easy for the faculty and instructors so that actually gets done. The next thing you know we need to do is how do we do this and also make it meaningful.
So what we've done is we built and a lot of programs do this by the way I think people have learned over the years. So it's pretty standard now. We build a matrix of we look at all of our required courses and all of our elective courses and we assign we try to assign no more than one outcome to any given course.
Now we can't do this for all courses because we lean very heavily into our design and our lab courses because those are the courses where it's more appropriate to measure things about teams, communications, design designing experiments, ethics. So those are outcomes two through six, two, three, four, five and six. That's five outcomes. But luckily we've got two lab courses and we have two design courses. So we're able to break it up so they only have three outcomes to do.
Some programs wait until the last year and dump this all, all seven outcomes on the capstone design course, which makes sense because you're measuring with our graduates the extent to which they've learned it. But it makes it unsustainable in my opinion. Right. It's even more work for the person who's already doing the hardest class. Exactly. And that's just not fair. So we've broken it up.
So our idea is let's collect a relatively sparse data set, but do it every single term so that we end up with a massive amount of data after six years. We do this 12 terms. And I have to say it kind of works well. So that's the first thing. And we also only do assessment in our junior and senior classes. We assess outcomes one through six in our required courses. So regardless of path, every student is assessed in our required courses because they all have to take it.
Then our elective courses, because we want to distribute the load again, we have outcome seven is done in all of our which is lifelong learning, learning, different methods of learning, all that. And so we have all of our elective courses measure that outcome. So regardless of which elective courses our students take, they're being assessed. So we do assessment of all outcomes across regardless of path for all of our students. So once we've done that, we need one more thing.
How can we make this even easier for our for our instructors? And that has to do with the actual mechanics of how they first they have to create. We expect our faculty to create the assessment measures. We have a choice. We could hire assessment professionals and have them tell us what the assessment should be. Or we can ask our faculty. I happen to think our faculty are better positioned because they're teaching the course. They know what they want their students to learn.
And after all, don't we want all of our students to be able to solve complex, open ended problems? And all of our faculty do that. So they should find something within the context of their course. So it's meaningful for them for their course parameters, but use that to measure the particular outcomes. So what I've done is I made a series of videos. One short video for every single outcome, explaining to the faculty good, good approaches to build an outcome assessment.
And that assessment measure, this measure for the assessment, it can be a homework problem. It can be an exam problem. It can be an activity that's not even graded for the course, whatever the faculty member wants to do. But if it's outcome one, for instance, it must involve a complex problem. And I show them the definition of what a complex problem is to ABET. And simply say, if you look at that, it's really just an open ended problem. And our faculty are great at writing open ended problems.
So every faculty member, and they usually only have to do this once, because if they teach the same course over and over, they can keep using the assessment tool if it's good. And they can talk to other faculty if they're inheriting a course and borrow theirs. That's OK. But in each case, we have some faculty member to make a document where first they write down what is the outcome they're assessing. Because you want it first and foremost in their minds. Then what is the actual assessment?
And they write down that problem in great detail, exactly what the student would see, what you're asking the student to do. And then finally, after that, they write a short little paragraph explaining why they believe this is an excellent outcome, a measure of the outcome they're trying to measure. And again, that's just to make sure that it's present in their minds and they've actually thought about it. Once they do that, they put that in a document and make a PDF.
And at the end of the term, they upload that document. So they'll get a link. This comes from our college because we've developed this system across all of our 12 programs. And the college has our matrix and the every program has an ABED coordinator on the ABED coordinator for our program. So every term I review the list of who's teaching what courses that are being used for assessment. I confirm that with the college. They send a special link to each instructor with everything pre-filled.
So what term it is, what course they're teaching, their name, all that stuff that's all in there. It's done for the faculty member. All the faculty member has to do is upload the PDF of what was the metric. And they have to upload the scores of the students. This is where it gets tricky. So we want to know the actual unique name, the student identifier for every score. This is going to be critical for another onerous thing that ABED has made us do. ABED demands that we disaggregate the data.
How does that mean? That means that we can only consider students who are in our program when we do the analysis of our results. They don't want to contaminate it by a graduate student is in the class, by a student from another department. I still don't understand why, but we have to do it. If we know the unique names of every student, we know that information. We have a big data warehouse that's got all that stuff.
And in the back end, we can easily filter the students who are in our program and not making a faculty member do that is insane. Because if you have 60 people in your class, you don't know who they you don't know what program they're in. And you don't even want to know what program they're in. You want to treat them all equally. You don't want to have a bias. Oh, they're graduate students. They should do more work or oh, these are students from, you know, from.
Well, I won't say the name, but that other major that we like to pick on, you know, that we know we're going to you don't want to know that you want to know that there are students in your class. So and plus, it's really hard. A faculty member is going to have to download, you know, look up what program they're all in. What a pain. It's easier to just do your whole class and dump it in. And now we're launching a new way to do this.
So if you have a spreadsheet, you just highlight the two columns, the unique names and the score. And we ask all faculty to put in a score normalized zero to 100 so we can combine the scores with other courses to see how they're doing across an outcome. They just highlight those things and copy. Go to a line. I never knew you could do this in a Google form right in the line. You just paste it and it's kind of like comma separated values.
But in the back end, they use regular expressions to parse the data to put it back into a spreadsheet. We're doing this because these spreadsheets we were getting from faculty in the past were all over the map and it became really ugly for the back end people to deal with this data. So this should be much better at any rate. The way the reason it works, because the first thing you might think of is, oh, my God, that's a FERPA violation. That's student records. You can't make
that public. And we don't. But the only way it works is because our university signed a legal agreement with Google to make sure that the instance of Google that we have is FERPA compliant. So we're allowed to use Google mail from the university to talk about student issues and all of that. If your university doesn't have that agreement, you'll have to find another solution. But it works really well for us. And that student identifier is incredible, as you'll see in a few minutes.
So that's all. But from a faculty's point of view, they just give that assessment, collect that data, upload a PDF of their metric and upload the student data. And they're done. It takes very little time. And believe me, I remind everybody to make sure they do that assessment so they have that data.
And then it works. And after we've done this now for six and a half years now, we have 100 percent compliance, except the very first year, one of our 80 year old faculty members who didn't understand Google Forms didn't do it. But that's pretty good. Yeah, I have to say on the user side. Being tasked with three outcomes per semester due to the lab class, it takes me an hour, maybe an hour and a half tops to do this once a semester. And as you said, I just I have my assessment items.
They're already in our learning management system. So I just grab the student scores, normalized to 100, grab the student IDs, paste columns in a sheet, upload to our collection form. And I'm done. It's really quite painless. Yeah. And from the ABET coordinator point of view, I just go to the Google spreadsheet that they build and I can instantly see who's done it and who hasn't. I can click on the documents and check them very quickly.
And if there's a problem with a faculty member, I just go visit the faculty member and talk to them and help them make it better. So it's great. So the really cool thing is when it's all done, there's this program called the A BET. It's called Tableau that I'm sure many places use. It's, you know, a database program that ingest spreadsheets and you can have you can build a user UI that does different filtering on what you see.
But the data that we care about, what we want to know is we want to be able to see a histogram of the actual values of the scores that our students got, where every single piece of data is on that histogram. In the old days, we used to report an average and we would say if the average is above this number, we're good. And of course, A-BET evaluators through no fault of ABET, but because evaluators like to come up with new stuff, started saying, yeah, well, what about the standard deviation?
And what about the modality? And it's like, yeah, well, what about it? And, you know, eventually it's going to become part of ABET evaluator lore that if this isn't done, you're going to get a shortcoming and there's nothing you can do about it. So how do you protect yourself against what I call the ABET virus that rapidly, you know, because, you know, 13 people, visitors on a team hear about some great idea, which has nothing to do with the real thing.
And then the next year they go to 13 different programs and they infect all those new teams. Well, when I did this, it was and all of a sudden it's growing exponentially like a virus. So how do you protect yourself against that? Well, you know, we're scientists. We know that the best thing to do always is just to look at all the data.
And so what we plot is the histogram of all the data so we instantly can see what the standard deviation is, what the modality is, if there's skewness, we can look at the tails, you know, all of that. We just see in an instant. And in a way, it makes the analysis of the data very straightforward because you just look at it and you can tell. So what we do is for each outcome, some way of a spreadsheet and the spreadsheet has little boxes. And you first you choose what department.
And when you choose the department, you can choose which outcome you want to look at. Then you choose which terms and you can check boxes and choose, you know, like one term or one academic year or two academic years or three, whatever you want. And you'll pull in all the data. Then you'll see for that particular outcome, here's this curve, which is a histogram. And there's no curve drawn. It's just the actual data. And it gives you an average.
We have to filter out zeros sometimes because when students drop a class and it still shows up in the instructors grade books. So the zeros usually are not meaningful. Sometimes they are, but whatever. And then you do the magic because, like I said, ABIT doesn't accredit departments, they accredit programs. And your department might have a course like we have our structures course MSE 350 that has a huge number of students who weren't in our program in that course because we offer a minor.
And one of the requirements for the minor is our structures course. So we have students from biomedical engineering, from aerospace, chemical engineering, from mechanical engineering all over the place that are not in our program. So when we have another checkbox that says which program do you want to filter by? And we can choose our material science and engineering undergraduate program.
And you'll want you click that button and you watch the numbers drop because now you're only getting the students from your program. And it allows us to then prepare these histograms so that I can go to a faculty meeting and I do this once a year. And I show the entire faculty how are our students doing on outcomes one through seven for the last academic year of just students in our program. And we look at the data, we discuss it and off we go. And that's continuous improvement.
We've used it as input if it's not going to help us. That's OK. At least we've considered it as input to our program at the level that's appropriate. So that's what we do. Yeah. Well, the process sounds great. I love. I actually enjoy that annual faculty meeting where we look at the data is we're visually interrogating this database of student performance in a variety of different ways.
And it's quite interesting to see where the program is doing particularly well and where there are improvements to be made. So I'm a fan, but that's just my opinion. Would you say that this method is working from the ABET point of view? Are they satisfied with the process that you've described here? Yeah. Well, we just had our ABET review last fall and we have 12 programs that are accredited and 10 of those 12 got NGR. No general. I forgot. NGR is perfect. No shortcomings.
And so, yeah, I think it worked really, really well. Were there problems? Yeah. That's why the two programs that got dinged for shortcomings got dinged. And guess what? They deserve to be dinged because they just adopted our process. But forgot a really important part. They forgot to document how they did continuous improvement of their program. They just said, we did this and it's not going to help. That's not OK. You must show continuous improvement of your program. And they forgot to do
that. Now, they had done it. They just didn't document it. So they had to go back. And of course, both of these programs had pretty major curricular reviews and changes. And they just didn't document it or write about it. So it made them complacent because it worked too well, I would say. So don't get complacent. You still have to make sure that you document and continuously improve your program. And this is not going to get rid of that part at all.
Well, it sounds like the process works as long as you follow the process, which sounds almost tautological. I'll have to logic that one out for a minute. Well, how I think about that, let me ask this. You've mentioned that we sort of started this process six years ago, seven years ago now. What do next steps look like for our continuous improvement of our process for continuous improvement? How are we going to get better at inoculating ourselves against ABET?
So when I look at these histograms, I think when anyone looks at these histograms, although we have something like 90% of our students are above our threshold for minimal acceptance, you can graduate from the University of Michigan with a 2.0 GPA. That's like a 60% for the way most of us grade exams. And so that's our minimum level because that's just being honest, because a 2.0 can graduate you. But when you look at the data, there's always a few stragglers in the tails that are below that.
And you have to ask yourself, what about them? And so this has actually been a good process by looking at these histograms over and over. It's made me think about what about that group? Even at our university level, we are very, very proud of our graduation rate at University of Michigan. Our five-year graduation rate is like 93%. That's the envy of the whole world.
Only a very few schools can say that. I remember going on college visits with my daughter and the people standing up and saying, "We are very proud of our five-year graduation rate. It's 65%." And I'm like, "What? How can you be proud of that?" And I came back, and when I talked to some of the deans about that, they said, "Yeah, we're unusual."
Even with our amazing 93%, and by the way, some of the 7% are accounted for by people who dropped out of engineering but still ended up graduating from other colleges at our university, there's still the stragglers of the 3% to 5% that do leave our university. I just went to a second provost seminar on teaching where the provost stood up and talked about this and said, "What can we do about that 3%? How can we make sure that they graduate?
That should be our focus." And why shouldn't that be our focus in our program? And maybe we can use the ABET data we're collecting to help us understand that because guess what? We know who those people are. We haven't accessed that data yet because it's a little complicated. You still, I believe, we would need IRB approval to actually do a study. I'm not sure. I'm going
to have to find out the rules. But a new tool just came available that might let us easily do this without hiring education researchers or data specialists. And that tool is MAIZY. MAIZY is U of M's generative AI bot. That's a private generative AI bot system. So it means we can look at that data without exposing student data to the world. We can stay FERPA compliant. MAIZY has a way to link to a SQL database, which is what our whole data warehouse
is built on. And so I've been talking to those folks and we're going to see, can we ask the generative AI bot questions like, the students in the tails for outcome one, you know, can we do a longitudinal study? Because we have several years of data from when there are sophomores until they graduate. Do we see them improving? Do they graduate? What happens to those students? Can we talk about race and gender? Can we talk about first generation students,
Pell Grant students? What can we learn from the data of the students in our tails? That could significantly help us improve our program. And I'll tell you what, if we did it, I'm not putting this in our self-study because I'm not doing this for ABET. I'm doing this for ourselves because of what I said way back. First, we want to improve our program for ourselves and worry about ABET later. But I think it's a wonderful exercise, a wonderful way to use our data.
I should also mention that our Center for Research on Learning and Teaching, CRLT, which is solely focused on improving education in Michigan, when they saw our system, they came and asked me, can we use your system where we put in, you know, our own questions? Because they're doing curricular revision pilots with a few different programs or departments because they do care about departments. And so they're going to be using our system for their purposes, even though it has
nothing to do with ABET. So I think it's really, really cool. We have a lot of data. We've got data from, you know, 13,000 students in this last pass. In any given term, we're looking at 3,000 students. That's a lot of data. And it's only growing because I think other programs have learned how valuable this is. So that's what we do for our ABET outcomes assessment. And it's made getting through criterion four a breeze.
I really love how this program essentially started as, let's make sure bridges don't fall down. And now we're at a place with it where we can interrogate this really rich, robust data set to ask questions about equity in our programs and to say, how can we better serve the students who we are not currently serving well enough? I think that's fantastic. Well, that is all the time that we have for today, but that was a pretty good day one deep dive into
ABET. I think we might have to revisit this topic in the future now that we're asking interesting questions about what we can do with everything that we've learned from these self-studies and how we can use it exactly, as you said, to make our own programs better for ourselves and for our students. And I should mention, so, you know, I agree, Tim. I'd like to, you know, maybe bring on, you know, some people from ABET. I'd like to see if Jeff Fergus
can talk to us. Jeff Fergus has probably been one of the most influential people from the materials community at ABET. And I really think his principles and values align closely with mine. You know, Jeff was the head of the EAC, the Engineering Accreditation Commission. He's been in charge of training for ABET. So he's very, very well known at ABET. And so I think it'd be great to get his ideas because, you know, one of the reasons we're doing this podcast is to try to share best practices with
our materials community. And to that end, I hope everybody comes to the North American Materials Education Symposium. There's the plug. It's a plug because on Friday, I'm going to be doing a free workshop on, I will show everybody our outcomes assessment
stuff in detail. You get to play with our Tableau system and I'm available to give, you know, to answer any questions, not from a official ABET standpoint, because I don't have any official standing in ABET, but just from a colleague in materials who would love to help all of our programs get through ABET with a minimal amount of work and do a maximal job on their ABET. So come to our symposium, get your chair to send you and we'll help you with ABET for free.
Excellent. Well, thanks for being in the hot seat today, Steve, and to everyone else out there in the world. We'll see you next time on another episode of Undercooled. See you later.