277: California Is Outsourcing the Bar Exam to Kaplan?!?

Welcome to the Bar Exam Toolbox podcast. Today we are talking about the California bar's decision to outsource the drafting of the 2025 bar exam to Kaplan. Your Bar Exam Toolbox hosts are Alison Monahan and Lee Burgess, that's me. We're here to demystify the bar exam experience, so you can study effectively, stay sane, and hopefully pass and move on with your life. We're the co-creators of the Law School Toolbox, the Bar Exam Toolbox, and the career-related website CareerDicta.

Welcome back to the Bar Exam Toolbox podcast. Today we're talking about California's recent decision to outsource the development of part of the next bar exam to Kaplan. So, fun times, Lee, huh? Oh my gosh.

Can we just talk about how I was on vacation when this news broke, and I kind of missed it because I was on vacation. And then somebody sent us a note about it and I had a moment where I really kind of heard the record skip, when you're like, "What is happening? I don't understand." I was like, "That doesn't compute. Even with my vacation brain, that does not compute." Alright, well

for a recap, for those who are not yet familiar about this insanity, here is what is going on in California. Well, the California Bar says it's running out of money - kind of per usual - but more so than normal, apparently. They want to get into remote testing at test centers. The NCBE, who writes the standard bar exam, particularly the MBE, which is the most relevant piece we'll talk about today, won't let them do that, for probably fairly obvious reasons.

Oh my gosh, there are so many problems, I don't even know exactly where to start. Well, let's start with the fact that we have students who are studying for this test already. Whether they're foreign-trained attorneys or people who are studying part-time or took the last administration off, people are now prepping for a test that hasn't been written yet, and we don't exactly know what it looks like at all. I mean, I guess we know what the essays look like, sort of.

Well, those are not changing, to be fair, right now. They will change eventually. For the next test, at least, we're only talking about the multiple-choice. So right now, if you're going to practice your essays, your PTs, you're good. You can keep doing that.

Yeah. Now, multiple-choice though, as we know, it's very hard to write multiple-choice questions. We have done this exercise and it is not easy.

I kind of find it hysterical that that's what they decided to start with. It's like, "Oh, we're going to just write 200 multiple-choice questions." All of which take a long time if you do them yourself. I don't know, maybe ChatGPT can just write them, but we'll get into this. Maybe a few copyright issues all around with all of that. But yeah, these are not easy to write. And the other thing is, they're not easy to test and validate.

No, this is why the NCBE always includes experimental questions as part of every administration. They are testing questions for validity before they include them on the test. Also, you need to mix up the questions. You can't just have the same 200 questions set and be like, "Oh, we'll just do that in February and then we'll like mix up the order and do it in July." I mean, people are going to start memorizing them. That's a terrible idea. Is that what they're going to do? They don't say.

I think we're not alone. Everyone is like, "What the mmmp about this?" It's a family-friendly podcast, but, really, seriously, like, WTF? There are so many things about this that just are, frankly, kind of insane, like, do we even know how the next test will be administered? Is it going to be at a test center or are you going to have to go to in-person? We don't even have that basic information right now.

What about people with accommodations? What sort of testing center are they going to take their test at? Is it going to be at these testing centers? Where are these testing centers? Where are they? What are they? I don't understand.

I mean, I remember back when I took the GRE a long time ago, I just went to a local neighborhood testing center and I sat down and I took it, and it was adaptive, they gave me harder questions if I got them right, that kind of thing. So I think that ultimately is the goal, which, again, fair enough if you can figure out how to make that work when everybody needs to do it kind of at the same time where the questions are going to be leaking out.

I know. And then I also feel that some bar takers are going to probably also file lawsuits about this. I mean, I don't know what the grounds would be, but there could be reliance, discrimination, they didn't validate the test. I don't know, but I feel that some attorneys are sitting down and making a list of possible claims, because this just doesn't sound right to me, in my advanced legal knowledge.

No, the other kind of fun aspect of it is, Kaplan has to get out of the market, obviously, for test prep in California, but people have already signed up, presumably, to use their stuff. Are they just going to get a refund and like, "Oh, sorry, you have to go to some other company now"? I mean, the whole thing, it's just beyond.

It is really bizarre. And why, why, why do you have to do it right now? I understand that the Bar is running out of money. I mean, I don't, because I'm still licensed by the California Bar and my bar dues are a fortune.

Right.

As you know, because you see my bar bills.

But at least you get to actually be an attorney. I'm inactive and I still pay like half of what you pay for literally nothing. Nothing. I think maybe they offer me life insurance policies. That's it.

Well, at least you don't have to do your CLE, which is coming up for me, and I'm going to be even more salty about doing all my CLE and then paying those bills, because the Bar can't manage its own money and make good choices. On our work chat, our Slack channel, we've had some discussions about this when the news came out, when I got back from vacation and was like, "Wait, what? What? What has been happening?" And I feel like it's a universal confusion.

Yeah, I feel like we're kind of seeing, what is it, Fyre Fest plus California Bar, whatever. I don't even know what to call it. All combined into one really poor set of decision-making. It also has the feel to me of like somebody's on the golf course who happens to work for the bar, who's on their buddy's golf trip with somebody from Kaplan, who's totally like, "Oh man, now we can, like, disrupt the bar, man. This AI stuff, ChatGPT. Yeah, we can totally just knock this out for you, man."

Yeah, for what was it? Eight point something million. On one hand, that's a lot of money. But on the other hand, that's a lot cheaper than administering the bar? How much money are you spending on administering the bar?

I mean, that's a separate question that maybe we should take up with other people. But yeah, it does not feel very well thought out. I really feel for the people who are trying to navigate this situation, because if we're flipping out, what about someone who's planning to take the bar exam in a few months? I mean, literally the next exam. It's like, if you're taking the bar again and you're taking this exam, and this exam does not even conceptually exist right now.

Yeah. I mean, results come out in November, we're recording this in late August. So, November is not that far away when you look at the calendar. Yes, it still feels like summer outside, but as my daughter was remembering, Halloween is coming. November is going to be here before we know it, and we are hoping that we have some examples, and then we're just supposed to study off of the old framework? I mean, the whole thing just feels so uncertain and really chaotic, and unnecessarily so.

I honestly feel like it's even more chaotic than it was during COVID, which at least there, there was an excuse and a reason. But it was also super chaotic for people and very, very challenging to revamp how you're thinking about the test and where you're taking it. We had to do a whole podcast on like, how do you take the test if you don't have access to write something down? We don't even know, in February what that's going to look like.

I know. Then you also think about the added stress of all of those folks who are going to sit for the bar, and then they have to figure out how to curve it. I mean, that's the other thing. Like, everyone's going to take it and they're going to be like, "Well, we just see what happens"? I mean, I don't

understand. Yeah, I think actually that's kind of what it is, like, "Oh well, we'll just figure out how to scale it after we give it." Because that's what I'm saying, there's literally no time to validate any of this. It's not like you can go out and test some questions with a bar population. It's the next exam. They just signed the contract like two weeks ago.

Yeah, it's really weird. So, we know that Kaplan is writing questions, even though they are not a test-writing company. I also took the GRE on a desktop computer back in the day we probably took it around the same time. I remember that, but that's not how the bar works. You just can't roll in on any given Tuesday to sit for the bar exam. So, they still have to come up with these test centers. And then, there might be litigation.

Literally, the press releases are referring to the fact that they're probably going to get sued and that there are indemnification clauses and limits on legal liability in the actual contract. I think it's probably fair to say that no one on any side of this believes that they have resolved the copyright problems.

So, basically the issue is, the only winners in this whole thing are whatever BigLaw firm the State Bar and Kaplan have hired to consult on this potential litigation and probably represent them when they get sued.

Maybe some potential class action attorneys who are licking their chops at this. But then again, who pays out when they lose? Oh, probably the bar if they lose it, then that's us.

Right, which is not fun. I hope they have insurance. Oh no, the whole thing is so stressful. Okay. Well, this feels like one of those COVID podcasts we did back in the day,

I know, where we're just like, "What is going on here? Why?" We just don't to tell

We just don't know what to tell you. Well, I guess the thing is, you've got to just accept the chaos, because here's the thing: No individual student has any power to get the Bar to change their mind, I don't think. I think you're just at the whim of the Bar. The law schools are going to be upset and talking to the bar. The bar doesn't care what we say, but they're going to get feedback, I'm sure. But they've signed this contract, so they're probably going forward with it.

Right. think at this point, there aren't going to be any real practice questions. I mean, they might hand out a few, but I think you just have to assume they're going to be similar to the MBE questions and use a tool for that. I literally don't know what else you would do, because like we said, it's hard to write these. No one who does MBE prep specifically is going to be writing their own, because what would they write them based on? They've never seen them.

Right, right. They've never seen them. And I think the thing is, even within prep companies, some people find that UWorld MBE questions feel slightly different from AdaptiBar MBE questions, which feel slightly different from BARBRI MBE questions. They don't all feel the same, because they were written by different teams of people with different parameters. So, we're not going to know exactly what those questions look like.

Right. Hopefully they release something saying what content's even going to be on it. I mean, are you just to assume that they're using the same exact content that's on the MBE subject matter outline?

I don't know. That's cool. Yeah, it's so fun to not know.

Also, if they are, isn't that kind of stealing? I don't know. Not legal advice, but just a curiosity.

It's weird. It's weird. And not to spread rumors, but I feel like back in the day Kaplan got in trouble for copyright infringement with the NCBE.

Maybe.

I have this memory. It was a long time ago, like back in the early days of us working together. But when they were doing PMBR, which was their like MBE, that some of their questions got a little too close for comfort.

Right. But that

could my bad memory. But it's very shady, because you're trying to write questions that mimic the real questions.

I mean, it's going to be very hard. Do you feed them all into ChatGPT and then just be like, "Oh well, we edited them a little bit"? I would be concerned about that if I was the NCBE's copyright lawyer. I haven't read our contract with them recently, but I'm pretty sure that would not be allowed.

I feel like that's not allowed. I think we aren't even supposed to edit those questions at all. I think we're supposed to leave them as is, and say we licensed them. I'm pretty sure that's what it says. I mean, we licensed them a while ago and I haven't read the contract, but I feel like it says something like that, like, "Don't mess with our questions."

You can kind of assume it would.

Yeah, yeah. Okay, the other thing is that the law schools have to be freaking out, because bar pass rates are such a huge big deal, especially in California, for California law schools. I would say that if you have already graduated and maybe you're a retaker for one of these 2025 administrations, maybe circle back with your bar support office.

Yeah, I agree with that. Basically, if you are taking one of these tests, you need to keep your eyes out for the prep materials that they say are coming at some point in "the fall". Hopefully that's sooner rather than later, we'll see.

I mean, the fall is quite a long season, as I was discussing this with my six-year-old yesterday.

Yeah. Arguably it could start in a week and change from when we're recording this, on September 1, or maybe not. Are we going by the lunar calendar? I don't know. Who knows? At some point, some materials will likely appear, you should probably look at them. I think the schools hopefully are going to be having as much information as they can possibly get and obviously talking to whoever they can talk to. But the whole thing is just such a ridiculous mess at this point, let's be honest.

Yeah, I guess you just have to go into it with this idea that you have to play the hand that you're dealt. This is kind of a crummy hand, so you just have to do the best with it and know that they are probably going to curve it all and everyone's in the same boat. That's the good news.

The only thing I was thinking, maybe they won't even get around to writing all 200 questions and you'll get a shorter test. Something like that could happen. I mean, who knows?

We've all been in these meetings with high-level people who are just like, "Yeah, we can do that." You I have definitely been in some of these meetings where it's like, "That would be so easy." And then the people who actually do the work show up and are like, "Actually, that's a lot harder than you think."

Yeah, this is actually really hard. They're people who do PhDs and things like validating tests, and there's a whole process. There're some best practices. Best practices, probably not give the test six months or less after you sign a contract. I mean, guessing that's not a best practice.

No, and people litigate over the SATs. I just feel like this is also ripe for issues of discrimination. That's why you're supposed to validate these tests, to try and minimize these things. I mean, is that part of the indemnity litigation clauses? I don't know.

No, I think they just cover the IP, actually, from the limited information we saw. What could possibly go wrong with a bunch of lawyers involved? I mean, you just look at the NextGen exam. People think that's moving too fast in certain ways. They have been working on that for literally years, and it is not even coming out for another several years. There're a couple of years now at this point.

And they're trying to calibrate them and give the test so people can write answers, so they can see what people are doing with them. One of the things they're doing is getting groups of people together and giving them these tests. That's what they're doing.

They've had meetings and committees and all kinds of stuff. They're having discussions about what the content is and fighting over it, and then coming back together and then having another list. You can't do that in a few months.

No. Also, they're having webinars for law schools and people who work in test prep, and they're doing interviews with folks like us to talk about it. They're out to educate everyone. And this is not saying I think the NCBE does everything right, because I don't. But I'm just saying that that is what they're doing, which is what you do when you write a new test. I just don't understand it. I don't know, it's all very anxiety-inducing.

It is, it is. But again, back to what we control, what we don't control. I

true.

We do not control what the California Bar does, if anyone has not learned that yet. They just do things and people react to them. I just feel badly for the people who are trying to have some sort of orderly preparation period and know that they needed more time and they thought they were doing the right thing by maybe skipping the last test and waiting till February. Now they are in a world of chaos.

Yeah. However, I will say that if you have struggled with multiple-choice questions, working on any multiple-choice questions is probably still a good idea, because even though they're not going to be the same, many people struggle with time management or getting distracted by answer choices or not carefully reading the facts. There are so many things that we find that folks do struggle with that are going to translate across tests.

I know. I feel like we're just going to assume that it's the same seven MBE topics, unless there's a way that could be infringing copyright. I feel like you could use the same material, because that's black-letter law, it's not really copyrightable.

I know, but about study guides, the ones that do all the subjects and all the stuff that's covered? That's probably copyrighted.

Well, yeah. I mean, people presumably will still buy stuff from the NCBE, because I can't imagine that California is going to put out a full subject matter outline of rules at this point.

No, but then I think the easiest way to get around some of those copyrights would be to test the California-specific subjects.

But that would be total insanity. Let's face it, they have nothing to work off of with those.

That's true, they don't even have a model for that!

Also, that would be even more unfair than ever. All these people who are going to find out in November that they didn't pass, they've got at that point a pretty limited time to study. And there's no way that you could just switch all the multiple-choice topics. That would be even more insane.

Yeah, I hope so.

I mean, maybe I'm giving a little more credit than is due here, but I'm like, "Surely not. Surely you wouldn't do this."

I know. I mean, this whole thing is pretty crazy.

Yeah.

It's all just so bizarre. I'm so sorry, everyone. I'm so sorry. I didn't think this was a possibility.

This wasn't on the 2024 bingo card.

No, it was not. It was not. I was so confused when I got the note while I was on vacation. I was like, "What is this person talking about?"

Right. Well, it's kind of like what they did with a Baby Bar maybe like half a year, a year ago, where suddenly they're like, "Oh, by the way, that whole test is multiple-choice. Moving on." And everyone's like, "Wait, what?"

I know.

I'm sorry, was this under discussion? What's going on here?

Yeah. And the Baby Bar is not just like a road to licensure, but for some students, it's how they get back into law school when they've been academically dismissed. And so, a lot is riding on the Baby Bar, and they're just like, "We're going to change it. Sorry."

Yeah. I feel like that's actually the most infuriating thing to me about this whole thing, is, these are actually people's lives and careers, and I feel like it's very disrespectful to just be like, "Oh well, we're just changing everything and we're not providing you any information at all, but I'm sure it'll all work out. Good luck, bye! And also, give us a lot of money to take this test."

I know. And then I also feel like you're going to end up with this chaotic situation of possible litigation. And then are they just going to waive people in? I mean, it's some of the same stuff we were talking about in 2020 with COVID. It's like, what are they going to do if this is a true disaster? And in that case, they didn't end up waiving a bunch of folks in, but they still had a test that was tested. Even writing these essay questions, they take forever to write these questions.

Which, we've all seen mixed results with our explorations of the AI tools out there.

I know. It's like, can you make

them write some questions? Sure. Is that question a great question? Maybe not. And I can't imagine there are that many people who really want to sit around who don't already work for the NCBE or something like that, that want to sit around and write a bunch of these questions, because it's really not a lot of fun.

No. Uh, okay. Well, this was a stressful podcast. So, I think everybody needs to take a deep breath. Everybody needs to take a deep breath and we'll get through this together. It is going to be rocky, guys. We'll figure it out. We will keep sharing news once we hear it. I mean, who knows, but no more vacations for us. We just have to stay online and see what happens. We'll keep talking about it, and I think everyone's just going to have to kind of hang in. It's a little bit like COVID.

for you to get a license. Some people will pass.

Some people will

pass. Some people won't.

We're going to help you try to pass. A lot of other people are going to try and help you pass. Take some deep breaths. It's a good time to create a meditation practice.

Yeah, get your self-care in order. Everybody, thanks for listening. For more, you can check out BarExamToolbox.com, which is full of helpful tips to help you prepare and stay sane, hopefully, as you study for the bar exam. You can also find out about our courses, workshops, and one-on-one tutoring programs to support you as you study for the UBE or California bar exams. We'll put it on our email list too, if you've got any updates, so sign up for that.