Welcome to the Bar Exam Toolbox podcast. Today we're doing a special episode on the mess of the California bar exam. Your Bar Exam Toolbox hosts are Alison Monahan, that's me, and Lee Burgess. We're here to demystify the bar exam experience, so you can study effectively, stay sane, and hopefully pass and move on with your life. Together, we're the co-creators of the Law School Toolbox, the Bar Exam Toolbox, and the career-related website CareerDicta. I also run The Girl's Guide to Law School.
If you enjoy the show, please leave a review on your favorite listening app, and check out our sister podcast, the Law School Toolbox podcast. If you have any questions, don't hesitate to reach out to us. You can reach us via the contact form on BarExamToolbox.com, and we would love to hear from you. With that, let's get started.
Hi, this is Lee from the Law School Toolbox and the Bar Exam Toolbox. I'm here with Alison and we're here to talk about what is happening in California. There really are no words.
Yeah, I think that's accurate. I mean, we all thought this was going to be bad. In case you don't know, if you didn't take the California bar, they just kind of made it up on the fly for the last, I don't know, four to six months. We knew it was going to be bad. If you go back and listen to our original episode on this, we predicted it would be bad, but I really think it was worse than anyone expected.
Yes. If anyone wants the full saga, you can check out our YouTube series on this. We'll link to it in the show notes, where we've been doing updates. But the test did happen and it was bad. And now we are here in the aftermath. So, it is also important for us to know what date we're recording this because yes, there are still unanswered questions here on February 28th, just days after the test. Emails are still getting sent to people who sat for the test and there is still chaos.
The chaos has not stopped. So, we're here for you, but you need to know that this is the date of which we are sharing information, and you need to just own that part, I guess.
Yeah, yeah. I was pretty shocked that on the Friday before the exam, I went and looked at the California Bar website and they had totally changed certain things about how things were going to be operating, what the software was going to let you do. It turned out it couldn't really do that anyway, but hey, they thought it might. You used to launch software, I used to be a programmer and launch software. This is really not how you do this.
Not how you do it. No, this is not how you do it. There are dry runs, there's a process for this. I used a few different processes, and none of them were this. basically.
of them were, "Oh wait, the exam's over and we still don't know how we're going to let certain people take the exam." Because the reality is, we don't have that many students. But we heard some crazy stuff, just from that small pool of students. We had a student who on the essay day taking it remotely was just never able to start the test.
And then they told her that was just too bad, and she could take it in July.
Although now it looks like they've changed that. It's pretty clear they knew this was going to be a total mess, because they had some makeup days already scheduled as the 3rd and the 4th. So that went around for a while. That's not actually what's happening. It's later in March now that they're having people maybe retake - or take - if they didn't take this the first time. Apparently, they were planning on just using the same essay questions. Oh my God!
Which I'm sure are all over Reddit by now.
They're all over the Internet. Within minutes of people leaving, they're talking about them. The idea that you would just use the same question a week or so later who is in charge of this?
Okay, so, for those who have not been following this, let's regroup. So, just to bring everyone up to speed, California decides to go rogue back in August.
To save money. Can we just flag that, because they wanted to save? I think it was around $3 million that they needed to save. I don't know if it's every year, every administration, whatever. That was the stated goal.
Right. So then they say, "You know what would be a great idea, is we should just hire Kaplan, who's a test provider, to write our multiple-choice questions, so we can give a remote test, because the NCBE who writes the MBE will not allow it to be given remotely." Maybe this is why.
And they just then hire this other vendor called Meazure Learning to administer the test and use a software that is called ProctorU, and a whole bunch of people signed up for the test - many more than they thought were going to - because it was remote and you could take it anywhere in the country - actually, within the world.
And they didn't test it particularly well, and they had to open last minute test centers, and everybody went to take the test, and we have a whole lot of stories about what happened. But they are very consistent stories, some of them being, couldn't take the test at all - whoops-a-daisy, functionality just stopped working. We heard stories from students that their machines were just having error messages pop up periodically, that they lost Internet connections.
And these were using the Meazure Learning laptops at a testing center. These weren't even at home on your own laptop. This was using the machines that were given to you in a testing center. There were still technical problems. So there were technical problems at home, there were technical problems at the testing centers. It was insane. An hour to get into the testing software, even to be able to start the test.
Yeah, I think that was one of my favorites, where they said, "Oh, too many people are trying to log in at once." You knew how many people were going to be logging in. Obviously, this was not load tested. Why not?
Yeah. On the multiple-choice day at some testing centers, things went down for like 15 minutes. There were open questions about whether the timekeeping software was keeping accurate time - also disturbing. And now we have all this discussion that maybe they're going to reuse or not reuse questions to give another test in March to some people who weren't able to submit enough finished questions. But we don't even know what those questions will be. Will they be different? Are they vetted?
Or is there someone at the bar writing questions right now, just so they have something to do in the end of March? Not
even the end. Middle.
Oh, middle. Sorry, middle March.
Two weeks. We all might remember a few years ago when they accidentally released the topics in advance, but this, I'm like, "Well, is it going to be on the same topics? Or is it definitely going to be on different topics?" Is it fair if one person takes a Contracts question and one person takes a Civ Pro question? How is that possibly fair? And I loved the email that we saw last night where they said, "Well, if you submitted like four things on the essay day, we're just going to
go ahead and grade it because we can just do that." And I could see that argument for somebody who's a clear pass or someone who's a clear fail. If someone did very well on all of them, they're probably going to pass. If they look at their MBE score, the quasi-MBE score, and it's good - okay, that person probably is going to pass regardless. Same thing if somebody's going to fail. But the problem area is the people in the middle, and this is a lot of the people that we work with.
They might be 30 or 40 points off from passing the first time they take it. Those essays make a big difference. If they do really well on one or really poorly on one, they may pass or fail. There's literally no way that is fair to anyone who is on the border.
No. And the performance test, which can be a strength for some people, because you don't need to know any law to do well. So if you're someone who struggled with memorization, or you're an attorney applicant, because a lot of attorney applicants count on that performance test to boost their score, because you don't have to know any law - that was a piece that had some pretty terrible problems, and if you don't get to have a performance test, that's not going to be good.
That's whole of the test. How can they just be like, "We're just not going to grade that part. It's not really relevant." Then why are you giving it?
Also, just the fact that this was such chaos. We heard from students that took it in testing centers that said that the proctors had no idea how the software worked. They are waving down proctors for help, nobody knows what they're doing. They're like, "We don't know. We've just been brought in", because they're just the random proctor that they found.
When the software went down during the multiple choice, it was down for like 15 to 20 minutes, and people started whispering - as one would do when it looks like the test is combusting - and they're like, "We're in an active testing situation. Nobody can speak." And you're like, " Okay, this definitely feels like a solid testing situation." I mean, the whole thing is just insanity. And they are just hemorrhaging money, they're offering refunds.
They were offering refunds up to the test and they're going to probably give fee waivers for people to take some of the July exam. For those of you who don't know, to sit for the California bar is a lot of money. It's a very expensive bar to sit for. Fee waivers add up.
It's not even just that. They're literally refunding people like non-refundable hotel reservations. I mean, the number I saw thrown around just for that was like $3.1 million. So that tells you we are not saving money here. That's not even counting having to actually schedule, what was it, four large centers where they said they weren't going to have that sort of thing. Not to mention, I'm kind of guessing there may be some lawsuits.
I don't think this is working out well from a financial perspective. No.
I think we found out right before the test that they had to renegotiate with Meazure Learning because they didn't include accommodations testing in their original contract.
And this really raises some questions for me. And I brought this up, I think, in our very first episode of like, "Who is behind all of this? Who is making money on all of this? What tech bro golfed with somebody at the Bar?" It's pretty clear they did not put out RFPs. There was no time. How would you have done that? I obviously don't know that, but I'm guessing that they did not do an RFP process on this. So the question is, how is all this going down? Who's benefiting from it?
Who's benefiting from it? Who did the contract reviews? These contracts were clearly not written well. And who owns this problem? That was one of the things you and I were talking about yesterday when we were talking about this. Is this the Bar's failure? I mean, clearly. But who is the person to stand up and say, "I'm where the buck stops. This was my choice." Is the Supreme Court in California, who technically the Bar
reports to going to just be like, "Oh, no problem." I have to pay my bar dues and I am I'm really waiting as long as possible to give them money, because I'm so mad. But they also hold the key to my kingdom. I have to pay my bar dues, I don't have a choice.
Yeah, I was thinking about this earlier today. I'm like, shouldn't there be a test of minimum competency for running the bar exam? I don't know. I feel like they're just certain, as we would say in our ski training, certain "non-negotiables" that need to happen for you to pass. And for them to pass administering this exam, there need to be things like, people are able to access all of the questions. I don't think that's asking too much. People are able to submit answers to the questions.
People shouldn't wait an hour to start the exam. I vividly remember sitting down for the test in the big testing center, and that 15 minutes where you were waiting, where they hand out all the paperwork and you're sitting with your laptop, and your anxiety is just through the roof. I cannot imagine doing that for an hour or longer.
Or sitting at your house, where they've told you you're supposed to log on and you just can't do it, and that goes on the whole day. Come on, that's not okay.
No. I also heard some reports - of course we haven't seen them - but that these multiple-choice questions were not great.
We haven't even gotten to that, which was one of our main concerns in the beginning.
That was in the beginning, we were like, "What about the multiple choice?" All this other stuff has eclipsed the fact that there are multiple-choice questions that don't seem to be on par with what people were expecting.
Well, and also the fact that they had to send out a redline correction of the ones they submitted for people to practice with, again, a few days before the test. I'm sorry, what? And one of our tutors was using those questions and he wrote us, he's like, "One of these is completely wrong. It is totally incorrect."
If we're finding the errors, that's not a good thing.
No. No, I have heard from various people that the questions were really kind of weird, not good, a lot of the answer choices didn't make sense. I don't think, shockingly, they were really up to par.
Well, because it wasn't vetted. Listen, we create products, we make things, we vet them with other people. You and I create a new something or other. It's usually read by other lawyers on our team to give us feedback. Feedback is how you do work, really. That's how you do it. And there was no opportunity for feedback. You know the other thing we haven't heard a lot about lately? Oh, The Experiment from the fall.
What happened with that? The exam happened. Are people getting extra points? We don't know. Nobody's heard anything.
When are people going to know? How is this going to factor into all of this new grading scales they're doing, some sort of statistics, gymnastics to create something? I don't know. It's just really sad, and that's why I'm just waiting until the very last day to pay my bar of dues, even though they don't care. Whatever, I'm a sucker. I'm going to have to pay them because I want to keep my license.
Yeah. I mean, let's be honest, the whole thing is kind of a cartel on all sides. So, we were talking the other day, really, probably they should just go with the UBE and the NextGen, but then California lawyers are going to be upset because they don't want reciprocity. You don't get the UBE without reciprocity, basically.
Nope. So, I don't know what's going to happen, and the Supreme Court's been silent. Somebody's got to be calling the Supreme Court, right?
Also, the other thing is, what are they doing in July? Are they going to reuse Meazure? Hopefully not, but okay, are they going to pick a new platform? Time's already short. We're already short on time for July, and they were supposed to be having Kaplan write the essay questions by then, too, I think, in the original idea. This is crazy. We don't literally know how people should be preparing for the July exam.
Yeah, I was talking to someone this week who was talking about prep, and I had to be like, "Well, I will have this conversation with you, yet I cannot tell you what the July exam will look like." so.
It's just insanity. It's literally crazy.
It's insane. And what it really all comes down to it, this is supposed to be for the public good. The whole point of these bar exams... I mean, there may be various reasons for the bar, but one of the reasons is to license lawyers of minimum competency, so that you can be pretty sure that somebody who passed the bar is qualified to practice law. I don't know that the public is really talking about this, but it's just not good.
This isn't what the Bar is supposed to do, and the Bar is running out of money, even though we pay obscene fees for bar dues in California. It just feels like the whole thing needs to be taken over by someone else.
Who's competent. Minimally.
Who's competent. Yes, minimally competent. Yeah, and as we referred to, this is not the first time
the California Board of Bar Examiners has screwed things up, but this is really next level. I mean, this is worse than COVID, which to be fair, wasn't entirely on them. The other thing about this is, it's entirely self-inflicted. There was absolutely no reason this needed to happen. They could have just had a normal bar exam. People would have taken it. People would have passed or failed. No one would have been suing them and complaining and whatever. It didn't need to happen.
No, this is all about them getting in a fight with the NCBE, because the NCBE wouldn't let them use their questions remotely. And then the bar decided it needed to give a remote exam to cut costs. And here we are. I don't think any costs were cut.
I do not think they were cut. I think they were elevated. I
think they were elevated. And frankly, there should be lawsuits over this. All the different reliance damages. I mean, my contracts law is a little rusty, but a lot of people spent a lot of money and took time off of work. And law firms are going to lose money because people aren't going to be able to be licensed. Law firms count on people being licensed.
Yeah, people may be losing jobs. A lot of people, if they failed in July, February is their time to pass, and you're going to lose your job in this mess?
Somebody should be responsible for that. I mean, who's going to work at the bar right now? Who's just showing up and being like, "We should write this next email"? I sat on those Q&As and listened to the Q&As, and I know that folks at the Bar were trying. But at some point months ago, somebody should have said, "Hey, maybe we shouldn't do this. Maybe we should just go back to the way it was", or something. But this solution was such a mess.
You and I both worked in software rollouts and development. Sometimes you pushed a rollout because it wasn't ready to go, it was riddled with errors. You did not send anything live when it wasn't ready. Come on.
Well, anyone who says, "Oh well, we just did our best. We couldn't have known" - just go listen to our very first episode. This is not shocking.
No, and honestly, I sat on those Q&As, where thousands of people came to lodge complaints. They 100% knew things were not going well. The mock exams did not go well. The Experiment did not go well. This is no surprise to anyone. It shouldn't be a surprise. Maybe somebody was surprised.
I mean, it shouldn't have been a surprise when you're like, "Oh, we'll just do all this in what, five months, six months?" It was obviously not going to go well. That was not enough time, as we said.
Yeah, I know. So, it's a lot. So, as we mentioned, there is this makeup that was just announced last night, on the 27th. And you're going to get an email offering it to you.
I don't even know that it's really an offer. It seemed to me like if they want you to do it and you don't do it, then you have no shot at passing.
Well, I guess they're only offering it to you if you can't possibly pass. So, I
Well, it seemed to me they're offering to you basically if they don't have enough that they might be able to grade. So, if you either didn't get any of the essays because you were never able to connect, or basically at least two of them were not processed or didn't upload or whatever, then they're going to have you come and do this makeup day, or days. It was two days. But yeah, stay tuned, I guess. Check your email. Yeah, if you think you're one of those people. And people work,
travel. It's spring break for families coming up. People
might have had a bar trip planned. People do bar trips. Does no one remember this? You leave the country for a month after you take the bar. I
did that!
I mean, the idea that they can just be like, "Oh well, just come back in a few weeks"- that is also extremely disrespectful.
Yeah. I guess the question becomes, where do we go from here? What do you do? So, if you're listening to this and you sat for this chaos, you are welcome to vent to us. You can send us an email, you can reply on our contact form. We're collecting the venting. We don't have any power, but we're here to hold space for you, because we're so sorry that this has happened. I think that there's a bit of wait and see.
I think that they did grade the exam for COVID, which was also a bit of a mess, and people passed, and so there are still going to be lawyers that come out of this, that are going to get licensed out of this, in theory. If you don't get licensed, hopefully you explore all your remedies. Maybe there'll be an appeals process. I don't know. I'm encouraging our own students to write everything down that happened to them in the exam room.
I do think that with time and reading everything on Reddit or whatever, your own experience can get a little convoluted. So, if you don't want to write it all down, do a voice memo on your phone. Just capture what actually happened to you, because if there is some sort of appeals process or lawsuits or something, you need an accurate description of what happened. And so, maybe that's something productive you can do, is to capture what happened.
Right. And I think that might also just help people process. You've got it down, you can put that in the box and then start trying to move on, because the reality is, the Bar is probably going to do nothing. We know how this goes.
But yeah, I think people just have to start sort of trying to move past this, which is really hard when there's this much uncertainty and they're sending out emails being like, "Well, we haven't really decided what to do, so we're just going to keep you hanging in for a while."
"Oh, but we're sorry."
"Sorry we're totally incompetent and we expect you to be competent, but hey, whatever." It's just, I find it so infuriating because it was all so easily predicted. Yeah,
so easily predicted. I mean, we started talking about this in August, about what a mess this was going to be. And so, here we are in the end of February and we are looking to July. We're about the same number of months out as when in August we started talking about this. And yet, we're really in the same place, sort of, except now we know how bad it can actually be.
So, if you are still in law school, looking towards this July bar, it's okay to be nervous, but I do think you just have to proceed accordingly and then we'll figure it out as we go. I mean, if you're doing some early studying to practice performance tests, to practice multiple-choice questions, I still think that's a good idea. What you don't want to do is just throw up your hands and be like, "Well, they're not going to be adequately prepared. Then I don't have to be adequately
prepared." Because that's still not going to help you reach your goals.
Yeah. It is kind of terrifying that, as you just said, we're about the same amount of time from July as we were when we first heard about this Experiment. I'll be real curious to see what the next few weeks hold in bar land in California.
I don't know. I still think the Supreme Court's going to get super mad. Maybe they're going to do something, I don't know.
I don't know. I mean, I feel like the easiest option is, you have to administer the NCBE questions until you can show that you are able to do it another way, and you have obviously not shown that.
Yeah, I think they need to revert. That's what I would say. If I was the California Supreme Court - which I'm not - and they are never going to listen to this podcast, but...
You never know.
If I was, I would probably say, go back to the old way. Rent the convention centers, do it the old way, until you can truly test something new.
Yeah. Now I was thinking I could totally see some of this podcast showing up in footnotes of a lawsuit or something. But hey, you have the whole record. Go to YouTube if anybody says it wasn't predictable.
It's just really disappointing and sad, and we really just feel for everybody who's in this mess. You all worked so hard. You worked really hard, and you tried to show up and you tried to do your best and you tried to roll with all the bumps. And it was unconscionable what happened. I really think it was.
No, I feel it's so unacceptable. Bottom line is, it's absolutely unacceptable what
happened. Yeah. We're here for you, you're welcome to vent to us, we will keep you posted, we will come back on the podcast and our YouTube channel with any updates. I think that my last final thought is, if you took that exam, check your email very carefully and make sure your bar emails are not going to spam, because who knows what they're going to send out next?
Who knows? Literally, who knows? The whole thing is still so crazy. I mean, they're still literally, I think, at this moment in time, as we record this on a Friday afternoon, still administering some of the tests to people who have accommodations.
Oh my gosh.
Maybe they'll have some meetings over the weekend. I don't know, maybe not.
I'm guessing they don't work on the weekends.
I think normally, but I'm also saying maybe this could be an emergency type situation where they need to call everybody in. But we can't clear our schedule for that. I don't know, does everyone else clear their schedule for two weeks from now?
Yeah, I know. Anyway, keep us posted, we'll keep you posted. We're thinking about you. Hang in there. And we'll update everyone as we figure out what comes next.
Good luck, everybody. And tune into our YouTube if you want the latest updates, because that's probably where they'll be. Good luck!
Good luck!
With that, we are out of time. I want to take a second to remind you to check out our blog at BarExamToolbox.com, which is full of helpful tips to help you prepare and stay sane, hopefully, as you study for the bar exam. You can also find information on our website about our courses, tools, and one-on-one tutoring programs to support you as you study for the UBE or California bar exam.
If you enjoyed this episode of the Bar Exam Toolbox podcast, please take a second to leave a review and rating on your favorite listening app. We'd really appreciate it. And be sure to subscribe so you don't miss anything. If you're still in law school, you might like to check out our popular Law School Toolbox podcast as well. If you have any questions or comments, please don't hesitate to reach out to Lee or Alison at lee@barexamtoolbox.com or alison@barexamtoolbox.com.
Or you can always contact us via our website contact form at BarExamToolbox.com. Thanks for listening, and we'll talk soon!