Special Episode: PowerSchool CISO, Mishka McCowan

Mark has done a lot of work on this over the last couple months. We're not sure what he sold or what he promised to anyone to get this. But we have some representatives from PowerSchool. We have Mishka, this is CISO from PowerSchool here with us today to talk about the last several months and the incidents that they have incident that they have gone through over that period of time.

So, Mishka, thank you. thank you for joining us. Chris and Mark and I are very thankful for the time that you're going to dedicate to us this morning and that you actually were willing to come on.

And you start to respond and you start to work the problem. So the first thing was, like you guys have, we've got playbooks for when something bad happens. So you get the instant response team spun up. You get them working the problem. We've got CrowdStrike on retainers. So you reached out to them and say, we got something going on here. We're going to need your help. Just for listeners who maybe haven't been through a security incident before,

the first thing is always, is this real? And you get your fingers crossed saying that it's a mistake. It's not real. It's a hoax. It's whatever. But you go through and you validate that. And this wasn't. So we figured that out pretty quickly. And from there, it goes on to containment. You try to figure out, hey, we got a bad thing happening. We got an intruder in the system. Where are they getting in and how do we stop it? You go through those containment

steps. And that was actually, we got to that very rapidly and were able to pinpoint, okay, this is how he's getting in and this is how we can keep him out. The hard work came after that with, okay, what happened? What's the scope, right? Because there were a lot of servers involved and a lot of logs and a lot of

stuff to go through. Yeah. We, you know, the, the, the normal instant response team is usually pretty lean and you've got some, you've got security experts, you've got subject matter experts, and that's usually it. But in this case, I had a ton of people working on this and my security team, as well as, you know, engineers to say, okay, we've got 15 terabytes of data to go through. We need to, we need to write some scripts. We need to do some data analysis.

And, and literally we, we spent days, you know, just running the, you know, write a script, run it, look at the results. Do some validation, make some changes and repeat until you get to the point where you're like, okay, we have a good handle on what happened now. So from a high level, that was kind of the first Week, 10 days of just going 15 hours a day, all hands on deck, just trying to figure out what happened to some level of confidence until we could communicate it out.

So it was Dr. Pepper and energy bars, man.

You're going to tap that teacher that's volunteered to help you before on different things. So it's interesting to know that a very large corporation, similar incident, you're now commandeering people that normally play different roles on to an incident response or security team.

But you're pulling all these people in, just tagging them in and saying, okay, I need this. I need you to take a look at it and go do this and report back. So, yeah, it's not that different. I mean, yeah, we don't have too many librarians running around. But I absolutely get that for a small district or even a medium district, it would be really hard to do that if you don't have... That incident was, you know, at least one person that's kind of been through an incident and has the training.

So we, we kind of, we kicked around a lot of ideas internally. And at the end of the, and at the end, we kind of said, you know, we, we got to do a webinar. We have to get people in the room where they can give us some feedback. They can, and we can give, we can talk to them at scale, right? Because there were a lot of people involved. So that's why we, with the help of our internal people, they got me kind of

prepped to go out and talk to humans since security dies. They don't normally let us out that often. But in this case, we thought, okay, they need to hear from our CEO, Hardeep. And they need to hear from the security side because this is a big deal. So we kind of compiled all, we got all the information we had at the time because you never have like 100% confidence in everything that you have that early in the investigation. 10 days is really fast for something of this size.

So we did, I think, four webinars. We cycled through about 9,000, 10,000 people in total.

Um for anybody who was on the first one i have to apologize it was it was kind of i was i was kind of rough on that one because um i'm trying to answer questions and it and they're coming so fast it was like trying to read the matrix they're just zooming by and i'm trying to be mindful and catch ones that are broadly applicable because some people are putting in things they're really good questions but they're really specific to that district you know

my locality my district You know, my setup for my sis, et cetera. So you're trying to go through and find questions that are, you know, broadly applicable, right? Like, you know, what is things that everybody on can get something out of. But at the same time, you're looking, watching these and thinking, wow, the next part of this is going to be interesting because I have to go talk to all these people next and address their questions. Yeah.

So it may have felt a little uncomfortable at the time, but it was almost a relief to hear a little bit more information, knowing full well we're going to have to wait and see what the rest of the timeline looks like. Can you describe a little bit about, you know, what was your thinking? You've talked a little bit about the webinar. Now you've got to pivot towards this individual school support. I mean, what was that like? And, you know, what is your kind of communication

practices there? Because I would also imagine, too, you've got a massive organization. You've got to start to make sure that they have the answers for all those questions.

So we spent a lot of time internally saying, okay, these are the talking points. If it doesn't fit in one of these, escalate it up. And what happened is me and the ELT, our leadership, everybody below them kind of cleared their calendars. And what would happen is if we just, you know, completely clear, just start scheduling things on my calendar and on, let's say, Paul Brooke, our chief customer officer, and Hardeep, our CEO.

And we just started doing these meetings, you know, meeting after meeting after meeting. And you would try to find things where you had maybe groups of people, like a user group was a good place to get multiple people on one meeting so you could answer a bunch of questions. Because a lot of times it would overlap because it'd be in the same sort of area. Yeah. But you just do these and it would start about 8 a.m. Eastern time and it would go till... It's usually 5 p.m. West Coast time, right?

So it's long days where you're just kind of sitting here going through these and saying, okay, what's up next? Trying to get some sense of what are their concerns? Is it something for me? Is it something for legal? Is it privacy related? And you just cycle through these meetings and you try to answer them as best as you can. But the problem is you don't necessarily always have the information. A lot of times what they want is what comes next.

And that's the hard part. Like, oh, you've done the notification. Let's talk about credit monitoring. Let's talk about how you're going to notify the individuals. Let's talk about the notices that go to regulators. And a lot of times you just have to say, that's coming. And that's incredibly frustrating for the person on the other side of the table who's receiving that message.

Like you don't uh but and i understood that and i always tried to be as transparent as i could about it and and just talk to them like i would want to be spoken to if i were in their in their seat right yeah yeah uh so yeah i mean that that was it just trying to you cycle through these all day every day for a few weeks well

Or is that just nature of the beast? Was that intentional? Does that make sense?

And it wouldn't give us any other information. That was it. We had one letter we could give to, you know, so I'm going to HR and saying, okay, we need to contact these people who are internal. And they had, you know, because it's some of the, it was them. Sometimes it was their kids information. And they had tons of questions, right? And I had nothing. And the insurance company was giving us no more information, right?

So I've been on... And that's one example amongst many where you're kind of on the other side of the table, either professionally or personally, where you get a letter in the mail and it's one of these things. So I tried to put myself in that saying, okay, I'm on the other side of the table now giving information. How can I be as transparent as I can to give them as much information as they can, because that's what I would want if I were sitting there.

you just talking to you in the last 10 minutes makes me realize that I think a lot of that transparency and that empathy was driven probably by you and likely Hardeep as well in your senior leadership team. I don't think you see that very often, especially from technical people.

Someone had made this great document to start looking up logs, to start doing investigations for the local folks to start getting ahead to do their discoveries. Did you know that that was happening, that that document was out there, that techs were doing their own research and kind of coming up with their own findings?

Because while you want to be transparent, you also want to be very careful about what The veracity, you know, how confident are you in what you're saying, right? Because if I tell you something and you go talk to your parents and students and I walk it back, you now have to walk it back. So all of the, it's not just us having to walk something back. It's you. And that puts more stress on you. Your parents are in, your parents are going to believe you less. You're going to believe us less.

So it's a very fine line. We have to walk in making sure that, okay, let's be very deliberate about what we're saying. Not because we're trying to hide something, but because if we have to walk something back, it has massive repercussions, not just for us, but for our customers. So, yeah, so to answer your question, yeah, we absolutely saw that. I love the fact that people helped each other. There was a lot of good information out there. And that was helpful. So I commend the folks who did that.

Thinking through your role you you know we we often as k-12 techs if we're working at the district level we know that the teachers are going to the principals are going to throw their data wherever they want they're going to you know they're not going to necessarily think about data security you guys are managing a platform which is a platform where schools and districts can do whatever they want and put in whatever information

they want but then you have to then deal with the repercussions of that there's a way to do this and there's a way to support our customers and what they want to do and how they want to use PowerSchool. But at the same time, how do we all do this safely?

Right. So part of it's just awareness on our part, just talking and having conversations with customers about this stuff because they just might not be aware of it. And part of it is, you know, we're looking at this as itself to say, okay, can we make it easier to kind of see and delete what's there? You know, that's something we're actively working on.

But at the same time, again, fine lines you have to walk because different localities will have different regulations on what to keep and for how long. So, you know, we don't want to overstep and say, yo, you should delete all this stuff because they may not be able to delete it at all. But, you know, we want to engage in the conversation and have, you know, share whatever learnings we have to share and listen to other people's learnings because they're a great source.

You know, our customers are a great source of information on maybe what we should put in the system or what we should talk to others about.

You brought up some examples of how PowerSchool had to trust and do the repercussions of third-party providers.

You know, where do we as school districts go from here? What are the questions that we should be asking our partners and our providers?

How can you evaluate vendors? Because it can be a little daunting. Because if you go to finance and say, okay, who are all the vendors we're paying? You probably come back with a huge list. We do this internally at PowerSchool. We came back with this enormous freaking list of vendors. So my advice would be this. Start by triaging. Try to winnow out. Maybe look at the ones that you don't have to evaluate.

The people who re-stripe your parking lot. Probably not someone who's going to keep a lot of data for you. You don't need to look at. So try to get those, push those away and say, okay, we don't need to evaluate those. Then look at your technology vendors and put them into two buckets of, hey, they hold sensitive data for us or they don't. In the don't bucket, it's probably things like, hey, we've got a subscription to this site where we pull down stock photos for our newsletter.

Not a huge risk, right? Not something you need to go through. Um so what you're left with is kind of that a smaller list of core vendors that yeah we need we should probably talk to these folks to make sure that they know what they're doing and um there are a couple different avenues for doing that and i you guys are pros i mean you you know this uh so i'm just so just sharing it for your listeners what i would do is i would start with two things.

I would ask for a SOC 2 because if you're not familiar with the SOC 2, they can be really long, gnarly reports. They're not fun reading. And I don't advise anybody to sit down and read them cover to cover. You won't get that much out of it. But if you flip to the back, the auditors will always put any findings, anything that's out of true at the back of it. In the last few pages, there's a section say findings, and it might be nothing. It might be few there.

Look at that. Look and see if it's something that might concern you. The other thing is if they don't have that or if you want a little bit more certainty, you can ask for something called a HECFAT. Apologies, I don't remember what the acronym stands for, but it's a questionnaire that was developed specifically for K-12. It's pretty standard.

It has a lot of standard questions on it. I believe the last time I looked at one, it has a You send it to the vendor, they fill it out, and then it has this automatic scoring mechanism that will tell you that you can look at like a chart and say, okay, they're strong here, they're weak here. And you can kind of make a decision to say, do I want to do business with this vendor if it's all really low?

Or if there's a specific area you want to ask them about, like where they've scored low, you can do that. And it doesn't take a ton of expertise to do that. And it kind of puts the onus on the vendor for a lot of this stuff. We get requests for this stuff all the time.

I'm curious to know from your side, what were the kind of the things that you recommended?

Um d dpas which you guys had signed dpas so you know it i guess a lot of the worry and a lot of the frustration from from districts comes in that when you have a vendor and and we can remove power school from that from this statement when when you have a vendor doesn't matter who the vendor is it could be my my filter vendor if if they're socked to if they if they're willing to sign a dpa that says they're doing all of these things. They're meeting these standards.

They've done the survey and they scored grade A choice through the survey. And it still happens. Something bad still happens. And then everybody's got egg on their face. That discussion naturally evolves to, well, what else could have been done? And I don't know what that answer is. And I think that's where a large part of the frustration is coming in.

So supply chain is not a solved problem by any means. And I don't have, I don't have a ready pat answer for the best way to handle supply chain, that is applicable to every institution from your LAUSD to your smallest district that's servicing just a handful of students. It's hard. It's really hard. And part of it is when we do, we have to evaluate our vendors too. I mean, I literally have three people that's a big part of their job in doing this.

And mostly I rotate them through because it's really hard and they need focus and they need breaks. To do that, it's something I struggle with with my vendors with as well. Because how do you get certainty on this from the outside? Because I'm not able to pen test these vendors. I can't hire someone to go in and pen test their environments. We're kind of reliant to some extent on trust. And part of the trust is built on, okay, there's some visible pieces of it that you can look at.

There's some third-party attestations that you can look at. And then part of it's just the, you know, what is your interaction with them like? I mean, yeah, we got, you know, to use the Mike Tyson quote, everybody has a plan until they get punched in the face. We got punched in the face hard. And we, you know, how do we respond to it? And how are, you know, how do we respond kind of with, you know, we've talked about the webinar and other stuff, but how do we respond going forward?

What else are we putting in place? Are we going to be transparent with the, you know, are we going to learn from this and put other things in place? Short answer to that's yes. And we can go more into that if you want. But that's kind of what you have to do is you have to engage on multiple fronts with your vendors and then do your diligence.

trust and to understand the level of security. But at the end of the day, we need to govern ourselves. And, you know, I dealt with that within my my little district saying, well, we had some data in there that we did. We did not purge and we should have. And that's the kind of stuff that I think internally the community can start to surface from this is that we can be better at data minimization, at really thinking through.

I like your analogy of the guy who's striping the lines in the parking lot. Although I will say the lawyers will take, you know, vehicle insurance way more seriously than data insurance. But, you know, we can we can categorize the different vendors that we work with and then we can minimize the amount of data that we store and collect and transmit to vendors and do it in a safer way. So that's something that we as a community have talked about internally.

And it helps that you're able to come to the conversation or come to the table and have this conversation with us. So thank you for that.

And so just to get to that system now for employees, it's got to, you have to be on the VPN, which means that you have to be using, and to get on the VPN, you have to be using a PowerSchool laptop or VDI, a virtual desktop. Then once you're, and then to authenticate to both those things, you need, you need corporate SSO and MFA. Then once you get into, into, you know, then you, that's just to get to it,

to log in with your credentials. Then once you go to connect to a Sys, there's other things that we put in place. Probably the largest one is on the Sys itself. Admins have always had the ability to turn things off, to turn off the maintenance. It's called maintenance access, and they could turn it off. And one of the first things we did is we just like for all of our hosts, we just turned it off.

And then for the release that happened in January for the sys, what we did is we turned off the ability to leave it on in perpetuity. So what we did is instead of just being a binary on or off, it's now time-based. You can put it on, and as soon as an admin enables it, the clock starts to tick down from whatever they set it to. It could be one day, it could be up to 30 days, and then when it reaches zero, it turns off automatically. No one has to remember to do it. It just gets turned off.

And what that does is it lowers the potential blast rate. So we put in all these layers in front of it to make sure that no one can get in. But once you're in, you have to plan for someone getting in. And you have to, you know, if they get in, you have to contain the blast radius. And what this does is it means that for most of the sys at any given moment, that access is off. So you can't get in. Now, that has some interesting side effects like for our support department.

So for us to get in at any given moment to help someone or do a services engagement, we have to call and say, hey, we got this ticket here from you. Can you let us in? And they turn it on and we go in and it gets automatically turned off. So it was a really different model. Now, I think some of that I talked about in the webinar, and that was fairly soon afterwards. But what happens after that is you take a step back and say, okay, let's look across the environment with a fresh eye, right?

Because you don't want to make an assumption that says, yeah, that thing over there, we know that's secure. No. Again, to use the Mike Tyson thing, we got punched in the face. So let's not assume that there's not a right hook coming from that system over there. So what we've done and continue to do is to go to all these systems and say, and to reevaluate with fresh eyes and say, okay, is it really,

you know, what can we do to make this better, right? What can we layer on top of it to make it better? Because there, you know, we've got a lot of, you know, Sys isn't the only product we have. We have a lot of different products. And we, you know, there are systems like this to enable support to get into those products. So we focused like a laser on those to make sure we had those same sort of controls in place for those. You've got your VPN, which requires the laptop and the SSO and all that.

Make sure that we have these same blast radius things in place with those kinds of systems. And then stepping back further, you start to look at your infrastructure environment. We have very large footprints in AWS and Azure. At any given moment... During the day, when auto-scales all the way up, we'll have between 10,000 and 11,000 servers running at any given moment. That's a lot, right? And when you spread it across however many accounts in AWS and Azure, that's a lot of territory to cover.

So we step back and said, okay, what do we need to do to make sure that nothing like this happens on the infrastructure side? Because this was about credentials for this. but we didn't want to get hyper-focused on just identity and credentials. We wanted to make sure that, okay, let's look at infrastructure. Let's look at kind of our internal processes and procedures internally. Let's look at these other things, always with the question of how do you make it better?

So every meeting that my team goes in, the RISC team that goes in to have this, it's not just evaluating what's there. It's the, okay, what else do we need? What else could we put in? How could we make this better? So really long answer there. And I tried to be, you know, give you some specifics as well as the general, but that's what we're doing is kind of going house by house, you know, system by system, network by network and saying, how can we make this better?

And just implementing, implementing, implementing. it.

And, uh, you know, for, for the folks that are listening here that have invested heavily in PowerSchool, what can you say to them?

It's your data you should have a voice in protecting it so we want to have that forum where you know someone you know a smarter security guy than me can stand up and say have you thought about this nope but that's a good idea well let's take that back you know we want that kind of feedback you know we we don't want to assume that we've got this we want to engage we want to be we want everybody to be part of the solution on this stuff so that that's a

big part of what we're doing in addition to kind of what I described, kind of the internal facing, how can we make it better stuff, being transparent with that and getting feedback from our customers.