Live from the somethingcool.com Studios this is the K-12 Tech Talk podcast it's a very special episode I did not do my homework I don't know what episode number this is gonna be because I will tell you this is a very special episode tonight uh I broke out some of my own bourbon from my own Bourbons stash, not Jeremy sending me. I have poured myself a glass of bourbon tonight I don't know Chris what are you drinking tonight.
I went to Sonic and I got myself a Nerds Slush with strawberry and lemonade Mark uh I don't know what you are
drinking. I'm not gonna uh say what you are drinking but it looks like you're drinking some some form of uh hydration right yeah it's a it's a big night we we bust out the big stuff tonight because we have a really cool guest tonight 108. 108 so this will be 108 I am drinking blantons out of my own uh private reserve because this is such a special episode tonight Mark why don't you do the honors since this is a friend of yours uh and you did you did the legwork to uh twist his arm to come on
here with us uh you do the honors so tonight we have uh Soheil Katal who is the current CIO of the Los Angeles Unified School District in California suhail uh I've known suhail for a few years and the over the last few months of Hale's gone through uh the one thing we all don't want to go through and that is a cyber incident uh and so he's here tonight to talk about that just as a full disclosure there are certain things that we're going to talk about tonight
there are certain things that we're not going to talk about because this is an open interview with suhail and we know the district and we know who we're talking to so we don't want to do anything that creates any undue risk for the district so we're going to talk at a high level of what happened at LAUSD in the response that the team took uh and we are going to keep certain things confidential just so we don't create any more future problems for suhail so let
me turn it over to suhail right now to introduce yourself tell us about how you got to LAUSD uh and and how you got into your position all right thank you Mark um although I'm at work so I'm not gonna have any dreams so with that uh I'm sorry cancel uh Chief Information officer uh for Los Angeles Unified School District um that's very interesting question how I got into the CIO in LAUSD uh because as they call me I'm a call with CIO ccio that's what I call myself there you go
that's a new title yeah yes uh actually I got to the cereal position in LAUSD uh a year before the pandemic it but where it was mostly in the transitional work getting that you know you always have your 90 days plan hundred days plan all of those and at the moment that you're working to implement those plans uh it's very difficult to um to focus and get your hands around the organization as a whole so although I was with the organization before I visited at the role of the Deputy CIO I
was able to have a better understanding and uh on the infrastructure side my background was mostly in the private sector when I joined LAUSD about six years ago when I joined LAUSD and then that helped me to through that transition to get the better understanding of the infrastructure um post uh when the pandemic hit obviously I started focusing most of my effort like Mark and the rest of my colleagues across the country to become mostly the focus on device distribution
connectivity and all basic need of I.T rather than I would say the real role of the I.T as a Chief Information officer to provide more intelligent and Equitable access to the data it was mostly about the critical access to the technology hardware and systems for the student community and then uh obviously eventually uh we when we came out of the Cyber when we came out of the call with uh mostly hybrid learning I'll focus on bringing instruction back in the hybrid environment and eventually go
back 14 class learning that's interesting you think is it turning the lights on it wasn't it was mostly coming back with the new brand new infrastructure sorry I think my choir department is that if you need to evacuate that's fine we got you we got you that's La that's what you're doing everything it's better to be fire department than something else sure so yeah uh so overall that was the my experience with uh getting to LAUSD and working uh during the pandemic and post pandemic as
you indicated we started with this uh unfortunate incident and uh I'm going through the process of recovery and everything else so real quick uh for those that don't know I mean if you're in North America you know Los Angeles you know La you know the LA Unified School District but for those that may not know they're listening somewhere overseas Australia Canada wherever else give us an idea of your student population uh how big your IT staff is um kind of give us give us that idea
so it is very interesting when uh when I came with the to the LA USD from private sector I was I used to work for United Technology and United Technology was a conglomerate Corporation although it doesn't exist anymore through my surprise Raytheon bought the company and now it's part of radio we had 250 000 employees when I was in the private sector art of the UTC and when I was looking at my active directory and domain control system I used to see these 250 000 as a massive number sure
it was massive at the time and the company with 150 billion dollar revenue and 250 000 users when I came to LAUSD the first thing that I said wow and my jaw dropped I saw one million user inactive wow so that tells you a lot that tells you wow they don't see education the way that they should see because they consider our budget or everything else but reality is the infrastructure in education especially K-12 again when we say education everybody talking about
higher it and everything else in J12 is very different than anything else you can imagine why not in La USD would like to I don't know maybe it's a proud thing be proud to practice with break capacity of any infrastructure component that you can imagine to get them to the limit for the vendors manufacturers to enhance their product but yes we have about one but right now at this point we have 1.5 million users in an active directory system in our identity systems including
active directory it's about uh 600 000 uh 500 000 student K-12 about 50 000 Early Educational students about 150 000 adopted students and we have about uh hundred ten hundred thousand hundred ten thousand uh employee and contractor plus we have about half a million uh sorry 700 000 parent account in our system we create account an identity for parents to access the system so when you add all of this that tells you the size as far as the population one metrics I always like to talk about
is the uh our infrastructure so that we have 1300 the schools across the city um we run fiber optic and networks for this uh 1300 School the last number I remember we have 29 000 miles of cabling across our school if that's your one time around the earth plus from here to New York so I can come up see Mark if I want with that it's like this guy's bragging right now I don't know well I I think some of these numbers are going to come back to bite him later on
in this description so I love it I love it right before the fall yes yes uh but it's very uh impressive uh one key in the metrics that probably relates to Cyber attack and we should be maybe I should bring it up over here we have 16 um petabyte of the storage uh data and uh on top of it we receive 35 million um at the essentially I would say threat or attack every year wait say that number again the attack number get 35 million with Detective so you're detecting what what are you
defining as an attack obviously DDOS fishing any type of detection that IDs and ideas okay not this so hey I'm not talking about incident incident hopefully by incident never got a bet so yeah right yeah wow okay okay so so let's pivot here because you just had some very impressive numbers um but the one of the reasons we're really excited to have you on is because you've gone through our worst nightmare you've gone through an incident so if if you could take us through
um I guess you could call it D-Day what what happened when you first got the call and you're alerted to something's wrong no we call it neither you're correct but uh you haven't coined that date with the minute and the time of the incident uh and literally coined it and I will probably gonna send you one um so it was a very interesting time you know that usually most of the attack happens in the long weekends and this was no exception uh it was a Labor Day weekend
uh the the and that usually those are the time you uh essentially wind down your staff people taking time off mostly uh so yeah we have a 24 7 uh Network Operation Center um and then uh about I would say eight nine PM 9 p.m that you noticed um unusual activity in one of the uh one of the separate systems from RIT infrastructure it was a facilities infrastructure and uh no surprise sometimes you see an organization there are some autonomy and some type of the Federation across I.T
department and not everybody have a centralized it show and it was the same scenario um there was a separation but we still saw the activity in the movement and we were notified and um that was a very hectic uh uh thing that you know uh you're dealing with when you see um essentially let's say bad actor trying to move laterally when you see bad actor May uh try to run an encryption on your system and everything like that so obviously when we saw those activities uh we it was isolated to a
separate system um and the team uh tried the escalation process an escalation processor was to work really rapidly and fast it was actually that was one of the successful part of this incident that helped us to mitigate the situation to the extent that was possible and uh the escalation got to the data center director Data Center Director to um uh to the uh we call it shared Technical Services the sheriff Technical Services ultimately escalated to me so from 8 30 to I would say 9 30 10 I got
notified about incidents about an hour and a half as the team were working on the incident to identify the impact by the time that I got it uh I was literally driving back to the boji which is our uh essentially Command Center our uh one of our datas every time multiple data centers but that was one of our data center I was driving over there to meet with my team and we were recruiting one thing you want to make sure when you are in the Cyber incident relying on the
virtual connection and platform that you were relying in the past probably is not the best option uh you need to have a proximity access to the infrastructure and at the same time you need to have other bad communication system to keep those communication active so we activated the other band communication and at the same time uh we were trying to regroup to be able to make faster decision without being interrupted by communication uh nightmare that everybody always deal
with so uh I got to the uh to the to our data center to baudry and we started uh grouping And discussing whether the movement that this as they were happening so when you see those movement uh the first thing you need to protect is your crown jewel uh and the Crown Jewel for us was essentially our it infrastructure versus facilities infrastructure so what we did we activated our firewall system uh to isolate and quarantine facility system to access to anything to the rest
of the it first structure the second thing that we did uh which um remembered when bad actor getting to your system uh they are in your system it's not like they are uh and you're activating firewall rule is not going to prevent them from getting in because they're already that's why you need to come up with the immediate action to stop them from spreading beyond what they are and that's what we eat we made a rational decision at the moment uh to completely cut down our access for
all of our assistant and I'm not talking about firewall I'm talking about Network system access from the from both Data Center and through the internet when we stop that activity obviously the bad actor lost anyway even through the artifact even if they had compromised out of and they wouldn't be able to get it that's what I'm trying to say we'll probably start well when they a bad actor in other scenarios like uber case they uh they break into your uh emergency systems and
everything else you don't know what's happening the first action is mitigate immediately so you understand the impact and that was our action we stopped the network activity we saw internet access and everything then we went through the uh through the routine process that you usually do for any cyber incident essentially going through the detection respond eradicate and pull nine yards of the cyber security practices it's funny to hear you say that that one of the first
things you guys did focus on was was removing access like that because the mark and Chris will tell you that I I have a running joke that I've told my guys if something like this happens and we get those calls of hey you know my machines encrypted or we're seeing this weird activity we've got a pair of scissors next to our fiber that goes to the internet and that that line is getting cut um so it's funny to hear you say that because that's in my head that's the
very first thing that that we're gonna do um yeah that's that's just it's not a joke like you're super serious about it oh yeah absolutely like there really are scissors right yes yes there are in the data center yes so so so this is what at what time of the day is this this was a long weekend it was the Friday night before Labor Day weekend at what time are you cutting off the entire system to be exact at 12 52 uh of the midnight when we passed the midnight 12 52 we cut
off the internet and actually that's the time because Bitcoin uh in our uh response stick going because I know but it literally we tracked it through the log you know there was involvement on law enforcement they were working with us fbic saw and uh DHS they are all working with us um including local law enforcement agencies but the main focus was tracking the logs and activity to see uh inactive in the adult around the data exploration and anything else and that time with everything is solved
then one thing was very unique about this uh obviously our superintendent uh it was never and we still never want to name the bad actor and if you're thinking of their name out because we don't want to give them value we don't want to give them credit we don't want their name to be important because they are criminals no matter what and uh but reality is uh they admitted uh that they fail uh in uh essentially bridging us remember there are two sections when you do the
um essentially this type of activity the step one you want to encrypt and [ __ ] the organization and accept to do the data exploration the step one obviously they failed their encryption was not successful to the extent that they were expecting us to prevola and step two it was a data exfiltration which the total data that they uh Express rated from our system that it was published is about 400 gigabyte and you know that that size of your laptop hard drive compared to 16
petabyte of storage of data that the district had in public information from facilities contract and there were unfortunately some of the other sensitive information which was very limited and the impact was reasonable so you you said you cut everything off at 12 52 a.m and and you mentioned that law enforcement was involved FBI says uh were were those folks involved that early in the process when you guys made that decision if you can say that we're like were there folks on site already or
you guys had already made contact with your kind with with response folks that you knew with with the agencies uh were they there that quick you know they are they were not they were on your own when you are in a Cyber attack uh it doesn't matter what what level it is it's a DDOT side of anything we are the first responder we are on our own until we do uh essentially we do the first response when it comes to the detection part that's where the uh other resources comes handy your cyber
Insurance resources the federal resources or contractor your vendor that's why really really in detection form it is was it was us and when I'm saying us when you worked in the middle of the night how many people you can pull in probably five six people right right response incident response state that they are protecting the district and getting that key decision being made in that early minute is very crucial to do the rest of the activity so the incident command and everything else so so two
calls you had to make that night that I'm curious what it was like versus your family basically to say I'm not coming home for quite a long time and the second is probably the superintendent so what were those calls like or can you describe a little bit about you know how you had to notify those around you around what happened uh first of all uh probably you tell my wife she's not gonna like this but I didn't call her I texted her I said I'm not stuck don't wait for it
that's all about me so uh but definitely I called the superintendent literally on the phone and uh a superintendent immediate to respond that we've uh I shared the situation with him um our superintendent coming from the background of knowing the uh criticality of the cyber security based on his past experience in Naomi and every other experience that he had in his life so he knows how crucial is this early moment early minutes uh and it wasn't surprised that they called it because when the day
that he arrived in Los Angeles for the first time when he accepted the job the first thing that he asked from me was a briefing about the uh cyber security infrastructure of the district and you received the first briefing one of the first briefing in the first week was Barney wow so he was a tank dip he was very aware and wanted to be involved in it and he still is therefore I didn't have him up again I had Direct access to his personal cell phone and I was able to reach him
directly and escalate the situation um definitely he that's comes the trust trusting me to leave it up to me make the decision that he makes the communication and even engaging the long the federal and everything else it was my call it wasn't anybody else's call that's what goes back how much you can build that relationship with your uh CEO with your president with your superintendent not to become like 50 people's decision when you need to make a decision during the
incident but in one of the lessons learned from this that actually was part of my uh uh when I had when I was in the panel and talking with the FBI team it was exactly that you needed real Incident Commander that can make a decision on site when everything is not working the way that it should so somebody need to make that calls and the decision and I was the one making the decision obviously I wish at the time I add the ciso and my season would have made up I did an app so I made that book but
yeah um which actually that's well later we're probably gonna talk about it that was course correction that we got but uh definitely superintendent was uh getting engaged really fast and the recommendation was to engage uh federal agencies and other resources that they can help us mitigate this situation so Ed talk about those first 48 hours or three days what what was the that like the level of obviously anxiety is going to be super high um you know just just that that
environment being in the first 48 hours I assume within the first 48 Hours we've all read stories about you know the the FBI is coming on site I even read a report at one point that that someone picked up the phone and called the White House I would love to be able to say that I picked up the phone and called the White House um so talk talk kind of about those first 48 hours and what that um response was like once people started showing up to help how that went
so it was really fast so one thing you well you should always pay attention just me picking up telling FBI hey I have a security incident doesn't mean FBI gonna show up at your door definitely there are gonna be some conversations some due diligence some escalation and that necessate your presence so because um the team in FBI that they're engaged in the Cyber response is a very small team but they are amazingly I would say I don't know I don't know how to describe it but probably you've seen
them probably in some Sci-Fi movies rather than from you may feel I'm exaggerating but reality is that this was amazingly good the technologies that they were using they were beyond the technologies that our organization like us or even any other private sector could claim that they can use and they are and beyond that they are very skill and when they know what they are doing they are uh fast in detecting and remember these are they have case expert that knows the bad actor they bring that
uh therefore they can study it where to look where to identify the footprint versus you if you do all of those without the team experience like that it's like you're shooting in the dark sure you don't know where you're gonna get or how I'm gonna find the information so they were very instrumental to involve but uh yes uh the contact with the FBI made actually by our superintendent he made we made the call up to the to the Department of Education to the White House and the
federal defense were ever the earliest stage of the work and due to the severity of the issue that was one the size of the LA USD was important for them not to [ __ ] the organizational art class but I think one more thing that uh helped to bring this FBI faster than any other response was the fact that we stopped the attack mid attack and we communicated that with them so there are two things so obviously if you stop it why would they come in so you already stopped it the reality is
um we were able to preserve very instrumental uh and promise the first time I'm publicly saying that very instrumental information for uh for federal government investigation against this bad actor wow two decks I cannot give you more detail what but a lot imagine when you are running uh encryption system an encryption crashes what would you get from that encryption would you some very uh crucial evidence and even some uh some type of reverse engine and everything else interesting in a sense
it was very instrumental for FBI to be involved we were helping them they were helping us so it was both way Mutual conversation and that's why typical with an FBI engage or any other law enforcement engaging investigation there's they have a short they're flowing they come in like two three and they do it two three days daily uh believe it or not they were cycling through agents uh because they were getting exhausted day and night but by the way it was 24 hours work it wasn't
like oh it's 5 PM let's go home it was 20 cars away there were really different shifts people were working uh we had our own partner Microsoft also was a good partner in this part we engaged the special unit inside Microsoft that they're responding to Cyber attack um and what I learned for the first time about this team I didn't know that they exist and really they were working with any agencies including government agencies to help them during Cyber attack they call them Dart
and that dark team was also very instrumental to work with FBI and in hand during the investigation so you just mentioned that people were there 24 hours like you know like like you said nobody's quitting at five o'clock or six o'clock when going having pizza and beer um what was your time on site like what are you you're sleeping in your office if you're sleeping at all what what was that like uh not just me I'm my deputies and others we were all here um live yes maybe I would take like one
hour towards just not in my office we were just you know whenever you have a high priority project what would you do or their pizza for everybody sure it's a party for three or four days and nights I'd be eight enough Pizza that I ate pizza I was taking nap in the office getting back to work uh one night I decided after three nights that I couldn't get enough of sleep uh and probably if you search uh the incident and see my picture I go after it three days we had
our first press conference and uh and when I went to that press conference um the reporter I think it was uh one of these news agencies that took my picture put it in the media they took my picture from the below my chin and I'm looking at this guy you should Google it and find it that's very funny picture and literally it's like I'm praying God just help me let me get some sleep I mean all they do was that I needed rest so I I was driving back home to get a little
bit of sleep and come to come back fresh the next morning after three days so we know we uh essentially did the detection we did the response we were working on your eradication and that was the time I said okay I can get and I had my death YouTube to take over the uh the war room we created the world we asked what environment was it the entire floor was locked down nobody in and out there was a specific area dedicated to the feds to do their own investigation there was
another area for myself my team War Room superintendent was involved in that war room also he was coming to get the update so we we will be able as transparent as possible way the next day that we were working and yeah that was the only time that they went wrong and I remember it was like about an hour or two hours of sleep and then I got the call I need to come back because there was a finding that it was divided yeah critical escalation and you know that when you're in the middle
of this investigation you'll find things that you need to address right away and that was that was it I came back to the office I was in the work for the next three days so six seven days in a row I was working oh my gosh one just on the picture one shower after three days I came back and then it was an American flag in the background and he is looking up to the sky exhausted exactly that wraps up part one of our interview you'll have to come back to hear the
rest we do want to thank our sponsors for their support of this podcast they make great episodes like this possible in particular visor that's v-i-z-o-r they can help you with Chromebook management and inventory in pretty much any Asset Management you got in K-12 if you have the task of creating reports for title funding or any kind of funding where you have to track where the funds were spent advisor that's b-i-z-o-r can help you check out visor.com K12 Tech
talk to learn more and get 20 off also thanks to Fortinet they do more than just firewalls but they also have a top-notch kick butt firewall you can email fortinetpodcast fortinet.com and Chris there will tell you all about it make sure you tell him that we sent you extreme networks is a proud sponsor of the podcast email Dominic D mayor extremenetworks.com some for specs and quotes and he will hook you up and then two more we got something cool.com email Jeremy at somethingcool.com about
information about their cyber security offerings and finally last but not least provision Data Solutions you can hit up Jeff Derrick and Ryan those guys are awesome see you next time