¶ Initial Breach and Sophos Introduction
Hi, I'm Jack Reese Sider, host of the show. Back in 2018, an interesting cyber attack took place.
Yeah.
It's it's kind of a a funny thing. I mean it was it basically came onto my radar the second month I was working at SoFos. Oh.
I should introduce you to Andrew.
Yeah, so I'm Andrew Brandt and uh throughout the time that the research was going on for this story, I was the a principal researcher for Sophia. Uh but I am now a principal threat researcher for a company called Netcraft.
¶ CyberRome Source Code Theft Details
So one of the things SoFos wanted Andrew to do was research novel threats and write about them on their newly established SoFos blog.
the team that I was on eventually didn't exist. I was the only person on it. And one of the analysts reached out to me through the company chat and said, Hey, I've got a great story for some really cool research. I'd like to write it up and have you publish it on the blog and do some edits on it. I said, Great. uh tell me more and he he told me the story, uh but the one thing he didn't tell, or what he said he couldn't tell me, was uh who the target was.
So he's like, Okay, fine, send me what you got, let me research it and I'll write about it.
It started with So there was a sales office and they had a bullpan, like you have a lot of you know, in a lot of sales offices where people are on the phone, you know, trying to sell the product. And so they had like this leaderboard that was on a on a computer screen that was running off a little uh Linux computer. And that was the first machine that got infected and and The threat actors managed to pivot from that, you know, Intel Nook, which is like a tiny little computer that's
small enough they can mount on the back of a TV monitor that's hanging on the wall that they were able to pivot from the NUC in and find access to the repository where the source code was and then get into that and then to do the cloud snooper attack on that cloud service that where the where the source code was. It's just mind-boggling to me. Like the amount of effort involved in pivoting from this to this to this to get into this and then to build this like backdoor that allows them access.
It's it's amazing to me.
Oof.
The attackers got access to the source code. But why? Was this an insider trying to seek revenge? Were they stealing it in hopes to sell it to someone? Did they steal it so that they could copy the product and steal their intellectual property? At the time, nobody knew what their motive was.
🎵 Music
These are true stories from the dark side of the internet.
🎵 Music
I'm Jack Resider. Is darknet.
🎵 Music
This episode is sponsored by ThreatLocker. If you've listened to Darknet Diaries for a while, you've already heard of Threat Locker. I've talked about how they lock environments down, deny by default, zero trust, all of it. But the problem they were solving changed because attackers changed. don't break in like they used to. Now they just log in with real credentials, real sessions. Nothing that looks out of place. And once they're in, they're treated like they belong.
So ThreadLocker took what they already were doing and pushed it further with their zero trust network access and zero trust cloud access. So now access isn't just about logging in, it's about the device, the connection, and whether any of it should be trusted at all. If you want to see what zero trust looks like when it's done right, go to threatlocker.com slash darknet. That's threatlocker.com slash darknet.
This episode is sponsored by Meter, the company building networks from the ground up. If you employ and work with IT engineers, you're gonna know how hard it is for them to do their job well. What your business needs is performant, reliable, secure networking infrastructure. But what you get is IT resource constraints, unpredictable pricing, and fragmented tools.
What you and your engineers need is a modern platform you can all trust to support your business. Enter Meter. Meter delivers a complete networking stack, wired, wireless, and cellular, in one solution that's built for performance and scale. Alongside their partners, Meter designs the hardware, writes the firmware, builds the software, manages deployment, and runs support. That means less time your employees spend writing to multiple vendors and more time working and improving your IT systems.
Meter's full stack solution covers everything from first sight survey to ongoing support, giving you a single partner for all your connectivity needs. Thanks to Meter for sponsoring this show. Go to meter.com slash darknet to book a demo now. That's spelled M-E-T-E-R. Meter.com slash darknet and go book a demo. So hackers broke into a company and copied the source code for that product.
The way we managed InfoSec there for a while and and and kindly too, it was the type of network that was, you know, in the process of being brought over to a a kind of set standard.
This is Craig. He helped clean up the intrusion.
My name is uh Craig Jones. I'm the chief security officer of Ontinue, but several years ago I was actually the senior director of information security inside software. I mean, if you don't know Sophos, we're a UK based cybersecurity provider that has everything from kind of EDR, MDR and through into firewall products. Um and at the time they had three different firewall products, one being Cybrome. Um the other one being a German based firewall provider and the new Software's firewall product.
So essentially they were collapsing two products into one and the new one being Sofa's firewall.
Yeah, Sophos's main product is their firewall. This is a network device that will act as a wall between a protected network and an unprotected one. Out of the box, nothing is allowed to pass. You have to tell it exactly what you want to allow through, because the point of a firewall is to stop unwanted traffic from coming into your network. And believe me, there's a lot of unwanted traffic that's always trying to get into our network.
And in twenty fourteen, they bought another company called CyberRome, which was also making an interesting security product.
That product, you know, we were flattening that product to make it into something healthy. You know, like CyberRome was very much purchased to be the development house for the new software's firewall product. You know, there's some super hot developers there.
And it was this newly acquired CyberRome network, which was the victim of this attack. Someone had gotten into CyberRome and was looking for their source code and found it for one of their products, which Craig and his team had to go clean up that intrusion.
There's some really cool stuff that that those actors did. You know? There were several points where I sat down and thought, damn, these guys really know what they're doing. You know, and I think for me, there was one where they'd actually attempted to intrude the network in several different ways. Um Mostly at the same time. And what was really interesting about it is we could tell that there were two or three actors working together in different consoles.
And one of the things they did, which is kind of funny actually, was that they'd gotten hold of a a a secure shell key. And one had obviously copied it and another person was trying to type in the password for it. And we could tell in the logs that they They were mistyping the password, you know? And, you know, the person who'd obviously taken the key had had obviously tried to brilliate onto another person and they were mistyping this thing. And it was
I it it was kind of crazy. You immediately knew then that this wasn't just like a dude, you know, this was a serious operation.
The attackers had really unique methods for getting in, not methods that were publicly known at the time, super sneaky and crafty ways to get into a network. And they got in through multiple ways. And then when they got in, they were able to move laterally in really unique ways too. So unique that the Sophos team had no idea that was stuff was even possible. It was like exploiting bugs in the way AWS handles identity.
One problem though is that they didn't have enough monitoring at first to know exactly what these hackers saw or took. They assumed because they got access to the repository with the source code that they took the source code, but they were unsure. They had to enable a lot more logging and monitoring to fully eradicate them from the cyber realm network. Andrew wrote this attack up because it was so interesting and new and published it on the SoFos blog, but didn't say who the target was.
¶ Discovery of Asnarok Attack
Yeah. So uh flash forward.
Two years go by. It's now 2020.
You know, we now have the team up and running. I've got a couple of people working with me. We're publishing a few blogs a week and I find out from internal people within the company that there's a security incident. And the security incident started with a text port call. where someone sent an email to their support technician and said, Hey, my firewall is showing this URL in the user interface and I didn't put it there and I don't know why it's there.
Hmm, sounds like a minor problem at the surface. This firewall had a configuration which showed what IPs are allowed to access it and manage it and configure it, and a strange URL was showing up in that list of IPs. It didn't make any sense as to why it was there or why anyone would ever even put it there.
So the s Sophos has a firewall called the XG firewall. At that this point it was just called the XG firewall. And The firewall has its own operating system. It's right, it's running a version of Linux in it. It has a UI that's running on the front of it so that you can manage it.
At the same time, someone outside of SoFos submitted a bug into Sophos for this same issue.
Um I think it was April twenty first. They had um they had a well we actually had an external bug bounty report. As a SQL eye injection. And what's kind of weird about it was You know, they I remember the user um actually claiming to be from Australia, um, but they had a a Chinese name, you know? Now at the time we didn't have amazing telemetry from any of the software's firewalls. Um we had kind of base telemetry which gave you like um
It was really designed for product managers to understand what features that um, you know, users were using. So they understood where to put their kind of limited resource time into, right? Well, so we had that and we we had a really good idea of like you know, where all of the CEO numbers for these devices sat and their IP addresses associated to it. So it's always kind of interesting to correlate the IP with the intended location of the the researcher.
So w we got a researcher's devices, one that had never been turned on before, which was pretty suspicious, you know, had never been registered. It was a a serial number that just come from a a web trial. um of a VM and We find the IP actually related back to Chandao in China.
Okay, odd. Someone from China with a trial license of the Sophos firewall found this bug and reported it to Sophos? And Sophos did in fact pay the bug bounty for this.
It was about ten thousand dollars, I think. Hmm.
Someone got paid a pretty penny for reporting this bug to Sophos at almost the exact same time that they were seeing it being exploited by devices in the wild. Strange timing.
¶ Asnarok Vulnerability and Impact
Oh, we we called it um Arsenal.
So the team investigated this bug further. It was present in the front-end web user interface of the firewall. To configure this firewall, you can use a browser and access it that way. Well, the web UI of this firewall had a SQL injection vulnerability in it.
Basically, in one of the form fields of the firewall, like maybe the username field or something, an attacker could enter in some commands there, which would glitch out the user input handling mechanism of the firewall and allow the attacker to inject their own commands into the
into the database of the firewall where the configuration's at. And this was a really bad bug for Sophos to discover. Their devices are supposed to be blocking hackers from getting into the network It's the vulnerable device which is allowing hackers into it. This is not good at all.
And they found that essentially every firewall that was facing the public internet was affected by this uh this bug.
These firewalls weren't just vulnerable. They all had been hacked into. Exploited. Someone probably scanned the whole internet looking for these particular SOFOS firewalls, and then ran some kind of automation script to go infect them all.
We'd we we kinda worked out that there were a huge amount of devices um affected. I think in the AIND uh FBI report that came out about this, I think they mentioned eighty thousand. Um I'd has a guess it was probably more, you know?
Dog, eighty thousand Sophos firewalls hacked into. But just because someone put a URL in place where it shouldn't be, that's not all that damaging just by itself. So the team investigated what that URL did, and that's when they started to panic. The URL would trigger a Git request in order to update the Sophos firewall itself.
But what was really weird about it is that we It was a W get to a domain called sofficefirewallupdate.com.
And Sophos didn't own that domain. So it tried to blend in like it was supposed to be there, and it fooled many of the people even at SoFos who just figured the update domains changed. But my goodness, this meant suddenly 80,000 firewalls were looking somewhere else for updates and not to SOFOS?
And it's kind of strange because we we actually monitor all domain registrations. It's kind of part of our kind of core security like ops function. So every single like cert that was registered, every domain that was registered, we we kinda pop up and, you know, anything infringed on software's IP we'd attempt to pull back, you know.
And it was one that had popped up like a little while ago, but nothing had kind of come of it, you know. But actually seeing this thing in in operation was quite like quite quite jarring, you know.
And I don't know if you fully understand what this means. If a malicious actor is able to send your firewall software updates, then they can put in whatever they want. They can give themselves full access to the firewall or they can log all traffic going through it. They can poke a hole in the firewall and let themselves right into your network. And then from there they can just infect your whole network with ransomware.
The thing that is supposed to block unwanted traffic is no longer blocking anything if the attacker wants it that way. Not only that, Sofos was worried that they had lost capability to update any of their firewalls properly.
Yeah, so so effectively what what they could do, I mean, the truth is anything, um, but what they really were after was system configuration and passwords. Now, I've always suspected that this was something that they expected to run quietly, for them to kind of pull that configuration, the passwords quietly. And then for them to kinda delete any presence they ever had on those firewalls. And then for them to have a really easy and simple access campaign.
¶ Sophos's Unprecedented Hotfix Deployment
Jees, so the attackers took copies of the configurations from the firewalls and then passwords from it? This was a pretty darn scary event for the SoFos team to handle.
So it was very much a I like an incredibly tense situation where We first up had to get a hold of one of these devices. Um, you know, we s set multiple teams up to to work out what happened. Um and to really do some in-depth incident response on this. Um, we were incredibly lucky, you know, we had the entire arm of like sophos labs to to help us kind of reverse engineer this stuff.
Okay, step one, fix the bug that made these things vulnerable. And step two is get the bug fixed on as many firewalls as soon as possible. They were able to complete step one pretty quick, but step two was a little bit more tricky. If you buy a firewall, whether for your home or a large enterprise,
Typically you've got to update it yourself, just like how you have to do your own software updates on your phone or computer. And Sophos firewalls are no different. The customers are the ones who have to issue updates for this thing. But to SOFOS,
This was too critical of a bug to try to tell 80,000 customers go update your firewalls. Because I'm just guessing that like less than 50% of them would do it in the first month. There's just not enough time, or it's not a high enough priority for them to fix it. So Sophos decided to do something they've never done before.
They pushed out a a hotfix to these firewalls. A hotfix is like a little software patch. that can run in real, you know, in real time and it they can live update all the firewalls remotely with these hot fixes. It doesn't require the firewall to reboot to be enabled. And they felt like they had analyzed the attack and s figured out exactly how the threat actors were uh, you know, leveraging their access and they closed those loopholes with the with the hotfix.
This was the first time Sophos ever issued a hotfix. To one of their customers' devices.
Now they they had built the facility to do hotfixes and they had not really used them before this. So there there had been no real reason to do it. But I think they had built in the f the the capability to do these hotfixes. anticipating that there might be an opportunity to use it if there was something that was a real problem. And it was fortunate that they had rolled this out in the previous firmware update that, you know, just before this attack had taken place.
Yeah, I think this is a really big deal. Like it makes me wonder if there's language in the small print of the terms of service that says Sophos reserves the right to make configuration changes to your firewall or or update it whenever they want.
I think that's what's important as well, is like this isn't something that's just kind of done. And it's not something that's done willy-nilly, you know. Um, and y you're right. I mean, it does feel kind of offensive someone coming in and tampering with my stuff, you know? But but effectively it's written into the the EULA, um like the end user license agreement.
Um and and candidly you kinda need this. And I think that's where a lot of firewall providers actually fail, is the fact that they rely on end users to patch everything. And and candidly, so many filers are just bought and they're never updated, you know?
Gosh, I really don't know where I stand on this. I was a firewall admin for my previous employer for 10 years. Those Cisco firewalls were my baby. I knew everything about them and would review every single change that ever took place on them. And I don't think I would like it if Cisco just decided to patch them one day without my consent.
Like some were in hospitals that were mission critical and some hadn't been patched for years because they were so finicky and any change to them would just make them wig out and crash. And when I had to update them, I wouldn't do them all at once in one big swoop. I'd do them one at a time and hold their hand and make sure that nothing broke after the upgrade and everything came back up as expected. So if a security vendor just slapped a hotfix on all my firewalls that I was in charge of.
I would freak out. What? We did not get approval for this change. We aren't in a maintenance window. We don't even know what changes you made to the firewall or what's happening. How can you just come into our devices and make changes without us knowing? I would be upset.
¶ Ethical Dilemma of Remote Patching
Like I wonder, did the SoFos team get approval from their lawyers before issuing a hotfix to their customers like this? Is this even legal?
Yeah, I mean that's that's a great question. I I was not privy to those discussions. But uh but there were uh I'm sure there were discussions like that about, you know, what is our legal liability? What are we allowed to do and not do remotely on these devices. Um, I believe ultimately the decision was made, and I'm not sure if there were lawyers consulted on this or not, but made a lot of sense.
that the the harm of allowing the firewalls to basically try to r ransomware the inside of networks was probably greater than the risk of somebody complaining that uh oh you made a change to my firewall without telling me first. So they just went ahead and did it.
Yeah, I mean I think not only that, but it's like this idea that the vendor can come in and change my device in in any way. It's not just like crash logs that are being sent to it. Wow, what else can you do? If you could put a hotfix in, can you see the password? Can you see the connections? Can you see can you come in and do other work? Can you uh update to different firmware that has uh malware on it or something like could you do things that
Yeah, you know, in and you know, you you start your mind starts going like, could you do things that the NSA wants you to do and s go and spy on this customer or something like that, right? And so when you're a firewall admin, you're like, No, I have to make sure that this is No other you person in the planet can access this but me and the other people on my team, because you can't risk some like a back door. But it's basically a back door that you had.
Yeah, that's that's entirely c accurate. And you're not wrong. Um and these are these are devices that are typically placed in a in a position in the network where they act as the barrier between the outside and the inside worlds of of your the the networks. And I and I recognize that that is a risk. Um however, and it is also worth noting That this is exactly what the bad guys were doing at this moment. They were installing malware inside the firewall. So How do you fix that?
🎵 Music
I could just imagine the headlines at this point. And just I I don't I don't I my question is, did any bad news come out to be like SOFOS VO found vulnerable, tens of thousands of customers impacted, um, huge vulnerability. Hacker has complete control over their firewalls patch immediately. Like that could make the stock tumble, that could, you know, really hurt business.
Uh yeah, I mean it could. And that was one of the reasons that I was brought in basically on day zero of this happening. Um the company realized that they had a public a potential public relations nightmare on their hands and they needed to communicate. communicate uh as as openly and as forthrightly as possible everything that they knew and everything that they were doing to fix it. Um and you know, credit goes to the people, you know, in leadership at the company who decided that uh, you know
possibly against the, you know, conventional wisdom at the time that they were gonna go public with with everything we knew about this attack. Um, it was not a common thing at that time, but uh As I said, maybe yeah, I've I've worked for a long time doing this kind of in this kind of role where I do investigations and then, you know, publish about them to the public to warn people about bad things that are happening on the internet. And it's been my experience that
more uh information that you get out, the the better protected people are. And that um being radically transparent the customers who are affected. It also warns the public that like, hey, this is something that you need to be aware of in the future. And um it might also put the threat actors on on notice that uh hey, we're we're watching you. And we're taking action to stop you.
¶ Linking Attacks, Nation-State Suspicions
As the SoFos team investigated this more, they learned that whoever did this attack had to have really in-depth knowledge of SoFos firewall. Like there's no way they should have discovered this bug unless they had access to the source code, which wasn't publicly available. And and that's when the pieces started clicking into place. The part of this firewall that was vulnerable was code from the CyberRoam firewall that was moved over to the SOFO's firewall.
And two years before this, as you know, there was an attack on CyberRome. And what server did the attackers get access to? The one with the source code for their firewall. So they started to think, holy crap, this is a very serious threat actor who's been attacking us for years. They spent tons of effort getting into CyberRome's network to steal the source code, only to study it for bugs, and then launch a massive attack on our Sophos firewalls?
Whoa. What do we even do with this information? To think your products are the target for a major cybersecurity campaign like this? This is starting to smell like a nation state actor is behind this. Who else has that much time and resources? And what the heck was the deal with someone from China submitting this bug the exact same time that Stofos discovered this? Very strange.
One of the things that we've been kind of working on, but even before this uh situation was, you know, pulling in our uh telemetry or firewall telemetry, the the kind of basic telemetry I was talking about earlier into Splunk. And I remember talking to uh Mark who who was just this amazing Splunk engineer in my team. Um
Can we go back on that day? Like can we find out like when this first started? Because I couldn't quite work out the exact moment in time or the first firewall that was hit by this this uh as Narok um attack.
And then I went back up.
that data go back? And then Mark said, Well actually I think I've got like three months worth So we kinda ruled this thing back three months. And there was one single device that had been hit like a month or so beforehand. Like sometime in February, if my memory serves me right, and it was just really strange. So it was kinda registered to like a Chinese uh one six three address and it sat again in in Shangdu.
Chengdu China again? That's where the person who submitted the bug was from. So they took this firewall, and again this one was running a trial license, which was actually just a software based firewall running in a virtual machine.
And it's a virtual machine because SOFOS isn't allowed to sell their firewalls to China due to export controls. So really nobody in China should even have a Sopos firewall. Their suspicion was that the attackers were using this virtual firewall to practice their attacks against.
develop them, and then unleash them against the world. Because Sophos has the ability to run in a virtual machine with trial licenses, they can just spin one up real quick, try attacks on it. If they mess up the firewall, they can just reboot it, take it down, and bring a fresh one up in minutes.
¶ Profiling the Threat Actors
We find this trial license and They were kinda all associated to a 163 address and a moniker that we we called G Big Mao.
Okay, interesting. They looked up who registered that trial license, and this gave them an IP address, a username, and an email address. And the username was Big Mao. So now you pivot on that name. What other SoFos products has GBigMoo downloaded?
kind of pivoted on him um we find that he actually started to experiment with this um with this database Or SQL eye injection like a month or so ago. And we kind of found then looking at his IP address, again, we had phenomenal tip telemetry here. that he was looking at different knowledge base articles around our kind of previous CVEs issues. He was looking through our um forum system. to s look at maybe other potential issues or places that he could maybe pivot and work on.
Then they took a look at his email address and wondered, has this email address been used anywhere else in the world? So they do some OSINT investigation to see if this email is known anywhere else.
And we find that he was uh an actual firewall researcher and he published like a number of different um like vulnerabilities. We could see him on kind of Linux boards and
Yeah.
But you know, publishing various different router vulnerabilities. Up until about twenty eighteen and then he went silent. You know, he'd been really, really busy up until like twenty eighteen. Now we kind of found out that he was working for a company called Xuan Silence Information Security Technology. Um mostly because doing some extra OSN. we found that his username um appeared in many like Chinese hacking groups and lots of CTFs. So like capture the flag type event.
Um, where he'd been registered towards this company as well. So we found kind of corroborating evidence from a couple of different places that this was the same guy in the same company, you know? Um Again, l located in uh in uh Shendau in in China. So we we found a really clear picture of who this person was. Now his external obsect was pretty good. You know, like you would not have been able to find him that easily, but because we could see the internal telemetry.
and get the license information, kind of connect the dots, we could actually pin these devices to to him and his usage.
But for...
¶ Deployment of Kernel Implant
we had to do at that point was find out more about these devices that were being used for research. We found that from the limited telemetry that we'd started to gather with the first hotfix. But what we realized is we actually needed more. Like we really needed more detail, faster detail to like a greater depth to understand what these guys were doing. So We we developed a kernel implant in house.
A kernel implant? That's a that's a nice way to say it. I guess when the good guys make it, it's called an implant, but if the bad guys were to make it, it would just be called malware. But essentially a kernel implant is a hidden piece of software that they develop to sneak onto their firewalls to covertly and sneakily spy on what the firewall is doing.
Yeah. There's a lot of interest within the company. Well, we know that there's these firewalls that have been registered to people who have. uh non corporate or non uh enterprise level uh email addresses, uh like free webmail addresses. The firewalls are checking in all from Chengdu. Uh we know their serial numbers, so we know the exact count of the number of firewalls that are being used.
Uh in these places. And we could see from some of the log telemetry that the uh the threat actors are running commands. that are testing how these exploits are gonna work, but we don't have the exploit code itself.
Thank you.
So the security team decides they're going to build something that they've just called the implant, or sometimes they call it the kernel implant. Uh and it's a it's a small elf binary that it gets distributed only to the machines. that they are specifically interested in uh taking a closer look at. So these machines that they believe are being operated by threat actors, uh, where they're doing these commands that are way outside of the boundaries of normal firewall behavior.
And these things are capable of doing more than just uh sending, you know, log entries. They're able to uh pick arbitrary fields from the file system uh on the firewall and send those files back. So that was how uh in some cases the the team started throwing these kernel implants onto some of these firewalls that we could see were being used to do this experimentation.
And they were retrieving all sorts of very malicious and pretty dangerous files that were being dropped on these machines by the the people who were developing these exploits and were testing them out uh in advance of uh of attack.
Wow that is wild. This is going to take me a minute to fully grasp.
Mm-hmm.
Developed an implant and sneakily put it on one of their customers' devices to essentially spy on them. Is that going too far?
To call it malware is kind of a uh it's kind of a misnomer. I mean, I I'm not gonna defend the overall argument here, but I I will just say that like there is nothing malicious about wanting to know what someone who is doing malicious things with your product is doing. You know, it's kind of a It's an ethical gray area.
¶ Justification of Implant Capabilities
I've got a caveat this with we only ever deployed this to devices where we would be absolutely certain that they were a threat actor device, you know? There's
Not just threat actor controlled, but threat actor owned. Like this is where they're doing the research.
Exactly. So so number one, like we never deployed it to any properly licensed devices. Um the second part is like we only ever deployed it to Chinese devices. Um, we we just didn't sell firewalls in China. So there was really unless you're a company maybe bringing one from external, there's no real reason for you to actually have one legitimately in in China. So under the the EU law, you know, we we could take steps to um
to protect the firewall, you know, and and gather intelligence. And that's covered clearly under the ULA.
No, the other one. So that's what you got in there forty people in the room. The lawyers must be in there too. Like are we allowed to hack into these devices? that we think
It's a serious conversation we had. Yeah. I mean, it wasn't just a small one either. I mean, I don't think people have ever done this before, you know? Like we sat there debating this for thing for hours, you know, and really hours, because
You know, there's some serious ethical challenges around this. You know, it's not. You know, what what happens like if we find a guy, we you know, we we record him, we see him doing it and we we send it through to law enforcement, you know, like a we facilit you know, th there's so many crazy things that we discuss there, you know, it's
Yeah.
It's it's a conversation that I I never thought in my entire career that I would I would have, you know. Um yeah. I mean candidly too, I never thought legitimately in my entire career that I'd ever deploy a a kernel implant either, you know? Um but it it was it was certainly interesting, you know.
Well, I've never heard of a security vendor doing anything like this. Adding in stealthy secret implants to spy on their users? In my opinion, spyware is malware. And gosh, before hearing all this, I would have said, that is going too far. But now now I'm not sure. My ethics are really being challenged here.
Again, you know, we I had amazing access to just quite incredible engineers. They built this uh kernel implant that allowed us to basically move Sophos firewalls from like a normal update path to like a specific update wing. And we would then deploy this specialist kernel implant in a in a normal update. And you just wouldn't see it.
But what it allowed us to do is like grab anything we needed from the device. So for example, things like you know, files, if there were entry updates, it would kind of record anything that was kind of written to specific um writable directories. And it would start to give us our a really good idea of what they were doing, what they're writing, why they were doing it. But some of the really cool things that we actually got from it were quite unexpected.
So for example, we started to pick up on the devices around the firewall. So we'd you know, capture uh all the MAC addresses of devices connecting to this firewall, we'd also capture MAC addresses of the things that also sat in the network alongside the firewall. And then we suddenly realized that actually this is this is huge. This isn't just like
SOFOS Firewalls. We've seen other vendors' devices on the same subnet alongside the software's firewall. You know, they were looking at all sorts of devices. You can probably pull from the top of your head, thinking about things that have been attacked in the past couple of years, the devices that were in the rack alongside that software's firewall, you know?
Oh wow, so the firewalls that come to mind for me are like Cisco, Palo Alto, Juniper, Checkpoint, Fortinet. And he says he saw other vendor firewalls set up alongside their firewall in this threat actor's lab.
Now just being the, you know, person who's telling the story of what happened, uh We were observing, you know, in the world, uh not just Sophos firewalls, but every firewall vendor getting hit with zero days, uh, their customers being, you know, attacked in various ways. And there being no way to resolve this and certainly no way to anticipate it. Now, whether or not other companies are doing the same thing.
No one else has disclosed that. Um, but I d I don't think it's outside the realm of possibility to Think that maybe some of them were.
Oh man, this is now tugging at me in new ways. If every firewall vendor is getting hit with the same type of attack, and SoFos is the only one being transparent about what they're seeing and what they're doing to mitigate this.
¶ Counterintelligence and Domain Seizure
Then yeah, I give them a lot of credit for that. Here's the test, I think, for whether your company is evil or not. First, it has to be transparent to its customers. Let them know exactly what kind of configuration changes, updates, or spying or data collection you're doing on your customers' devices and in what circumstances and what's that you're being used for. And second, be proud of whatever it is you're doing around that.
If you're a company which is making changes to the customer's products, but then not telling them and secretly adding spyware, but making it so top secret that not many people on your team even know it exists. Then I think you might be evil. If you're afraid to let the public know exactly how you operate, because you think it's going to look bad on you, or maybe because you think it's not even right, then either stop doing it or go public with it.
And sofos came to the conclusion that while this is not an ideal situation, this threat is novel and sophisticated in ways nobody's ever seen before. And not only that, whoever was doing this, they're being unethical themselves. So Sophos had to deploy a novel and sophisticated approach to defending their device. And while it's not pretty, at least they came out and told us about it through Andrew's blog post.
And they're basically saying, hey, we're in the middle of a nasty street fight here, and the gloves are off until we can neutralize this threat. And again, I give them a lot of credit for that. Nice job. So at the same time, they were developing this implant to eavesdrop on the hackers. They were also in the process of studying those domains which were found in the exploited firewall. The hackers pointed all the firewalls to two domains to get updates from, which were not owned by Sopho.
Yeah, well there was so there was Sofrost FirewallUpdate.com and Sofros ProductUpdate.com, which uh were registered at different registrars and hosted in different IP spaces. Uh but because they were they both had sofos in the name and they were part of this attack. Uh Sophos went to ICANN and did the domain name seizure process on those domains so that they could pull those down and uh start to uh they wanted to sink all the domains and see what was connecting into them.
How do you seize a domain?
With lawyers and money. And uh, you know, it's uh it's a r a really serious thing, you know, like attending court in in Delaware, I think it was, you know, remotely. Because at the time, don't forget that this is The thick of COVID.
Geez, that's another thing that's wild to me. The fact that you can take over someone else's domain if you can prove that you're the one who's the rightful owner of it or should be owning it, but they gave enough reasons to the courts, who then demanded that the domain registrar give SOFOS control of the hacker's malicious domains.
Server used by the threat actor actually sat in the Netherlands and it was one of these bulletproof um Hosting uh providers. So we were super h lucky that um, you know, through the NCSC in the Netherlands they w they were kinda an intermediary with the kind of Dutch National High-Tech Crime Unit. And once we kind of realized how this was panning out, the Dutch National High-Tech Crime Unit just jumped on this. And they um managed to get hold of this C2 server, so the actual physical Linux bot.
I guess it wasn't bullet proof then, huh?
Well yeah, this is the thing, you know. So they managed to grab hold of it. And um I mean we were super keen to
How do you even so how does that happen? You convince the Dutch authorities. So you're just a company in the UK. You're just like, hey, we we make this product. You can't just call up the Dutch police and say, go get that server. We need it. And then they're like, we're on it.
Well yeah, I mean you'd think, but then You know, luckily or unluckily for us, there were a couple of Dutch customers affected, you know, by this attack. So that allowed us to be able to register a crime and then get assistance.
¶ C2 Server Takedown and Analysis
And we did this globally, you know. Um, we really used all of the resources available to us. Um So, you know, this obviously took time, you know, I think right now this is like three or four days after the attack. Um but the NCC in in the Netherlands were incredible and the Dutch guys there were just
Super helpful. I mean, we wanted a copy of that threat actor device. Like I wanted to see that Linux box and understand what they've done. I mean, obviously it was it was evidence now, it wasn't owned by us. So we we couldn't get a snapshot of it, for example. But they allowed us to to basically you know, work with them and analyse the box live uh on a screen share so we could actually understand the scale of what had happened, you know. And
We'd seen the these threat actors scripts for scanning the devices, the outputs that they'd taken from the firewall, you know, how they'd set this thing up, you know, kind of Chinese characters and notes and things throughout the device. Um What was actually surprising was that everything was kind of set up manually on the C2 server. I kinda expected them to deliver the C2 server with some sort of kind of DevOps bizarre.
Um, but it was just basic. You know, it was like a Linux box and someone who copies subscripts to it, you know? Um, but they were amazing. I mean the the NCSC in the Netherlands just gave us so much help and really helped us focus what we you know, where we needed to look and and the kind of scope and scale of all of this.
At the same time, they got control of the domains used by the hackers and sent all the traffic they were getting to a sinkhole and logged it all.
It's just fascinating to think that like I don't know, a Netgear, a Linksys, uh, you know, some other commercial product was checking in to sofa's firewall update.com. It kinda it it's It's is almost screams of like, well, you know, we we could be bothered to register this domain for Sophos. We're not gonna bother to register it for this these other companies. Like we already got the domain, we're just gonna keep using it for these other things.
I couldn't find a single article by Linksys mentioning any of this. Nothing at all. Netgear put out an advisory saying a Chinese threat actor is attacking their products. However, they say they are not aware of any Netgear devices being exploited out in the wild. Which, if they don't have any telemetry from their customers' products, then yeah, of course they're not going to know if any devices are being exploited. And that's what's challenging me here.
Should the firewall vendor be collecting logs off its customers' devices in order to better understand what devices are actively being exploited? Or should that be the responsibility of the customer? In many organizations they have their own security logs and even a team to monitor those logs to look for threats. But things like Netgear and Linksys are typically home devices, and it's very rare for people in their own homes to be monitoring their logs looking for threats.
I looked it up. Negar actually does quite a lot of analytic collection from their customers' devices. They collect IP addresses, geolocation, how often you use the firewall, what you use the hardware for, what channels your Wi-Fi is set to, and what devices are connected to it. It's surprising with all that analytics collected that they didn't spot a single device being exploited by these threat actors.
And this is what frustrates me. When my home router is sending all kinds of logs to another company, like what devices are connected to my router? Really? I hate that. I want the devices in my home to be private and not sending tons of data to somewhere without me even knowing. Because if Netgear has that data, then it's likely a lot of other people have it too.
¶ Round One Conclusion, Asnarok Named
But then they also registered for the for the kill switch, they registered Ragnarok from Asgard, right? And Ragnarok, of course, is the Norse mythology end of world. And it was fascinating that the that was how they, you know, used that nomenclature and that language behind it. Uh, because by this point, we already had some folks who were using
Marvel characters, superhero names in their user accounts that they were, you know, that they were using for for downloading these firewalls. So we had a guy who used the handle of T Stark. uh who was who was involved in some of the exploit development and had registered a bunch of these virtual firewalls. And now we're seeing, you know, this is the time frame when the T V series Loki came out and when the Thor Ragnarok movie had come out as well.
And it's just fascinating to imagine that these guys who were doing this stuff saw themselves as some kind of, you know, superheroes or maybe they just like put themselves in the shoes of like that maybe they're just, you know, maybe they're like up there with God. And that they can, you know, engage in, you know, a a hammer that can throw lightning from a distance at a at an enemy. Just fascinating to think about.
So this is why Sophos called this particular exploit Asnarok, a combination of the words Asgard and Ragnarok. And all these efforts on their side paid off. The implant gave them incredible insight into how these attackers were developing their exploits. and were able to write fixes for the next exploits before the attackers could even launch them, which is incredible to be in the hacker's machine watching them in order to be one step ahead of them.
Good job, Sophos. This looks to be a pretty hairy threat actor that you're dealing with. But little did everyone know that was just round one. We're going to take a quick ad break, but stay with us because round two gets even hairier.
This episode is sponsored by Drata. Let's face it, if you're leading GRC at your organization, chances are you're drowning in a sea of spreadsheets every day, balancing security, risk, and compliance in an ever-changing landscape of threats and regulatory frameworks.
can feel like running a never-ending marathon. Enter Drada's agentic trust management platform designed for leaders like you. Drada automates the tedious tasks, security questionnaires, responses, continuous evidence collection, and much more. Saving you hundreds of hours each year. With Jada, you can spend less time chasing documents and more time solving real security problems.
With Drada, you also get access to a powerful trust center, a live, customizable product that supports you in expediting your never-ending security review requests in the deal process. It's perfect for sharing your security posture with stakeholders. or potential customers cutting down on the back and forth questions and building trust at every interaction. Ready to modernize your GRC program and take back your time?
Visit drada.com slash darknet diaries to learn more. That's drada, spelled D-R-A-T-A. Drada.com slash darknet diaries. Yeah, so that kind of wraps up round one. You you identified, you fixed, you cleared, you found all the ones that didn't get fixed, you found fixed those and took down the the whole infrastructure that was doing it. Done. That's patched like permanently, one hundred percent. There's nothing that no customer has that's not patched. We're good.
So everything I've just described to you happened over four days.
You know?
Which is just yeah, when you think about it, I mean it's it's insane. It's basically one of the largest, widest instant response operations on Earth. And we did it in four days. Wow. Um and I I still think about it now. I mean it's like a crazy situation. But we were lucky with an amazing team. It was uh, you know, think things aligned, you know.
Amazing. Well, that's gotta be um one of those four days that is permanently in your head, like a light bulb experience of of of work. Like I I a lot of people are being on the show and I say, Tell me about the worst day of your life and would you say that that's probably it?
I wouldn't say it was the worst day. I would probably say it was uh it was an experience, right? I mean, I l I remember thinking at the time, Oh my god, this this just can't get any worse, you know, and uh Every time we'd kind of look at this, there'd be something else. Or, you know, I remember as these devices were checking into telemetry, we'd just see the number of of affected devices grow. And I remember feeling like just this gut-wrenching feeling of like
🎵 Music
¶ Round Two: Baja Attack Emerges
Within about I don't know, six to eight weeks after the hotfixes were rolled out, um the threat actors had figured out what the hotfix did to make it impossible for the Ragnarok attack to work. And they had done a workaround. They had just, you know, bounced their attack around the thing that the hot fix was able to, you know, in a very rapid way cluge together to make it not work. Uh they clued together something that got around that hotfish.
And wham, round two officially begins. More SoFos firewalls are getting hit with a brand new vulnerability, one that SoFos had no idea was even possible. But Sophos was ready. They even developed a specialized team just to handle this, Exops. So XOps jumped on it. They saw what the vulnerability was, they wrote a fix for it, and started immediately trying to patch the firewalls.
Team starts to realize okay, we we need to uh give these things names because if we're gonna be having these uh attacks happen in uh you know in sequence in short order, uh to just keep straight, we need to come up with names. So they they decide to use uh the names of locations around the Pacific Rim
uh as the code names for these internal attacks. So they they give this attack a nickname Baja. It doesn't have anything to do with Mexico. It's just they just decided that they want to talk about it in the sense of, you know, it's on the Pacific Rim.
Which is uh
region of the world where volcanoes and earthquakes happen, right? So it's it's a place of turmoil.
So internally, so folks realized this attack is bigger than a single attack. This attack is linked to multiple attack campaigns against our product. So they called this whole series of incidents Pacific Rim campaign.
¶ Attacker Adaptation and OPSEC
So what the threat actors figured out when they were doing this uh the the development of this Baja attack is they watched Sophos and and they watched how the hot fix mechanism worked. And they learned how to uh develop a a a new exploit, but also uh they started to develop technology and technique to get around hotfixes. So they would they figured out how hotfixes were being uh deployed on firewalls.
and they were slowly starting to turn off features inside the firewall that allow the hotfixes to launch and run and do their fixing. Now this time they're putting just regular old web shells on the firewalls.
A shell is like CLI access to a computer. A web shell is having remote CLI access to a computer over the internet. And what the threat actors did this round was simply give themselves remote access to as many SOFOS firewalls as they could.
And this also removed the need for the attackers to use command and control service because they could just log in directly to the firewall whenever they wanted and do whatever they wanted to it, which again is a huge problem. You should not allow attackers to enter your firewall On the internet. This is like the security guard of the building suddenly being remote controlled by the bad guys.
In June, I mean we we've seen this um this attack happen, obviously, you know, it was a an Apache um module uh issue. And it was chained as like a a local privilege escalation. Um So it's basically again any device that had a one uh facing web portal um w could be affected, which was a lot of devices.
The threat actors set up these web shells where they just needed a username and a password to log in, and so the Sophos team tried to crack that password, but they couldn't for some reason.
Actually I think we unsuccessfully tried to crack the hash of the password, but I think eventually we found out that the the actual password is was Gucci. Um I I we ca come across this a while later because it was it seemed to be a common password for Chinese threat actors to use the word Gucci. Now I've no idea why. Yeah, we find I think at the time it was about 175, 200 devices um that were affected.
Okay, so one thing you want to do in your investigation is just try to see if there's a commonality of what firewalls are being exploited like this. And that might give you a clue as to what might be next or who's behind this. So they start looking to see where these firewalls exist in the world and for which customers.
Yeah, so so this one was very much targeted. You know, the first attack was very much uh Bream pri You know, th this was specific devices aro around the kind of Asia Pacific area. I think, you know, like Taiwan, Pakistan, places like um Philippines, you know, were very much targeted, completely different to the first attack.
And, you know, we we kind of found that, you know, this one had had delivered payloads that'd been used in kind of earlier tax as well. So again, you know, two Linux shell scripts. So we were able to kind of connect it back to uh a specific actor, you know, we we obviously seen these specific files and hashes um on on the device that we've been tracking. And then eventually we see it being used. Now, what was kind of interesting about the way that they would develop these is that
We we'd kind of see them starting to work. Now obviously they'd be working during Chinese hours, they work nine to five, and you know, would see them with amazing OPSEC externally, but the OPSEC they had on the box. was atrocious. So they would be for example um working with crash dumps. And you could set up the this SOFOS firewall that if you ever had a kernel crash or a crash of any sort, it would email you the crash log.
to your email address. Well, these guys would use their personal email addresses. So imagine the the actual firewall is registered to a completely anonymous person and then we have linked email addresses and Gmail addresses. Um inside the firewall telemetry,'cause I guess it was probably quickest and easiest for them to grab the stuff from the personal mail, you know? And it was super easy for us to like OS in exactly who these people were.
¶ Vulnerability Patterns and T Stark
They they start looking back in time at the telemetry that they collected and they discover that this was another bug that someone had submitted a bug bounty for and gotten payout on. And Here it is being used in the wild, like just days after the the payout happens. So this is starting to get to be a pattern and the the attacks are, you know
Widespread. People are, you know, getting notice about it. So I get called in and have to you know, decode how the whole attack works and do another flow chart similar to what we did with Asnarok to do the the Baja attack.
These two names keep showing up again in their analysis of these attacks, which are G Big Mao and T Stark. These are the people who registered for trial licenses of Sopho's firewalls. They were in China, and the malware would show up on their device first. which would indicate this is where all this is originating from.
Well, you know, uh one of the one of the things that we can do. So you've got this telemetry tool that you can you can do basically wide scale threat hunting within the firewalls themselves. And so you can do things like, Okay, well we recovered a piece of malware off of
the very first machine that was that belonged to a customer. Um, let's see where else this malware exists on, you know, the universe of firewalls that are out there. And that was how they found T Stark. So T Stark's firewall was the first one where they Uh they found a copy of the the not just the same malware, but like the binary identical, like the actual same
uh on this guy's firewall. It had been there for two months. So he'd been experimenting with this piece of malware. Uh while the Aznarok attack was happening, he was he was basically planning the next Like in the middle of us dealing with the aftermath, they were already developing the exploit and building out the payload for that attack.
Um and then the other thing that was really interesting was that uh we found a bunch of other stuff on this T Star guy's firewall. Uh his firewall had a bunch of malware on it that was designed to run on the map. And on iOS, on on iPads and iPhones. And there is no conceivable reason why there would be like a Mac executable on a inside of a lint a SOFO's firewall. It just there's no reason for that.
So that was an interesting find. And we didn't really understand what that was being used for, why that was there until much later.
¶ Nation-State Confirmed, Fourth Zero-Day
Yeah, what was that?
So this all happened in June. Uh starting around August, September, uh Sophis had started to communicate with other companies in the field, uh, some of whom did Uh forensic analysis uh you know, f post post attack analysis for their customers. And one of these companies is called Volexity. And Volexity reaches out to Sophos because They had a customer with SOFO's firewalls and they were called in to do the investigation on the Baja attack and they had also discovered.
uh mac os and i os software in their firewall and vollexity came to sofos and said hey guys is this what why is this here? We had no idea. But it turned out so Velexity had figured out that the threat actors who were dropping these software on the SOFOS firewalls that they were investigating, um, that the the owners of those firewalls. We're we're operating a charity that supports the um the Uyghur diaspora.
And the Uyghurs are a a an oppressed minority in China. They're they're uh they believe in uh Islam and they practice their faith, but they are uh strongly discouraged from doing so and they've been put in prison camps and You know, there it's the story of the Uyghurs is outside of the scope of this podcast, but the the point is is that there's really only one organization that actually cares.
about these two groups of people, uh, you know, about surveillance of these two groups of people, and that is the government of China.
And during that time they kept a close eye on the activity of Gee Big Mao's firewall. and they would see it would just get infected with a new vulnerability, which was like the fourth zero day vulnerability on the SOFOS firewalls. Zero day vulnerabilities are ones that SOFOS don't even know exist. They've had zero days to fix this, basically. And for me, this is the point. Where I suddenly see the scale of all this.
The first attack was scary already, but four zero days on a security device discovered and leveraged by the same threat actor? That is a lot of time and resources put into finding ways to attack SoFo's products. This isn't just a group of kids or even some kind of cyber criminal which is focused on making money.
When someone can spend this much resources and time focusing on getting into a very specific thing and spend years doing it, that's typically a nation-state behind it. The skills and patience were so impressive here.
¶ Hardening Firewalls, Ongoing Battle
Which meant so false. had a lot of work ahead of them to fix it.
Absolutely. You can imagine like the the amount of work that this spins up, like and the way that it kind of balloons out of control, like as you discover that you know more and more pieces of The open source code base that you're using are being exploited in different ways. Yeah, who has time for all of that? Like if you if you if all you're doing is just fixing these patches, like that could be a full time job. But you're also supposed to be building out a product that has new features and
response to coup customer requests and all other things. So Yeah. It i i it at a certain point it just becomes oppressive. Like the amount of patching that you have to do and the analysis involved in that and, you know, fixing the firewall takes just as much QA. Um, you know, that it takes takes time to build build things that don't break. And these are these are critical I don't want to say they're critical infrastructure, but they're protecting critical infrastructure.
Yeah. I mean in reality, you know, we're at that point that, you know, the the sophist firewall itself need needed some hardening. I mean, that part's fairly clear. And there was an internal mission going on um where dev resources may pivoted to try and harden is certain elements of the operating system and and web portal to to really help us um
Portal. I tell you, man, that the more ports you have open, the more vulnerable you are. And if you have a web portal, you're going to have a million different ways. to mess with that thing. And you are my my f when I was a firewall admin, I was very adamant about zero exposure to the internet. No SSH port, no web portal, nothing is allowed. that the w it the internet should be ac able to access this firewall. If you want to get to this firewall, you have to come at it from the inside.
Exactly. And I wish every Ferwell admin acted like you, Jack.
Yeah.
But anyway, you we we we had people who just put the firewall on the internet and they put the web portal out there. Now there was some legitimacy around putting your web portal out there because you had the admin portal, which is separate to the web portal, and the web portal was where users picked up SSL profiles and and you know, things like that.
I mean, it is wild to think that someone or some team out there is working feevishly to find vulnerabilities in your product and then to have an implant on their firewall so you could watch them develop their exploits. And the threat actor had no idea there was an implant on there watching what they were doing. The SoFos team did a really good job at hiding it, so it would be really hard for them to notice.
It was really well hidden, you know. So um so you know, we we did start to get some really good telemetry and start to know these guys at And honestly, um We were really obsessed with it. It was almost like obsessional. Um we would just wait for this telemetry to come in and then we would be all over it. You know, we'd start to dissect what they were doing, how they were working, you know, if they'd add any new IP addresses, we'd start to OS into and we'd start to build a picture.
of who these people were. There were multiple threat actors that we were watching any one time. You know, it's kind of funny because like, you know, I often think that, you know, external threat intelligence is very much like um almost Like uh uh astrology, infosec astrology, you know, where people are kind of connecting a technique to a specific threat actor group. Dude, we had names. We could tie them to companies, you know? And then we could tie it to
to threat actor group attribution, you know, it was a really weird situation we were in. We had visibility that was just unreal. I I I remember like at one point we seen one of the actors searching for a flat.
So we started to work out that he, you know, he was looking for a flat, like he was a normal dude. You know, he's going about his everyday life, probably sitting there bored in the lab, you know, having run the same test ten times, thinking, Oh, you know, I'd really need to sort my housing situation, you know. And we're there like building this picture of his life and honestly we we were obsessed by it. It really became like obsession ops.
Yeah, because since Craig had control of the firewall in that guy's lab, he could essentially see all the traffic going through it, which gave him a unique look into this person's life. And with these new insights and closely watching everything that was going on, the Sophos team were able to quickly create fixes for the vulnerabilities to minimize the impact as best as they could. So with all these vulnerabilities fixed, Round two of this battle came to a close. Sophos had a lot of bruises.
But I think they won the battle.
¶ Identifying Chinese Defense Contractors
Yeah, that's it for for round two, but you know, there there's several parts.
That's um
That's is kind of useful. Number one, round two really validated our our use of telemetry. It was the first time that we'd really used our implants. The other aspect to this as well is we we'd become really adept at finding these uh threat actor devices. So we started to work out that obviously we'd identified this actor called G Big Mau.
But w all in all, we were dealing with about seven different actors that we could see. And, you know, some of them were doing the same thing, but in different locations. So we kind of worked out quite quickly that they're working for individual Chinese defense contractors. Because when you think about like a government department, they're not going to duplicate the same work because effectively it's all the same people working, where a defense contractor
Everything is valuable to them. If they're the first to an exploit, that's super valuable. So what what we found then is we found these multiple companies. And one of the simplest ways we actually found it, funnily enough, and this sounds so basic, is that we would look at devices that would be continually going up and down firmware versions. Um
Threat actor devices would constantly like putting lace firmware on, roll it back, new firmware, roll it back. And they'd do this like, I don't know, maybe five or six times a day. Whereas like normal
Farble.
operation. It's like it's new firmware and it's left. And then in a month it gets new firmware and then it's left. So these things just stood out like a sore thumb. So it suddenly became really easy to find these threat actors, you know? The more telemetry we had, the the easier it got, you know? And we started to really build a wide assortment of threat actors in China, the locations they had, and of course, you know, their their
honestly Pisspoor obsec that they had it on the device itself just allowed us to start building up really quite wide profiles on them. And over this period we would start to like really get an idea of how they were targeting things. Um And it was very much like um seeing them do something, build an attack, know that this was coming.
and having to wait for it to be deployed, you know? I mean if we went and pre-patched the devices continually, they would have noticed. They would know that the game was up. You know? So we kinda waited to understand what was happening. Would wait for the first indication of deployment of whatever they were doing. I kinda run in patch it almost immediately, you know? So we had like probably one of the craziest and a like forward going threat intelligence.
That's crazy. Threat intelligence is simply the understanding of what threats you will face or have faced. This is why I think it's really great having records of all attacks that your company has ever seen, because it's incredibly valuable helping you defend against future attacks. But in Sophos' case, they knew exactly what threat was coming next, and were a hundred percent prepared for it the moment it would be seen.
That's really slick. That's threat intelligence that's on a whole new level. But even after two huge rounds of attacks against Sophos firewalls and discovering four zero day exploits on them, The war wasn't over. The threat actors continued to develop more and more exploits for Sophos firewalls.
¶ Targeted Attacks and Victim Notification
Yeah, over time the threat actors were increasingly they were targeting specific organizations or specific groups.
Yeah.
Uh they you know, they had identified who all of the customers were in those early attacks because they they smacked all of the firewalls at once and grabbed some data.
Oh my gosh, I didn't even think of that. So if we back up and look at the way all this has progressed, first they hacked into CyberRome, only to get the source code for SOFO's firewalls. which gave them inside information to basically bug hunt. Then they infected eighty thousand SOFOS firewalls with malware, taking all their configurations and information about the firewall itself, and then combed through that, looking to see what targets are interesting to them,
and now they're being super precise about who they're hitting. This campaign keeps evolving.
From twenty twenty one onwards, it really pivoted towards a very sharp focus to discriminate attacks, you know, really highly targeted hands on keyboard attacks um against like specific entities. So for example, government agencies, critical infrastructure. research and development organizations, healthcare providers, everything from kind of retail through to military, even finance, you know? And again, all focused in the APAC region.
What a nightmare. I cannot imagine all these places getting hacked into through my security device. All these companies bought SoFos firewalls to protect themselves, and it was that very firewall which allowed Chinese hackers in. At some point did you reach out to some of these victims to say Hey, I think Chinese government is attacking you.
So so that's one thing we did really extensively. Um well two things. One is we'd reach out to the customer. Um and again it was this was part of our philosophy of making sure that, you know, there was no further damage or no no hurt. Um and as well, we we would reach out to either the localized law enforcement or if we had great uh ties to the local, you know, CERT or NCSC or or Who whoever the the local cyber authority was. Now, in the UK we had some amazing connections in the NCSE.
And they would help us facilitate these these connections out to to all sorts of certs and bodies and you know they they were incredibly supportive of us.
Yeah, I mean what's that call like to call up a government uh a foreign government? Uh I know you're just talking to the sysadmin there, but still like hey, uh You guys are getting hacked.
It's it's pretty strange, you know, and and not only that, when we sit there, you know, obviously through translation very often, explaining what we've seen and what happened and who we attribute it to. It's a very strange experience, you know. Also not as strange as calling up another firewall provider.
telling them that their box is being tooled over by a Chinese threat actor and them asking us, well, how do you know? And not really being able to tell them how we know and why we know, but we definitively know. That's a bit of a weird experience also.
¶ CyberRome Product End-of-Life
At some point CyberRome gets hacked into again.
Well, it turns out that the Cyberoam code is the predecessor to the XG firewall code. So CyberRoam was the company that Sophos bought and their product became the XG firewall. So when back in twenty eighteen we're talking about how the threat actors had stolen the source code.
You know, they were using some of that still to find additional vulnerabilities. And they found a vulnerability. At this point, CyberRoam and the XG firewall were were in parallel operating, but CyberRoam was about to be phased out. It was about to be end of life. And the threat actors found a vulnerability that allowed them to create an admin level account on the box with just a SQL injection query that.
That was pre-authentication. So they could just they could just hit the the SQL server that was running on the firewall from the outside. and run a command that was able to get it to add a user with admin access. And then they could log in on any CyberRoom firewall or, you know, that they wanted to with that credential. Uh and there was no easy fix for it. And because the product was close to end of life, Sophis just decided to rush it to end of life.
and get everybody who was running a CyberRoam firewall to upgrade to the latest XG and put that one to bed because it was it was the point where If we had to start, you know, tracking attacks against CyberRome and XG firewalls, that would have taken the entire like all of the entire teams resources all the time. At a certain point it just made better sense to end of life the product early.
It does make me think though, if they were trying to get into CyberRoom to get source code, uh, they were probably trying to get into Sophos's network as well, trying to get source code.
I mean, yeah, I I that's an interesting thing to hypothesize about, but I have no idea about that.
Uh how you should say the sofos firewalls are so good that they're block those guys, don't worry.
Well, I don't work there anymore, so I don't have to defend them. But like I do think that you know, Sofos did have it did seem to have better security practices than CyberRoam did.
Wow, so after the threat actors found an exploit in the CyberRome product and were actively exploiting that, Sophos just decided to kill that product altogether. Now Andrew tells us it's because it was already on its way of being killed, but I don't want to diminish the idea that a cyber attack can have the effect of killing an entire product line. That's a pretty big deal, if you ask me.
¶ APT31 Attribution, Escalating Tactics
Anyway, somehow the French authorities investigated the cyber roam intrusion and publicly announced that the attack was carried out by APT thirty one, which is a Chinese state for the
Sponsored.
hacker group. So yeah, if it wasn't clear by now, it should be. The Chinese government and military are the ones who are behind this attack campaign known as Pacific Rim, which has been going on for years at this point.
we started to see these actors working on more and more attack types, especially um T Stark. You know, we we we found him working on like a a root kit at the time. It was called libxelinux dot so and we managed to capture it from his device and it was like a customized user land root kit. Um
So that that was actually a real win for us. I remember feeling like, okay, yeah, we we've really got a great view of what's happening on these devices here now. Now we managed to grab these devices from the T Start device. But like a week later then, he's got a completely new um like um injection there, like a new vulnerability um in WebAssembly and it's kinda unknown to us. And effectively what he was doing was he was
In this where
assembly um vulnerability, he he was injecting like an iframe into the proxy as things move through there. And um We found that this thing, like I think it was about two weeks or so after we found it, had actually been deployed in Tibet. Now, this was we found this on this device in Tibet for for an organisation that was basically providing uh support to Tibetan exile. So, you know, he basically moved from ten days to deploy
Yeah, and I can't remember which uh I don't know who said it. I feel like a president said something like, you know, a a business isn't going to be able to take fire from like a scud missile or a rocket launch. And so we can't expect them to be able to take on attacks, cyber attacks from the uh nation state actors as well.
And at at this point you're you're starting to feel confident that this is a nation state of thres attack on your company to disf to and at this point there's five or six different zero days. Discovered on you.
¶ Exhaustion and Cyber Warfare Reality
I mean, uh that's gotta be some of the most heart wrenching s gut sinking feelings t to say, Okay, I don't know how we're going to ever stop this attack. This might go on forever. Like like what is your response to this mentally?
Honestly, I remember at that point just feeling exhausted, you know. Like this has been months and months and months of fighting these you know, w what is effectively the the P the P L E, you know, for for all intents and purposes. And the truth is like who else help helps these organizations? That organization Tibet had nowhere near enough resource to be able to deal with it?
They were lucky that Velexity had been doing some pro bono work there. We'd reached out and helped them as well. But in reality, like if if it hadn't been for our graces,
They would have been stuck. And it really comes down to this weird intersection on the internet of lawlessness. Like like there's just so many areas that just are not covered with with anyone. I mean the UK, you know, we have the the the serious organized crimes unit and we have the NCSE who protects us in the US at the FBI and the NSA and you know.
many co countries just don't have anything. And this is the part that actually surprised me the most. Like who do these people call to? You know, we we we felt like heroes, but in reality, like, who who are we to deal with this? You know, we we're kind of woefully underqualified to deal with uh A threat actor at that level, you know, I mean this felt like almost a military operation. Yeah.
Suddenly your war room doesn't feel so at uh up to snuff, right? Like you're you're like, Man, w we're we're nowhere compared to their war room.
Exactly, like, you know. Um and I I think that's what what what what surprised me is like we were really on the edge of like what is effectively cyber warfare. And it started to really tip into that feeling with this. But it was it it was it was certainly interesting. And, you know, as a whole, you know, seeing that that payload being delivered there and understanding the purpose why they delivered the payload.
having seen it being built on a device in Shendu like like ten ten days, two weeks previously, it was just one of those crazy moments of like, Oh my God, like we really see this super
Now when Sophos would issue a hotfix or patch their firewalls, they would tell their customer what the update was for, like bug fixes for several security vulnerabilities. To learn more, visit our knowledge base. But Sophos discovered that the threat actors, T Stark and G Big Mao, were also accessing Sophos's site, logging in, and reading the knowledge base articles too, to see what got passed.
And they were reading exactly what Sofos had fixed and then developed exploits to get around those patches.
¶ Cat-and-Mouse Game: Disabling Telemetry
So the SoFos team had to get increasingly vague with what got fixed to avoid giving the enemy information.
And I suppose that
That's a form of counterintelligence. Being very careful what information you give your enemy, but it kinda contradicts what I said earlier about don't be evil, right? If you're not being transparent and you're hiding what it is you're doing, then you might be evil. But in this case they had to hide it because they didn't want their enemies to know this. This is so difficult to navigate.
And at that point, the threat actors understood how the hotfixes were working and what telemetry Sophos was collecting off these firewalls, and so they developed an exploit to disable the hotfixes and to stop the telemetry from going back to SOFOS to detect which devices were infected. and they took extra steps to hide their presence.
The threat actors are developing exploits and they're developing malware and they're uh coming up with new techniques for breaking into firewalls. And the implant is revealing all of that stuff to the security team. So behind the scenes, the security team is
Right.
And because they have this ability to send the hotfixes
you know, not necessarily to every machine, but maybe to every firewall except the ones that the threat actors are using. They can fix the whole universe of firewalls except for the ones that the threat actor is using. And I think After you've tried to deploy your second or third or fourth attack and it just doesn't work and you're scratching your head because it works in the lab, look, I can show you it I demonstrated it to these, you know, guys in the, you know,
higher ups at the company or whoever is telling me to do this attack. Um that it works and and but you know in in the wild it suddenly doesn't work. I think after two or three times of of shooting blanks, you're you're gonna start to wonder, like, hey, is there something else going on? And they started to look at, you know, well, what is this inform you know, what's a firewall collecting about us? And
Are we inadvertently revealing as as bad guys to the good guys what we're about to do? So yeah, so they start looking at telemetry, they start looking at log collection. and process lists and they're trying to build out the capabilities to be stealthy. Um, it's maybe distracting them from building custom malware. Or developing new exploits, but they have to spend a little bit of energy on uh, you know, it puts them on the back foot.
And for the first time I think this is like one of the cases where you can say, Yeah, there was some there were some challenges and we had some bad days early on, but uh you've we're forcing the threat actors to have to make moves to counter us. And actually that feels pretty good.
¶ Continuous Malware Innovation, Bootkit
Um tch, it's this this uh story just goes on and on. There was another root kit found. This is rootkit number four, libsopos.so.
Yeah. So Lipsofost was the the very custom uh rootkit. It was able to and and again, yeah, deleting logs, delete you know, hiding its presence on the machine. uh trying to do everything as stealthy as possible, uh low v volume of outbound communication, uh and persistence. Well they're they're experimenting with everything. And and the They've been g it it seems to me that the threat actors have been given carte blanche to just
Try and experiment with all sorts of different things. So during this period from 20, late 2020 to the end of 2022, we're seeing a huge variety of different payloads, of of exploits. It's bad. It's bad out there. Like the it's it's kind of like the Wild West and you never know where something's gonna come from.
🎵 Music
At some point they saw the threat actor was trying to develop a Eufie boot kit. This is malware, which infects the firewall at the BIOS before the operating system even has a chance to boot up.
you know, if you if you can get a boot kit into the UFE BIOS of of a device, um there's nothing that you can do in in the, you know, user land of the of the operating system. to remove it because it can it's it's running at a level the beyond which the operating system cannot reach.
Yeah, a boot kit like this would remain on the system even if you deleted everything and reinstalled the entire operating system again, since it lives in the part of the computer which loads before the operating system loads.
Uh this was actually kind of scary to find this uh experimentation happening on one of the threat actor devices. They they were really trying to figure out if they could get this boot kit to run on a firewall. And they they ended up breaking the firewall. Uh it didn't work. And uh after we discovered what they were trying to do, um the sofus engineers figured out how to, you know
changed the firmware on the firewall um at that low level so that it wasn't able to run. And they they implemented that in a in an update. Uh, but that's that's the scariest thing on all of this. I think the UFE bootkit malware on a firewall is the holy grail. It's where you you've got malware on a firewall and it can't be removed.
the firewall has to be thrown in the trash. It's scary. And, you know, we've already seen that there's been other firewall vendors where their recommendation was Unplug this box and put it in the trash because it is not safe. Um So it makes me wonder because we never get the details from other reports about what happened.
Whether this was successful with other vendors and whether they were testing this with us and it just failed because we were watching them and you know, stuck a wrench in the works just at the right moment and made it too much of a pain in the butt for them to keep trying and they just moved on to the next guy.
¶ Lingering Threat, FBI's Most Wanted
This was very much the k the kinda end of my involvement in this because I I actually uh left Sophos at this time and went to work for m the company I'm currently working for now, you know? But I mean, from that point I I kept in really close contact with my my colleagues who were there. Um and we were sharing intel uh as things progressed, you know.
But I mean, there were kind of two further published engagements, um, basically one in May of twenty twenty three and then one in March of twenty twenty four. And then it kind of came to a head, you know. Um Which I actually kinda w was kinda disappointing in a sense for me because I think I think very often that this stuff
hasn't stopped. I mean, the devices are significantly more secure now. Um, software's putting like an inordinate amount of time, effort and money into hardening the devices. Um I would actually hazard to say that they're th probably the one firewall company that actually is secure now. Um You know in in in all seriousness though, it's you know, I think it's one of those aspects of you know, you learn from your mistakes. I mean, Sophas being incredibly open and clear about this.
I mean kudos to them. I mean, uh, you know, being open about it and you know pub you know, publishing your mistakes and also, you know, publishing what we did and how we worked through this is is is super unique, you know, and you don't see any other firewall company talking but And we know for sure that this stuff was happening across a multitude of other devices. The tree says it's probably happening right now to some other firewall uh providers.
We just they just don't know. They don't collect telemetry. They don't have the hot fix mechanism that allows them to forward defend you. And uh yeah, it's an issue. It's still an issue.
One of the actors involved in all of this, we talked about him earlier. His name is, you know, use the handle G Big Now. Um, that we eventually figured out his real name. pictures of him. And the guy appears on the FBI's 10 most wanted list today. Uh his name is Guan Tianfeng, and he was the researcher at this company called Sichuan. Uh Sichuan secret.
Silence Technology Company.
Yeah, Siswan Silence Technology Company Limited, right? So This guy made it his career to break into firewalls and find vulnerabilities and then pass them off to people who would take advantage of them. For all of his efforts, he's in his early thirties. He has a ten million dollar rewards for justice bounty on his head and he can never travel. Outside of a non-extradition country in the world ever again without fearing for arrest and
extradition to the United States. And it just makes me wonder if if it really was worth it to him. In many respects, uh seems like an seems like a nice guy. He at one point he had his heart in the right place. So G Big Mao in his early days of working in this field. Mm message boards trying to get firewall companies to fix their stuff. Um, I can't imagine what happened to turn him, to make him break bad in this way.
It actually says in the FBI's cyber's most wanted poster that this guy hacked into 80,000 SOFOS firewall. And just because I'm curious, I took a look at a few dozen other FBI Cyber's Most Wanted posters, and strangely, I don't see any other person listed for hacking into other security vendors. So again, hats off for Sophos for taking this stride actor so seriously and getting them on the FBI's Cyber's Most Wanted list.
The story as as we published it finishes in twenty twenty four. Not because the attacks stopped, but because at a certain point you just got to put a pin in it and say, we're going to stop here because if we keep talking about this, it never ends because the attacks have continued. Ever since. Nothing has stopped. And if if there's anything to be said about this, is that the cadence has picked. Uh if it's broadened its scope, uh we're seeing every In very similar ways.
🎵 Music
A big thank you to Andrew Brandt and Craig Jones for coming on the show and telling us this incredible story of how SoFos got targeted by a Chinese state sponsored three.
Right actor.
This story is dang scary to me since the playing field is so unfair. A single company versus a superpower like China. And not only that, a superpower that's lawless and feels absolutely no shame from breaking the law. You'd think that after their main guy was arrested by the FBI they'd pull back and maybe apologize, but
No, no.
They increase their efforts and are hitting harder than ever against so many security vendors too. Hey, I really want you to become a premium subscriber to Darknet Diaries. All I'm asking is for you to buy me a cup of coffee once a month. This is my full-time job. This is how I make a living. If I suddenly stop making this show, would you be sad?
If so, then you probably find it valuable. And I hope you support things that you find valuable. If you become a premium subscriber, you get ad-free episodes, bonus episodes, and coming up later this year is a new podcast I'll be releasing, and you'll be the first to listen to it. Because it'll only be available to premium subscribers for a while. So please visit plus.darknetdiaries.com to support the show. Thanks. This episode was created by me, the lead firewall offender.
Jack Resider. Our editor is the port knocker, Tristan Ledger. Mixing done by proximity sound and our intro music is by the mysterious Brakemaster Cylinder. I named my firewall Linebacker because it's great at blocking and tackling. This is Darknet Diaries.
🎵 Music