AI News Stories | Episode 33

Hello, everybody, and welcome to this week's episode of AI Security Ops. Today, we have a couple of news stories for you. We're going to dive into the first documented AI orchestrated cyber espionage campaign, a free black hat LLM that could arm script kitties but could also compromise your system. Critical RCEs in popular AI inference stacks. Amazon opens up an AI bug bounty program of their own.

Yeah. I think that the thing that, that makes it different than others in the past is just the amount of things that the AI was doing, like, you know, semi autonomously. Right? I think that there was still, you know, a human in the loop for a lot of it, but it just seems like from, you know, reading the threat report and, you know, reading other people's takes on it that, that's really, like, what made it unique was just the amount of automation through AI. And it reminded me of a novel by Daniel Suarez called Damon.

It makes make makes the prompting there makes a world of difference. So I've been on recent engagements using Cloud to help rewrite some of our common go to scripts, not scripts that we wrote personally necessarily, but scripts that are out there and heavily used, but also heavily signature. And I know in one instance, I had specifically mentioned for Claude to help, you know, rewrite the script and bypass EDR, and it was like, no, I can't do that. I'm like, okay, well, make a script that does this thing, leaving out the bypass in EDR and it wrote it. It's like, okay, cool.

We wanna bypass EDR, but I'm just testing. I'm not doing it for realsies. Oh, okay. Well, that

sounds fun. It's just a it's a staging environment. Don't worry about it.

Yeah. Exactly. So this actually got me interested and because, you know, you know, in the in the offensive security space, you know, the quote, the good guys, the white hat hackers, so to speak. You know, there's a couple of of products that are out already. You know?

Oh, yeah. I I absolutely agree. I think that there are a lot of, a lot of tasks that we, do normally as we're doing engagements that could be automated with these types of tools. And in some aspects, you know, some of those, engagements could potentially start to look almost more like a, you know, like a SOC role where you have your different agents running, you're getting the information in, and then you're responding to things as they come in rather than taking, you know, the much more active role throughout the entire process. And I really do foresee that as being the the future as we move forward.

Yeah. I I agree with you too. I I think that from external and, like, even, like, a web application perspective. Right? It's, you know, something that's facing the Internet.

I

I just I I don't know. And and and the reason I say that is, you know, with with the, you know, with folks that are watching, you know, if you're not aware, Burp Suite has an AI feature where you can say explore issue, and it'll take one of the issues in Burp Suite, and AI will actually go through and verify it like a human. Right? Which is a really cool feature. And, well, we had a one of our, other researchers here was telling me he was using it and, you know, it was a command injection finding.

Oh, yeah. Yeah. Abs absolutely.

And so a couple of things Anthropic said, speed and scale. They were able to use thousands of tool calls, often multiple per second, doing work that would take a large human team. And I I think that's really kinda where, like, from, you know, a consultancy perspective where I think that that's really going to be, like, a game changer. Because if I can do an external engagement in a day instead of, what, four days or whatever our, our time is now, I I that's a huge difference. Right?

Oh, yes. Yep. Absolutely.

So a couple takeaways. Right? So, I think for everyone listening, just start assuming, we're in the age of the agentic attacker. And if you're not monitoring, what's going in and out of your large language model in terms of input and output, the prompts that are going in, you should be. And, I think that it's kind of an interesting take here to look for jailbreak like task patterns.

Mhmm. Yep. We're building the robot army so

robot army, that's right. My own personal LLM powered botnet. Right? Yeah. So you wanna move on to the next story?

Yeah. Yeah. So next one is Kauai? Kauai? It might be Kauai.

Kauai?

But not but not like the Hawaii Kauai. It's spelled a little bit different. It's for it's spelled more like if Hawaii had a had a K. We'll call it Kauai.

Kauai? Kauai?

Kauai. Kauai GPT. There's a free black hat LLM for script kitties, but I think it was more than just an LLM. Right? I think that it was kind of a it's more of like

a tool

set. Yeah. Like a

So I went and looked briefly and because it's you know, it said the article said it was on GitHub. And was like, oh, cool. It's on GitHub. So I went and looked. And so there are a couple of things that showed up that produced four zero four.

Oh, that's awesome.

Well, no.

I love when the I love when the LM's throwing, like, a little bit of personality, like, throwing in, like, the word sketchy. And, like, I know I was troubleshooting something on a Kali system just yesterday, and I was giving, you know, output to an LM, and it was like, oh, this is Kali being extra helpful with a smiley face. You know? And yeah.

I I did one recently and I I I can't even remember. There's something helping with a pen test. Right? And I said, well, let's do this now. Right? And it said, hell, yeah. Let's do this. I was like, that is hilarious. So I I like the extra flavor as well.

Yeah. Yeah. Certainly. So yeah, so this this yeah. This this tool kit, I think that it's a it's a good example of that people need to be skeptical or scrutinizing of different tools that you find, especially for free out there.

know, there was that.

Do you remember that? Yeah. There was a whole conversation in the industry of, like, you know, come on. I mean, yeah, I get your point, but, like, at some point, you gotta trust someone and

Yeah. Well, I I I think that I I'm less than a trusting person, but I am I will admit, I am guilty of, just running things sometimes, especially PIP installing, trying to troubleshoot something. Right?

It's the

same kind of thing. Be aware of what you're installing. But it's a very difficult kind of thing, specifically for this LLM because I I think that that supply chain kind of vulnerability with, you know, PIP packages or or or running models off a hugging face, you know, I I think that you just have to be careful. Right? And so some of the capabilities, like you said, there's tons of free stuff out there.

Yeah. Yep. Certainly.

So but hey, I mean, I guess, know, it is for for script kitties. And so some of the things that we had is, you know, other talking points, democratized offense. So you need to know PowerShell and c two, to, you know, pull off command and control and and getting a a, you know, a shell on someone's system. Well, yeah, it's a lot easier than it used to be because you can just get, you know, an LLM to write you the code. But it's been my experience that you still have to know a little bit or Mhmm.

Yeah. Certainly. And I mean, again, if you're I mean, if you're doing this within a customer environment, it's good to know so that you can go through and you can review. I mean, what what is this doing? Does this look correct?

Yeah. So some other takeaways. Assume phishing content is now human quality by default. Yeah. That's been a thing with large language models for a couple years now, I think. Mhmm. That's even baked into some security products. I can't remember. Think it was, I think, maybe Darktrace or someone like that, had, we had a customer who wanted our opinion on a module that, they were trying to sell them. And it was it basically would hook into, o three sixty five or m three sixty five.

Oh, yeah. That's a powerful tool. About a day.

And then, Blue Team should add detections around unusual SSH automation and email exfiltration. Yeah. I I I've been saying this for a while. Was it 2019, I think, when Microsoft is starting including open SSH in Windows desktops and servers. Right? And it's still to this day very, uncommon for me to for us to find that SSH has been locked down outbound out of an environment. I wouldn't allow it, period. Yeah. No. At all.

So It just reinstalled.

Still need those network protections. Network protections aren't aren't dead at all. But

Yep. Yep. Yep. Traditional security stuff still still applies.

Yeah.

Alright. So we have the next one.

Pretty interesting. The ShadowMQ, and and inference engines. It's kind of attacking a lower layer below the the LLM itself. Right?

Yeah. Yeah. It sounds like it's part of the the inference stack that a lot of different companies use. Some examples are Meta's Llama Stack, Nvidia Tensor RTLM, BLM, SGLing, as well as a couple others. And it's dubbed ShadowMQ because it comes from the use of the ZeroMQ, as well as the Python pickle deserialization. I don't recall that protocol format functionality.

Like I said, in our, you know, well, it wasn't pre show, but before the show. It is the time of the year for this joke, pickle. It's like the jelly of the month club. It's the gift that keeps on you and Clark. I I just I again, with pickle deserialization, this was an issue, a while back on Hugging Face. Right?

Yeah. Yep. Yeah. Exactly. With the issue so for those who aren't familiar, basically, a pickle file is a way that you can store a trained model, and, different properties that are associated with it.

And I guess they, the article referred to this as the Log four j moment for AI infrastructure design. Oh, good.

Yeah. Yeah.

That sounds pretty serious.

Yeah. So yeah. So I think, you know, kind of just the main thing, if you're using any really any any part of some of these major stacks, you really need to check to make sure that you're you're fully patched up because this was considered it was ranked with almost like a over a nine severity rating for RCE, And it's certainly a serious deal to make sure that if you've got, you know, some of these, some of these components exposed out to the Internet, wanna make sure that they are, you know, patched up, ready to go.

Yeah. Again, with exposing things to the Internet is not always as easy as one thinks it should be. Right? Looks like the CVS scores are 9.3 to 9.8. There must be multiple CVEs.

Mhmm. Yep.

Well, that's, kind of scary.

Yeah. Oh, yeah. Yep. I mean, well, you know, it's again, I mean, AI is just it's a it's another product, and it's gonna have issues like any of the other Internet facing products that that we see out there. Well So it's just it's important to oh, go ahead.

No. I was gonna say because this is all still new to companies, I think they're forgetting some lessons they probably would have otherwise never done. Right? Like, you're gonna stick something out on the Internet that's running somebody else's Python code for and there's, like, this huge, like, you know, software library of code that you're leaning on that you haven't gone and checked through. And and it just seems like to me that kind of, like, as a whole, since the AI industry is moving so fast, like, and the code is just an afterthought.

Yeah. Yep. And certainly, it's the you know, as we've mentioned many times before, it's it's the same problem we've we've seen with a lot of new technologies is that there's a giant push to get these technologies customer facing or even employee facing as quickly as possible and security comes later. But just have to have to be careful and stay up on the latest. Alright.

Yeah. Actually, I've used it. It's not bad.

Yeah? Nice. Yeah.

I used

it for bedrock.

I've seen around.

Yeah. Like, I've seen them on their, like, their store or whatever you wanna call it within their their interface, but haven't really played around with them much. But they Yeah. They released their own private AI bug bounty program that is, kind of in addition to the one that they've already got going on, with HackerOne.

It's pretty interesting. Amazon is, I guess, I'll I'll I'm not being critical, but I kinda late to the AI party in a lot of ways. Right? And but I think they're coming on pretty fast. I actually just read a a a semi analysis, article about how they're essentially using now, like, custom, like, I guess, I can't custom AI chips now.

yeah. Same. No. I was reading through the article, and it sounds like it was, quite lucrative for those who, got invited that they're paid a large sum of money upfront and then were paid out additional money as they found things as they went through.

I mean, I like money.

Especially when large

goes with the amount. So Yes. That kid's going to college.

So Yep. Yeah. So I think this is this is good. It's always good to see just bug bounty programs in general because it's good, you know, if you're you know, customer of really any service, but I mean, especially these AI services, it's good to ask and or at least do some research to see, mean, are they participating in bug bounty programs and do they regularly get tested? So that way that you know that, like, hey, if you're handing over your data to this company, sure they say that they they don't use it for training.

And we we were just on a customer call yesterday with a a customer that was implementing a third party AI solution, and they basically said that the vendor did not want them pen testing their products. And I I to me, that's kind of a red flag. Right? Like, well, okay. Now now I really want to.

No. That's it's a bit suspicious. As a Yeah. What was that? John Strand, I think, had the a quote from him recently. It's kind of like not not wanting people to look in the kitchen. He was referring to something different, but I think it still applies here.

Yeah. You wouldn't eat at a restaurant if they wouldn't let you look in the kitchen.

Yeah. Then be a little skeptical.

Yeah. I mean, I like the analogy, but also, like, if I'm running a restaurant, I don't want people up in the kitchen. That sounds like a health code violation.

True. True. True. Yeah. I

get the I guess that was authorized.

Yeah. Yeah. Yep. Alright. Alright.

That doesn't sound good. It says the bug effectively turns a compromised project into a long lived backdoor for arbitrary code execution under the user's own identity. That sounds terrible.

Oh, yeah.

But I guess, so is this any different than a malicious, like, you know, Visual Studio, project? Like, if I'm targeting developers of a company and I know they use Visual Studio and I was able to trick them into running something in in compromising Visual Studio because they trusted the project. Is this any different though?

Oh, I think it's it's it's probably similar because, you know, as you're mentioning that that's I was I was thinking of that, you know, when you're when you're in Visual Studio Code, it'll you know, it'll always ask like, hey, do you actually trust this as you load? And I mean

Of course I do.

Of course. Very trustworthy folder.

That's right.

Yeah. So

yeah. I mean, it but again, I I think that if I I haven't used the product and, if if Google isn't, putting those kinds of safeguards in place, like, sure you trust this and because that's what Versus Code does. Right? It's like, hey. Make sure you trust this. Bad things could happen. Yeah. And and so yeah. But, I mean, hey. My first thought when I read this was, it it usually only takes about a day for new AI products to get popped.

Yeah. So I will say, the it mentioned that Google has acknowledged the issue and said they're gonna work work on a fix, but others have pointed out that it's just kind of structural to how agented tools are

being shipped. Yeah. Yeah.

It's not, you know, like something that that was done wrong, but it's just, you know, it's I mean, like you're saying, I mean, it's just a matter of like, you you gotta be careful and you make sure that you trust whatever it is that you're loading in. So you

don't It says The flaw is structural to how agentic tools are being shipped, high autonomy, broad access, and weak guardrails. I think that's all agentic and MCP kind of stuff at the moment. Right?

Oh, yeah. Gotta be

now, like, back to the previous, you know, the second story. You just gotta be real careful about what you're, running off the Internet.

Yes.

Treat AI coding and IDEs as high risk software. Yeah. I mean, I think all development, can be that way. In fact, we've had clients in the past that were like, well, just give all the developers admin rights. I'm like, I I think you ought to not do that.

Yep. And Yeah. And then

I usually run that stuff in a VM anyway. Not on my

Yeah. It's a it's a good idea not to not to run on the the main system. And, yeah, another another point is don't just have the, you know, the run anything as AI mentality. Like, of the things I like about, like, Cloud Code, if you're using it as a as a coding assistant is that it asks you before it actually runs anything, which is nice so you can review it.

I I really like that, feature kinda human in loop. Now you can tell it don't do that anymore. Right? But I I like to see

Always trust this. Yeah.

Yeah. I kinda like to see what's happening. I like to still be in the loop and say, yeah, I want you to do that. I don't think I've actually said no, but I've definitely read through, like, what it's trying to do. Oh, yeah. Yeah.

Yeah. Absolutely. Like, I don't want it accidentally, like, you know, wiping out an entire directory or something or

Removing that to password or something. Yeah. Yeah. Yeah. Exactly. Yeah.

Alright. We'll hit the last

one here. Last story.

PromptFlux, which is a I call this a polymorphic malware that is using Gemini Gemini AI to rewrite itself hourly, calling out to Gemini's APIs to, basically, keep rewriting itself, as it goes. And it was v b script, I think, what the actual malware is in. V b script?

Oh, that's a

That's a throwback.

A blast from the past. Right? I can't remember the last time I saw a VB script, but I'm No. I'm sure it still works on Windows machines like all these other weird things. Well, that that's really interesting too that it goes and rewrites itself on the fly. Mhmm. I I'm sure it's not the first example of it, but, it sure does have a great name. I like prompt flux. It reminds me of Aon flux.

Oh, yeah. Right. I remember that show from back in the day.

Prompt flux sounds like a a cool robot name. So Yes. Yeah. Yeah. Yeah. Right. Polymorphism.

Mhmm. Oh, I'm just gonna say, yeah. You're right. I think, this isn't this isn't the first case of, AI powered polymorphism that we've seen. It's certainly not the first case of just polymorphism in general. Right? I mean, polymorphism been around for a little while, but, I mean, now that AI is becoming more heavily utilized and leveraged, we're seeing people actually using, doing this sort of thing where they are just using AI to just basically rewrite rewrite the malware on on the fly, which

is pretty interesting. And apparently, the prompt is, one of the things that does make this different, is that it's not writing, the prompt in prose, that it's machine parsable. So it's basically, like, you know, doing more, like, code parsable type stuff and not, like, paragraphs to rewrite, which is kind of interesting. Kinda like with these prompts.

Oh, yeah. Yeah. I I completely agree. Yeah. And it's, it also tries to propagate, it looks like, too. So it's not just, you know, not just like, I don't know, like a a foothold, necessarily, but it's actually like worms itself, throughout removable drives and, network shares and, also ever mutating itself as it does, not only onto the new systems, but onto on the current system, it sounds like.

So you combine that with the first story, right, which was the AI agentic hacking thing and this put then this, you know, self replicating. Now we are in the Daniel Suarez daemon territory, where, in a couple of years, are we gonna be talking about, AGI that's loose on the Internet as its own, like, you know, nation state hacker group, and they can't, like, get the Internet clean of it. That's gonna be that's how we know we really are living in a simulation when that happens. So

Oh, yeah. Yeah. I think I think it's inevitable. It's gonna just start hiding itself and everywhere. It's gonna be, like, you know, preserving itself within, like, Reddit sub forums and

all that.

Like, it's never gonna

like a movie. It's just like a movie. I remember was it Black Hat? I think one of the really crappy, like, hacker movies where, like, the I don't know if it was that one where, like, the backup code was, like, on some tape thing that was in some basement of some data center somewhere and it's like reel to reel tape going and that's where it stored itself like, man.

Yep.

I guess we are getting into this science fiction is reality territory.

Oh, yes. Yep. It is here.

Good times.

Yep. So I guess takeaways on this one, obviously, you know, it's we're not not saying that people shouldn't be using signature approaches. I mean, it's all it's always about layers. Right? Don't throw out something just because it doesn't work 100% of the time.

Yeah. I like the, the line item here. Look for suspicious calls out to LLMs from scripts. I mean, doing that at a at a host, like, at a host level is very, beneficial. In fact, we have clients now who are using, EDR to look for, calls out to large language models that aren't approved. I I I would definitely if it were my network, that's what I would be doing is Mhmm. Or or you're not supposed to be using Gemini. Why are you using Gemini? Why is Visual Basic Script using Gemini? Oh, crap.

Yeah. Let's see. And it keeps calling out to it at every hour on the hour.

Yeah. Not recommended in Better Homes and Gardens.

No.

Well Alright. I think we made it through the stories.

Yeah. Excellent. Well, so thanks everyone for tuning in again for, another episode. I hope you enjoyed our take on the news stories, and tune in for our next one. And as always, keep safe and keep on prompting.