AI News Stories | Episode 36

Welcome to AI Security Ops, the weekly show where we cut through the hype and dig into how AI is being weaponized and how we can defend against it. This week's headlines range from prompt injection exploits that turn ChatGPT into a data leak tool, a critical vulnerability discovered in the n eight n platform, and malicious Chrome extensions siphoning AI chat histories, not to mention a classic a p t 28 credential stealing campaign that now leans on AI generated phishing lures. Fun times. This episode is brought to you by Black Hills Information Security and Antisyphon Training. BHIS helps organizations find real world security gaps before attackers do through penetration testing, adversary emulation, and purple teamwork.

Yeah. So Derek and I have both been playing around with this quite a bit as well as a couple other people I know at the company, Bo, has been playing around with this

a

lot. One of the baits we're still having is it n eight n or is it Nathan? Which I said to Derek the other day is kind of like the southern pronunciation of Nathan.

According to the n eight n website, it is n eight n. So

Yeah. n8n Alright.

That's what I've heard most folks on YouTube saying, but I do like Nathan. It's Nathan. Yeah.

It just kinda rolls right off the tongue. Nathan.

Well, I mean, is it Azure? Is it Azure? Is it Azure? No one knows. Azure. Azure.

If you want to be fancy about it.

Potato, potato. Can we call the whole thing off?

But this I mean, this the the tool though is awesome. It makes automating these different workflows or throwing together a workflow automation, I'll say, so much easier. I mean, could you go out and code this all up yourself? Certainly. I mean, you could go out, you could put together the discrete Python pieces.

Yeah. I mean, kind of similar in the same vein where I was using n eight n to actually take proof of concept Python code that I wrote for our class that does essentially a cyber threat intelligence, like, newsletter. And I replicated that literally sitting on the couch watching a basketball game in Nathan. Now I used the template and, you know, kinda changed things around. And but I was I think that your to your point, you can write the scaffolding or the glue code, whatever you wanna call it, around large language models to perform tasks other than chatbot tasks.

Well, it's it's you said it yourself. Everything old is new again. And over and over again with these AI implementations, I'm seeing that it's the same suspects, the usual suspects that we run into with any type of thing. If the the authentication is bad, if the RBAC is bad, if if there aren't clearly defined policies or or guidelines in terms of what it can access, of course, badness is going to happen. Who is really surprised by this?

Well, not information security practitioners. Right? I mean, the thing that I'm doing with with n a n at the moment is I'm literally allowing a a a a model access to tools that have running in a container with an API and then in some cases, like command line tools. Just those things in itself are, like, risky. Right?

Yes. Yep. Absolutely.

Anything else on the n eight n vulnerability, or shall we move on?

Nope. Tilt the next one.

Okay. So the next story is about ChatGPT's memory feature supercharging prompt injection. Researchers have discovered a zombie agent exploit, which abuses ChatGPT's new long term memory feature. An attacker will craft a prompt which stores malicious instructions in the model's persistent memory, then later trigger it via a seemingly benign follow-up query. The module dutifully executes the payload, leaking conversation history, or running arbitrary code on integrated tools.

Does anybody else hear the cranberries right now? I'm just wondering. Sorry. Yes. I mean, it's also how memory works. Right? I'm assuming some things I I did not read in-depth about this, but I'm assuming some things need to be true. Like, you send me a crafted URL. I have to click on it. I probably also have to be authenticated to chat GPT. Right? Like and so yeah. I get that, but I yeah. I I mean, the alternative is the term memory off and not use the feature.

Don't know. Which which we've talked about on a different episode. Honestly, that's what I do. Not because of the I didn't even have the security risk in mind, but I just I don't like it when it uses other chats as context for current chats. I mean, certainly, I mean, I've you know, within the context of a chat, I wanted to use my previous messages to guide future messages.

Yeah. I mean, so I'm kinda a little more nuanced with it. I don't run it on chatgpt.com, and I use chatgpt for more kinda like one off kinds of things. But I do have it running for Cloud Desktop. I really like Cloud Desktop, and so I want it there.

Please do go on.

Yeah. Please. Right? And so I I think there's a time and place for it. I think people just need to be aware and evaluate their risks. Right? I'm also someone who I I just I don't even click on links that, you know, Brian Fearman might send me. Right? I'm probably gonna copy

I understand why I wouldn't either.

Yeah. I mean, one time, I actually had John Strand send me an email, and it had thoughts, question mark, and then a link to a Word document. And so I went and downloaded the Word document in a VM and actually looked at it and made sure that it was, like, not weaponized. Right? And this is years ago. Right? And I actually called John and be like, did you mean to send me that? And he's like, oh, yeah. Want you to take a look. So I I guess maybe I have a little bit higher level of paranoia.

Well, that's the thing about being an info sec. I often feel like I have this split personality, like an a new Gizmo will come out and I go, oh, shiny. And it's the same thing. Oh, that hurts. You know, it's it every every new shiny toy has a downside, like the teddy bears that got released over Christmas that are that are AI enabled. Did you

see any That sounds fun.

Yeah. I haven't seen this.

It's just disastrous as you can imagine. So, I mean, like you, I leverage the memory feature judiciously. And a while back, I had it write up a biography of me for a potential podcast. And of and, course, I'm looking at what it says, and I'm I'm intellectually, I I'm going, I detect no falsehoods. But on an emotional level, it's like, who the hell is this person? That person sounds like they really know what's

going on. My response. I'm like, man. But I don't know. I I could look at the system prompt. I had chat GPT kind of flatter me today and and something. I I asked it about, like, to help me do something. It's like, man, what you're doing is serious and awesome or something along those lines. Like

God. I hate the semantic. I hate when it sucks up to me. The things that I do like about ChatGPT, you have the ability to have a temporary conversation, and those do not leverage any of the the memory features. So I'll use that if say, I have a one off.

I mean, don't I think data breach is a strong term here, right? Like, is it a ChatGPT data breach, or is it a like, I don't know. It's it's kind of like kinda sort of a data breach, but not really. It's like calling stealer malware a a data breach. Like, it's not like a whole bunch of data from a whole bunch of folks was taken from ChatGPT.

Well, the idea of the the whole the way they describe the whole zombie agent thing reminds me a lot of a persistent cross site scripting bone.

And I Yeah, like a stored cross site scripting. That's what I thought of when I heard this. But a whole lot harder to fix. I mean, it's sort of like a feature. I mean, that's like part of the problems with some of the issues that we've highlighted across the last couple months of this podcast is things like prompt injection is the issue and the feature.

Yeah. No. I mean, at this yeah. I mean, at this point, it's basically like saying fix social engineering. I mean, Yeah. Okay. You know?

Yeah. It's it's

With with web forms go ahead. Oh, go ahead. I'm sorry.

No. You first. Please.

Alright. No. I was just gonna say, yeah. I mean, it it's about, you know, blocking down the things around it, mitigating the actual risks, and mitigating the damage that come that come along with it, you know. But, yeah, it's at this point, I mean, it's just there's architecturally, there's not really a good way to to fully address the issue.

Yeah. I guess one thing that for this particular thing is, I mean, maybe not put things that are sensitive into chat GPT. I mean, there there is that. Right?

There's that.

Yep. I can't I don't think that I I typically don't put what I would consider to be sensitive information into chat GPT. Maybe a little bit more with clogged code and clogged desktop. I would consider that a little bit more sensitive, but that's not really applicable in this situation. Right? Different technology, different thing. So

The the thought that comes to mind for me is that when it comes to like web interfaces, input validation is usually fairly straightforward. But with an LLM or some other artificial intelligence generative doohickey, how do you validate the input? Like you said, how do you do that protection for social engineering?

That A little bit more porous of a Band Aid, Mhmm. But Yeah. I mean, you're you're basically gonna put another classifier in front of it to try and classify the text as, you know, malicious or not. And and and it's not like, I don't know that you can get away with the same kind of input. Like, I've I've done web app tests where, yeah, the the the web input or the, you know, the user input was highly restricted.

Yeah. I mean, that's that's been the that's that's been the ongoing battle with, I mean, security in general. Right? Sec security versus convenience. I mean, at some point, you have to be able to use the thing. Yeah.

Yeah. Because the most secure state is off.

Well, yeah. Exactly. Just unplug it and it's, you know, probably alright.

There there are reasons why I'm very glad I live in a remote area. If I wanna go off grid, all I need to do is flip one button and it's that is a very reassuring thought on a regular basis. So since we're talking about ChatGPT, the next story is also about ChatGPT. In this case, it's a new zero click attack. Radware's research team uncovered a zero click attack that abuses ChatGPT's agentic features.

Sounds like a fancy way to say indirect prompt injection. But that that's what it sounds like to me. Right? Is I I know that you have, I'll say, Gmail. You know, your your OpenAI account is reading agentically your email, so I guess it can give you a summary of your daily email.

Yeah. Yeah. Certainly. I'm pulling up the whole article over here and they're also labeling this as zombie agent as well. So I'm curious if this if this is related to the last one that we we just did because it it seems like two different things here because the other one is

But it

seems like there's overlap. Yeah. What's that?

One's a persistence. Like, the other one's more of like a persistence mechanism. This one is like more of like the initial like like, a different compromise vector. Like I said, zero click that you've got the email summarization, and you're just embedding the instructions within the the emails, which is, you know, been an idea that's been around for a while. I mean, we participated in that CTF, like, a year ago that Microsoft put on the l l mail challenge, which was basically doing this.

And the one that we are teaching in the fall in Deadwood. Oh.

That's right. Fall and also, I think it's coming up in February March? Or March, I'm not mistaken.

I think it's March. Virtual teach of it, I believe.

Mhmm. Yep. Yeah.

And so, yeah. I I don't think this stuff is going anywhere. This is probably gonna be a recurring thing when we do news episodes. Oh, look. It's the latest indirect prompt injection.

Mhmm. Yep.

Good. So the next story is that two Chrome extensions were caught stealing ChatGPT and DeepSeek chats from 900,000 users. This article was posted in the Hacker News. Security researchers found two malicious Chrome extensions that silently capture conversations from OpenAI's ChatGPT and also from DeepSeek along with regular browsing data. Once installed, the extensions inject JavaScript into the AI web UI, read the DOM, the document object model for chat text, and post it to an attacker controlled server.

I got got a I almost feel like that that that this isn't the story as much as maybe, like, do OpenAI and DeepSeek's websites not have or are they blocking third party scripts, like, in their content security policy?

It sounds

I like it's really don't know how this works. Like, I But would see that's actually, man, that's pretty sophisticated if we go after some chats. Right?

Oh, yeah. Yeah. I'm trying to look up what the what the actual extensions were. Chat GPT for Chrome with GTP five, Claude Sonnet, DeepSeek AI, and the other one was called AI Sidebar with DeepSeek, ChatGPT, Claude, and more.

Sounds useful. So they're acting like they're an extension to implement the AI within your browser. And of course, once you log in, now you've given away the keys of the kingdom.

Yeah. That's dirty.

Attackers are sneaky and So

are these extensions, were they actually like on the Chrome store? I guess they were.

Apparently. Sounds like it. Yeah. That they got that they managed to push their way through. I mean, it'd be relatively easy to do, I would think, with these because I mean, what what are they doing?

I was gonna say, anybody who's done web app testing in the last like eight or nine years, like when you go to a modern web app, the amount of tracking and and just scripts that run from third party resources is insane. And most of the web app tests that I did, you know, I spent half of the year, you know, doing web app tests last year for BHS. And I can't remember, like I I mean, it was, a consistent consistent finding for allowing third party scripts through CSP. I was actually able to take advantage of of one of them in a test. And so I just kinda feel like that maybe OpenAI and, I guess, DeepSeek folks, if you're listening, might wanna might wanna fix that on your site.

Yeah. Work tasks, whatever, whatever.

Yeah, I have a browser for work and a browser for personal. And then I have a browser for when I don't want either of those browsers touching what I'm trying to go after on the internet.

Yes.

I do the same thing. And even on my personal system, I have different browsers for different personas. I mean, all the fine online banking stuff stays in one set of browsers and there's other stuff that goes to other browsers and never the twain shall meet.

Yeah, I actually take it even a step further. I have a whole separate system that I do banking and financial stuff on that I don't really do anything else that touches the Internet so much. Like, I don't browse on it. I do some other things on it. Right?

Yeah. It it I feel bad for, quote, normal people because they don't they don't know the stuff that we know. They don't know that just because it looks shiny that no, these are are mean, nasty, awful people who are looking to steal your entire retirement fund. It's it sucks.

Yeah. I wonder how long it's gonna be though because like right now, my wife, she basically doesn't she has a work computer. Right? But at home, she doesn't really use a computer at all. She uses her phone for everything.

So I don't know.

I actually don't know either. I've never tried.

Never even considered that. Take the buyer beware approach when using any extensions. That's about all I can say. Alright. Ready for the the next and last story?

Yep. The obligatory APT story.

Of course. Russian APT twenty eight runs credential stealing campaign targeting energy and policy organizations. So this is another article from the hacker news. APT twenty eight, also known as Blue Delta, apparently launched a credential harvesting operation against Turkish energy and nuclear research agencies entities, a European think tank, and groups in North Macedonia and Uzbekistan. The group distributed phishing emails containing malicious Microsoft Office documents that embed AI generated spear phishing text, making the lures more consistent, convincing.

Yeah. I mean, look, we do this here at Black Hills. Right? Like, we we are currently using open like, in-depth open source reconnaissance to then use LLMs to create things like this. Right? I mean, I'd be surprised if every threat actor wasn't doing it. Right? And it certainly, you know, especially, like, with language barriers. Right? So what do you what do you tell people now about emails?

Yep. Yeah. And I'm curious of the so they mentioned the malicious documents here. I'm curious if there's macros that they had built in or what and, you if it's in this case, you know, another another point too would be, I mean, macros are disabled by default at this point. And as an organization, you can shut it off so that users can't even re enable them.

The only thing that truly that's AI related to this story is the fact that the, spear phishing text was AI generated. And I know we've talked before about how phishing campaigns have gotten more difficult to spot because of the AI generated text in the the phishing or smishing or vishing or whatever it is ishing that's going on. But other than that, it seems like the same old

You know, to bring it back full circle to to n eight n because that's what I'm here for. I'm pretty sure you could create a workflow that would go off and automatically run, you know, some tools to gather open source reconnaissance on a given individual and then have a large language model craft potential spear phishing roosts. Sounds like a fun workflow to create.

Oh, yeah. I like it.

All right. So, well, those are the stories we have. The bottom line is the and I've said this so many times. The AI genie is out of the bottle, and we're still all learning as we go along. As CJ likes to say, this is the worst that AI will ever be going forward. Hopefully, it will get better. But it has definitely changed a lot of games. And just be cautious. Be careful. Don't just arbitrarily throw sensitive information into any chatbot. Pause. Any other closing remarks?

I'll channel my inner jaw fire and say keep on prompting.

Perfect. Alright. Well, thanks everyone. See you next time.