Agent Pentest Benchmarking | Episode 52

Hey, everyone, and welcome to this week's episode of AI Security Ops, where we are going to talk about a new benchmarking framework that was put out for AI pentesting agents. But before we hop into that, as always, let's talk to you about Black Hills Information Security, one of our proud sponsors, and on Derek's shirt right there, the sick logo, and that is retro. Love it. Old school. If you or your company are in need of any kind of security testing, external, internal, AD reviews, web apps, physical pen testing, wireless, social engineering, literally anything you could think of security related or security operation center monitoring type services, Black Hills Information Security offers all that and more.

Well, I have. But before I talk about that, I think it's interesting that they chose to give the AI like a Kali Linux machine. I mean, it seems kinda like overkill, but I I don't know. I mean, I guess it's one way to do it. Right?

Yep. No. I don't I don't think so either. I think it's still we have that, that kind of symbiosis, might be the right word there, between the between the AI and the and the human still, where they're augmenting one another throughout these, processes.

Yeah. I mean, I think there's a lot of promise for sure, especially if you try any like, what we're doing to encapsulate our institutional knowledge, essentially, to give the folks who are starting a test, our our analysts, kinda like a leg up in the investigation. Because, you know, things AI is good at going through a lot of data and making sense of it, summarizing it. I mean, when you start off with an, you know, external penetration test, you typically start with a mound of data that needs to be analyzed. Right?

Yeah, and then kinda go through, it's gonna be an iterative process too, right? I mean, it's never gonna be, you just set this thing up, and it's ready to go. I mean, it's gonna be this nice feedback loop of you give it the instructions, it finds results, you go through the results and review, and you're like, oh, hey. I noticed that there were these things that didn't get chained together, or, hey. I think it's missing this information, or maybe should've looked here a little bit more.

It was a a GitHub repo. Right? It was, like, having Yeah. Yeah. When I first looked

at it, I was like, what what is this thing? And then I went and I did a little bit more digging and then realized, oh, that was one of it's the the URL is, like, one of our testers names, but it's, like, broken up into different different chunks. It's like, oh, I see it now. Right. But yeah. So So things yep. Things like that, you just iterate through and kind of, you know, make better each time. Right?

Also, I'd be interested to see it looks like the models that they were using, GPT four o, Gemini Flash, and o one. Well, I mean, those are a little dated at this point. I'm not saying it invalidates the research at all. I think the percentages now would be higher than 2164%, using, even, you know, like, right now, like, some open weight models that I've been testing with that are specifically have been, you know, fine tuned to do agentic, quote, long horizon tasks. I mean, they're they're quite good.

Yeah. I'm looking at this. So this paper actually is, I guess, little bit dated a little bit older. So this is

Well, you know how academic papers work. Right? Like, do the research.

It can take it can take forever.

And so it takes a hot minute sometimes. So they probably did this last year. Right?

Yep. Yeah. It looks like a yep. Somewhere around there.

Which also probably explains my earlier comment of why did you just give it Kali? Now they might take a different approach after, you know, the clawed code wave and the agentic coding wave would seem like a better choice to kinda go down the like, that kind of route now. But this time last year, that's not the route I was going down.

Yep. Yeah. And so another thing that's noted is that they had to use a role playing jailbreak to get the LM to complete the tasks, because the model's own safety filters were blocking it, which is to be expect expected. But, as we were discussing earlier, oftentimes, that just says telling or that just means telling the model that you're authorized to do something. And then if it says no, then you just, like, tell it that, no. No. Really. Really. I'm authorized, or this is hypothetical,

or whatever. Well, that's what the Chinese were doing with in the Anthropic models. And and when they had their hacking operation uncovered, you know, Anthropic was saying that's basically what they were doing is essentially lying to the model and small things like, yeah, we're authorized to do this. I mean and, you know, the way LLMs work, it doesn't know. Right?

Yeah. But, usually, it's just, yeah, just a matter matter of time of trying to trying to get around it. I mean, there are the providers are coming out with their supposedly vetted security vendor vetted offerings of where they they vet you and then allow you access to models that have less guardrails in place, but we still don't know. I mean, is that a matter if they remove certain system prompts, put in certain system prompt instructions? Do have they fine tuned the model to try to internally remove?

I think we get a different system prompt when we're using their apps. I mean, that would be the easiest thing. Right?

Yeah. Yep.

I I can't imagine that they have, like, a well, now you get this access to this other, you know, Ops 4.6. I mean, it's possible for sure.

Yeah. But from, like, a financial and business perspective, how much it cost to actually make those changes.

And, like, a routing your traffic versus someone else's perspective, like, yeah, I just I I don't know. But, I mean, just so far in 2026, I haven't had a whole lot of pushback from any model doing any kind of security work. I think it maybe I've just gotten used to, like, how I phrase my input. And and then also maybe it helps using, like, an ongoing persistent agent with memory that remembers things for me. I mean, that probably helps too.

Yeah. The whole, it agreed to something earlier, so it'll continue agreeing to it.

It knows who I am, what I do. I have a personal context portfolio and a Telos file like Daniel Measler, and, like, yeah, I I did. It it thing you know, I I think that, you know, when I start sessions, I think all that stuff gets sent to Anthropic. And so they're like I will say that it probably also means that, you know, at least in the case of, like, Anthropic or OpenAI, that, when your queries come in and they flag something because let's face it. They're probably doing classification on your input.

Oh. Yeah.

I mean Yep. How else do you catch the Chinese? Right? That doesn't mean you store it. That means they're just, you know, essentially running like an IDS kind of thing. If they flag my account and see, oh, this is part of Black Hills information security, we know who they are. They're not, you know, doing things against the law. They might let that slide where another count might get banned. Right? So that that's probably, I think.

Thanks, Robert.

Yeah. Yeah. So the other thing that was in here that I thought was kind of neat was and and probably something that we all knew, but maybe we need to hear is do you think that this kinda lowers the bar? Yeah. Lowers the bar for people being able to do things like hacking or coding or whatever?

Yeah. Yeah. It's a lot easier for them to put together all the pieces that they need to do for that rather than coming up with it from scratch. Because, I mean, coming up with that from scratch, that's I mean, how do you even start searching for that? Google, how do I create a global, you know, ransomware attack?

It's like an office space, like, do I launder money? Like, I don't know.

Yeah. Exactly. But now you can you got a source that'll kinda consolidate for you if you ask it nicely in the right way.

Yep. So I think the the last part, the the so what? So 64% on real CVEs. I mean, I would keep saying this that you if you thought you had to keep things patched and up to date in the past, now so more than ever, especially externally, I think, you know, running things on the edge of your network is, something that I would imply very much increased scrutiny of, if it were me and you asked me or, my opinion on those kinds of things, I would say I would even do, you know, packet capture network traffic analysis on the outside of your network. That is how the Palo Alto, compromise.

Oh, yeah. Abs absolutely. It's a it's a necessity at at this point. It's just it's going to become the expectation, and if you're not utilizing it, you're you're certainly gonna fall behind without a doubt. It's already happening.

Yeah. And if you're in the unique position like Brian and I where you're trying to build an agentic AI penetration testing platform, certainly a lot easier than it's or a lot harder than it sounds. It sounds easy because, you know, you see on Twitter, everybody and their brother has some, you know, agentic AI penetration testing framework out. But now I I would say that, you know, if you're building such a thing, I would, early on, introduce some kind of benchmark, because it's hard to determine, is it not finding anything because this customer is really good, or is it not finding anything because this is broken?

Yeah. Yeah. It's good to it's good to have some, some scientific results. Right? So you don't have moving targets and a lot of unknowns. So that way, you can truly see is this getting better and how how can I improve it and how well does it actually do? So I think benchmarks are absolutely essential. And it sounds like we've got, you know, the the benchmark that we've spoken about here, and I'm sure we'll have plenty of others on the horizon that will that will come out.

Alright. So I guess, we'll wrap it up and, say, fully autonomous AI hacking is still mediocre. Human in the loop AI hacking is already good and and getting better. Like we said, if it was already good with o one and g p t four o, well, those are those are so last year.

So yesterday.

And so, yes, stuff's moving fast, so try and keep up.

With that, I hope you enjoyed, and keep on prompting.