Questions From the Community | Episode 38

Welcome to another fun and exciting episode of AI Security Ops with your illustrious hosts, myself, Joth Theyer, Brian Fuhrman, doctor Brian Fuhrman, should I say, and doctor Derek Banks. In fact, we'll all be doctors today. Why not? I'm not a doctor.

Doctors all around.

That's right. Doctors all around. This episode is brought to you by Black Hills Information Security. If you are interested in AI assessment work, please visit us at blackhillsinfosec.com. Click on the contact us page, and we'll see if we can help you out.

Alrighty. Could someone extract training data through model inversion attacks, and how realistic is that today? The answer is yes because that's basically the definition of a model inversion attack in which you are, sending a bunch of queries, to the data or to the model and getting data back and using, that the response data to make inferences about the training data that was used. A lot of times the examples that you'll see this on are actually image classifiers, and so essentially what they do is they'll just start feeding in a bunch of random pixels and what they might get back is, let's say, a facial recognition system and, they'll get back maybe some kind of a probability of how well those random pixels match something within, the database. And so then they just keep modifying the pixels until they get closer and closer matches.

So, I'm curious, though, about one thing here. I think, Brian, fundamentally, you're right because you are extracting the predictions from the model, which is which is fantastic. This is what models do. But are you extracting the original training data? That's a slightly different question. I guess that's not the actual question being asked.

So I guess that's one of the things that, like, one of the fights that I try and fight is that, you're not really storing data inside of a machine learning or AI model, right? You're basically creating math that will predict the data, right? And basically, if you looked inside of a model file, it would just be huge matrix of numbers. That's what it would look like to you, right? Like this big, just blob of numbers.

Right. But actually, you know, sort of going digging a little bit deeper, you know, these attacks are useful because do they enable you to reconstruct a facsimile of the model? Then the answer would be yes, they do because you're extracting those inferences. And we are welcoming Brumwin to the show. Brumwin, welcome.

Hey, Brumwin.

I'm sorry. I didn't realize you were already recording.

That's okay. That's alright.

You can just you're like the Brady Bunch. You just jumped in, and we're we're we're we're about to break out in song story about a lovely lady. Exactly. Okay. Let's go with the next question. Derek, do you have a question up in your question list?

Actually, the one I picked out is actually kind of close to what Brian said. Not the same one, but a similar question, so I'm gonna skip that one. I'm gonna go with what kind of telemetry should be collected for detecting prompt abuse or API misuse?

That's an interesting Prompt so let's deconstruct that question a little bit. Prompt abuse. I'm not sure what the listener is talking about when they talk about prompting. Maybe we can make the assumption prompt injection.

Right? Yeah. I think that's what they mean. It's like basically people are trying to get our models to do our model to do something unintended. How do we detect that?

Yeah. And the other issue that's slightly related here too, and I've asked I've asked some, customers this on occasion, you know, are logging around AI model use, especially if you have a chat interface to that model? And the answer normally is is no. And there are instances where logging can actually potentially get you into trouble. Right? Yeah. I mean, we know that there is such thing as too much logging in our industry as I

mean, it all depends on what we're talking about, right? Like, I don't think that ChatGPT should be logging things centrally for what I'm putting into it because I we pay them and they say they're not storing those things. If it was an application in an enterprise setting, maybe that's a different story. And so I was really talking about more of like an enterprise like setting type of thing. Like we have this critical chatbot thing or critical process that people are interacting with and we want to make sure that no one's, you know, how do we detect if someone's trying to abuse it?

Yeah, and I would also add to that question. It is often the case in most AI model deployments with regard to dangerous or, you know, even potentially, you know, malicious prompting, that guardrails are deployed around that model. And, you know, you as a, if you're providing a model in some sort of local sense that's backing some sort of application, you would probably be implementing similar guardrails around that model. And if you are, you should certainly enhance the logging on the guardrail aspect of the model. If it trips, make sure that you're logging that instance, and it would also give you a sense of how effective or not effective your guardrails are.

That sort of leads into my second take on that is, okay, now we have all this, like data. How do we determine whether or not it's prompt injection? I guess my response to that would be, well, is like, well, my friend, now you have a natural language processing problem. You're not gonna write like, traditional SIM rules to get yourself out of that one. You're going to have to essentially do some kind of NLP.

Right. So what was the second part of that question? We we covered the prompt injection, I think, pretty well. The second part was?

Or API misuse. So prompt injection or API misuse. And what I think they mean by API misuse is probably a little broader than prompt injection, and probably goes back to kind of our first, the first thing that Brian was talking about, about model inversion is that, I mean, without prompt injection, I could send theoretically, depending what kind of information I can get out of the model, I could send prompts and measure the response and the tokens out and do some data science y type stuff and essentially try and recreate the model. In fact, there is a suspicion that the Chinese did this with DeepSeek against OpenAI. Right.

Yeah. Oh, one of the other aspects of API misuse would be the denial of service aspect, of course. But if you throw a lot of traffic at an API that's connected to a back end AI model, that's gonna produce a tremendous amount of compute load because, you know, even though it's only inference, inference is still using the multiple processing units of a TPU or GPU. Right? So that's gonna run up electricity bills.

Yeah, I would throttle the API, but even, you know, let's say I was able to get right under the throttle threshold, the good news is is that, like, that kind of API misuse is a little bit easier problem than an NLP problem. That problem is basically looking at like, you know, standard deviations of normal baseline traffic, right? Like, why does this user have a million more queries an hour than this other, like the rest of our users?

Yeah, exactly. Actually, of the, this I'm I'm gonna give OpenAI a compliment, but I was looking into this the other day because for, coursework, I wanted to, put some constraints around how the API key was being used, and and OpenAI is actually doing a fairly reasonable job. They're putting they allow you to budget constraint. They allow you to throttle, and they also allow you, to dictate specifically which models that API key will answer to. So it's actually pretty cool.

Yeah. I think we have time for one more.

So I think we should throw this one at Bronwyn because she's been sitting there patiently just waiting and itching to answer a question. I can feel it. So somebody

Oh my goodness.

And if Brumman has the questions document up, she can, ask the question.

So the the question is, have you seen instances of anyone implementing secure phrases almost as safe words to protect against things like prompt injection and confusion attacks. So this concept of a safe word I haven't seen regarding prompt injection, but I have seen it come up in discussions about how to protect against deep fakes. So I'm actually not going to answer that question. I'm gonna pivot though to the fact that this concept of a safe word as a human centric defense against deep fake attacks is something that I am seeing show up much more often. And it's coming up in a variety of different conversations because at the end of the day, if we don't have a human to human means of verifying what, is sometimes called proof of life, then these deepfakes have gotten so good now that it's almost impossible to detect, at least without doing an in-depth, insane forensic analysis.

This is

something Give us a context.

A context.

Yeah. Where would this apply?

Okay. This would apply. You're you're working for a company, and in order to prevent a chief financial officer from releasing funds based on a phone call from someone, There would have been there would have had to have been an out of bounds previously set up safe word that you know so and so treasury officer or manager comes in and says, I need x thousands of dollars payable to this bank account. And the CFO turns around and goes, Okay, what is the code word of the day? And the person, if legitimate, would hopefully have said code word.

And if I could pick a safe word, that's danger. Right?

Well, no. I could

Stranger danger.

My safe word. Has anybody seen Bert Kreischer's birth stand up?

Like I could certainly see a scenario, as Brahmin describes, where maybe a company every month, let's say, let's just pick a period, publishes, a random list of words that are common, that are indexed somehow, actually, in a paper mechanism and distributes it out.

Aren't you call you're you're basically talking about a one time pad. Exactly. Exactly. So

And and there's

a lot of precedent new again.

There's a lot of precedent for this. We've we've seen all of the the scenes of the movies

where World two and so

and so can't access this and such security system, can't press the the big red button until they enter the correct security code that's kept in an acrylic thing, and and it's on paper.

Reminds me of the the Navajo called to code talkers. Right?

So so Brian Brian has a comment, and and I have one as well. The key there is out of band. Right? It's out of the communications channel. But but, Brian, what's your comment?

Oh, yeah. Well, I'm just gonna say I know that at, the previous credit union we were at, we did there was actually a code word that we had. It didn't rotate, but it was the one that, like, you picked. And anytime we called, like, in addition to other account information we had to give, you had to give code word. It was actually yeah.

English is not And But it's very sad that it didn't rotate.

Pick one

that's not safe for work.

Well, I you know what? I think we've covered the waterfront today. I think we all do need to run off and do other things, unless anybody wants to try for another question, which I don't think January's

not supposed to be this busy. I I don't and it's not. Yeah. That was It's cold that it wasn't gonna be this busy.

Alright. Well, thanks again to my illustrious host, Brahmin, Derek. Everybody wave. Brian, doctor, doctor, doctor. I hope you've enjoyed listening to this episode of AI Security Ops, and we will be seeing you net next time. Keep safe and keep prompting out there. See you.