Niv Braun on AI Security Measures and Emerging Threats

Hello, listeners. And welcome back to another thrilling episode of data driven. In today's episode, we delve deep into the fascinating and, let's be honest, slightly terrifying world of generative AI and security risks. Joining us is Niamh Braun, co founder and CEO of Noma Security, who's on the front lines of keeping your AI driven project safe from digital mischief. So grab a cuppa and let's get data

driven. Well, hello, and welcome back to Data Driven, the podcast where we explore the emergent fields of AI, data science, and, of course, data engineering. Speaking of data engineering, my favoritest data engineer in the world can't make it, today. But we have an exciting, conversation queued up with Niv Braun,

who is the cofounder and CEO of Noma. Noma is a security firm that focuses on effectively he'll describe it more eloquently than I can, but effectively thinks about security in the context of data and AI across the entire life cycle. Welcome to the show, Niv. Hey, Frank. Happy to hear you, bro. Yeah. It's good to have you. And and security is one of those things where I've been thinking about more

lately. Right? So my background was a software engineer and, you know, software engineers historically have not thought of security. Then I made the transition into data engineering and data science, and, traditionally, security is not really at top of mind, for them either. Now I kinda look at this, and I kinda look at the landscape that we're in where enterprises are deploying LLMs, generative AI solutions, on top of the predictive AI solutions, fast and furiously, and not thinking about

security ramifications. So what are your what's your take on that? 100% agree. I think that, it's even like the the the the current, like, timing is even more fascinating than the than just, like, a new technology. Because exactly like you said, like, Frank, like, we all like the data practitioners. We all know that, like, security is not, like, our top priority. And by the way, like, by, like, like, this is, like, how it should be. Like, we are focusing on the business and, like, drive,

like, drive, like, the business forward. And this is why we're, like, this is what we're paid for. The problem is that because we're not, like, in this kind of, like, mindset, we also, like, like any technologies in the company, also, like, create some risk. What we see right now is the LLM drive, which is pretty cool, is that for the first time, the security teams started to put

the focus and, like, the spotlight on the data and AI teams. Because until now, let's be honest, they were focusing only on the software developers and their SDLC and the CICD and all these areas. Like, we were, like, you know, like, in the shadow. And we were, like, able, like, to act like exactly like, like, like, completely freely as we wanted.

But now when, like, the security team start, like, to put the spotlight on the data and AI teams, what they understand is that it's not only this new kind of LLM threats, but also all the basic principles of security are not implemented in the data engineers and the data science teams. Nobody, like, scans all the code in our notebooks, for example, unlike the software developers that, like, all

their code is being scanned. Nobody helps us to find configurations in our data pipelines or our MLOps tools or our AI platforms, like Databricks, for example. Like, nobody, like, provide us this ability to to find it easily, unlike, again, the software developers that they receive all this coverage and everything. Like, on the moment that they have, like, the smallest misconfigurations

in their SCM or their their CICD, they will immediately, like, receive, like, a notification, like, helping them exactly, like, how to secure it. And also eventually, like, in the run time, in the runtime, in software life cycle, in classic like software application, we also have a lot of API security and web application firewalls tools that help us to protect the application in the runtime. But now specifically in LLM, this is, like, very

related also, like, to what you said. Like, there are new kind of adversarial attacks, all the prompt injection and model jailbreak and stuff like that. And, again, nobody, like, else would like to protect it, like, in real time. And I think that this is, like, one of, like, the main shift that we see today in this area. We understand that the spotlight moved to the data and AI teams, but we need to make sure that we

do, like, both. Like, we start with, like, a new kind, like, trendy, like, risk that we want to make sure that we are protected from. But also that for the first time, after a lot of years, we're starting also, like, to implement the basic security measurements needed in our area. But the most important thing, of course,

is to continue and, like, do it without slowing us down. Like, we need to make sure that, like, everything, like, all the different, like, security measurements that we take still provide us the ability to move fast, to enable the data sent the data science and the data engineering teams to continue and, like, innovate, but in a secure way. You know, that's a good point because I never thought about scanning a notebook for errors. Right? Shame on me. Right? Like for code

security I mean, not errors, but, you know, security vulnerabilities. That's not something that I have seen done in practice. I mean, the the closest I've seen where security has been an issue for anyone in this space is, basically using protected, you know, Python libraries, right, or or Python library repos, right, where they're those are scanned by, I forget the name of the 3rd party that'll do it where you just basically say you point your Python instance to there. Yeah. Because

I also think that Internal Artifactory. Yes, exactly. So like, what, because I often wonder, you know, people just like to install. God only knows what's in there. I can tell that, like, it already, like, happens. Like, I don't know if you heard, but for example, like, like, like, pretty recently, PyTorch, for example. Right. PyTorch that we all know was compromised. We all know and love. We're most people

love. It was compromised. Like, specific version of PyTorch, a malicious actor succeeded to to put some code inside that basically, collected all the the the secrets and token that you have in the environment and sent it to DNS. Now we all know, like, how much like like, how many downloads, like, PyTorch have. And most times, where PyTorch is downloaded to through, like, to all these different, like, notebooks, wherever they be, JupyterOps, SageMaker, Databricks, like, we all use them.

And it I can tell that, like, it caused us to a lot of, like, problem. I can tell, like, like, like, firsthand, like, we saw, like, a lot of organizations that were compromised because of this attack. And it happens all the time. And by the way, if you mentioned, for example, like, if you already, like, touched the point of, of open source, now you have also Hugging Face, which is completely different area. Now it's not only Open Source packages. It's all these different Open Source

Hugging Face models and Hugging Face datasets. And there, all these internal artifact are completely useless because they don't even scan these models. It's completely different technology, completely different, like, heuristics in order to find it. And, therefore, you start to see kind of, like, trends for for the attackers. They started to upload a lot of backdoored and a lot of malicious models

into Hugging Face. I can tell you, like, we personally, we already, like, detected, I think, almost, like, 100, back or the malicious models, on Hugging Face because it's a wild west. Right. Because how do you because these these models, first off, they're physically large files. Right? So that there's that's a factor. Right? I don't know how Hugging Face makes money. I'd be curious to have someone on the show talk about that. But, you know,

they're doing the service. And, how would you even scan? I mean, that's a good question. Right? What types of vulnerabilities have you sent have you found so far? And how does one even scan, like, a safe tensor or g file? Like, how do you what's what's that look like? Right? Obviously, I'm pretty sure, you know, McAfee antivirus doesn't have a thing for that. But, like Exactly. But, how do you even do that? I'm just curious. Yeah. So this is, like,

exactly, like, the problem. Like, it's even, like, in in in the models, like, it's even, like, a a more, like, the the risk there, like, is more, like, clearer because as you know, a lot of time, like, these models in hanging face are even, like, in pickle. And, like, pickle is, like, by design, like, insecure, like, file. And so binary dump, right, of, like, the memory space. Yeah. Like, in the deserialization process, like, basically, you can, like, put, like, any kind of, like, malicious,

action that you'd like, that, like, the attacker can. So we see, like, different attacks. Like, most of the attacks come today, like, from pickle files. Some also, like, not even, like, in the deserialization process, but also, like, in the model code itself. For example, like, if you ask for a specific example, like, share something that we detected, like, recently. We found, like, a very, let's say, a popular, open source, LLA model that we all

know. But we know that, like, a it has a lot of, like, different versions. And one of the version was actually a docker

that took the original model, wrapped it up with few lines of code in the model, which what they did is that every input to the model and every output from the model was also sent to the attacker, which basically just received full visibility and observability to all the runtime application and production. So, like, all the organizations that, like, use this model. And performance wise, the data scientist, of course, they cannot, like, detect it because performance

wise, it worked perfectly because it took the original model. So nothing to be suspicious about. If we want the data scientist, every new open source model that they like, like in Hugging Face, they'll start, like, to open, like, these files and the binaries and, like, to start, like, to looking, like, in their own hands, they're manually

for, like, a for a for risk. First, like, of course, like, we understand that this is not their expertise and, like, it it like, we want to be secured, but, like, like, even, like, worse, we just spend all their time on security. And I think that this is, like, the worst stuff. Actually, it's not the worst. I think that, like, the worst, and this is also, like, something that, like, I saw recently in several

organizations is just, like, to block everything. Organizations that, like, understand, okay, Hugging Face model, it's, like, true, like, a secure, like, in secure area. Let's block it. Let's say, like, to all the data scientists in the organization, you're disallowed to use HAG interface model. I think this is, like, the worst. That seems like a mistake because because the people are gonna find a way. Well, 1, where you can't stop the signal. Right? That was a line from, a movie.

They can't, kudos if people know who that what movie that is. But, you know, if you block Huggy Face, people are gonna find a way around that. They're gonna put it on a thumb drive at home and then bring it in. So percent. This is, by the way, also, like, what you see, like, with this kind of, like, internal Artifactory. You see that, like, once you get to you you create for the r and d or create for the developers or for the data scientists, you create some level of, like,

friction. They will just find a way out to, like, bypass it and to to lower this, this friction. Right. So so couple of questions. One, I've seen, improper naming Not improper naming, but but basically using, names, like, that's looks similar to what should be. Yeah. Type will split. Type type splitting. That's it. I've seen that, which is kind of, I guess, kind of, you know, dollar store approach. But also, how does how does it if you wanted to look through these model files, as

far as I know, they're just I just looked at them. I just see binary stuff. Like, how would you look for malicious code in there? Because I think you're right. That's not a skill set the average AI engineer or data scientist would have. Yeah. So, basically, like, you need, like, to manually kind of, like,

parsing it because, like, you have, of course, like, the the binary file, but most times, it's not only, like, the binary file. You label for, like, the the code file that run, like, run the model, and you label for, like, the, in case it's, like, pick a, like, the deserialization process, that you can, like, parse and then, like, to see, like, the code there. But then you

need also, like, you know, like, you have, like, 2 phase. 1st, you need to to parse it, you know, like, to see, like, the code, but then you need also, like, to be able to read code and to understand which one is valid and which one is malicious, which is also, like, completely, like, you know, like, you need expertise in this area. If you see bash commands, is it okay or not? Do you see access to the

Internet? Okay or not? Like, you you need, like, to have, like, some, like, detectors in there that, that know how to do it, like, build by by expert or something. So how would you even detect that if you found it? Like, how was this found? Was this just somebody looking in network packets? Or, like, what how was it discovered? I'm just curious. Yeah. This specifically was, like, by our

security research team. Okay. Yeah. That's like, looks a lot if, a lot like all the time, like, you know, all these different kind of, like, open source and third party models in order to to help our users to make sure that, like, everything that they use is is valid. And again, most importantly, without slowing them down. They can just, like, download and, like, run, like, with everything that they

that they want. And in case, we see something that is, that is suspicious, we know how to detect it and to to help them to to secure it. Interesting. Interesting. Because I know a lot of people, you know, they they've been downloading these models from Hugging Face. And just taking it on faith, and I've heard that these things don't call out to the Internet. Mhmm. And I fell into that. And then I kinda had this moment of paranoia where I'm like, how do I know?

I mean, the only way I'm a I'm just a humble data scientist. Right? Like, so the only way I would think about it would be to have a firewall rule that would block network traffic going up for that box. And I'm sure there's probably workarounds to that too. I mean, are these attacks are these attacks that sophisticated yet? Yeah. Yeah. And, like, also, like, most times you don't, like, the data science, like, they don't want, like, to permanently, like, to close, like, the Internet, like,

the outbound because also, like, the application needs it. And also, like, the, you know, like, the the in order, like, to download, like, the dependencies and the models you needed. So most times, like, just, like, to block the Internet, it doesn't solve everything. It was, like, more, like, in the past that everything was, like, network based only. Today, when you have, like, also, like, the applicative layer here, so it's, like, a bit more sophisticated.

But yeah. Wow. So the safe tensor format, as I understand it, what you know, you basically digitally sign or somebody digitally signs the contents of it. Is that is that a correct understanding? Yeah. So it's end up like a like, in general, first thing, of course, that, like, a safe denture is, like,

much more secure. Okay. I already like by design, and as long as we as the industry will go, like, more and more, like, towards this road, because today, like, we still see, like, tons of light pickles. But as long as we progress, like, all as an industry, we'll already, like, be, like, in a bit better situation. It's not

perfect, of course. We still see some issues. And, of course, organizations still need, like, to have some security measurements and processes to make sure that, like, they're aware of what, like, Hang in Face are using. But I think that it's already, like,

going to be a bit better. I can tell you something that, actually, like, recently one of our one of our partners told me, which was pretty cool, very similar to what you said that you start, like, to feel a lot of concerns about this area. VP data science of a very big like, Fortune Fortune 500, like, very big, like, corporate. And you kind

of, like, the the head of, like, the older data science, like, groups here. And they told me, you know, Niv, I I already know what I'm going to be fired about, like, in a in the next, like, 24 months, and it's going to be about that. I know for sure, like, we're using, like, so much, like, Agiface models. I know for sure that I'm this is, like, the reason that I'm going to be fired, like, one day. Because today, like,

we're using it, like, freely. We are also, like, very creative. We're not, like, only using, like, the most popular LAMA model, but, like, we're to, like, take advantage of this great advantage of the platform, which is, like, the amount and the diversity of the model that you have there. But I have no no doubt that we create so many risks that we're just, like, not exposed yet, that I'm going to to pay with it,

like, with my head. So it it's it's pretty cool because it's not it's not always that you see, r and d and business owners that are so concerned about security even before the security team arrived to them. But they're already aware of this risk. And it's something that we start, like, to see more and more because, you know, it's just like it's it's too obvious. Like, the the the window is open and everybody see it.

Yeah. I I would suppose that's in in a in a very kinda strange way that's bit progress, right, where people think about security beforehand. Like, even if they don't know I mean, I think this this this VP, you know, is pretty spot on. Like, what concerns me about the widespread adoption of these models and particularly Hugging Face, so there are no knock on Hugging Face. I think whatever you get your models Mhmm. I mean, we just don't know. And these things are just complicated.

Right? I mean, they are by design complicated with 1,000,000,000,000 of parameters. In some cases, I guess, 1,000,000,000,000. But also, you know, they have this ability to even even if everything worked out well, even even assuming everything is fine, right, in terms of the operationalization of these things, There's still the chance that the model itself and its training was poisoned. So, like, I I mean, like, there's just so many because when I my wife works in

IT security, and I was all excited. It was about a year and a half ago. I I was talking to her about LLMs and stuff like that and chat GPT and and and those types of things. And I was like, oh, well, you take all this data and you train a model and you you distill down this graph and this and this. And then she's like, that sounds like a big attack surface to me. Yeah.

And I was like like, data poisoning in the classic one and data poisoning can be, like, in in in 2 levels or, like like, someone like poisoning your data or exactly what you say, somebody just, like, this way, like, create backdoor in, in third party models and open source models that then, like, everybody downloads. Right. Right. And we wouldn't know, like, what's the I mean, the defense against that seems very intricate. Not impossible, but very delicate and intricate.

So in in in classic application security, there is a great practice called SBOM. SBOM is a software billing of material. Basically, it means that, you get, like, in specific format, kind of like visibility to all the different software components that build your application. One of the things that now we're also, like, part of the building is a official framework of OWASP, the nonprofit organization

around security of AI and machine learning. And what you have there is for the first time you have like double layer of visibility. The first one is just like to understand what models I'm even using in the organization. Everything, like what models like, include in my application. It can be open source models. It can be self developed models. Also, by the way, not only not

only LLM, of course, also like vision, NLP, like everything else. And also third party models that are embedded as part of the application, they are not open no. They are not open source. For example, if software engineer add API call as part of the application to OpenAI, in this way, they embed

LLM as part of the application. This is also like one of, like, the models that you are using, but you you you want to know this is all my AI and model inventory that I'm using as Spyro as part of the application. And in addition to that, you have even the deeper context there, which

is also like what you referred to. It's not only this is the list of the model that I'm using, but for each one, you want to understand on what dataset it was trained, what data maybe also like it has access to in case it's in production, let's say, with RAG architecture. You want to understand, like, the deep context

of all these, like, models, what I'm using, but also, like, what happens, like, in this specific, like, model. Sometimes it's, as you said, for to to understand what data was trained on a model before, like, I'm starting, like, to use it by 3rd party, a lot of time is even, like, internally in the organization. Because once we start to train a lot of models, we want to make sure that we don't violate any policy that we have in the organization, either it's for compliance or

security. For example, one of the things that, like, we are like, I keep, like, hearing a lot of time from, from security and legal and privacy teams is that, look, we instruct all the organization not to train any sensitive data, PII, PCI, PHI, any other sensitive information on our models. But except instructing it and speak about it, nobody knows if it happens. And we don't provide also our data teams tools that will help them to

detect it in case it, like, it happens, like like, not in purpose. For example, I can tell you, like, one of the thing that we saw very recently. Big organization, a huge Fintech company, that data scientist unintentionally trained all the transaction of the application on one of the models. Now it's a, like, crazy big violation there of, like, compliance and security. The data scientist did this unintentionally. They

truly, like, didn't know it. If they had something that, like, would help them, like, the basic visibility that you mentioned before, it will truly, like, help them to start, like, to continue, like, innovate and just, like, in case something like bad happens, to be alerted in that. And so I see that, like, the the data training is also, like, very, very important point also internally and not only the external data train on the external models that we're embedding and

downloading. So you mentioned, OWASP. So just for the benefit of folks who may not know, because most of our listeners are either data engineers, data scientists. What is OWASP? And what is the

I think it's with the OWASP 10? Yeah. So OWASP in general, it's a amazing organization that, is like a nonprofit one that helps basically, we combine a lot of people together, gather together in order to make sure that all our industry is much more secured with a lot of different security initiatives in a lot of different aspects, mainly of like product security, but not only. Product security is like application security. It's building security.

Specifically in OASP, you have several different types of projects. So for example, one type of project is the OSP10, top ten, that basically takes different areas and define the top ten risks in this specific area. So it can be top ten for API, top ten for CICD. And now there is also like top ten for LLM.

Addition framework, like, there are a lot of like different tools. Specifically, if someone wants to understand a bit more about like the wide landscape and the risk around AI and machine learning, the framework that I would like recommend on, highly recommend on, is amazing and very comprehensive called the OWASP AI

Exchange. A group of people, again, gathered together, that covered not only LLM, but all the basic principles and risk in data pipelines and MLOps and start from the building and up to the runtime and start from the classic machine learning and up to Gen AI, very comprehensive, very also practical, which is very important and speaks in both language, on both languages. On one hand, of course, security, but on the other, also like very oriented for data and machine learning and AI practitioners.

Interesting. Interesting. What what do you see well, here's what I mean, I'll have a lot of questions, but one of them is, do you think the 0 what do you think the 0 trust approach is a good starting point? I don't think it's the answer here like it is kinda everywhere else. But do you think that, that type of philosophy of don't trust anything? Right? Kind of like, I mean, is that because you you mentioned this early when I talked about network firewalls, right, where the old approach of thing

is just pull the plug or set up rules. And that used to work, but there's plenty of other ways around it, Both I think kind of low skill, mid skill, and certainly high skill ways around that. What do you you mean then 0 trust is meant to address that. What are your thoughts on like I mean, is that the pro is that the mindset that either security folks in this space would have to take on? Like, it's more if they well, they probably already have. Right? Yeah. I think you're,

like, I think you're actually, like, the the you you you perfectly defined it because I believe that 0 Trust is exactly like you say, it's kind of like a, like, kind of like a mindset. It's not like a very accurate, like, technical approach, but it's kind of like more like a a philosophy with some level of implementation.

I believe that, like, the right mindset and, like, the right framework to look on a on a security for AI and, like, all the building and also, like, the runtime is basically to take all the different principles that we are all already aware of. Like we are all, like I'm saying, like the security industry, we are all already aware of on classic software development, building and runtime, and to implement it on the

data and AI lifecycle. For example, if we mentioned, like, code scanning, so code scanning the notebooks, we mentioned open source, so checking all the all the Ag interface models. But it's not only that. For example, one of the things that, like, we see, a lot of attacks that we, like, we had recently in the security area are around the

CICD. A few years ago, there was a big attack called SolarWinds, that basically, yeah, so you know it perfectly, just for the audience that, like, are not familiar with the specific details in, like, very high level attacker that exploited and misconfigurations in CICD tools. And this is basically how they succeeded, like, to start, like, this whole huge attack and breach. Now one of the things that, like, it taught us all as an industry is that until now we were focusing on, like, securing only

our code. Now we understand that the code is not enough. We need to make sure that the building tools are also well configured. So we start, like, to see a lot of, like, tools that help us to make sure that we don't have misconfigurations in the CICD and the SCMs and all these different kind of tools. But when we are going to our domain, when we go to the data and AI teams, as we know, we just use different

stack. We use all these data pipelines and model registries and MLOps tools and platforms like Databricks and Domino and Snowflake and stuff like that. The configuration, as we know, is not like neverwhere. Most time, it's even wider. This is why it's not managed by DevOps. It's managed by us, by the data teams. It's managed by MLOps teams, by data infra, by data platform. And we're doing a lot like, a great job in order to optimize all the configuration for the

product. We're not security experts. We don't want to be security experts and, like, start, like, to spend a lot of time in that. But nobody else just like to very easily find all these different kind of misconfigurations. And this is also a threat and, like, attack vector that we started, like, to see a lot in the field today. I can tell you that, like, we see tons of attacks around different misconfigurations in tools like Airflows and Databricks

and stuff like that. And I think this is also like a very, very important, like, mindset, like, to be in. And in addition to that, of course, we have all the all the runtime and all the adversarial attacks there. There are specifically, if I mentioned in the OSPI exchange, so OSPI exchange covers everything. The OSPI 10LLM specifically is more covering this LLM, like,

specific risk. And then you have, like, all the adversarial attacks, like prompt injection and model jailbreak and model dn out of service, model dn out of wallet, etcetera. So basically, the mindset should be we already know security very well. We already have, like, these principles. Until now, we just haven't implemented them on the data and AI teams, tools, and technology. And this is exactly what we start, like, to what we, like, need, like, to start to do. And this is what we see

also that, like, you know, like, now we have no reason. Like, we all see, like, these different kind of attacks. So we start to see that all the organizations were, like, starting to to already, like, walk the walk. Wow. Yeah. I I often wonder too, like, what you mentioned the pipelines being a vulnerability or an attack surface. Right? Like, or a potential vulnerability.

I often wonder now, like, when, you know, we're looking at agentic AI, right, where these things aren't just LLMs, right, producing text or going through these materials. We're giving them, you know, abilities, right, to influence pipelines, right, to to or to whatever. Right? Like, that just seems to me like a giant security risk. I mean, telling someone you know, there's there's multiple ways to break an LOM. Right? Like, obviously, there's the the the $1 Chevy

Tahoe. Right? Where the guy did that. Right? Pretty low tech approach, pretty brute force ish. But I often wonder, like, well, what what sorts of things are agentic systems gonna open up? Like, what does that look like? I think that this is exactly like where we

we will start, like, to see, like, the very big LLM, breaches, that we'll have. I believe that, by the way, my belief is that the the how does the attack start will still be, like, in a lot of cases,

very similar to what we see today. But the impact of the attack will be much, much, much, much, much higher because now like the model cannot only like, promise you a $1 a car, but you can throw, like, I already like send the order, can send the car to you, can like book your hotel, can do like everything there, can share with you, like, the data of maybe, like, other customers in the application because it is, like, a RAG architecture, and it is also, like, different, like, tools

that provide him the ability to maybe even, like, write different codes to the application. And then it might also like start like different types of remote code execution. As long as we are going to provide to these NLMs more privilege, more access, more tools, more abilities, the impact of the risk

that they will be able, like, to cause will be much higher. I still believe again that that pack vectors are going to start from more or less, like, the same areas, like prompt injection and model jailbreak, but they they eventually, like, the outcome of these attacks will be much higher. I could see that. Because we're giving them actuators, so to speak. Right? Like we're not we're we're

giving them agency. Right? Like where they could actually do real damage as opposed to because one thing in saying you're gonna give somebody a $1 Chevy Tahoe. It's quite another to actually place the order, sign off on the invoice, and then ship it. Right? Yep. And what if you'll do, like I don't know. Like, you'll you'll start, like, to see it also, like, in banks and in investments. They will start, like, to transfer

your money. They will start, like, to invest, like, to buy stock. They will like, the the the the amount of, like, potential impact here is, like, a crazy high. I believe, by the way, that eventually, this is going to be one of the things that, like, we'll see also, like, slow down the adoption, not less than the than the technology or, like, finding, like, the right use case. Yeah. No. I could see that. I I just think that we're just setting, as an industry.

We're setting ourselves up for a huge exploit that we haven't figured out is already there yet. And so so what what can AI engineers, data scientists, data engineers do today to make things better? I know we can't fix it because we don't know what's we really don't know what's broken. I think that's one of the frustrating and kind of fun things about security work is, like, it's not that there's no vulnerabilities.

You haven't discovered any vulnerabilities yet. Right? There are no unknown there are always un there are always unknown unknowns. But if you have an unknown unknown or a known thing, you can you can say that you pretty much figured that out. But there's this whole aspect, which I don't think data scientists fully appreciate. I think they can understand the concept of the unknown

unknowns. But in terms of the consequences of it, I don't think I think it's gonna take 1 major solar wind style issue or CrowdStrike style issue to make people conscious of of that. But how do we how do we prepare ourselves? Right? You can't stop the hurricane, but you can board up your windows. Right? Like, you know, how do you Yeah. I and I totally

agree that, like, what's going through, like, to to shake every everybody will be, like, the the first SolarWinds or, like, the 4 log 4 j attack that we see, like, in these areas. I think that, like, I think that you broke it very well and that we need to relate to both categories. 1st is, like, the known, which already, like, exist. Like, we know that, like, you know, like, we see that as scientists. Like, we are not a scientist.

And we see that one of the the things that, like, we see in in in our code in compared to software developers is that we don't give a tip on, like, everything, around security. Like, you'll see, like, tons of exposed secrets in plain text. You'll see tons of, like, test and, like, the sensitive data just like playing. And, like, it's state, like, exposed, like, in the notebooks. You'll see that we download, like, any dependencies without, like, like,

even, like, think about it. Even so that, like, yeah, it looks like maybe, like, a bit suspicious and stuff like that. So it's it's far from from the basic. Let's make sure that, like, what we know that is not best practice, just, like, start, like, to implement it. And then regarding the unknown unknown, so, of course, like, you don't know how to handle it. I think that, like, as you as you said, you can start to prepare yourself. How do how do you

prepare yourself in security? It's basically to be very organized and to to make sure that you have, like, the right visibility and governance. As long as you have, for example, like, you know how to build, like, your your AI or the machine learning bomb. You know all the different, like, models that are built or embedded as part of the application, and you have, like, the right lineage, which one was trained on which dataset, etcetera.

Once, for example, that now let's say we'll continue with the examples of of Hugging Face. Like, a new Hugging Face model is is is now, like, published as a like, someone, like, found that it's, like, malicious. You because you prepared yourself and you have, like, the right visibility, you are able to go and very easily search exactly, like, if you use it and where you use it in all your organization. And this is also

because you prepare yourself. This is exactly what happened, like, in Log 4 j. In Log 4 j, it was like a dependency that found as a critical vulnerable. And a lot of organization, what they spent, like, most of the time is to try to understand where they even use this Log4j. And they seem that, like, if you prepare yourself, you are like, if you are organizing everything, you'll already be very, very, like, ready for the for the

attack of, like, the unknown unknown. And, of course, everything in addition to to, you know, like, learning and, like, educating yourself. If you start, like, to understand, you'll go to, I don't know, Databricks, for example. A lot of people use Databricks. You'll go and, like, start, like, to see what are, like, the best practices of how to, like, configure your Databricks environments and what are, like, the best practices

there. It's something that you can, like, find very easily, like, in the Internet. You don't need, like, to to do it, like, from scratch. But I'll say that, like, you you know, like, it's still, like, when we are aware of that, it's not still, like, the the top of our mind as the data practitioner to start looking, like, in our free time for this kind of concept. Right. I mean, that's a good point. Right? The fundamentals are still fundamental. Right?

You know, making sure, you know, you track what your dependencies are. Right? So that way, if there's a breach in a hugging face model, like you said, you'll know right away whether or not it impacts you or not. Also too, I think you're right. This isn't top of mind for AI practitioners. Right? Even when I code, like, an app, my met my thought process are very different than when I'm in a notebook. Mhmm. It's just different wiring.

Yep. And by the way, it's kind of like, it's kind of like a paradox because most times on the notebooks, we are connected to much more sensitive information than on our ID. Right. No. Exactly. So it's kind of it's like the worst, one of the worst case scenarios. Right? And and you're right. Like, people wanna work with real data, and they they just assume that if they're on a system that's secured and internal, they they, they don't have to worry about such things,

which I think you're right. Like, with these systems that have access to sensitive data, these pipelines, I mean, it's one of those things where we need to start thinking about this. And what would you do you think that there's a, like, a career path for, like, an AI security engineer? Right? So it's not just a security engineer, like, in a traditional sense. Right? But also a someone who specializes in AI related issues. You think that's a growth industries? I

have, like, no doubt that we are going to like to see more. Like, we already see these kind of practitioners in the field. I have no doubt that it's going, to be more and more frequent. And in addition to that, I believe that, like, even in the future, it's it's going to be even, like, several different, like, roles. For example, one of the things that, like, a lot of people that we work also, like, very closely with are AI red teaming. Right. It's not even,

like, just like a AI security engineer, like, general one. Specifically around, like, credit teaming because all these kinds of adversarial attacks on models are very different, requires different techniques, different tactics. And the red teamers are the ones that, like, to, like, learning all these different types of adversarial attacks and how to, like, check your model,

in your organization. And by the way, specifically in this area, I do feel that it's kind of, like, top priority and like top of mind also for the data science team. Like you do see that on LLMs, once they are deployed into production, the data scientists, they are kind of like understand that there are a lot of risk there and they are starting, like, to take also, like, responsibility even completely, like, regard regardless of the security team to make sure that, like, we we

we reduce some of the risk there. Now the risk is not only security. The first thing is security, like, to try and, like, make sure that you are secured from all these different adversarial attacks or that you know how to detect sensitive data leakage, for example, as part of the response and stuff like that. In addition to that, it's also a lot of time

like safety risks. You want to make sure that once you deploy LLM into production, your model doesn't give any financial advice to your customers, doesn't give any health advice in case it's not your business. So you then have, like, these kinds of responsibility, or example, like in the Chevy example that you gave, that you just, like, you don't just, release free cars or flights or books or a tail off, like, anything

like that. So I think that because the the the the amount of potential risks are so high on the run time. In this area, I believe that, like, the data scientists already understood that this is, like, under their responsibility. They see it also as part of, like, being a professional data scientist. If I deploy this model, it has, like, a lot of, like, accuracy, but, like, it creates all these different kinds of risk.

I would define myself as not a super professional data scientist, unlike on the supply chain, unlike in the notebooks that if I code a code that is not secure, I wouldn't say that, like, it's not professional. I would say that, like, it's okay. You're just, like, focusing

on the business. So I do believe that we start, like, to seeing this shift also, like, in the mindset of the data scientist because of the risk of the Gen AI, but now it's also, like, like, a move to to all the the development and the building practices that we have. Yeah. And I think data scientists are acutely aware that LLMs are just taking they mean, we talk we we call it hallucinating when they get things wrong. But realistically, they're

always hallucinating to a very real degree. Right? It's just they happen to be correct. And what these things are doing under the hood is they are looking for patterns of words. Sometimes those patterns of words are wrong, obviously wrong. And sometimes they may give out sensitive information inadvertently. So I can talk at least at least there's some common sense out there when they when they do realize these things are higher risk than I think

we've been led to believe. Yeah. Actually, I love this this finish. They are, like, hallucinating, like, all this time. Sometimes they really find it as wrong. Like, they do the same thing as always. Right. Right. The they don't know they're hallucinating because they're just operating normally. And so when they go in a different direction and I've noticed that, you know, kinda like a little bit of, you you know, off by a little bit, and then then then it generates an off by a little bit, off

by a little bit. I ran an experiment with a hallucination, and I read it through I ran it through a bunch of models and each one of them didn't do any fact checking, which I mean, realistically, I wouldn't expect that. Right? In the future, I think that'll be kind of table stakes. But, you know, it would just go through. So I took a hallucination, fed it through notebook l m, which then

create even more hallucinations. Right? So it took this little genesis of something that was wrong and then made it even crazier wrong, which I think is an interesting kinda statement and and and also is a risk. Right? Like hallucination on top, compounding other hallucinations. And I don't think we've really seen that yet because we've only really seen for the most part, I've only seen one kind

of model in production. But if you have these models that will kinda work together as agents or, you know, whether they're agents that do things or agents that it's different LLM discrete LLMs that talk to one another. They can get things wrong and make things worse. I mean, I haven't I think it's too soon to tell either way, honestly. Yeah. But, like, the

like, theoretically, like, it makes a lot of sense. I think in general, like, we don't see, like, a lot like, we hear a lot about Gen AI. I think that, like, the level of adoption and the amount of business use cases that, like, businesses

found are not that high yet. I think that, like, the most of the usage today is done by, like, consumers, like, like, directly, like, from, from the foundation model providers, like OpenAI and stuff like that for day to day, like, jobs, like, you know, like, reviewing mails and stuff like that. The the big businesses are still trying to find these different, like, use cases. I do believe that the that the agents are going, like, to open a lot of different use cases

around it. Right. Right. I could I could see that. And I think I think it's just too soon to make a statement either way. But I think grounding yourself in the fundamentals is probably always a good idea. Mhmm. And probably a good a good approach. So so tell me about NOMA. What is is it NOMA? I I don't wanna make sure I pronounce it. NOMA. Okay. NOMA. Security. What does NOMA do? Is it security firms that focus on this space? You mentioned red teaming. Is that is that a sir service you offer?

Yeah. So NOMA basically is an like, our name is Nomo Security. The domain is Nomo dot security. So it's Oh, okay. Sorry about that. No. No. We're good. So, so, yeah, what we do is, like, secure the entire data in the AI life cycle. Basically means that we truly, like, cover it end to end. Like, we enable, like, the data teams and the machine learning and the AI teams, to continue and innovate while we are securing them without slowing down. And this is like the the like, we are built from, like,

data practitioner, like, the company. So this is, like, our main focus, meaning that we start, like, from the building phase. So if we said, like, notebooks and hugging face models and all these different stuff and the misconfigurations are on all the different stack and all the envelopes

tools and AI platforms and data pipelines and stuff like that. So we are connected seamlessly on the background, and, basically assist the the data teams to to work securely, without changing changing anything in the workflows. And then also, like, we provide, as you said, the red teaming. Before you're deploying the model into production, you want to understand what is the level of, of

robustness and security that the like, that your model has. And what we do is we had, like, a big research team that, like, builds, simulated, thousands of different attacks. And then we dynamically start to run all these attacks against your models, showing you exactly, like, what kind of, like, tactics and techniques your model is vulnerable to, and exactly also how to mitigate and improve it to be more robust. And then the 3rd part is also the runtime.

We are mapping, we're scanning all the prompts and all the responses in real time, making sure that you don't have any risk on both sides. The security, we are detecting all these different kind of, like, a host and a little, like, adversarial tax prompt injection, model jailbreak, etcetera. We check also the responses for sensitive data leakage and stuff like that. But in addition, also the safety. We see a lot of organizations that the data scientists, as we

said, they understand the risk of deploying models into production. And this is why not even, like, the security, but more like the the Chevy example and, like, the the health advice and stuff like that. So they built for their own, model guardrails in order to make sure that they are, like, controlling what are, like, the topics that the model is be able like, is allowed or disallowed to communicate about. And what we do is basically to save

them also like this time. We also provide them, like, all this runtime protection already, like, as a service. You can define exactly what kind of, like, detectors and in native language, what kind of, like, policies you want to make sure that are enforced. And then we also, like, protect it in the run time. So, basically, we just, like, cover you, like, end to end, start from the building and up to the run time. It starts from the classic data engineering

pipelines and machine learning and up to gen AI. Interesting. Interesting. It sounds like something I think is totally, I think, a needed needed service and and skill set. Because you're right. Like, I mean, there's just so many risks here, and the hype around Gen AI is so over the top. It is gonna be revolutionary, but maybe not in the way you think. Right? And I always call back to the early days of the dotcom. Right? Where it was pets.com. There was,

you know, this.com, that, you know, like all these crazy things. But the real quote unquote winner of, you know, .com was some guy in Seattle selling books. Mhmm. Right? No one no one I mean, selling books. Like, really? Like, not, you know, and it's it's interesting to see how I think I think that the the obvious use case for chat for for LLMs thus far has been chatbots. Right? Customer service type things. I think that's really only the

the the the the the surface of it. I think for me, what I've seen is most impactful is the ability for natural language understanding and their ability to understand what's happening in a in a block of text. And I think that that has enormous potential. I agree. A lot of risks too. Right? Because what if, you know, what if I I mean, to your point. Right? You wanna make sure these things stay on topic. Right? Like, I don't if I'm talking to a financial services chatbot and I say, hey, I have

my my leg kinda hurts. Right? It's, you know, the risk of moving into health care, like, it's just kind of, I don't how mature are those guardrails? Because I've not really seen a good implementation of it yet. Yeah. So, you know, like, I don't want to to give ourself, like, a compliment, but, we Oh, you guys are pretty good at it? Yeah. Like, we're pretty good. Like,

we were, like, you know, like, with fortune 5 100, with fortune 1 100.

Not in vain. But, yeah, I believe that in general, specifically, like, when we speak more, like, on the guardrail side, I see that the most important thing is to make sure that it's, it's building the right architecture to be very flexible and easily configure for the organization because eventually, like, you know, like, each organization is completely different needs, completely different context to the calls, like, in their customers, internally to their employees.

So everything should should be, like, very easily configured, but very flexible. Interesting. Interesting. I wanna I I could talk for another hour or 2 with you because this is this is a fascinating space. Where can folks find out more about Noma and you? I you think it's Noma dot security? Yeah. Noma dot security. Can't believe that's now a top load pain, but, and, any any, NOMA dot security, you're on LinkedIn, and, anything else you you'd like the folks to find out more?

No. I had, like, a great time speaking with you, Frank. Great. Likewise. And for the listeners out there, if you're a little bit scared and a little bit paranoid about generative AI and LLMs, then I think we had a good conversation. Because I think we need a little bit of that fear in the back of our heads to guide us and maybe think about security issues. A little bit of thought ahead of time will probably save you a lot of problems

later. And want to lose some. That's that's all I got, and we'll let the nice British AI, Bailey finish the show. Well, that wraps up another

eye opening episode of data driven. A big thank you to Niamh Braun for sharing his expertise on the critical intersection of AI, security, and innovation. If today's conversation didn't make you double check your data pipelines or rethink your Hugging Face downloads, well, you're braver than I am. As always, I'm Bailey, your semi sentient MC, reminding you that while AI might be clever, it's never too clever for a security breach. Until next time, stay curious, stay secure, and

stay data driven. Cheerio.