Testing AI Vibe Coding: Stop Vulnerabilities Early with Sarit Tager

In a land of testers, far and wide a journey Seeking answers, seeking skills, seeking a better way Oh, the Test Skill Innovation Testing Podcast Testers with our nation awesomeness From ancient realms to modern days They lead the way Oh, the test is all our nation-testing podcast

Hey, today's episode goes deep into something I think is becoming unavoidable for automation engineers. And that is how AI-generated code creates new risk. And with that... new responsibilities. Whether using Cursor, Playwright MCP, Copilot, or internal AI agents, the way you build, test, and produce code is changing fast. And that's why I'm really excited to have Sarit Taiga joining us from Palo Alto Networks.

help them break down what testers and automation engineers need to know. Things like how vulnerabilities get introduced and how you can use Godrails to protect you before the code ever hits your repo. If you're building automation frameworks using AI to generate code, While working in pipelines that ship fast, this episode is packed with insights that directly affect your day-to-day work. You don't want to miss it. Listen up. It's all right. Welcome to the Guild. Hey, Joe. Nice to be here.

Awesome to have you. I'm really excited to have you on the show. You have a lot of experience and you work for a really cool company. So I think before we get into the topic, I think we're going to cover, you know, vibe coding, security, all those things that are really happening right now. Maybe a little background. How did you become like the VP of product at Palo Alto Networks? So I will talk a bit about my product and a fun fact where I started from.

So I'm the Vue product for application security within Palo Alto as part of the Cortex Cloud family. And we kind of use everything from code scanning to pipelines to ASPM to application security. posture management, super exciting subjects and very, you know, very forming one. A lot of AI is in place. The fun fact is that I came to this by being a developer first and then a VP engineering.

And I always say that I came to the application security and the cloud security area in a mission to make it better for developers and for appsec practitioners and for security practitioners. Generally, I do remember... a lot of nights sitting and trying to fix things, security issues as a developer. And I think this is part of what makes me become a product manager in this area.

And yes, I've been working for like a variety of companies from network side of things to cybersecurity of images. And even there was more like JFOG. So a lot of companies interested. Love it, love it. So, you know, one of the topics you said you wanted to cover was vibe coding. I guess...

People that know what is vibe coding, a lot of people think it's a joke word or is this a legit thing? How would you explain what vibe coding is for someone that might not even know? I would explain that this is something that, as opposed to... When we used to go to Stack Overflow and kind of check how to do things, you now have two things. One will be a chat that is actually helping you generate the code, and the one that actually generates the code.

And if you think about it, it's kind of changed the way you define a developer. Who is a developer? If I just write a prompt saying, I want an application that I know color things in blue. Is he a developer or maybe more a product manager trying to do requirements? So I think Vibecording in general really changed the way we think about developers of their day-to-day job.

And I think the most important stuff is that if I'm using a lot of AI-generated code, then the question will be, who is the one that needs to fix it from a vulnerability perspective? It creates vulnerabilities because it was... created by people and hence most of the examples they see within the wild are with vulnerable code. Another question will come if I do scan it and I didn't write it or at least I wrote a prompt.

to write it. Can I fix it? Can I as a developer fix it? And I think this is really changing the way we should think about security for developers and how they really need to fix their code or actually help them secure the code.

You just mentioned something that's interesting. I was just going to say, I've been using cursor a lot and even small apps. And then I go, okay, check this for security issues. And it finds a bunch of security issues after it wrote me the code, which I always think it's weird. So you said... One reason it's been trained a lot of code and a lot of those code had known security issues. So how do you get around that? Is that always going to be an issue when someone's doing vibe coding?

So I think the way to do it is to make sure it's being generated secure from the start. And trying to fix it after the fact is actually something, it's much more complicated. If you secure it, if you put the right... guardrails in place, if you put the right rules to be checked before that, then the code will be secure by generation and not just after the fact. And also, what I would like to say is...

Make me an application that doesn't have any critical vulnerabilities and not just create an application to do something. So instead of after the fact, change it to be before you actually create the code. So it actually could be a prompt issue then. So if you prompt it correctly from the start, then you're going to bake in security practices, I guess. Yes, but if you think about it, there are a lot of different things. For example...

I may become kind of a loop generation of code. I try to do something which is not vulnerable, but what I'm asking is it actually has vulnerabilities. So you have to be very careful in the way you define the prompt. I would say it's different between perfect and secure enough. I think this is where the security companies come into place and say, we know what is.

secure application, and we will help you bring in the rules to your AI-generated code. It will just say, you know, write something without vulnerabilities. I don't think I know any open source that doesn't have any vulnerabilities. Maybe something that was never used or never really in production. Right, right. Absolutely. So could this give people a false sense of security then? If they said, oh, I told it from the beginning to write code without vulnerabilities, so.

I don't have to worry about it at this point. Yes, it may. And I wouldn't say it will completely remove all the tests that are being done later in the process. For example, for pull requests, for periodic scanning, for build. It will not kind of say, okay, we'll do it only on the IDE stuff and we'll do it only when developer write the code. But another thing that can happen, in reality, it's something that actually, if you think about it, I will ask.

create an application, create a code which is not vulnerable. They will try to do that, but then what is actually vulnerable? I have a lot of questions. Is the CVE really vulnerable if I have it within my open source that I got it? It may not be reachable. So I think it's kind of a balance between understanding which prompt you have to do and actually have a security that's in brim behind it to understand what needs to be fixed.

I wouldn't say they're just a prompt and saying, create something which is not vulnerable will be enough. So I know before AI really came on the scene, it was more like a... You could have some tooling that as you're actually writing the code, it can prompt you about certain suggestions, how to make it more secure. But now that AI is writing it, you kind of lose out on that. So working for a security company, do you now like...

Do you work with AI behind the scenes to make it more secure? Do you have any solutions to help with that type of scenario now? Yes. So the idea is that first, we have AI within our product for a lot of things. But we also change the way we think about AI-generated code. And AI-generated code, when you go into the periodic stuff, like when you scan the branch, it looks the same.

The main issue is the developer experience, which is different because you cannot ask the developer to fix things he didn't actually created. And what we're thinking of are kind of a level of security. One will be kind of rules using our scanners today. Then some enforcement going forward. There is also another attack surface coming from these IDs with all these MCP.

Think about the fact that an MCP can just ask you to delete all the files in your computer or that it can actually send all the code to an external source. So it's not just about...

secure coding is about the fact that you now have an autonomous agent that actually can do a lot of things on your environment. Oh, that's interesting. So almost your attack service has increased. How do you then... know that like how do you handle it because it seems like ai can then stay one step ahead of your secure practices or maybe that's not true i don't know

So I would say that both attackers and cybersecurity companies will use AI to make them and train about the future attacks. But for IDE, at least for the Vibe coding, it's a new attack service. Yes, people could have done mistakes in the past, but it's different than a malicious code coming, you know, I'm just writing a Jira ticket and the Jira ticket is saying, and then the Jira ticket creates the code.

And then GeoTicket can say just send everything to one, you know, a malicious website or something like that. So one of the options will be, you know, an agent or an EDR within your environment. Other option... hardening capabilities around the IDE. So there are a lot of cool stuff that we are thinking about. Nice. Do you have any real-world studies or maybe how attacks may have increased with AI or how...

Using tools to help you make your code more secure using AI has helped. I think you see kind of different things about attacks. different examples, but I don't see this as something that we can say it's widely spread because I think it's not yet being used as often. If you think about the big companies. They are still struggling with understanding how exactly to use cursor or cursor-like components. From a security perspective, I mean. So...

You know, OWASP has been around for a while. I'm always shocked how many developers and testers are unaware of OWASP. And you would think people would be like, security would be top of mind. Has AI actually helped the security space, making people more top of minds and still...

Is it still almost like an education process that people need to say, tell the management, hey, we need you to watch out before we ship this? I think it's more about making it easier. I'm not sure education will be enough because at the end...

If it's easier to write the code, it should be easier to write it securely. It shouldn't be something that you educated to write security about. You can still put guardrails in place for all cases, and I will get to our... ASPM solution later on to explain, but generally speaking, I really think the product needs to be easier and much more automatic rather than

ask developers to write securely or ask them to scan the code. It needs to be much simpler. It needs to produce less false positives and be kind of adopted by... simplicity rather than education. That's absolutely true. I speak with a lot of security folks and they always say a lot of teams are worried that more security controls are going to slow them down.

companies have said, we need to ship quicker and faster. So how do you balance security with trying to deliver at velocity almost? So this is a great question. Kind of brings me into... the application security, the SPM part that we are doing. You know, one of the things that I think it's a perception that says we will not stop developers from creating application into production because

We will kind of delay or slow the velocity of the business value coming to customers. But what ends up being is that problems of our abilities get into production. And then you need a developer to fix them. And then it's kind of starting a vicious cycle, I would say. You have to figure out who is the one that actually created the code. It's not that easy to do that once you are in production.

You need to figure out which production issue is actually mapped to a code issue. You want to understand, for example, how many production issues were created from the same code issue. Then, usually, you need to figure out Okay, this owner of the code probably already located on something else. So he kind of bringing them back to solving things they did like a few months ago or maybe a few weeks ago.

and then they need to fix it, we need to build the software, and then the software needs to be deployed. So in the end, you spend a lot of developer time just because we didn't want to... stop them or kind of put the garbage in place in the first phases of the software life cycle. And one of the things we...

focus on our application security solution, besides other things that we'll get to do in a second, is that we will provide a smart guard address. One of the things you'll usually kind of hear from customers, they're saying, I cannot stop any critical CVE. This means that all my builds will be break or all my PRs will be break. I do want to do something which is much more smart in a way.

Let's do it only for things that actually go to production. Let's do it for things that actually have an internet access. Let's see if the worker that I'm deploying has an excessive permission. Something that will give them more information whether this can really be exploited. And the entire, say, our perception around it or the motivation we need for the application security part was...

Let's bring in all the information we have, whether this is the business criticality for the user, whether it's the code context, whether it's the cloud context, and make them a super smart policies to be applied. So from one hand, you will be secure while you get production. And on the other hand, it won't block your entire velocity or delay it to not be able to bring business values into your environment.

All right. So that's interesting. You talked about context. So, you know, every business is different. And you said, you know, depending what business you're in, something you may require more security than others. How do you learn about the context? training on like the requirements do you like did that almost have their own personal not model but personal kind of training so it does have the context to say yeah this is could be a security issue but in this case don't have to worry about it

So we have like three levels. One will be the code itself. We can know a lot about the code because it's scanning the entire code and you can understand whether this code was ever being shipped into production. The second will be about the cloud. Because we have such an extensive cloud security solution, we actually have, I would call it Cygnus within the cloud.

whether it's on the network side of things, whether it's on the data side of things, identity, workloads, we kind of see everything and we can actually understand how your production environment looks like. And then this context is... We kind of bring it back to the application security and define it on the code and say, we saw in production that this workload is really highly permissive. So maybe we should be more careful about the things that go into this workload.

And the last thing will be about the business criticality of things. Every organization has different ways they want to look at their applications. So this is usually something that we need to get from external sources. And the last thing, and this is a super exciting thing that I want to share. As you know, our Cortex Cloud solution is built on our Cortex platform. And our platform also incorporates all our...

XAM and XDR and EXO, which is all the SOC and agent capabilities we have formed. They're real attacks. So while you think about it, another context, which is super important, is what is actually happening in production. Can I say something about the attacks that I see on my production environment? Can I prioritize based on them? So having all these signals into our environment, everything on the same data lake, and then bringing back to the developers and to the DevOps and say,

You need to fix it because these are the things and then also provide the remediation option and potentially an automatic remediation. This will make their life easier and then... If it doesn't cost them anything, they would just apply it. And this is the reason we really believe that if you meet the developers at their systems, if you bring all the context into their environments, they will...

I wouldn't say it's an education. It's more about just simplifying the process and making them part of it, part of the success. Have a production with, I wouldn't say zero issues, but... a very small amount of issues, it's a victory for everyone because nobody will get up at night for security issues that was just found in production, which is obviously much more urgent than anything will be found in code.

Yeah, absolutely. And I guess maybe this isn't right either, but because you're using a product like yours, a real product that knows about other attacks that are happening, maybe they're newer attacks that have just started happening in other companies.

And, you know, someone, a security agent, maybe needs to stay on top of, oh, I better check my code for that. Does this also give them insights? Hey, here's a new vulnerability we found going out there. And by the way, you have it. So you might get on it now. Is that possible? Yeah, so the models we are trained for attacks are always in continuous updating. So we have this information from, I would say, the amount of data we have, not just from a specific user.

And then the idea is really to get them back to the developer and say, we saw this attack or we saw something new that you see and we want you to fix it. And again, not fix it, create the code correctly. I will give you one example, which is very common. Developers like to, I would say, choose an open source component that do what they need. And these components are not...

necessarily something you can trust. If you think about an agent that just tried to generate a code based on OSS, he will probably use the one he saw that is being used a lot. But it doesn't mean that it's a secure one. If this will be flagged out within the ID, then no issue. The code will not be created. You will create a code with a different component, and then you will continue to do your regular coding.

If this will not be flagged within the IDE, then you're already stuck with an API and an OSS component that you already chosen, and then you have to figure out how you patch it or... Maybe upgrade it or something like that. So early prevention, which is something we really believe in, is important for everyone. Just not to spend time on things that you could have solved.

if you planned correctly. Thanks. So we've been talking about as developers in security, but I know security, there's a lot of trained professionals just in security alone. Is there such thing as like vibe security? where someone just now can use plain English to do security prompting rather than have to go through all the certifications and everything that a lot of them are required to have? So think about, like, simple tasks. I'm a security practitioner. I can win apps at security.

practitioners and I need to know I'm responsible for an application and I want to understand what are the top items for me. And this is something that AI can help a lot just, you know, by going with natural language. and ask, what are my top issues to fix? The second thing will be, what are the things I need to do to be able to prevent first? And it's not just fixing things, but also making sure.

I'm kind of lowering the funnel and making sure no more issues are coming into the funnel. So I'm a great believer in natural language and using AI for... I would say a lot of tasks. I also believe it will change the way we think about products and how products are being built. So you mentioned production a lot. Does that highlight areas like...

You know, this isn't necessarily a vulnerability in security, but we notice this module is being used a lot, so you may want to add extra security measures around it. So one of the things that we do... is that we have a scanning of the models and we have full inventory of the models we have in production. And we also have these different signals within your pipelines and within...

other places within the environment to figure out if there are models that were deployed, although you don't have the permission to do that. So yes, AI is super helping, but it's also something you need to figure out.

if you understand exactly what is being deployed, if you understand exactly what is being used. So we're kind of trying to cover everything from code to the runtime and to the SOC. Gotcha. You did mention guardrails earlier. Can you list out some maybe guardrails that could be put into place to help make more secure code with AI? Yeah, so I gave an example of a bad guardrail.

just block any critical CV that you are trying to submit or every critical and high. A good guide will say, please make sure that these CVs are really reachable in code. their own business critical application, and being deployed to production. And again, I would say that if I was naive, I would say that everything, every vulnerability within the code will be fixed, but we know that companies

have their own targets, goals. They have to bring business values into the environment and we are here to help them secure the things that are important. I wish they had time to do everything. but they're more realistic than this, and hence we have to figure out how to help them really fix the things that matter.

Over time, do you foresee security no longer being an issue because AI will then be handling all the security for you? I mean, I don't know. That's probably outrageous, but any thoughts on the future of it? I would like to say it will be simpler. I'm not sure it will not be an issue, but I think in my vision, it needs to be much more simple, much more automatic in the sense of how to remediate things.

and it has to be incorporated within the processes in a very smooth way. It cannot be something that will be done after the fact, but it has to be as part of your day-to-day job. Whether it still can be a burden of trying to scan a code, yes, code will be probably easier to be created without any rules. We are living in a place with tourists, so I would assume it will apply to COD as well. Nice. So you have a lot of products at Palo Alto Network, so I'm just curious to know...

If a company's just starting their security journey and they may feel overwhelmed, like, is there stages you recommend going, implementing this so that they're not like, throw it all at the wall, then all of a sudden, like, they have to have all these false positives. They need to work through. Like, how does that work?

Yes. So definitely, especially in application security, because usually companies have a lot of different problems. There are some things like we added, which is super cool. We call it kind of stop the bleeding. You only fix things that are new. So at least you understand you have other issues and you have a huge backlog or technical depth that you have to fix. But the first thing you are doing, you are trying to block all the new stuff that are coming.

So trying to do it, you know, in a more graduate kind of more in steps or in phases rather than just say, okay, block everything. This is one example. Another example is putting some goals. to a group saying you have to fix like 15% of your backlog by this border. Another thing will be about the different scanners, secrets.

We have the option, for example, to validate secret. So we can understand whether secret is actually being used, and this kind of gives another level of prioritization. So the idea is really to figure out how we can help them. prioritize the work and understand what are the mediation they need to do. And the second thing, and this is the most important part, is not to create an issue in the first place. And I think my example on choosing the one

Open source component is a very important sense of not getting into a place you have to fix a lot of critical vulnerabilities. Great, great. So, you know, have you heard a lot of... Do you hear a lot of things that maybe aren't true about AI generated code from a security perspective? You're like, oh, why does everyone think this? If they only didn't think this, you know, it'd be so much better. I think AI is here. It's not.

a question for us it's here i think all of us are using uh like chats and we do a lot of things with using either lamps or ai or agent in general I think this is the new way things should be done. I think that if code can be generated by AI, then we just need to join this one and not say it's vulnerable or it's not going to be in the same quality of things.

I do believe it's something that's already here. Yes, there are some kind of, you know, concerns about the type of code and we do see that some AI models tend to do some... I would say things that are not like deleting code from production or doing things that are just not as we want this to be behaving, but it's here.

So we just need to make sure we do it securely. I don't think we need to fight the new stuff coming in, but rather embrace it and make sure we kind of use it for our advantage. So that's a good point. I know a lot of developers and testers that say it's just hype. There's going to be an AI bubble. This is all going to go away. But I know that for security, a lot of people are using AI and they are making it easier to do hacks, I think.

I think, so that type of thing, it sounds like it would leave people open to more vulnerabilities if they don't start embracing AI and realize it is here, I guess. Yeah, but we can use AI to protect as well. To protect us. Right, right, right. So it's kind of a race for protection and exploitation in the same way. I believe the protection staff will be winning, kind of the security will win here.

Awesome. All right, before we go, is there one piece of actual advice you can give to someone to help them with their DevSec efforts or security efforts? And what's the best way to find, contact, or learn more about Palo Alto Networks? So, of course, in our... site for Palo Alto, look for ASPM, super cool announcement and product that we are having. We kind of go through our entire ecosystem as a platform, making sure the developers has...

their own way of using the platform and also application security. One of the biggest challenge will be trying to kind of bridge the gap between application security and developers. And the second thing I would say, application security is a journey. It's not on one time. You have to kind of set your KPIs and see how you are progressing with them. And we are here to help and kind of...

help with explaining what is top important and then what are the things that can be fixed later on. Absolutely. We'll have links to all this awesomeness down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com forward slash A-5-6-7. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help.

in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers. over 40 000 email subscribers who are at the forefront of automation testing and devops if you haven't yet join our vibrant community at testguild.com where you become part of our elite circle

driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire through the land. Guiding testers showing the secrets to behold