Welcome to the VCD Roundtable Episode 47. We want to talk about the importance of hardening. So hardening software. So we added the software at the end. Oh, I didn't know that. I thought... The topic is around hardening, but OK. If it goes around VMware products, maybe I can help as well. Hi, this is Matthias, also a part of the VCD Roundtable stream again. Today's guest speaker with us is Fabian. Fabian, for the introduction. Hello. So I've been here many episodes ago.
I don't know, maybe one of the first and never came back. But now if things get hard, I'm back in the team. Fabian, architect at comdividion. I do a lot around architecture and consulting service providers when things get tough. Nah, just kidding. No, but the topic around hardening is something I dropped in during our Kickoff because we see that a lot of service providers need to be prepared against any kind of cyber threats and attacks and also audits. So be prepared for the audits.
That's why hardening is an important thing. And maybe I can drop the one or other story to give you guys out there some feedback around that. Absolutely. What's also interesting, based on the questions we get from many customers and service providers, is do we need hardening only if we need to get certified or is it also relevant in normal environments? Yeah, I mean, hardening is a general concept for many years to change your software probably in a way that differs from the default settings.
Yeah, there were many years ago when... was telling you how to harden your Windows systems and all kinds of software can be hardened by disabling features, by also giving you guidance around operational steps. You need to patch your environment. You need to make sure people are trained. All those things come together when we talk about the topic hardening. That's why I always see it as a three-part thing. One is the architecture around solution. That must be hardened.
One is the configuration, how is it applied? Those hardening settings. And also how is it lived? So how are daily procedures hardened against security? This is something you always need to observe as a complete item from my opinion. And if someone promised you, "Hey, we deploy Aria Operations or VCF Operations" and you have automatically your environment hardened or get reports of the hardening status. In my opinion, that's not true. It's one important toolkit for it.
But there's much more to hardening than just installing something and saying, "Now it's hardened." I wouldn't phrase it that simple because honestly speaking using Operations Manager and the implemented policy or the implemented framework with the hardening configuration is pretty straightforward. It just reflects Broadcom's default view: which configuration should be taken or which feature could be reconfigured to have a hardened infrastructure from a Broadcom perspective.
That is one of many approaches. It might fit your needs. It might not, but you get a proper report based on Broadcom's default view, how it should be done. If you don't agree, if you have a different approach on hardening your infrastructure or you have different security and recommendations/needs, you just need to reconfigure the framework within Operations Manager to fit your needs. Then you have a proper report based on your infrastructure and you can start reconfiguring everything.
Yeah, totally right. But from my point of view, as you know Ops Manager has a limited view on things. If I think about the environment of the service provider who might also have NSX, Advanced Load Bancer or I don't know Cloud Director in place, that's typically something the Operations Manager does not get. If we talk about those, let's say hardening relevant items like "Hey, your Active Directory must be secured." Let's say, immutable against any kind of attacks.
That's also something we cannot. So we have only this limited view and I really love using those built-in compliance options from Aria Operations as a starting point. But still, and you did a great point by saying, "Hey, it's everything related to risk and everyone needs to decide fon their own if they want to accept the risk of not applying this item or maybe this item does not fit for your specific use case environment."
And from my experience over the last years, I did a lot of... I was involved in a lot of audits around Enterprise solutions. And the important thing is you need to have something in place. That's the first thing. So you need to do this kind of hardening. VMware has done a great job over the years together with the community of creating those hardening guides; in the past they were called security configuration guides.
Those big Excel sheets where you have the default setting, that's what it should be. That's the argument why you should have that. And what we always did, what we still do nowadays is taking that extension for certain customers and have the discussion -- item per item with the customer. "Hey, how does it look like in your environment? Is it a risk we take?
Shall we live with the operational downside of enabling lockdown mode?" Which obviously makes sense from a security perspective in many scenarios, but sometimes from operational perspective, it can be quite annoying. So we need to get the risks together, calculate what's relevant for the customer and then make a decision.
And I think that's a very important point you just covered, because even though a guide tells a customer or a CSP setting ABC needs to be reconfigured to whatever, you can still come up with a decision against the recommendations. Say, "I'll configure a feature in a certain way because of..." And even if you have an audit... so an auditor has no chance to make any... He cannot force you to reconfigure it to a certain configuration.
But if you have it configured differently based on common guidelines, you always need to have a justification in place. You need to take a proper decision and justify why is it configured that way? And as long as you have a justification, you're good to go. Because if you have a justification, you made the decision and you're perfectly aware about the pros and cons. Yeah, depends on the justification. I don't encrypt because it's complicated. [laughter]
Would be the... I mean, from my point of view, most auditors I work with, most of them, they audit a broad area of IT. They're not experts in VMware. But they react differently if they have a question and want to see something or some documents, artifacts, whatever. And you take something out of the box. Here, that's a complete list. It looks great with bullet points and score and justification. You show it to them. You then take an extract from the Aria Operations report.
So they see there is a mechanism in place to prove that certain configurations are in place. This is something that really helps you during the audit. And for sure, for your own sake, it's up to you to still make the best design decision that secures you, but still keeps operations alive. And this is really relevant from my point of view. That's why, I mean, if you do those security engagements or hardening engagements, three to five days of architecture is easily gone.
You have one, two, three days of workshops, then everyone goes back together and make maybe different decision about those 10, 15 complex points. You need to, let's say, also, or try to get the proof for certain statements. Yeah, maybe that the backup is encrypted. Simply something you cannot see. You need to assume that the backup is encrypted. But still, when we go out there, I ask them, hey, give me some proof or some configuration proof that the backups in the back end are encrypted.
And this is really relevant. So those engagements are really important, but also take a little while. And that's what I said. You cannot simply deploy Aria Operations. Say, now it's hardened and let it run. You need to discuss that. It's an architectural decision you need to take around the environment. And even if you have some configuration monitoring in place, like Operations Manager, for example, it's just a supporting tool, because you need to make the architecture.
You take the decisions, you justify. The monitoring is just a tool supporting you to prove that you implemented what you have discussed and designed. That's one point. But what we also often saw on the health checks, for example, that there is hardening or hardening was done, hardening was documented. But then they had an issue and needed to open SSH or direct access to hosts again and never configured it back.
So from that perspective, I think, it is necessary to have proper control of all mechanisms and changes made during the hardening process to ensure they remain active. And totally, totally true. And one example that always comes to mind is, for example, I'm a big fan of segmenting the VMkernel ports of the ESXi, onboard firewall. We simply say only management networks can interact with the ESXi VMkernel, which is easy to do and gives you direct or reduces the attack vectors.
And even though there's some malware in your infrastructure, they cannot reach the ESXi network. Except, I mean, you could, if you have some within data center firewall, you could do something similar. But it's one way to make sure the packets don't arrive from unknown networks at the ESXi. And when I look at all... fortunately, over the last two years, I've known many companies had some issues with crypto attacks on their VMware environment.
Fortunately, none of my direct customers were attacked by that, or it was broken or encrypted. But the primary way I got engaged afterwards, it was always the same thing. Some older version, non-patched version of ESXi, they were reachable from one of the desktops that had some malware installed. So client network, server network, server network, they could scan the network, figure out ESXi host, figure out older versions, and did some attacks.
And the other perspective was always around this active directory integration of the ESXi host. So also one thing, when the ESXi host was integrated in Active Directory for use authentication, there were a lot of attacks happening over this path. And today, I think in the hardening guide for security, there's still the recommendation to add the ESXi host to the Active Directory, to have a proper directory or non-route-based log on as possible, even though that's the official statement.
And you also see in the CIS guidelines for security or other documentation when you want to reach a certification, I always argume," hey, let's not do that right now." There were so many bad things happening over this way. We do log them out. We segment the ESXi from the network, and we make sure we have some alerting in place when the root user is used on the ESXi host.
So you have Log insight configured, making alerts if roots are doing any actions out of our role-based authentication mechanism, because that is really important to make sure every action is accountable to a specific user or system. But that's... all the topics you've mentioned. They're all part of a proper hardening, because hardening is not a tool. It's not a solution. It's a process, right? It's something-- It's a lifestyle. It's a lifestyle, right? That we're not going down there.
Yves is not here, Fabian is here, and we're drifting. So it's a process you take. It's an approach, and also monitoring, and many different things that come to your mind might be part of a hardening process. Like you said, if a user is used to log into a system, or you mentioned the AD thing with the ESXi host, it's also pretty fine.
But I think even if a recommendation exists and you come up with, "no, I don't think that this is a good idea," like joining an ESXi host to the Active Directory, you just justify it. The downside of it is, and just being really blunt, is there are many auditors out there without any clue-- No, I should not say without any clue. That's not good. Not being in depth with a certain product behavior.
We'll tell you in the report, your ESXi host is not joined with Active Directory, so you get a minus score for something which does not make any sense because they have no clue. And then if they have a good name, if the company has a very well-known name, they are always right, even though they are wrong. Absolutely. But the good thing is, those are the items, even if everything's correct, those are the things they can find. But they feel good. We feel good.
But still, the Active Directory thing, and if anyone is here listening, I would be really interested how you are dealing that around the ESXi hosts, because I have a clear opinion just based on what I saw. Because I saw things, and I don't want to see them again.
I think if you're coming up with your approach and how you would configure infrastructure, I think as long as you have, as long as you take the decision document the decision, that's the proper justification, you're good to go because that documents that you're perfectly aware of what you're doing. I think that's one very important point around the whole hardening and those topics. True. Sasha, how can Cloud Foundation help us with hardening? I want to hear some sales talk, come on.
Yeah. It's the best solution. Absolutely. So you get a lot of configurations pre-configured by VCF, so that's interesting. But I think the one topic also for me, a big part in hardening is to work with certificates. Work with public certificates, make sure that you have a trusted CA behind your certificates and not working with self-signed certificates out of the host. And having a hardened certificate authority. Now we start again. Yeah, I mean, that's also a thing. Absolutely.
It doesn't matter if your certificate authority is compromised. That's a huge thing. I always get the feeling when you go down the security in Hardening Road, it's opening the box of Pandora because from one item, you get to the next one, ask, hey, what kind of CA certificate authority do you have? Is that something we can assume as being reliable and secure? We don't know to be honest. Especially if you talk to smaller corporations where that was set up by, I don't know, Erwin and Dieter.
And they left the company 20 years ago. You know, Erwin and Dieter, that's a common German name, just for Mike and Jane, if you're from the US. But still, this is something where you then also say, okay, perfect. When we are done with our, I would say, assessment or hardening sessions, and everyone is really hard. No, that's not the right thing. And everyone is really getting items to work in the VMware infrastructure.
There's also a lot of to-dos to figure out if we assume the right things around Active Directory, certificate authorities, and other topics. Erwin and Dieter. Come on, you said we can cut this later on. So therefore-- Well, we'll let it slide. So, cool. But again, it's also the combination with the monitoring tool. Because you need to have a report, because we are all just human beings.
And if we are tasked to configure something, and even though it's very well prepared and documented, and we receive a monkey, see monkey, do instruction set, there is still the chance that you're fat-fingered something. And that's the reason you need to have a pool in the backend, monitoring the infrastructure, being aware what should be configured, and compare it with what is configured. And, Sascha, I love the example you came up with earlier on with the SSH.
So that's a common use case, like, "oh, I need to troubleshoot something." I enable SSH. What is perfectly fine, right? And at the end, because you disabled all the warnings, because they drive you bonkers in the UI, you forgot to turn off the SSH service at the end of your troubleshooting session. So latest and next day, the compliance report within the monitoring tool, which is Operations, you shall have only one Operations Management tool, pops up saying, "I found this...
"design and deployed by comdivision" Yeah. ... here is a new host, and this host has SSH turned on," and then you receive an alert, saying, all right, I detected a configuration drift. Please help me turn off SSH. And even though you have an incident, so you violate one of your policies, but if you have a monitoring tool, you get an alert, and then you can react on, like, "oh, I forgot. I turned the SSH servers off." That's a proper approach, and that's perfectly fine.
The worst thing is you are not aware of that a configuration drift happened, and you are not compliant with your old policies anymore. That's the worst thing. Yeah, totally something we need to keep in mind. I mean, the good thing is over the years, the core VMware products, and if you now look into the security configuration guide for vSphere 8, there's not so much more in it anymore.
The product team did a great job over the years to bring the core hypervisor to a level where you should typically, or you don't need to tweak that much. I think Mike Foley, who was also in the product team, he's very often shouting out on LinkedIn that he was very keen after getting the hardening into the product. We see that VMware is getting better and better in a lot of areas. That will also be much more better than with VCF 9.
I'm pretty sure about that because now they all try to make it secure right from the beginning. But again, there are a lot of things we need to decide. Second factor of indication, for example, that's nothing you can bake into the product. We need to make a proper decision, and then also how to configure it concretely. Even if that's not a real word, I think. But also Cloud Director.
I mean, we are big old fans of good ol' Cloud Director, and also there are still a lot of things we need to tweak afterwards to make sure it is hardened according to the recommendation that VMware put it out also many years ago. Maybe not Broadcom. Maybe the recommendation which comes from either comdivision or even better from the customer themself. Because that's one thing we do with our CSPs and customers is interview them. What's your expectation?
What are your business requirements towards security hardening? And those bits and pieces. Another interesting topic. Let's move away a bit from the base infrastructure. And let's maybe for the last few minutes focus a bit more on the workloads which are running on the already successfully hardened base infrastructure. So we have guest operating systems. We have applications and a ton of stuff running. And if we are now back into the CSP game, it's a CSP.
You need to make sure that you have proper network implementation. Because you have no clue what your customers are doing inside the virtual machine. They install the guest user and you just don't know if the customer applying all the security patches as recommended by the operating system vendor for example. Yeah. And remember those site channel attacks a few years ago. You know where they said, "Oh, disable hyper threading."
Because there were chances of whatever this is something that could be. Could be a real... So winning the lottery had a higher chance than that one. But... I'm not sure. I mean there were attack patterns where you could extract certain things. And this is something you cannot control as a service provider. The thing here. What was the name of it? Spectre and Meltdown. Winning the lottery has a higher chance than that. No, I prove it wrong in the show notes. Do we have show notes?
We will get some. So I can tell you on the side why. But the thing is you have your... And that's not a guest OS thing. It was a hypervisor attack. So two different things. But you have no clue what your customers are doing inside the virtual machine. And if they're patching their OSs, if they apply security patches to the applications.
And we are perfectly aware, because virtualization is a big invitation to keep expired software running year after year after year out of support, out of everything, not supported anymore. So many companies do that. So as a service provider, you never know what's happening inside the infrastructure. So it's even more important from our perspective to take care that the underlying infrastructure is properly secured.
And also you are logging all the traffic between tenant networks and the CSP basic network infrastructure. Separate. And additional to that and additional to hardening. So really take a look what options you have to help your customers, your end customers, to bring more security inside their environment. So take a look at advanced threat protection with NSX, which you can buy as an add-on. So that's one point. And the other point is take a look at advanced ALB. So advanced load balancer.
So for all the web services your customers hosting. So you can activate web application firewall on top of it and make your environment or the customers' environments more secure. So ATP, so I first mentioned ATP, that's a different ballgame. It's an amazing feature. It's there and even with DPUs it gets usable. Because you're offloading everything to the smart name. And use those features because I haven't. So I've been in IT for more than 25 years.
So over time, year after year, you just see more different different patterns. It got more and more and more. There was no year where you had less threats running around the internet than the year before. So it gets worse and worse. Okay. And then in the end, you always need to have Operations in the back end for monitoring. Awesome. So if you are not hardened already, what can we do? Sascha, how much is our hardening? Standard offering. What do you say? You need now to turn on the red light.
And then... Wait a sec. So if you need... Not red enough, I would say. So if you need it really hard. No, wait a sec. Guys, you are aware that it's a podcast. Most people won't see the video. Really? We're now waiting for the audio description. That's why we talked about the red light. Yeah. So for all the audience who's not watching the video, the light in my room is now pretty red.
And joking aside, if you need any guidance around that, if you feel your VMware environment could use a third party look up around the topic of security hardening, just drop us a message and we can make you pretty quick and offer to help you nearly on all continents nowadays, isn't it? Sure. Yeah. Feel free to ping us. Feel free to drop us messages on LinkedIn, as an email, over our webpage, etc. and we can sit together and talk with you about what we can do for you.
How can we solve your problems and provide hardening or a design for hardening for your environment? Yeah, also from my side, thanks for tuning in. Famous last words, I think hardening is very important. Also monitoring and implementing the stuff, but always keep in mind that you keep your risks and hardening aligned with your business requirements to have a solution because in the end you need an infrastructure solution which supports the goals the business has.
Otherwise it doesn't make the best sense out of it. Short outlook Ep. 48. I heard a rumor. I heard a rumor that Episode 48 will be around Orchestration Automation. Yeah, I want to come back. So that's the rumor. So we won't discuss VCF Automation because that's not released. So Orchestration Automation will be around Cloud Director and Orchestrator. That's the plan. Interesting. I'm looking forward to this one. Who will present, not me. I'm sure you. He's Mr. Orchestrator. Come on. I know.
Okay, thanks for tuning in. Have a great day. We're more than happy to help just ping us. See you in the next episode. See you.