Rancher Labs and Kubernetes with Darren Shepherd

We were talking about the recent Docker announcements about Docker hub limits, which of course now in 2021. We've all moved on beyond that. And Dockers made a bunch of announcements about new features and paid Docker, desktop, and all sorts of other stuff. But a year ago we were catching up with. Darren on all the projects that they're working on, including our ke and rancher and K three S or keys. And all the other stuff they're working on Rio. And there's just a bunch that we talk about.

So I think that this show is still really great. A year later, I listened to the whole thing and I think it's super great. What rancher is doing in the space, and if you're at all new to Kubernetes, we talk about the different. Kubernetes distributions and how rancher plays with that and where it's best to use rancher. The rancher, gooey that we all could just call rancher as well as the rancher, Kubernetes distribution. Which are technically different things and where Rio fits.

And then we go down some great discussion of the gap. Between Docker Compose, which I've talked about in other live shows recently here in 2021 and how it's getting a sort of a reboot and a whole new life and a bunch of new features. And so Darren and I talk about the excellent product that composes and how there's a gap between that. And Kubernetes and some of the ideas they have for solving of that. So it's sort of a great retrospect on where we were a year ago.

In comparison to where we are now. So enjoy this. You're old but extremely still relevant conversation with darren shepherd the cto and co-founder of rancher labs thank you so much for being here, Darren.

So I'm glad to have you. . I have some questions of my own that I need answers to that I'm hoping that Darren can shed some light on you can also follow up by the way, follow him. You can find them on Twitter. He's constantly tweeting. And he's, it's always good stuff. It's always good stuff.

Yeah. ECR or GCR, can I do that basically white label it cause we don't want to run it or run it ourselves. But but what we're finding out is this is actually a really tricky one is the amount of data that we're serving up in images today is absolutely ridiculous. We had no idea the amount of bandwidth that is coming out of it. Of Docker Hub today. Yeah.

I think, I feel like Docker is trying to do the right thing, but obviously like it's a very expensive thing. So we all sympathize with them cause it's like, it's a super expensive and you know, people are paying for it, but yeah, it's a tough, how do you satisfy all these needs of open source? It's a

We'll actually have a bunch of those blog posts. If you're interested in this show, we've talked about it a lot. So if you go back and just search Docker Hub at bret.live that'll take you to YouTube, and then you can search on the previous shows and we've probably had three or four shows just talking about this for like 50 minutes or something. But I was interested in that tweet cause anyway, so random topic.

Let's start off before we get to the questions, let's start off with your main product Rancher. And so Rancher lab is a company Ranchers, one of the products. What is Rancher and how, how does it relate to Kubernetes? I guess? Cause people often ask me like Rancher versus Kubernetes and that's like a weird question. Yeah.

And so what that Kubernetes is we're not very particular about what, flavor of Kubernetes you get. That's what differentiates Ranchers. We really view Kubernetes as a commodity. So if you're getting EKS or GKE or AKS, or you're running it, you want it to set it up yourself through kubeadm or, you use one of our distributions, like RKE is our distribution for enterprise k3s as our distribution for edge.

So Rancher is going to help you basically deploy those deploy those distributions or manage them, do the security set up, GitOps, deploy applications, the way we view it as it's like, we want Rancher to be your portal to the Kubernetes world. We're the entry point. Like if you're just starting off today and you want to deploy a cluster Ranchers, really great spin up Rancher makes it really easy to deploy a cluster and you can get running.

If you're a large enterprise or you're more mature, Rancher will scale up to those use cases too. And yeah.

But to me, that's a little bit like building your own Linux kernel. It's really only for the people that know what they're doing. And so I'd like, and I, in my courses, I talk about Rancher is one of the ways for deploying on a server. And it has sort of, a lot of it makes a lot of these decisions for you and puts it together. But how has that sometimes when people are getting started, they hear something else called RKE. How is that ranch different than that? Or how are they related?

But if you're on premise, like on premise, you're running your own servers. You need some options. You need an answer there, and that's where RKE comes in is like, we give you a distribution. That's going to be good for, data center on prem. And then also, very similar to that, we have k3s which is another another distribution. So that it's just one, one component in the bigger picture. It is not required. It's optional. A lot of people like RKE we have lot of people using it.

But we don't, we don't try to fight and say, RKE is the best distribution ever, because it's like, well, like we just give you one because you need one, but if you get one, some other way, that's fine. Like our goal is really to just help you manage Kubernetes and, yeah. Right,

You can have a consistent approach if it's on premise cloud edge, whatever. So our 2.5 release strengthens our ability we have better integration now with k3s S to, expanding into the edge. We've better integration with EKS, so you can manage the cloud, Amazon cloud better. But one of the major features that I personally worked on that I really care a lot about is the integration of Fleet and Fleet is our GitOps at Scale solution.

So this is a GitOps solution that will scale up to a million clusters. It really came out of our work in the edge space but that's been directly integrated into Rancher. So now when you install Rancher, you have a built-in GitOps solution that that, worked great for one cluster works great for a hundred clusters, also works great for, a million

I went into kind of great detail on the architecture and whatnot because the key thing about Fleet is the multi cluster capabilities is how to deploy to a lot of clusters and how to manage the configuration and how to do that at scale. And so Fleet is like a very powerful multi cluster management engine. And then you kind of layer get on top of it and then it becomes an excellent GitOps. But Fleet is actually, it's a project you can just use Fleet directly by itself.

And then it's really nicely integrated into Rancher if you want a GUI. And it automatically, because you have to register clusters with Fleet. So the, all the clusters under management and Rancher automatically get registered with Fleet. So you can just add to get repo and it starts deploying software through that. So you can use it by itself, but you know, it's even better with Rancher.

So it's a very logical, simple thing of like, well, what if I just put those manifests and get, and when I change it, you know what I updated and get you just automatically apply it to the cluster. So it's like GitOps with Kubernetes is like a no brainer. It's like, oh, it's so simple. And that's why you see projects like flux, which are really good. It's a very simple approach and it worked really well. The complexities is just how you start how do you do this at scale?

And, but the basic concepts are. Are so simple. And so like when I talk about GitOps, that's all, I really care about there. And like, I don't even think, like, there's definitely a lot of problems with GitOpss too. It's not like, it's, you know, there's just never like a silver bullet for all these things but it has a lot of benefits now that it's like, it's clear. I mean, I'll tell you like some of the downsides is. Things we tried to fix with Fleet.

Is this like, as people scale up with good offs is like, well, they thought they would have all the centralized control and everything, but they ended up with like a million get repos and different branches and people are doing PRS and they're just like, well, what the heck's going on? Like, what's the real state of my environment.

And that was one of the driving factors behind Fleet where we're like, well, I can't just hook like every cluster directly to a get repo because I don't have enough centralized control and visibility, like, especially in an enterprise, like how do I, you know, what the heck's going on? So Fleet works as a kind of a centralized GitOps manager. So you can go to one place and get visibility of everything and then also control like our back and stuff. If you're in there. Right.

I don't like really to compete with other open source projects, but but it's like the architecture of Argo did not work. To get to the scale we needed. We our Argo is a push model. And that's, we know firsthand the complexities of that because Rancher two is actually written as a push model. And that will currently scale to a couple thousand clusters. Whereas with Fleet, we needed to get to a million.

So we had to flip to a, we Fleet, the architecture of it is really what we say is like a two-stage pole architecture.

Of to get operating all the things in Kubernetes. Sadly I, for those of you that are Docker and swarm fans, I don't I don't have any tools yet that anyone has really created that automate that, although it wouldn't be terribly hard, but it, I haven't seen it. So maybe you could be the one, maybe you and the internet, you could be the one open to open source.

Not necessarily Darren, because he's already probably got a list of 50 things that he needs to make open source projects, because he's constantly coming up with new stuff. We talked about that last time, actually, if I remember the question where I was saying something like what's your backlog of the list of things like, all these things that you've said I'm just gonna make a little personal project. And then it becomes a product that everybody wants. And it's like, what's the next one?

What's on your list that you haven't yet totally went abroad because you have any time. Hey, time to work

And I mean, I just have a massive back backlog of ideas or things or whatever. The one, one thing that's been fun with Ranchers as scaled as a company we're up to, I don't know. I don't know. I think we're like approaching 300 people. It's like two 50 or something like that. Yeah. It's gotten big.

Is we've I think we're getting better at like, I can basically kind of prototype and come up with an idea and then hand it over to a team who can actually support it because supporting, it's the maintenance of software that kills you. I mean,

And so now being in this position, I also realize it's like, as a maintainer or whatever, to, it's important to be nice, just give everyone the benefit of a doubt, it's like, yeah, it was just, there's no reason to GitOps.

Right. But I think I probably wake up every morning and think about that one repo that once someone asked about yesterday that I haven't done anything with in three months and that I should do something about it. So for those of you out there that have ever asked, we're probably thinking a lot more about your stuff, then you realize it's just a matter of managing all the incoming

That's still, check out Rancher to also manage it because we add a lot of functionality on top. But in terms of distributions, yeah, the cloud, you really can't beat, what AKS EKS GKE are doing.

So let's say what you can do is you can create an automation flow, like GitOps or Terraform or whatever, which is then interacting with Rancher, which is then deploying the clusters. And then Rancher will add on top of it, like monitoring. Also, if you want to go with like more Prometheus oriented, I mean, Azure is going to give you their kind of integrated monitoring.

But if you want to go with kind of Prometheus and that approach, which is quite nice because it's all container negative and all that stuff. So it's like Rancher will help you with that. Yeah. The way we look at it is like, you know, what kind of, once you get the Kubernetes base in there, now you can just start dealing with kind of a pure Kubernetes world. And then Rancher is going to help you with that, so we can, hook up like the good ops pipelines and things like that.

And then, so you have gatekeeper, which is going to be the Kubernetes specific integration. So if you're looking for a solution for enterprise or like for policy management or policy, offering an enforcement across anything, OPA is your way to go. If you're focused more on Kubernetes and you just care about Kubernetes, Kyverno is significantly easier to use. I very much like it. So it just depends on, on, on your approach. Cause like, to me, the biggest downside of OPA is Rigo.

I don't know how you say it, but it's their policy language it's based. It's, derive from data log, which is dry from prologue, which is so it's a different way of viewing everything. It's more of a query language, a query and evaluating data. So it's a little hard to grasp at first. So I'm a big fan of . I don't think at this point in the policy space, anybody should be making a large bet of just saying like, this is the only solution going.

Yeah. So I honestly would say we'll just start with Kyverno. And if that doesn't, if you need to extend, pass it then try this, try

That's why I asked everybody's opinion on there. . Like, what do you guys think? And I go jump into Docker Captain's room, which is always a great slack to be in. Cause there, there's a lot of experience in that room as well. Cause they're mostly senior, . DevOpsy sysadmin people. Yeah. Personal Docker registry equals nexus. Is that versus nexus?

Like, that's just,

Um, just run those and you'll get a Kubernetes cluster. But yeah I mean, it's, Prometheus Datadog assist egg, all the regular solutions. It's like basically how you want to monitoring monitor it. So if you hook up k3s to Rancher, you can deploy you know, Prometheus the Rancher and then you get the normal monitoring there. Yeah.

And so from the very beginning, we all went. We're like, well, why do I need etcd? Why can't I run on top of something else? Cause I don't want to run like our customers don't want to run a persistent system. Like they would much rather get it as a service.

And so the motivation for swapping out etcd with something else was always that I'd like to be able to run it on RDS because I can just pay for RDS and I don't have to worry about operations or if I'm in an enterprise I have a database team and they just do it for me, so I don't have to worry about it. So we actually started this work way means like five years ago we did this.

We contracted glider labs to develop a prototype of replacing etcd with you know, doing the shim of getting etcd to run on top of our relational database. And w and we basically proved that the idea was feasible, but the timing wasn't quite right. So it was this pet project of mine after that for many years, where I just kept hacking away at a database, how, like a database solution for Kubernetes.

And so it wasn't until k3s came along where we then found a perfect use case for it, because we could swap out FCD with SQL Lite and then it made Kubernetes much more lightweight, less intensive on the disc, much more, better, like much better, much more, better, much better for you know, edge solutions. And so that's where it came from. It's like people actually don't realize that. Really the amount of effort that went into creating that.

And so that code is actually it's in a project called kine, which is a project that I really would like. It's part of k3s you know, k3s is a CNCF project now. So that project is hosted with k3s in CNCF. But that project, I would like to see graduate, like move upstream to be a Kubernetes project, because you can run Kubernetes on top of any database. And I had mentioned Fleet, we only talked a little bit about Fleet and how do you scale Fleet to a million clusters?

The only way to do that is actually you have to run Kubernetes on top of on top of a relational database because the data store etcd just won't hold up. So this is like a super technical component. Something I've worked on for many years. I actually pretty darn proud of that. It, that it works. And I do see.

As the cloud distributions, like EKS and those things get more, you know, as that becomes more of the commonplace that all of the cloud vendors are basically gonna move away from running etcd. Cause they're going to run different. It's just too costly to run etcd. So it's like, you could run, we haven't actually done it yet, but it's like, it's theoretically possible to add a driver to kind to support like dynamo DB, and then it would be ridiculously cheap to run a Kubernetes.

I would recommend that for most people it only just becomes more of a problem when you're like enterprise and you care about, oh, you know, where your data is stored and stuff like that. But whatever it's Microsoft they'll, you know, they'll figure out some answer there too. So I have nothing actually against it for our use case. Like there's two requirements we were looking for is can I put a custom domain on it? Which I can't really, I don't think I can do.

And even though it's unlimited use for open-source projects, I think there's still actually a limit there's I don't think the bandwidth that we're pushing will actually. But we are actually, we're investing that's one of the options on our plate. Yeah, cause we still have,

When people sometimes think , GitHub allows all this for free? Why can't all this be free? And it's like, well, it's actually quite different to store, this stuff. And that's why all these companies, like NPM had to go make an enterprise model because NPM, as a free thing is quite expensive. I'm sure. But it's, but that's still just open source code. There's not a lot of compiled binaries in there, even though I technically, I think you can do it.

Now everyone and every server and every CII solution and all this stuff all has to now deal with auth on a thing that we never had to deal with this. I don't think it's a matter of, no one can do it. It's just, it's a new thing,

Yeah. I mean then Ranchers, Fleets get, so it's like our projects they cover most of the space. And like the only thing we don't do directly is going to be like monitoring, logging those types of things, universities like Prometheus or fluent bed. And but we still like support them. They're supported versions. You can get through Rancher and whatnot but as I introduce Rancher in the beginning, it's like, we really would like to be the, like your one-stop shop. It's like.

We are your kind of gateway into the Kubernetes ecosystem and then CNCF universe. So while, we're not necessarily the one, you're not, every project is not a Rancher project, but you're going to go through Rancher to get that. Right.

And that's been a. That's been a challenge for me personally, , because Kubernetes is definitely more difficult than let's say, like what swarm was doing. But that's where things like k3s S came from was like our attempts at trying to make this easier for people it's like the kind of running at k3s cluster is really no different than running a swarm cluster these days. It's pretty it's about that easy.

Now deploying your apps on Kubernetes is still harder than let's say, like swarm, but but we're, hopefully these things will get solved over time

Cause you know, if they bought 700 Docker enterprise customers, essentially when they bought the majority of company and on, as far as I know, close to 90% of those were all running. Swarm, As their primary yeah. Orchestrator, they could, they also might've been running Kubernetes, but they were also running swarms. So swarm is a prime, most of their customers. And so Mirantis I think was doing that to sort of calm the fears right. Of the whole, purchasing the customers bailing.

Cause they're worried. But since then they have continually restated. Oh no. We're adding now we're going to add CSI support. We're going to add jobs. Jobs is already up in upstream Moby, but the release cycle and feature set is still very slow. It's not like there's a team of 20, as far as I know, it's still like a person and a half. That's the developer team on swarm. It's not going to any, in any way k3s up or ever be the thing that Kubernetes is. Right. So

And now we've made Kubernetes easier to administrate, but using it is still hard. Um, But I do honestly believe it was it's the right move for like our users, our customers yeah, the stuff is a little harder to use, but the ecosystem is incredible around Kubernetes. And it's just hard to deny that. And I don't, if you continue with swarm, just make sure you have a good exit strategy because it's I mean, it's, there's just, no, there's no ecosystem there. Yeah yeah.

I would, I was just to say for the Kubernetes, if you want a really simple solution of , how do I just basically deploy applications Rio plus k3s Rio is a project from ranch or whatever, real plus k3s is a super, super easy way to run containers in production. Rio is a layer that we add on top of it, which is going to give you automatic certificates and load balancing and Canary deployments. It will pull apps directly from Git, it does some auto scaling.

So if you're really looking for that, just very simple, you know, almost closer to a Heroku, like, experience, but you're running it yourself. That's what Rio is.

When people go through my courses, the way that they're introduced to Docker, Compose, and then registry and like some things like that and hub, and they go through those basics and then to wet their lips on what orchestration is, because it's already built in a Docker, they learned swarm a little bit and there was a couple of hours of stuff in there. Yeah. And then for a lot of people, they just stopped there and it's good enough for them. And that's great.

I still actually run my own website on swarm because it still works. And my website is just a website and it has a couple of things. I think a let's encrypt proxy in front of it, or I'm sorry. A traffic proxy running less encrypt in front of it. And then a storage driver from an old project that doesn't even exist anymore, really for REX-Ray, I think. And, but that works for me and I haven't changed it in like three years.

But if I did it all again today, I would probably, you know, if I needed to run three or four websites and I just wanted, I had some nodes on DigitalOcean or something, I just wanted to have a little hobby. I might do K 3s or Rio. I does Rio run on top of, I mean, it's just a requires a Kubernetes communities. Yeah. And Kubernetes. Yeah. Yeah. So

And it's this great tool for people to understand containers and spin up environments and stuff. But there's like, I know Docker has some solutions, but they really haven't been adopted much of, running compose and production because it's a super hard problem because this is a space. I mean, I've worked with a lot of like, compose is great for developers. It's not great for operations. It's not great for enterprise. So how do you bridge that gap?

And this is something I've looked a lot at Rio was our first attempt at trying to bridge that gap. I don't think we actually succeeded there. We are actively working on kind of a Rio to where we're trying to address this, this major gap that I think exists.

And so it's like, how do you keep that simplicity, but then create like effectively an artifact that you can move forward. And yeah. Yeah. I mean, I really hope. Next year when we do our annual talk, I'll be talking about how we've addressed this problem. This is something that bugs me a lot, I wish.

And we struggled the same way in the training world too, because we're trying to figure how to train people on these things. It's like, okay, well, I'm going to teach you about this, but before you know this, you got to know this. And before you know this, you got to know this. And now even now you need to know 10 things just to get your app into production.

So it's like you have problems coming from both sides of , people start with Docker Compose and they understand containers. They don't know how to run it in production. And then in an enterprise it's like, well, yeah, we already decided to Kubernetes, and now we're forcing that on you and yeah. Have fun with that, as much as Docker was all about the developer, it was like all about the developer, like empowering the developer, Kubernetes, unfortunately is all about core.

IT It's like, that's why it's been so successful is that enterprise has adopted it and it's to a certain degree Yeah. I don't think it's been very good for the developer to a certain degree. Right. It's like kind of it's pushed the it's yeah. It, it put the ball back in its court, you know, control developers before were like, I want Docker and I'm so much more agile with it and they're pushing it. And then they're like, wait, hold on, give me two years Kubernetes. Now you deal with it.

So yeah, , Bret: I'm a huge compose fan and every one of my consulting clients, I almost insist that we actually like that's the first part of the project is we compose this thing. And it usually ends up with developer happiness, and then we start talking and then it's a completely different conversation to talk about server clusters and Kubernetes and shifting the manifest.

And the only thing I haven't been able to come up with is watching over and over again, multiple times a year with these projects, like seeing the ebb and flow of that is , if there was only something in helm or whatever that basically the developer gives it's composed file. It doesn't necessarily do a translation. Cause we've got those converters, but like basically what if it takes that compose and then it applies like the commission controllers the ingress, all the other things exactly.

On top of it, not changing it, you know, it's like, yeah. I'll tell you exactly the project that we're working on. Cause this is pretty much what you said is that we spun it out of Rio. It's something where it's called Dolly right now. It's kinda hard to find because it's in one of our developers repos but. Is that we're specifically trying to do that where it's like, what if I take a composed file and effectively turn it into a helm chart. So it's like, I can do the live development.

I can do Docker Compose up. I can, develop my application. I get all the cool flow of Docker Compose, basically. But then when it goes to deploy, I can basically turn that into a helm chart, but turning into a helm chart, I also then need that layer that you're talking about of like, how do I inject my production that's in, like, how do I add my Istio configuration and my OPA, it's like all the stuff that you're you need to be compliant for your organization or whatever.

What we saw was that like with Rio specifically as real has a runtime aspect to it, and the downside of Rio was developers, like Rio. Like it resonates with them and they like it, but they can't convince the core. It, the people running Kubernetes. You need to run the runtime of Rio, , because they have to get the operations team to buy into it. And then they're like, oh, well, what is this? Cause Rio comes with a service mesh built into it. And so they're like, oh, it's STO.

And like, now, then you'd know Istio and all this other stuff, or it's Linkerd linker, it's Linkerd by default, I'm big Linkerd fan. But but yeah, so it's like, basically, it's like, how do you create from the developer side and interface the developers, like, but that can turn into a, basically a standard artifact that then can be handed over the wall to operations.

As much as dev ops are supposed to be together, but like, it is still in, so it's like, how do you think create a standard artifact that can move forward? And then people who are like, I understand Kubernetes and I don't care about your developer world. They know how to deal with it. And so that's like the solutions like that, that we're looking at right now. So anyways,

So you have like good auditing, you know what everything is. There's a lot of advantages of putting things into a helm chart, like the downside of helm and what everyone always complains about is the template thing and offering the charts. And then there's complexities around CRDs and whatnot, but those are all misdirected. Anger. Yeah.

And it's just it's an exhausting a little bit, like there's very few people that I know that can be a good developer and then also try to manage all these other things. So I think it's totally legitimate to say that let's keep those ops focused things in ops. And it's not that we can't share in it. This is all infrastructure as code. Then everyone should be able to see this stuff. But it's definitely a hard area that I think we're going to be seeing more and more people trying to deal with.

Hopefully, I mean, at some point it's gonna have to be added to my courses because as the community grows and my students mature, they're all having to deal with these issues too. And I'm gonna have to come up with something and that's been the problem too, with Kubernetes. It's like, you make a course that people want me to make a service mesh course. I'm like, okay. Which one? , I can't do a service mesh course on all of them. And I can't do a CI course on all the CIs

And it's really exciting because what we've done with RKE2 is we've built. We've revamped. RKE on the same. It's almost like the same platform is k3s it acts and behaves a lot, like k3s it's just as simple to interact with as k3s but instead of choosing a bunch of kind of lightweight, embedded options that k3s does RKE comes with like your full integrate enterprise grade fatness. It doesn't use any embedded components everything's ran separately. But it's super easy to get up and running.

It's really easy to administer so I'm really excited about that project. And , it's consolidating our approaches down to it's like, if it's going to be, you know, basically RKE can run cloud data center, that's where we, and then k3s has edge. But no matter where you run, we basically have the same approach. So it's RKE2 and k3s. Uh, Be the same approach in how you interact with it, but they have different capabilities in terms of, more tailored to the environment where they're running.

And because I'm not like I'm not a big fan of , certification but the reason why. The thing that's great about this is it's not so much about like the certification of getting a paper. Like, I don't think that proves anything it's about learning the content or whatever, because if somebody asks, this was like a Twitter, I think it was maybe there was on Twitter. We were discussing this of like, well, what's the value of like getting the CNCF certification for, Kubernetes?

I think it's called CK a, is that what it is? So in my answer was like, we actually have seen a lot of value in the certification for it's your way to like, you've got a competent employee, a smart person, they don't necessarily know a lot about KU Kubernetes. It's just a quick start for them to understand and learn everything about Kubernetes.

And so that's what this is like, if this is not about getting certified, like, like, you know, that paper means something or whatever it's about teaching you the content. And you can have your people. Your team members can just go through this. And so it's been very successful. So when this level two coming, I can't say direct, like, I don't know. I know that they've gotta be they're working on it because it's been very successful.

So they're going to be adding more content to that, but the exact schedule, I don't know, but I'm glad you enjoyed

If you're going to do Kubernetes, right, you probably need to learn a couple of distros at least. And this is definitely the, one of the ones I would recommend. So do the Docker mastery course from me and then do the duke Kubernetes mastery course from me. So those will get you the fundamentals and then you're going to learn a distribution. And this is a. I've checked it out. I've actually watched some of the videos and I would recommend this. This is great.

While we're there look up Adrian's channel

So I can't, yeah, I don't have a great answer there, but I think it's a worthwhile thing to to do. We are scaling up that team. As Longhorn is now I think we said it was GA, but it's officially supported like through Rancher and so we have customers and stuff on it. And we're continuing to expand in that space as the more people are willing to put persistent workloads and they find kind of use cases where Longhorn makes sense. But yeah, not a great answer for that one.

So here's what really the one to follow if you want to, more know about that stuff or, yeah. So yeah, I'm excited for the new one. I'll probably end up buying the new one. That's built on the cm four.

So he's always looking for people to put in some PRS and all of his projects are really great. He also is really good at making great CLS for the tools. So making them easy, very similar to Docker experience

They're aggregating all the content or whatever, but I'll tell you, this is a major problem in my mind that even if you aggregated all the helm charts, the majority of the helm charts out there are not good. And so the qual, like this is one of the biggest problems with helm today is the quality of the charts is very poor. And this is where something at Rancher we're investing a lot in is like, today.

You have some upstream, let's say Linux software, but then that gets repackaged by distribution like Ubuntu. And then you can eat very easily, install it in Ubuntu and they maintain like that package and make sure it's easy. And it's integrated with other things that doesn't really exist today for help.

And so we're trying to go down that route with Rancher right now of how can we start maintaining a set of helm charts that are based off of the upstream, but we they're well curated in that like, cause like every chart today wants to install Prometheus for example. But how do we make it? So every chart we can, like, basically use the Prometheus chart that we also support or whatever. And so that was one of the things that we actually we've started.

This effort with Rancher Tuto five is starting to figure out how we can curate a set of charts. Just so these things easily installed. Like you can install and upgrade them. That's like the biggest things. So yeah, so that's, I think a gap in the ecosystem right now.

And I honestly think like people's view of what a Kubernetes distribution is today is really like, you just get Kubernetes, but there's really no reason that like a Kubernetes, like, like there's room for a distribution to very similar, like to what Linux distribution is really a Colonel and a set of user space packages.

While we say, this situation is a little different where we're saying, well, Kubernetes is the kernel and you can get that anywhere, but you need like a distribution of the userspace stuff, all the CNCF projects. So this is an effort we're investing in on the Rancher side of, you know, how do we create a kind of curated set of like all the CNCF projects? So they just easily install and you can get charts that you just, you don't have to be an expert to use them.

And the helm chart is fairly straightforward. You need to read the docs. But the docs are quite good that we have out there because you need to understand the TLS, you know, what options you want to do for TLS but so if you have a k3s cluster and you want to solve Rancher, then it's really just installing that to helm chart. So you're going to download the helm CLI and then run home.

And everybody wants to put in a GitHub issue for docs or whatever. So it's, I think it's always a challenge. , what do you think about distroless images? Is it okay to use them as long as Kubernetes SIG released suspended, the kubctl, debug feature.

Yeah. I, the maintaining, like the CV scanning and all that stuff is is really just a pain. So the less you can have in the container the better. So I think a district list is a great approach. Scratch, if you can do it, if you're doing that, I find basically the only people successful with scratch images go developers right. No,

Like you've got to, you've got to get good at just troubleshooting containers with all the stuff in them, including shells before you take the next step of trying to remove a lot of that unnecessary stuff, unnecessary, meaning that it's your app doesn't need it. And that's the challenge is that a lot of this stuff requires you to iterative approach to your learning, as well as your deployment.

So most of the customers, in fact, none of the customers I've ever worked with over the last five years on day, one of deploying things into production have ever done like a scratch or a distroless image because they're ops people, they're troubleshooting people. Everyone needs to learn how these things work, right. Approaches those lean approaches to these things make it even harder. So yeah, the debug command is exciting. The ephemeral image idea.

I have this thing I love Nicholas is a netshoot repo. He's got a great network troubleshooting utility. It's basically just the hodgepodge of all the Linux utilities for troubleshooting distributed systems. I recommend and use that a lot. So that would be my go-to once this debug command works now.

So that's the plan right now that new UI that we put in there is very ambitious. There's a lot of work and we're honestly, we're still refining the user experience, getting it it's a balance between trying to get a lot of power and flexibility of what Kubernetes provides, but making it easier. And we're still actively working on a lot of the things there. The next release, you should which we've a lot of six months. Of when we do like the major releases.

So you're looking at like, basically another six months before that UI will be done. And then a year before we delete the old one.

But but that use case of, basically running VMs There's still, a good market or, where I see the KubeVirt playing is like, in the OpenStack space where OpenStack was trying to go after KubeVirt is a replacement for OpenStack. And it's it could possibly be a replacement even for VMware, if you're simplifying your infrastructure,

So that's where it gets a little more tricky, but at this point you can find a lot of information on the internet already of like adding a cloud provider to two k3s like digital ocean or AWS or something. So it's really, it's just the components and stuff we choose. They're great for the user, but sometimes for like, I don't know, like like enterprise needs and whatnot, they want things to be separate.

So I'll tell you, we, we run all of our production Kubernetes instances on k3s because they're very easy to deploy and orchestrate k3s can actually scale massively far. We scaled it to like 5,000 nodes whenever the max Kubernetes is. Yeah. So you can get pretty far with it. . Bret: Okay. Can I orchestrate Longhorn with Rook. No, I don't think there's any integration there. Yeah. No, we haven't really worked on that because it's Longhorn itself is quite easy to run.

It's just basically a helm chart. And then once it's running, you just interact with it as a storage class or there's a GUI for it too. You can just point click. So that really provides for us. Yeah.

And then all the options are opaque of like how, what are the options of how to spin up an actual cluster? So then all the options become pro the provider dependent and they're specific to the EKS or UKs or whatever. It all comes down to the maturity of the actual drivers and the problem with the cluster API historically, and this has changed, but the problem with it historically is they focused completely on a VM based model where they were only spinning up VMs and then orchestrating the VMs.

So there was no integration with say EKS or GKE, and those are the primary ways that people, I think people should be running Kubernetes in clouds. So there was, there's just always been this practical thing of the actual value you're getting out of the framework, but the idea makes sense. And so we're committed to cluster API. It is very slow going. It's like, that project's been around for probably four years now.

And so yeah, it will slowly evolve once, you know, VMware is the one who's pushing it and very heavily once VMWare started investing in sort of picking up pace and because VMware is dedicated to making cluster API their solution. , I mean, they want that to work for their products or whatever. And yeah, so it's like Fleet, for example, has nice integration with cluster API. So if you spin up clusters with cluster API, that can be automatically registered with Fleet.

And yeah, so it's this ongoing thing, but it's really messy,

And it's like, well, no, because it's, there's an idealistic side to the way cluster API works, where it's purely declarative and it, doesn't, there's a lot of things where it's like, well, no, once you get into the real world, people actually want to behave slightly different and that doesn't, it's not compatible with it. So there's, yeah it's been a thing, but we're committed and trying to figure that out. It's just going to take I don't see it being feasible for at least another year.

The biggest, I mean, this is always a challenge with training is that training, that the closer you get to real world training, like 200 level and 300 level training, the harder and more niche that training gets to create and take. So most of us are living in the space of T of giving courses that are sort of introduction courses, maybe a few intermediate courses, just because you're I don't work on any one company project for deploying containers. That's the same.

They're all using different CIS they're all using different server. Different ways of creating VMs, they might be cloud or data center. Just to even get you close to that and it depends on your role, right? Like, are you an operator or are you just a developer or are you responsible for everything that relates to container? Cause now that means monitoring logging security. Like there's a ton, that's like eight courses of things that you'd have to take. It's probably 200 hours of video.

So I don't, my answer is, . I don't know. I don't know if Darren

This is for very large organizations and things where it's like, they have a ranch, for instance, was then managed. It's more Ranchers, but yeah, it's crazy. But yeah, it's actually weird

It has like a, it has support for cloud in it that has like a compose ish format built into it. So you can just define, these are the containers I run. So Rancher OS is a container optimized or oriented Linux distribution. We have K3 O S, which is a more Kubernetes oriented one. There's a question just came in right now with like any updates on K3 O S is that so if you've noticed the news over the last couple of months, we were recently acquired by Susa.

That's not quite done yet, so I can't speak to a lot of the, like, we're still in the process of that merger, but you can definitely expect once that merger is done, you can expect some updates on what we're doing in the us space. So something I'm very excited about, but cannot talk about.

All right, well, I know we ran way longer than normal, but I thank you so much for being here. We were killing it on the questions. We had 2100 playbacks already on YouTube. The we were up to like 150 people's concurrently on the channel. So I think you broke the internet on my YouTube channels. So good job on that. That's a new record. I think 150 concurrent is the peak I've ever been at. People love Rancher and want to hear more about Rancher products.

And that's great because a lot of, yeah, a lot of us in the open source and the container space had been fans of you guys since the beginning. So it's great that you're still doing great work and putting out new tools and always have things to talk about. All right. You can find out more of us on the, I built the cloud and the Bret Fisher. These are their little Twitter handles. So go to Twitter. We're both active there. We both answer questions and talk to people on there all the time.

Thanks again, everyone. We'll see you next week back here on YouTube live and thanks. See you soon. I build the cloud,