Introduction to the Alphalist Podcast -- Tobias Schlottke: Welcome to the Alphalist Podcast. I am your host, Tobi. And today we are talking about platform engineering at Sixt, a 100 year old company that has a very modern approach to platform engineering. Meet Bojan Dimitrov: CTO of Sixt --
Tobias Schlottke: And I welcome Bojan Dimitrov, the CTO of Sixt, who is, yeah, I mean, I don't know, a very I guess everyone knows Sixt, everyone had a car rented from Sixt, but not many people know about the tech stack there, so it's quite interesting to talk about that. Sixt is actually 8, 000 people and 800 in tech, which is amazing. Boyan, do you want to add a few
Boyan Dimitrov: Hi, Toby. Thanks for having me. That's actually a great introduction to us. As Eric Six would say, we are pretty much a tech company with a current attached to it. And, I mean, what you just mentioned, I think underlines the amount of passion and investment we do in technology in the way we work. Bojan's Journey in Tech -- Tobias Schlottke: Okay, so for how long are you with
Boyan Dimitrov: Clocking in nine years, as we speak. Uh, Tobias Schlottke: And you made it from working in DevOps basically to the CTO seat now, Boyan Dimitrov: kind of. Um, I, I've always been all arounder, let's put it that way. Um, when I joined Sixth, my mission was to, um, build up, um, Our cloud platform, so that we can move a lot of workloads at that point, uh, over there, we saw a big opportunity to, you know, move quickly and use new technologies for building better customer experience.
Boyan Dimitrov: But at the same time already, uh, my task was also to build a number of products. Uh, one of them, for example, back in the day was, um, Six Share, which we were just launching. Or just starting to build. So, um, I've always been focusing also previously to Sixth on both the system engineering piece, if you like, and how or where we run workloads and how we do that efficiently.
Boyan Dimitrov: But also, um, you know, how do we build applications? How do we solve business problems? So, yeah, in a nutshell, I have many passions. Tobias Schlottke: So yeah, full Boyan Dimitrov: If you like that term, yes, more. I would describe myself as a problem solver. Tobias Schlottke: problem solver. Okay, that's great. Yeah, we all love to solve problems. When did you start solving problems with computers and
Boyan Dimitrov: Well, I started, I definitely started breaking things in the very beginning. Um, I started using computers when I was, I think, second grade, um, back in the day. I'm from Bulgaria originally, and I think I got a 32 bit private back in the day, which looks like, pretty much like a, like a, uh, Maybe, Apple to like, green screen machine, um, obviously got cooked into games.
Boyan Dimitrov: From there, I was interested also how do you build games, got into programming. As I was in school, I realized that building websites was a highly interesting, lucrative, but also profitable business back in the day, so I moved to that. From there, I switched to a lot of other different things which were, you know, having my interest back in the day.
Boyan Dimitrov: Uh, how Linux works, how system engineering works, how to build protocols. Got into C, um, then dug myself out of that hole, went into, uh, more, uh, I guess, more popular back in the day languages like, uh, Python, like, uh, at some point, Golang. So I've been alternating and changing my hat, uh, since then, and also moving in between really hardcore hands on jobs, especially I used to, like, Solving problems for scale, uh, but also more leadership, creating or trying to create business kind of jobs, which we're also keeping interested.
Boyan Dimitrov: So yeah, that's my, uh, story in a nutshell. I've done about 12 years of different startups. Uh, last I was working in the UK and then I moved to Germany to join Sixth and here I am nine years and going. Tobias Schlottke: you're actually old enough to still know the center and the marquee tags if you sold websites in the early days,
Boyan Dimitrov: Yeah. I also know what a floppy disk is. And then a Corp 56 bit modem, which used to be how you connect to the internet back in the day with this weird noise. Tobias Schlottke: yeah, that was great. That was great. Yeah. Um, so. But I think, I think your career is also quite, quite interesting that you actually stepped up from, um, like working, um, like being all over the place to, to, to the CTO seat now. Um, how did, how did that happen? Um, like
Boyan Dimitrov: Pretty just organically or, very similar to my career before. As I said, I love problem solving and I've been given a lot of opportunities here at Sixt. As I was iterating on my first missions, I was given additional opportunities, other opportunities to solve problems in other areas. and as the time went, um, I was also given the opportunity to run the, uh, the tech business.
Tobias Schlottke: so you basically built that platform first and then, um, this was your masterpiece. Uh, and, and, and, and, and then you, you rolled it out as a CTO basically,
Boyan Dimitrov: Oh no, uh, that the story? back, uh, quite a few years ago. Um, we've been, we've been basically using it for our main workloads for, for many, many years by now. That's pretty much describes my first two years only. At Sext, um, afterwards, I focused on other parts of the business, building, um, customer facing products, uh, or looking into optimizing, um, internal processing systems.
Boyan Dimitrov: So I've been moving into different fields, always trying to see what is, what is, what is the most, um, critical origin problem that we, we, we've got to focus on as a business. Uh, as I said, I'm I'm naturally a curious person, so I definitely like solving different kinds of problems. um, eventually that path led me to what I'm doing today. Tobias Schlottke: Okay. Sixt's Legacy and Modern Tech Stack --
Tobias Schlottke: Um, before we, before we talk about platform engineering, I'd like to hear your honest, um, uh, thoughts about, um, a legacy tech, tech debt in a 100 year old company. Like, uh, what, what is, what is your, your state there?
Boyan Dimitrov: Dynamic. So, SICK started with technology a long time ago. Uh, it was actually, uh, Eric Sixtauer, Chairman of the Board, who was running the company for, for, as a CEO for, for, for, for eight years, uh, who put the first lines of code himself in the eighties, and he built quite a few systems. Um, and since then, um, and under his stewardship, SICK has been, or started its tech journey, so to speak, and has been focusing on building and solving.
Boyan Dimitrov: um, problems using own software. Even fast forwarding to today, the majority, probably over 90 percent um, of the systems that we run are internal, built internally, um, because of that, of course, we have different generations of those systems, but I would say probably 10, 12 years ago. We started with the mission to refresh, um, out of our tech stack from one perspective, obviously to get better performance, de risk certain pieces, but also, um, our business is constantly evolving and the expectations of our customers obviously keep rising.
Boyan Dimitrov: So we had to do, um, a better job to capture that. So we end up rewriting a lot of, um, of our technology and also, um, you know, changing a lot of our business processes, uh, as such. By now, 2024, major pieces of that is already done, so to speak, uh, and the majority of our business, or big parts of it, are really, um, Written in the last few years.
Boyan Dimitrov: So it's a pretty modern stack. Um, we still have what you would call as legacy running. Um, and those would be parts which are actually doing a pretty good job. Pieces which, um, also business wise probably have seen less change. And for us, that is just not a big incentive at that point to invest into that.
Boyan Dimitrov: Um, but. In terms of, you know, ratios, if you like, um, the majority of our stack is basically service oriented architecture written either in Java or Golang. And, um, you know, that's, that's, that's the stack we run. Tobias Schlottke: And a few pieces of Erich Sixt, uh, COBOL code, uh, or
Boyan Dimitrov: sure some of that is definitely left, um, but, uh, there is more than COBOL, right? We also have, um, we moved to PHP from COBOL many, many years ago. Yeah, Oh, you really used COBOL. Okay. great for certain things still. And if you think about it, in the 80s, It was probably the best choice given what was on the market back in the day, what kind of options you had.
Boyan Dimitrov: Um, but, um, I mean again, we've, we've reduced our footprint there significantly. Uh, as I said, that's not a recent movement. Uh, a lot of our software moved into PHP back in the day. Um, in a, in a very big monolithic system. Uh, which also served us well at some point, uh, scaling that, but also how the business was going and what the changes were at that point.
Boyan Dimitrov: pushed us to, to, to, to approach it differently, especially that in, at Sext, we have heavy backend, um, processing as well. So, um, over the years, we moved a lot of that now into, into, into more service oriented architecture to enable our teams to iterate quicker, but also, um, obviously more, um, you know, more suitable languages for the kind of problems we're trying to solve. Boyan Dimitrov: Um, in the face of Golang or in the face of, in the face of Java.
Platform Engineering at Sixt -- Tobias Schlottke: Do you believe in microservices? Um, really? Uh, um, because I, from my perspective, talking to many CTOs, it kind of evolved from like the micro micro to more like the, mid sized services in the last years. Uh, what, what is, what Tobias Schlottke: is your perspective on that?
Boyan Dimitrov: um, as an engineer back in the day, uh, I used to love the approach. I still do. But even back in the day when, you know, this was the biggest trend, I would say what micro is, was hardly defined. It depends who would ask. For some people, that will be something like, you know, what you go.
Boyan Dimitrov: application functions. Uh, and that's what would they describe as micro. And, um, for others, it would be like, you know, 50, 100, 000 line applications. So it really never, never really had a good definition. Um, looking at Sixth and how we do things and coming to your terminology, I would say we also adopted a more mid sized application approach, but that just has to do with.
Boyan Dimitrov: The way we cut teams and the way our, um, our, um, our problem space exists. Um, the challenge that we have and the fun part of the business is that we operate in over a hundred countries. And if you think about it, there is a lot of complexity, um, to, to, to satisfy their both from compliance, customer expectation, um, how different processes run in different countries.
Boyan Dimitrov: So. Having extendable business logic is a big part of it. But also on the other side, you've got to build software that you can, that has a bit longer life cycle that you can maintain. Otherwise, you know, you just end up in a situation that maintenance take over, takes over 50, 60 percent of, you know, team's roadmaps.
Boyan Dimitrov: And then, you know, they don't have enough time to, to, to, to innovate and to. Uh, you know, meet our customer demands. So, our approach was to invest in bigger services and reduce that way, um, the maintenance efforts. We've also invested heavily in automation when it comes to how those, how this service oriented or microservices architecture is, uh, managed and lived.
Boyan Dimitrov: We tried to take out a lot of the heavy lifting that comes with it. in terms of, uh, application lifecycle, monitoring, uh, alerting, um, and provide it as a service to our engineering teams. Because at Sixth it's a, you build it, you run it, kind of a, kind of a setup. So, our engineering teams not only build software and throw it over the fence to, to, to platform teams to, to run, but they are running it themselves.
Boyan Dimitrov: And, uh, For this to be successful and not, you know, to have overly big teams or to, uh, to, to, to spend so much time in heavy lifting, which adds zero business value and every single team reinventing the hot water. We approach this by standardizing a lot of the, the, the, the scaffolding around running microservices.
Boyan Dimitrov: Platform libraries, authorization frameworks, um, H Layers, as I said, Monitoring, and the Hopser Ability Stack. All of that is actually provided to our development teams as a service, uh, and it's maintained in many cases either by guilds or central teams, uh, so that they can actually focus on solving customer problems.
Tobias Schlottke: So, um, that sounds as if you. have like platform communication as well. And a concept, uh, how, how people actually, uh, actually figure out the best way to do things, uh, through, through Guilds. I just heard that there was just a buzzword from like, or not buzzword, but it's known from the Spotify model. I'd like to talk about that as well. But I first like to hear about the, the, the actual platform. Like what, what, what, what do you do there? And why do
Tobias Schlottke: you do that? I mean, You, you mentioned like monitoring, et cetera. Like, how does it concretely Tobias Schlottke: look like, let's start with the mindset, right?
Building and Managing Sixt's Platform -- Boyan Dimitrov: When we were building the foundations of what we call our sixth platform, which we codenamed GoOrange, um, for us it was important that we don't just build a platform in isolation because those approaches usually end up horribly, but as we were building this platform, we were already building a number of new products and migrating some very serious workloads from Our, our monolith back in the day, uh, onto, onto this.
Boyan Dimitrov: So we really approached this end to end and we always built as much as we needed to solve, you know, the, the state of the state, uh, that we were in product wise. The mindset we wanted to have is, hey, we are going to do a bunch of bets here. It's always a bet in terms of technology specific tooling, et cetera.
Boyan Dimitrov: Some of those will end up being right. And HOL. Others will be completely wrong and in one or two years they would either fail us or we would have to replace them because, you know, they went out of love in the tech community and for us maintaining that is just too much effort. Um, knowing that, we adopted a very modular, uh, approach.
Boyan Dimitrov: Um, we wanted to make sure that individual pieces could be taken out. with not too much effort, knowing that at some point we are going to have a pretty big scale and complexity, obviously, once, you know, 800 people are iterating on that. Our approach was to solve for the 80%, always batteries included, and for the rest of the 20 percent we would always offer APIs so that they can build on top themselves the solution that they would need.
Boyan Dimitrov: And basically this is, I'll give you a couple of examples for you to understand how that looks like in practice, but this is the approach we took for every single piece of our platform. On a more, let's take DevOps, if I give you a more DevOps example, it looks like this. Uh, when we were building the foundations of our CI city back in the day and how deployment works, et cetera.
Boyan Dimitrov: Um, how do you take, you know, code from, from a repository to build it, to get it into, into, into production and into different staging environments, et cetera. We built a system that, uh, was already integrated into the two main ways back in the day. Our teams were. , uh, interacting or, or, or, or, or, or, um, you know, deploying and managing their software, that would be ChatOps in our case, which is, you know, you have Slack and then you can deploy your application, see where they are, what versions they're running, et cetera over there.
Boyan Dimitrov: And the other way would be Jenkins for us, where people would be building their, you know, CHD pipelines, uh, directly there. Uh, and practice, you know. Brooklyn Deployments with other cases. So when we were building our deployment system, it had from the get go hooks and integrations to, to, to those, to those two approaches via an API.
Boyan Dimitrov: And also that same API, so that's all the 80 percent right? And that same API was also used afterwards for the 20 percent of our teams who might have a bit different way of, uh, you know, interaction and, and, and how they, how they, how they build and deploy software. Like there would be people who, for example, use GitHub, so they would be doing a lot in there and it would be linked to their flows.
Boyan Dimitrov: They would be using the same deployment system API, uh, and build something custom on top to make sure that, um, you know, it fits best their use case. But what this allows us to do, if you think about it is. Now we have a central piece, this deployment system, which is integrated via API everywhere. And as our platform team decides to iterate on it or change certain pieces of that technology, they can do that and automatically everybody gets the benefits, right?
Boyan Dimitrov: Instead of us, uh, you know, having five, six different ways of doing that in completely different code bases, maintained by completely different teams, and then. Um, you know, when you have a challenge or when you want to change something fundamentally, you end up spending years in, you know, migrations. This was pretty much the approach we took across the board. Boyan Dimitrov: Um, the other thing we did, yeah, go ahead.
Tobias Schlottke: but, but sorry to, to, to stick with that example. So if you want to get rid of Jenkins, for example, um, because it's no longer, um, a love technology around, um, or does it, does it matter at all? Or would you say like, does it, does Jenkins, as you mentioned it as a, as like the, the core of it, does it matter? Tobias Schlottke: Um, and if so, could you replace it because you build an abstraction layer around it,
Tobias Schlottke: basically with, with the APIs? Um, Could you move to Tobias Schlottke: GitHub Actions, for you, if you just look at it from perspective of, um, the deployment experience and tooling that we have. Um, hundred percent, right? Because if it's going to be, uh, I don't know, um, Argos deal, whatever you pick up there, um, we would still integrate it to the same deployment system we have if we don't want to change completely our user experience as well.
Boyan Dimitrov: And it will just work. When it comes to the builds, obviously we'd have to move them, right? That's a migration. But, um, It would still allow us to keep the same developer experience because as I said, there is partial, there are some people who, or some teams who would be interacting directly with Jenkins.
Boyan Dimitrov: But there are also many others that, um, just rely on the default automation that we have when you set up a new project. And they only interact with the so to speak end user interfaces being again, chat ops or some graphical UX. to deploy their software and build Scapan automatically just based on a default template in Jenkins.
Boyan Dimitrov: And if we were to move that, they would not know a thing because they just don't interact with that system to begin with. Um, from an engineering perspective, the way that looks like is, I want to start a new application, um, to solve a particular problem. Um, I basically just, just create that. I say, Hey, I want a new, um, I don't know, Golang application in, in Slack, just to get to, to one example.
Boyan Dimitrov: And from that point on, um, as you add a bunch of metadata, um, takes about five seconds, you get repositories being set up for you. Obviously pretty, a lot of metadata being injected in terms of, um, Who is the team owning this application? That relationship already gets created and that creates also a lot of permissions who can deploy it, etc.
Boyan Dimitrov: All the build pipelines by default get created and that's good. Let's take Jenkins, right? our core Golang framework would already be prepared in that repository for you to just start iterating on with already hooked up examples of how authorization would look like, etc. Next slide. The moment you deploy this application, even if it's a hello world, imagine you just do that and you decide to deploy it, you automatically get instrumentation, observability, uh, all of that stuff set up for you, uh, piped into your channel, um, dashboards being already shown to you for that application running.
Boyan Dimitrov: That's pretty much the experience we actually have at Sixth, and we've been having for many years. It's nothing new. Um, and. Our approach is that we continuously iterate on that experience and change things. And because we change that for 100 engineering teams in a way, that's how many we have, we always make sure that through automation we are able to iterate with it without having our engineering teams to constantly, you know, having to do extra work on that.
Boyan Dimitrov: Obviously for some things you have to, like, we expect our engineering teams, for example, to update their frameworks. That's just as you would, hygiene, right, as you would update your libraries. But when it comes to changing core platform pieces, we try to make that as least disruptive as possible. Tobias Schlottke: How big is your, I mean, you mentioned, or I mentioned, I think that you have 800 people working in tech. How big is the people taking care of the platform?
Boyan Dimitrov: So, Boyan Dimitrov: depends which part of it. We have one team, um, six, seven people really focusing on the software platform layer. Things like service discovery, um, our core frameworks, authorization libraries, things like this. Um, and then we have probably another 20, which are focusing on cloud automation and basically scaling our, our, our, our platform infrastructure, things like Kubernetes, Kafka.
Boyan Dimitrov: Key Clock, etc. The three Ks, which are the foundation of a lot of the things we do. Um, so, compared to, uh, our product engineering teams, our platform team is actually pretty small. The ratio would be something like 1 to 11, probably. If we really look at the core platform, of course, our platform team also is comprised of other engineers who are focusing on, you know, Uh, a little bit different areas as well, things like, um, security, a few other things, right?
Boyan Dimitrov: But the core platform, what we are just talking about right now, is actually pretty small. And that has been possible because, well, A, we have fantastic people. Um, so, it's not about the numbers, it's really, we have great people who can do a lot. But, to We've also invested heavily from the very beginning to automate everything, or as much as possible, especially those foundational pieces. Boyan Dimitrov: And that has served us well, you know, looking backwards.
Tobias Schlottke: So Tobias Schlottke: everything is like infrastructure and code, um, then Kubernetes, then, um, like automations for every layer. Um, and you use a lot of open source, if I see it correctly, like, I don't know, monitoring, for example. We didn't talk about concrete technologies, but there is something running that monitors your system on your, um, Boyan Dimitrov: Yeah. Tobias Schlottke: Premises, basically.
Boyan Dimitrov: uh, we obviously invest heavily in open source and use a lot of open source technologies. Of course, depending on, um, um, the problem we're trying to solve and what is the best for us. We also partner with, a lot with, uh, with different SaaS vendors. Um, again, um, for us it's the same, the same, the same kind of philosophy we have for how we want to enable our product engineering teams to iterate on customer problems.
Boyan Dimitrov: Uh, we obviously also want to enable our platform engineering teams to focus on the things that make a difference for us and not get buried into maintaining, you know, flaky, complex things. Thanks. So, from that perspective, we also try to, um, use as many managed solutions as possible and reduce our, reduce our efforts there. Tobias Schlottke: So talking about like application performance monitoring, for example, there you use something proprietary, I guess,
Boyan Dimitrov: Exactly. We actually use Instana, um, it, uh, it was a startup, Boyan Dimitrov: um, which got acquired by IBM recently, uh, but we've had, for example, a relationship with them from, you know, their early years product was developed heavily based also on. on a lot of feedback and challenges we've had and it fits the way we work pretty well. Tobias Schlottke: Okay. And you then host everything
Boyan Dimitrov: Airbus is our preferred cloud provider and the majority of our workloads are running there. Um, of course we also use other cloud providers for, for, for, uh, certain types of workloads. Um, it always depends what is the best solution for a particular problem, but the majority of our workloads are running Airbus. Cloud Strategy and Cost Management -- Tobias Schlottke: How do you make sure that cloud cost doesn't, uh, uh, like totally explode, uh, throughout the
Boyan Dimitrov: Um, that's been a thought from the very beginning. Um, I've been working in that space now for, what, 13, 14 years. So, In many different startups, I've seen how that could go, but also what can you do and how you should approach it to make sure that you leverage the best in terms of managed services, in terms of you know, time to market, but also are able to reduce your costs.
Boyan Dimitrov: And that usually boils down to, you know, this buzzword that is still probably used in many places, building cloud native. Uh, obviously it sounds cheesy, but the reality is. This is the best way to actually reduce your costs. A, you've got to know, um, how pricing works in the cloud in terms of, um, the main services that you're using there.
Boyan Dimitrov: B, um, the approach that companies take and how they, um, they, uh, how to say, Let's talk like this. Integrate, um, cloud technologies into their business, into their teams, both at that point business and, and technical teams, is usually tailing into how much cost they're going to be spending, in my experience.
Boyan Dimitrov: At Sixth, um, we wanted to do a couple of things, strategy wise, when it comes to cost. And it's always cost linked to execution, linked to, you know, in the end. Solving problems and creating experiences faster. They're kind of interchanged and we would focus on one or the other depending on, you know, the problem space.
Boyan Dimitrov: The, what we wanted to do in the beginning was, again, what is our focus? We always start from there as a company. Our focus as a company is to create the best mobility experience for our customers. Dot, right? So, from there, uh, When you go into tech, you want to make sure that in the end, your engineers and your teams, they really focus on, um, understanding their customer problems and, and, and, and, and creating solutions for them and not get bogged into, uh, infrastructures and a gazillion different technologies, getting, you know, distracted from that part, but more.
Boyan Dimitrov: They should use technology, uh, as much as needed. to create that experience. So, to achieve that, what we consciously decided early in the day is that we do not want that every single team creates their own cloud space, so to speak, and starts building their own infrastructures. This is an approach that is, or can be, successful, uh, in, in certain companies under certain conditions, but for us, Uh, this was evaluated at that point as heavily distracting.
Boyan Dimitrov: So, we decided to encapsulate major pieces of the cloud's complexity into, again, what we call our platform, and have that completely provided as a service to our engineering teams. What does this mean? Thank you. As an engineer, as I said, you will be responsible to run your application. But we will have created an experience for you where you don't need to care about EC2 instances.
Boyan Dimitrov: Let's take AWS as an example. You don't have to care about servers. You don't have to care about how to scale those servers. They will be there for you, right? You have to care about your application. You have to care about the performance of this application and understand that intrinsically. And You have to assume that everything else will be there for you.
Boyan Dimitrov: And that's pretty much how the, um, um, how the, how the responsibilities are split. basically just deploy into Kubernetes and the rest is done. just works for you, right? There are a number of patterns and automations when it comes to how you expose things to the internet and how you make sure that, um, in the end you have full observability of who you're, um, who is calling your application, uh, etc, etc, etc.
Boyan Dimitrov: But again, all automated for you, prepared for you, so you just use it. You just deploy as if you do, for example, on Heroku, and that's it. Uh, the, um, other point was obviously there are easier and harder, uh, handover points.
Automating Persistence and Infrastructure -- Boyan Dimitrov: One of the hardest ones for us that we took a few years to solve, it took actually a specific technology to emerge before we were able to do so, was how do you deal with persistence and how do you automate that as an experience, right? Boyan Dimitrov: That's, that's, that's a sweet spot. Thanks. Thanks. Tobias Schlottke: That was,
Tobias Schlottke: I just wanted to ask about going to go there, right? It's because, you know, running a container on Kubernetes is easy. How do you do persistence, right? How, how, how does, how is this not a ticket? And for us, it was a ticket for a long time, um, you know, because we, we, we tried many different ways.
Boyan Dimitrov: Um, our platform teams would be using Terraform to automate their infrastructures. Um, and we've, again, iterated on that also a lot. But regardless, um, We provided trainings, we got some engineering teams who are more, let's say, infrastructure savvy into that. Then just start opening PRs into Terraform for infrastructure.
Boyan Dimitrov: It always felt unnatural and against, again, the pattern that we were trying to create. So for a few years, when you need a database, uh, um, or database schemas, et cetera, you would have to, you would have to create a ticket and we were very annoyed. Uh, because of that, um, but at some point, um, a couple of years ago, uh, we finally were able to, to, to, to address this problem and create again, uh, or use our default recipe to solve it, which is create an API, uh, and, um, integrate that into everything else that we do in terms of automations to that.
Boyan Dimitrov: Our engineers can request S3 packets, they can request RDS instances, they can request schema changes and run them themselves without having to go through ticketing systems and things like that. This was actually pretty pivotal. We probably reduced the ticket like interactions between our teams by probably 20 30 percent which is big. Tobias Schlottke: But, but, um, just to understand it fully, you ended up using AWS native services for Boyan Dimitrov: We've always been using
Boyan Dimitrov: AWS from day one, uh, AWS Native Services, but the missing glue at that point was a technology called Crossplane. Maybe you have heard about it. Um, because what it allowed us to do was, from one side, it allowed our platform teams to basically embed best practices when it comes to how we want to provision databases, how we want to provision S3 packets.
Boyan Dimitrov: Uh, I don't know, making sure versioning is always enabled, making sure certain security considerations are always there, right? All those things that, uh, are standard in our company and we want to enforce and we want to make sure that they're there for the get go. But from the other side, it allowed us to expose a well defined API to Kubernetes.
Boyan Dimitrov: For that, to our, uh, engineering teams, which is how they interact with the system today to begin with. So in a natural way for them, so to speak, instead of them going into Terraform code, running Terraform code, et cetera. So this was the missing glue for us, which really clicked because, um, again, how we provide, um, how we automate the whole thing is via APIs.
Boyan Dimitrov: And, um, this allowed us to create this experience which, uh, which felt so natural for our engineers. And today they are able to do all those things themselves as well. Uh, Debating AWS Services vs. Self-Hosting --
Tobias Schlottke: And do you think it's smarter to rely on AWS services in long term, or do you consider every once in a while, I mean, it's also evolving, right? Persistence in Kubernetes and, and, uh, like great Helm charts for Postgres, et cetera. Did you ever think about that as well? Or is it just a too big, too big Boyan Dimitrov: how do you mean that? In the sense of,
Tobias Schlottke: Well, you could, you could, you could theoretically also host your own databases, right? You could also spin up your own instances for databases Boyan Dimitrov: ah, that's coming. Standardizing Cloud Services --
Boyan Dimitrov: Well, our strategy in general with cloud is that we really focus on standard services. Let's go with that way. A, much easier to manage. Hence, you know, keeping your, your, your, your, your, your team slim and reducing the complexity for them. Uh, B, much easier to upgrade. So, to give you a concrete example, we have a default go to for 99 percent of our use cases.
Boyan Dimitrov: If we are going to go for, um, you know, um, SQL like persistence, we would go for Postgres. Running the RDS because, again, that fits into, into the way we do things, but it will be Postgres. There will be not, there will be not 10 flavors of SQL, uh, solutions running around there. And if we go for a NoSQL, again, there will be one single choice.
Boyan Dimitrov: And that's how we do it. If at some point we decide to change something fundamentally, it makes it super easy, right? Because we don't have to think about all the quirks of all the different services that may exist. Maybe for some more proprietary things, there is also no alternatives, which makes your life very hard.
Boyan Dimitrov: In this way, we keep complexity low. And as I said, we solve. 99 percent of our problems, there is always 1 percent where there is something very unique, but at this point of time, obviously, first we, we challenge ourselves, did we understand our problem? Is it so different and uncommon what we're trying to achieve?
Boyan Dimitrov: Would that, would the, would the effort in adopting this new technology or this special technology really make the user experience better? The answer is yes. Then we do it. Um, and of course, if the answer is no, we abandon and we go back into, into a standard way of doing things. Tobias Schlottke: Understood. Evaluating Proprietary Platforms --
Tobias Schlottke: Um, and, and, and do you have, how do you, how do you think about, um, platforms, proprietary platforms that, that, that come out these days, like, like, I don't know, Humanitech, for example, is a German company that, that, um, actually builds platform, a developer platform as a service and IDP, basically. How do you think about that? Um, do you basically, do you see your, like your product or your, your core being competitive with that?
Tobias Schlottke: Like, don't people like ask for, I don't know, a nice interface where they can, can configure the instances they, they, they would like to run? Uh, do, do you have some sort of, uh, a layer that at least allows you to configure all the, infrastructure that you actually need for your services? Or like, how do I have to imagine that? Tobias Schlottke: I know it's a Very broad question. Tobias Schlottke: but
Boyan Dimitrov: we have, because we started eight years ago, right? And as I said, we have insanely strong teams, like, and we are very proud of them. We have insane amount of UX as well. Um, and it's an amount of automation, right? Um, So, again, you can see visually all your applications. You can see all the libraries that are running in production into those applications that you have.
Boyan Dimitrov: You have end to end tracing, seeing all the different applications calling you, what you're calling, uh, obviously any kind of golden signals you can imagine. I'm not only talking the things from the IPM. I mean, this very broad space. Um, And at this point, again, you can always do better enough because, and we keep iterating, but at this point I'm not sure there is something, um, heavily missing. Optimizing Cloud Costs and Performance --
Boyan Dimitrov: And on the other side, we've spent eight years in relentless optimization, which is again, how we keep also our cloud costs, um, uh, significantly on the lower side. Every piece of our stack is heavily optimized for purpose and for how we run applications. And because of this symbiotic relationship between our application architecture, Like, there is one way of service communication.
Boyan Dimitrov: There is very certain way of how we expose APIs and what kind of APIs we expose to the outside to our mobile applications, to third parties, to websites, etc. Those standards, they were defined for each other, so to speak, so would be very hard for us to just take something from the outside, plug it in, and get better results at this point of time.
Boyan Dimitrov: This being said, we constantly screen the market for nuggets, which we can, uh, integrate into our platform or replace pieces of it. By now, looking eight years back, what we started with has drastically changed in a very seamless way. But probably Kubernetes. Uh, that's why we Ks, it's just an internal joke, like Kubernetes, Kafka, and Kikrok are the three technologies that, you know, H2O has survived.
Boyan Dimitrov: Those were massive bets back in the day, all of them. When we chose Kubernetes, um, again, Mesos was still the more widely used, uh, approach for Kubernetes, for container workload scheduling or other point of application scheduling. NoMo was up and coming, right? They, they, they, they, the, the, the choice of Kubernetes was a bet, a huge one.
Boyan Dimitrov: Um, uh, and we were just lucky, right? Uh, we chose well, so to speak, but in end, you never know. Uh, other technologies did not h well, and they were, you know, they felt it into oblivion. They were changed, and we kept iterating on that. You know, and we keep doing that year after year. So, yeah, this is where we start just taking a platform from the outside, which is a Swiss Army knife and plugging it in.
Boyan Dimitrov: Um, at this point of time, again, given the amount of years of experience that we've put into it, would yield probably, um, worse results. Leveraging AI for Development -- Boyan Dimitrov: Plus, by now, um, we're at the point where The advancements, especially when it comes to, um, AI, is starting to get super interesting because it will be the thing that would allow us to go to do the next steps.
Boyan Dimitrov: Now that we have well defined frameworks, um, for our core applications, for how we build, I don't know, back office UX, design systems, all those things, what we are looking for at the moment is, um, How do we train models on that? Well, or, or, or use RAC, right? For, for, for, for, for, and use LLMs to speed up the creation of some of those boilerplates and start even going higher level and, um, start solving more basic to medium problems automatically, like What happens in our business often is we create a new product or a new feature, a new experience in a certain area.
Boyan Dimitrov: The moment you launch this at scale, again, millions of customers, many different countries, a lot of stakeholders. It doesn't matter if it's internal or external. The first thing that people would ask for is, hey, uh, I need my analytics, which by now we have in a pretty good automated state. But B, I need a back office, uh, uh, application to basically be able to configure this and that.
Boyan Dimitrov: That's for the different countries, all those things. We have a gazillion of those things. And as I said, we have a framework, how we build them as quickly as possible. But, you know, the next iteration for us is as a business owner, product owner, et cetera, or an engineer, I want to create this back office application, which shows me X and Y and Z in a tabular view based on those well defined APIs, go, right?
Boyan Dimitrov: And this should be up and running in 30 seconds. This is. Tobias Schlottke: That would be great. I mean, the Tobias Schlottke: good old admin interface is, uh, really, like, very hard to find a good
Tobias Schlottke: one, right? Like, example, for a couple of years, because again, we started with that stuff many years ago. Eventually, for example, Backstage got open source, a couple of others nowadays, in terms of frameworks for building back office applications. So been, hey, do we move? Do we migrate? Does it make sense?
Boyan Dimitrov: Because, you know, internally, we have the same thing. Do we open source our thing? So we've been busy a little bit with that. But now, in the last year and a half, we are more like, hey. Actually, forget about it. We have well defined design systems. We have well defined frameworks, backend, frontend, all of it.
Boyan Dimitrov: Let's just focus on how we faster iterate on them, right? That's pretty much, um, that's pretty much the, the, the, the, the stage we're at at this point of time.
Tobias Schlottke: So, to understand it correctly, like, your internal application stack, um, like, to build internal applications, um, like, many people use, like, I don't know, retool or, I don't know, in Rails, Active Admin or stuff like that. Like, um, is that proprietary? Like, do you, do you, do you build it yourself and now you're building? You're using LLMs to basically let product owners express, uh, their, their, their interest. Hey, I want to build an user interface to, uh, I don't know, manage
Boyan Dimitrov: That's pretty much it. Uh, that's, that's a step we want to take because again, if we think about it, by now we have, um, I don't know. Hundreds of those things running around and much of the components, what makes a back office tool, for example, they're kind of the same, right? There are, I don't know, 5, 10, 15 different components that we use across the board.
Boyan Dimitrov: And we have, again, also for our back office world, like a framework when it comes to user authentication, permissions, all those more common things, like color coding, whatever you want. And now we are at the stage of, hey, instead of just building them manually, how can we speed this up, right? If you think about it, it's more like, hey, if I start a startup tomorrow, what I'm going to do is, I'm going to do no code slash low code immediately, right, to solve a particular problem.
Boyan Dimitrov: I would not be building those things from scratch. No way, right? And, uh, we love this experience. We are way too complex for that experience, unfortunately, right? In terms of our scale and systems that need to be integrated, complexity of the process. But, luckily, in house is standardized. So, it's like, hey, can we offer a, you know, no code slash low code experience internally?
Boyan Dimitrov: Based on our stack, because it's well defined, again, all of it, standards are there, those standards are written in code, and it's a textbook case for, you know, LLM experience. And this is one of the areas that we are looking into at the moment. where we believe we will get pretty good results. Obviously, again, we heavily focus our data science efforts usually again on solving business problems, but when it comes to just looking technology wise for how we can speed up our execution, this would be one of those things that is super interesting for us.
Tobias Schlottke: Is that, um, the, the biggest change or innovation you see happening in, in, in your, on your platform particularly, or do you see other things ahead that, um,
Boyan Dimitrov: Oh, there are many things that excite me. That, yeah, um, that one for sure will be super fun and it's just a clear business case, right? It's just an obvious one. Um, but there are a few others which are, um, which are equally interesting. Again, for me, it's super difficult to, um, Just focus on the platform at that point of time, because I just look at our whole business and technology landscape and there is just so much, so much happening there.
Innovations in Digital Experiences -- Boyan Dimitrov: Um, we are building some pretty cool, uh, bleed digital car rental experiences. Um, where we really allow our customers to, to, to, uh, you know, go through the whole, um, to the whole experience. digitally, uh, which is super cool and there is so much technology behind it and it's a really challenging thing to, to, to scale, uh, worldwide.
Boyan Dimitrov: Then, um, you know, we're doing some, um, some, some, some, some super crazy stuff when it comes to our website architecture and experience. Uh, we probably have the fastest, uh, website in the industry in terms of user experience. It's all actually built on, uh, micro frontend architecture, which You know, we started looking into that a couple of years ago, and we're probably one of the few who really has, um, at that scale, that kind of technology powering it behind it.
Boyan Dimitrov: And for us, it's working pretty well. Um, we are interacting with our mobile applications via gRPC, which was for us a big achievement, and again, a massive speed up when it comes to experience, but also reduced so much, um, complexity, because in the past, Uh, we've adopted, um, gRPC for service to service communication internally, but we would always have to have a, or had to have, translation layers, uh, for our external clients.
Boyan Dimitrov: And now we can just go native, uh, which removes a complete layer in terms of complexity, uh, which makes me excited. Again, I get easily excited about a lot of stuff. Tobias Schlottke: What is the, the, the, the coolest technology that you recently discovered and that excited you as you, like, seem to be like, very excited about things? Like what, what is the Tobias Schlottke: The craziest thing that you saw thing? Yeah. Yeah.
Tobias Schlottke: Oh, well, the best thing, let's not solve the, let's say the craziest, but the best thing like, um, that you annoy your friends with. Like, Hey, I discovered this. Uh, use it, use it, use it I, I always have those moments there.
Boyan Dimitrov: I think I'm particularly nagging when it comes to mid journey. I'm a big fan, I have to say. Like, I knock all my friends about it. Because, again, for me, for example, personally, I've just, I just use it almost every day for, for, for, for gifts, for, you know, just sending funny pictures around or for, you know, creating stuff.
Boyan Dimitrov: etc. I just love this technology. And, um, the fact that, because when it comes to creativity on that side, art, I'm so horrible. I'm just horrible. And this thing allows me to just describe what I want and puts all my crazy thoughts in 15 seconds into something tangible. And I don't know, I've, I've, I've tried obviously a lot of other LOMs for the same.
Boyan Dimitrov: But with Bitjourney specifically, I think we have a very like minded, uh, uh, uh, thinking when it comes to, to, to, to, to final outcomes. So I'm particularly happy with it. Have you used it? Tobias Schlottke: Um, so I, yeah, yeah, absolutely. Uh, I, like whenever I touch it, I think I'm too old for Discord. Boyan Dimitrov: Oh, man.
Boyan Dimitrov: Oh, man. I love this. I've been in Discord before Bitjourney. I think that's why. Again, uh, I told you I started with gaming. I, I didn't, men, I never mentioned that I stopped it. Right. Actually, I never did. I'm still, even if I always don't have too much time for games, I, I mean probably, I don't know, 15 different communities there still.
Boyan Dimitrov: And, uh, uh, yeah, for me it was natural. I love this call Tobias Schlottke: So, but, but, but Slack in your company, Slack is still the, Tobias Schlottke: the key, key tool for communication. or, Uh, we use teams when it comes to video and when it comes to really conferences and calls and things like this. Um, but when it comes to chat and automation and engineering, it's heavily.
Tobias Schlottke: Yeah, yeah, sure. The good old problem, right? Like everyone uses Outlook and enterprise organizations and, and, and teams. And then like the digital natives really want to, want to use Slack and ideally, Tobias Schlottke: I don't know, Google
Boyan Dimitrov: I think in this case, there is just no way you can go, um, or at least we can go, uh, uh, and say, Hey. It just makes all the sense in the world to choose one tool. We try to use one tool for the job. Obviously the whole tooling space there, uh, keeps evolving. And at some point we might arrive, uh, at a moment where one tool really solves all the use cases.
Boyan Dimitrov: But at this point of time, I think the experience is vastly different. Like, um, you know, when it comes to video, um, I think Teams is just a fantastic experience. Or Zoom, right, as well is pretty good. I think Slack, um, is Discord as well, by the way. They're pretty good when it comes to chat. And they, you can't compare those two. Boyan Dimitrov: Uh, well, there are a couple of new open source things. Maybe, maybe that's unfair at this point of time. But,
Tobias Schlottke: There have to be like a few things. Like, I really don't want Salesforce to mess up Slack. I love Slack. Um, so, uh, I hope that like it, it, Boyan Dimitrov: well, we see how it goes, right? But so far, again, we, hope it, hope in engineering, we, we use it heavily.
Tobias Schlottke: Understood. Um, just like a final, like, loopback to your, your, your platform ideas and communication. Uh, as we, we're talking about communication right now, like you mentioned Guilds, or we, we briefly touched Guilds at the very beginning. Like, how Tobias Schlottke: does that work at Sixt?
Boyan Dimitrov: the way I mentioned it was not like in the context of, uh, what you may refer to as the Spotify Guild model, uh, nothing like that. I think it was the best term to describe what we're doing, so as I've told you, we have standard frameworks, standard libraries for certain things, because again, we definitely don't want to have, uh, ten different authorization frameworks running in a company, right?
Boyan Dimitrov: Then the question comes, how do you keep iterating their bug fixes, new features, etc. Obviously, we have a core platform team that is just focusing on that landscape. But as I also mentioned, our teams by standard, they're pretty thin. And how do you maintain something like this all the time, right? So.
Boyan Dimitrov: That's one problem. The second problem when you build things like this, which is so close to developer experience, is that you risk that what you start creating for your engineering team becomes foreign for them, right? Becomes just too far away, something someone else is deciding. So how do you get their feedback, their experience, their drive and energy into what you're building?
Boyan Dimitrov: And not have a silo where there shouldn't be one, right? So for those, um, for those areas, for those topics, we have a more what we call internally a GILT approach that the way you have to think about it is there is a, like, think about it like an open source project, right? There is a community of people from our platform teams, from our product engineering teams that are owners of this thing, so to speak.
Boyan Dimitrov: They co own it, they iterate on it together, and they have a process internally. How they, you know, version it, how they deploy, develop it further, what kind of features they allow into it, what kind of features they disallow. Um, this is how some of those things that we have, especially the ones which are at the boundary between, uh, our product and platform, uh, engineering organizations are maintained.
Boyan Dimitrov: I think this is a very successful model because it allows us to iterate jointly on this, um, and, uh, it reflects well the ownership nature of it, right? Tobias Schlottke: Um, and, and, um, like besides the, the code that is written, there's obviously like also a, like a Slack Tobias Schlottke: channel where people exchange
Boyan Dimitrov: are, you know, weekly, bi weekly, monthly gatherings on the topic. They have a Slack channel where they hang out all together. Um, pretty standard. Tobias Schlottke: Okay. Understood. So slowly coming to the end. Um, what is the, the, the top thing you recently learned and you want to pass on to other CTOs? Balancing Business and Technology --
Boyan Dimitrov: I'm not sure if it's recent, but I think as CTOs, the biggest challenge we have is that, um, we have one leg, uh, on the technical side and one leg on the business side. And both of those areas are very noisy, right? So, um, one thing that has served me well and I think it's just very important for that role is to be able to filter out the noise and focus.
Boyan Dimitrov: Uh, because otherwise you can easily get distracted and overwhelmed. Uh, there is so much happening in the technology space, right? Like, it's hard to keep track of all the things that are going on. Being announced, new technologies, new ways of doing things, hype, whatever the case is, right? And then, of course, on the business side is, is, is, is, is the other fun, right?
Boyan Dimitrov: New opportunities, new deals, new way of doing things, new business models, process optimization, the endless things that you can, you can be spending your time on. So, um, in the middle of that, obviously, you have your teams that you have to pay attention to, the, the, the people you work with. So it is definitely very noisy and my recommendation and the way I I try to structure my time always is, um, you know, filter out the noise, make sure you, you, you make up the biggest amount of time to the things you, you, you, you duplicate about.
Boyan Dimitrov: And, uh, for the rest, build yourself a structure, a network, what, however you can to filter out the signals and, you know, relieve the noise out Tobias Schlottke: So you have Tobias Schlottke: focus time in your calendar? Boyan Dimitrov: Two How much? I tried to not sacrifice that. Boyan Dimitrov: No. Tobias Schlottke: like a no meeting
Tobias Schlottke: day or? brought this dynamic. I would love to have every day no meeting day, but that's, that's, that's, that's not the world we live in. But on the other side, I'm pretty premeditated when it comes to what kind of meetings I take and, and, and, and, you know, why, why, why do we have to meet at all? Because otherwise you just become a, a, a, a, a, a, a puppet of your own calendar, right?
Boyan Dimitrov: It starts driving you. It starts telling you what you need to be doing next week, a month from now as well. And I think this is, this is a massive, um, uh, threat if you like. It's important that it goes the other way around. So you have to defend, you have to defend your, well, you have to defend your time.
Boyan Dimitrov: And Again, I try to have focused time every single day, but other than that, uh, this, this is the piece I try to have non negotiable, non negotiable with myself. The rest of the hours, I fight hard to, for, uh, as well, right? I don't, I don't just allow it to, to, to, to, to, to take over. Uh, I try to, to make as much space for the things that are important and for me to think and for me to, you know, make decisions.
Boyan Dimitrov: And look deep into, you know, topics that are important versus, you know, just meeting some, having a meeting which was organized three months back for a topic, right? Um, if you think about it, and if you really live as, you know, it's day one, uh, when you wake up, you have a pretty clear idea how this day should go and what is really important to, to happen.
Boyan Dimitrov: And that really could be the thing that, uh, even if you set up this meeting three months ago, that thinking can't be, you know, more accurate than what's happening today. That's how I personally think about it. It Tobias Schlottke: Yeah, I really share those thoughts. And I think it is really hard Boyan Dimitrov: is brutally difficult, right? It is brutally difficult.
Tobias Schlottke: To really Yeah, yeah, yeah, like people always like attack your focus time. Um, I, I still have a little, little surprise and an outro question for you. So, um, I want to invite you to a ride. So I got this special DeLorean edition that, um, Eric Sixt gave me. I think he's, he's using it quite a lot. Um, and, um, I wanna, wanna invite you for a ride, um, back in the days. Thanks. Um, and travel back in time to the year 2000. Reflecting on Career and Curiosity --
Tobias Schlottke: 2011, um, when you worked as a project manager at Pinewood Technologies. And, uh, we now, let's, let's imagine we now travel back like with that DeLorean, um, the doors open and you can now observe yourself, uh, back in the days. Tobias Schlottke: Um, and you now would have a chance to whisper something, um, to your younger self, um, like few years younger self, what would you whisper into a young a young Boyan's
Boyan Dimitrov: Those were the times when, uh, I was, I was doing my master's, I think, uh, that was, that was actually, um, I was doing this, uh, at the end of it. Um, fun times. Uh, a lot of partying. Boyan Dimitrov: Some gaming, a lot of partying. It was, it was, it was those times. Um, I guess,
Boyan Dimitrov: reflecting better on that, probably I would have whispered to myself to stay curious. Um, it was curiosity which, which kept driving me over the years and it's one of those things that, you know, it, it, it, it, it, it created who I am. It led me through many different rapid holes, but it also exposed me to so many different ways of both doing business, but also.
Boyan Dimitrov: So many different technologies and, uh, even when I was at Pinewood, right, uh, at that point I was, uh, I was, I was actually part of the platform team, uh, of the network team even, uh, and managing a, a monitoring project, I think. And, um, that was, that was such a good fun and I learned so much and it is the curiosity which kept driving me from, from, from afterwards problem to problem.
Boyan Dimitrov: So. Yeah, probably this is what I would have mentioned to my younger self. Tobias Schlottke: Maybe, maybe it Boyan Dimitrov: Yeah, yeah, yeah. Tobias Schlottke: as you're still curious. Maybe you really traveled
Boyan Dimitrov: Yeah, but you know, when you, at that point, I've tried, I had tried already a couple of things or a couple of businesses myself and failed on them, right? That was before my master's even. And I was, as Pinewood at that point was probably the biggest company I ever worked with to date. And I was just, when I joined them, I, I, I was just, I, I just, I think, had to shut down my website stuff.
Boyan Dimitrov: Um, I was just thinking to myself, is it really time for you to man up and, and get a real job? Right? And, and, and, and just start specializing, right? Just start specializing in one thing instead of, you know, being overly curious, and, um, going from one topic to another. Um. But yeah, I think I obviously did not, did not follow up that, that thought back in the day, right after this, I went into a completely different business again, that I had no clue about.
Boyan Dimitrov: And, um, yeah, I think that served me well over the years, but I certainly had my doubts if this is the right path for me. Tobias Schlottke: Thanks a lot for, for sharing your thoughts. It was really an interesting discussion, Boyan, and I hope we can continue. Continue at a certain point with a different topic. Uh, really loved the discussion with you. Uh, and, uh, yeah, hope to meet you in Boyan Dimitrov: Looking forward. Thanks for having me, Toby. Bye.
Tobias Schlottke: Thanks a lot. Bye bye.