#214 - Beyond CI/CD: Continuous Deployment Explained - Valentina Servile

If it hurts, do it more often, right? Continuous deployment is the perfect example of applying this principle and putting it into practice, because to me there is absolutely nothing I can think of that is more painful than deploying to production for things, right. So in the past we used to have like this manual approval stage where, you know, somebody will decide when we actually deploy

to production. So tell us why it is very, very important to actually remove this last critical piece. So to explain why it is important, I'm going to connect to the practice of Lean manual factory, right? And you also got Dave Farley to write the forward of the book. Write one of the co-authors of the You Know Seminole book Continuous Delivery. Every single aspect of your work needs to be thought as a life change in a production

environment. So it feels much more like being an electrician working on an office building that is currently in use, right? Well, I think it's a very nice explanation. So I think also maybe for some people who are still not yet in this kind of a working mode, right? What do you see? Some challenges. Surprisingly, there's a lot of situations where you think you might not be able to do it, such as regulated industries, but in reality you're able to.

Continuous deployment doesn't mean that you just deploy straight away. So tell us what is the minimum thing that you should have in a continuous deployment and so? Basically, there are a certain minimum amount of practices that you should definitely make sure that you have. Hello everyone, welcome back to another new episode of the Techni General Podcast. Today I have with me a lead software developer from Thor Works and also the author of a book titled Continuous Deployment.

So as you can tell, we are going to discuss about continuous deployment. How does it differ from, you know what we know, CICD continuous integration, continuous delivery. And yeah, happy to have you, Valentina, in the show. Hi Henry, happy to be here.

Right, Valentina, maybe before we go dive deep into continuous deployment and all that, maybe let's talk a little bit more about yourself. Maybe if you can share any career turning points that you think we can learn from you. So well, you already gave a bit of an introduction. As you said, my name is Valentina Servila and I am currently at Teched at Dotworks. I'm Italian, as you can tell from my accent, but I'm actually working in total Spain. But I did start my career in Italy.

So basically I began by joining a very small agile consultancy. That was my very first job and I had a very amazing training. So all of the juniors that joined this consultancy basically were told to study for three months. As soon as I began on my first day, I got told to read the Agile Manifesto, which is something that I still really believe in, you know, all of the agile principles and eventually led to me writing a book about one of those.

So yeah, maybe this is a very good reminder to know, pay attention to how you're training your juniors because it really does pay off. I think if I can use my examples, three years later I joined the ThoughtWorks in Spain and I started working with much bigger companies.

I started working with more enterprise type clients and you know, being a consultant in both occasions I was able to jump from client to client to client and see a wide variety of ways of working of different team compositions, of different practices. And I think that gave me a really good perspective on all of the technical practices that teams can adopt to be successful or not to be successful.

I also in between that managed to go on a long term assignment in Asia. So I was stationed in Bangkok for about a couple of years. And so it was really nice as well to see perspectives of how organisations work on the other side of the world in my case. But yeah, now I am back in Spain. I was back in Spain for a while. Now I wrote this book and let's see what's next.

This episode is brought to you by Swim dot IO and I'm excited to have its CTO and Co founder Omar Rosenbaum with me today to tell you more about Swim. Hi Henry, very nice to meet you and thank you for having me. So tell us a little bit more, what is swim dot IO? At Swim, we want to help companies understand their code bases. We combine static code analysis with generative AI to create comprehensive documents that help you navigate the code base.

As an engineer myself, I wouldn't want 10 years to spend so much time understanding existing code. I would want them to spend time creating and building new stuff. When you have code that has accumulated over decades, and especially in legacy languages that not many people are adapted nowadays, then the problem is even bigger. Swim dot IO is specializing into helping mainframe developers to understand their copies. Why mainframes?

We actually didn't start there. COBOL had been by some people obsolete for a few years, and I discovered that it's not really obsolete, not at all. There are more than 800 billion lines of COBOL code that are in production and they drive lots of the business in the world and we got more and more requests. From. Customers to help them understand the legacy code business that was written decades ago and get accumulated over a very long period of time.

So from your customers so far, what are the some of the success stories that you can share? So we worked with an analyst who shared with us that it took them a year to document a single mainframe application, and using SWIM they were able to document a similar application in a matter of hours. So saving that amount of time enables them to focus on other tasks. Thanks Amir for sharing with us about SWIM today. To learn more about SWIM, check out their website at swim dot IO.

Wow, thank you for sharing your story. I think one thing that you mentioned earlier, right? So train your juniors. Well, I think maybe not all

juniors are fortunate enough to have this kind of good training, good principles being taught since the very early of their career. So maybe for those who are maybe not so fortunate like you or maybe some of us. So what advice would you give to them such that even though they don't get the good training from the company or from the seniors they work with, how can they still learn a good principles at

the start of their career? I'm going to say, well, first of all, try to look at the literature because that has been really, really helpful for me. You know, in terms of good practices, in terms of how to be a better programmer. The literature has so much insight. Even if you cannot get it from your seniors or from your mentors, you can always fall back to that. Just don't be afraid to ask questions and be annoying, you know, try to be the one with the keyboard when you're pair

programming, for example. Yes, it will be a bit slower. Yes, you will have more questions than someone else would and hit more blockers than someone else would. But make the best of your mentors being there and don't be afraid to annoy them because that's the only way that you get to learn, you know, being hands on. I like your term be the keyboard, right.

So sometimes we may not be comfortable asking questions to senior and become like, I don't know, some people think they might be a hindrance to the discussion and all that. But I think, yeah, beginner's mind, I think that's always a very good skills to have in any kind of team. Yeah, I was going to say, and it definitely helps the team as

well. You know, you might think that you're annoying everyone by asking questions, but a lot of times it actually uncovers something more to the discussion, even what might be a really obvious question. So yeah, definitely don't feel like an annoyance. What you're bringing is really, really valuable, which is a fresh perspective. Right, I think that's really great tips, right? So for all the beginners, even maybe for seniors, right, Sometimes don't be afraid to ask

like so-called stupid questions. So let's go dive into your book,

right? Continuous deployment. For me, it's almost like natural. One day somebody will write about this continuous deployment, even though you know, like CICD has been, you know, around for, I don't know, more than a decade, I guess. So maybe in the 1st place. Tell us your story, the background, What inspired you to actually write this book? Yeah, And actually I brought the book. It's right here, continues deployment. You can get the physical copy on Amazon, of course.

So what inspired me to write it, basically, as you said, it sounded like a really obvious gap in the literature that somehow was missing. I don't know how nobody had thought about writing this book yet, but it's definitely a gap that I felt when I was working because being a consultant, I was really lucky to be part of some teams at some clients that were already doing continuous deployment.

They were doing it for many, many years, but sometimes we were on boarding more junior people or consultants that didn't have this experience. And even when they were really familiar with other great practices like continuous delivery, continuous integration develops, there was still a bit of a learning curve. So I found myself basically in a situation where I wanted to communicate what is the difference between stopping your commits at staging and every

commit going to prod. And it turns out there was quite a lot to communicate about it. And I had basically no literature that I could just send them or articles that I could just send them to, you know, just read up about it and be ready the next day. And so, yeah, based on the missing this kind of content, I decided to create it. So I started the by writing some blog posts and doing some talks

about the subject. And yeah, then it got picked up by Riley and here it is up. So hopefully this will help other people who have been in my situations, for example, in teams that are already practicing continuous deployment and you want to ember someone else or if you're one of the people joining such a team. So this is a pretty condensed version, even though it's 400 pages long, of everything that you might ever need to know about it.

Yeah, and he also got Dave Farley to write the forward of the book, right, one of the co-authors of the, you know, I would say Seminole book continuous delivery, right.

So I think maybe let's start from there, right? The definition it's always very confusing when we talk about, you know, CICD in the industry, right? Some people even already associate CD with continuous deployment. I guess maybe a little bit off back to the fundamentals, if you can elaborate what as continuous integration, continuous delivery and continuous deployment.

Thank you. So this is a topic where the terminology has undergone a lot of semantic diffusion, as Martin Fowler calls it. And a lot of people in the industry refer to CI and or CD as the pipeline, effectively forgetting that in reality all of these things started as

practices. So the pipeline is just an artefact that helps you realise the practice, but there is quite a lot more to it. So starting with some definitions, the first practice that came along was continuous integration as part of extreme programming. So the practice behind that outside of the pipeline tool was basically the act of integrating your changes into a mainline shared branch really, really frequently.

This had the purpose of always having all of the changes bundled into a buildable artefact that could be validated by tests for example, and so that the team could always show their progress. So it came with the practices such as nobody pushes in a broken build, we have to integrate changes frequently, not hold changes in really long lived feature branches and so on and so forth.

Then the pipeline, which is how implementation looks like or can look like of continuous integration, takes care of taking this mainline branch, building an artifact from it and running some automated tests. And that was a really big change at the time because that wasn't a particular way of working that existed so much before. And suddenly it was really, really easy to show progress and say, look, this artefact represents what we have been working on so far.

And it is buildable, it is testable, and it can be deployed because it contains every change that has happened, say in the past day or in the past week. That made the integration much more frequent and introduced the concept of the pipeline. But then continuous delivery came along and it brought even

more to the conversation. So continuous delivery added on top of continuous integration, the fact that not only your changes should be frequently integrated, frequently tested, frequently built, but also frequently deployed. And every single change that goes through the automated pipeline should be verified to such a greater confidence level that should be considered deployable to production.

So basically this was a really big mindset shift because then it came with other practices, say from the BOPS such as infrastructure is called and automating the deployments themselves, which didn't have to be manual anymore. But yeah, it was also a practice before it was the automated pipeline too. Then a further next step on top of continuous delivery, again to me and to many other people has

been continuous deployment. Where not only every change that goes through the pipeline should be considered the deployment to production candidate, but it should be actually deployed to production. So basically this to me was a culmination of this concept of the automated deployment pipeline, where every single step between development and deployment to production has been automated.

In the traditional implementation of continuous delivery for example, you might see that many many teams deploy automatically with their pipeline to some pre production environments, say diverse aging. But the deployment to production, while it can be done on demand, is still a manual action. So it's the push of a button to be done only when the team feels confident enough.

Let's say continuous deployment is pushing that limit away and removing that final manual gate in the value stream of building software and deploying it to prod. And so, yeah, it is definitely a combination of a journey that has maybe taken I'd say 20 years since Extreme programming and continues integration came along and is bringing automation all the way to the end of the value stream. I like that you brought up the historical aspect of how all this combination happened, right?

So from things like extreme programming, you know, DevOps, Agile and things like that. And I like the specific thing that you mentioned, right? It's not all about pipeline, right? Because people associate maybe CICD with pipeline concept, even tools like Jenkins, maybe no GitHub Actions and things like that. But it's about the practices, right?

The practices that actually lead us to use some of the tools that could help us actually adopt those practices, maybe slightly a little bit more about definition as well. People associate CIS less CD, right? They think they are kind of like a different thing. And now what is the term that you're gonna use for continuous deployment? Is it also the same CD or is there any other term? Actually, that's not really set

in stone yet. I've been debating it with my colleagues, some of them abbreviated to C deck just to distinguish it from continuous delivery. So I if you find it helpful, I encourage you to continue on that. But yeah, otherwise you can just spell it out. Continuous deployment, just not too confusing. So I think 1 aspect that is

very, very important that I would like you to also try to explain, right? This thing in the extreme programming, right? If it hurts, do it more often, right? So I think this is maybe one of the key reason why continuous delivery was kind of like invented in the 1st place. Maybe tell us why this is important for us, especially for those teams who are still kind of like fearful whenever they make code changes and they want

to deploy even to production. So I love that principle from extreme Programming. To me, that is the essence of extreme programming and most of agile practices themselves to be honest, is software has many more parts to it than just writing it. There's integration, there's testing, there's deployment and all of those other parts before

XP used to be pretty painful. So, you know, in an old fashion waterfall process, you will maybe have your giant development phase, then a big integration phase, then a BQA phase and so on and so forth. Now that used to be pretty painful because it was done infrequently, right?

So the principle of extreme programming basically says if something is painful, such as these phases of writing software, you should do them more often because doing them more often encourages their automation, which is a great thing. And in and on itself, automation allows you to first of all document and also make a process

which can be replicated. It is not relying on someone's knowledge on doing things by hand, but also more crucially, if you do things more often, like integrating software, testing software, there is less software to integrate and test and deploy each time. So when your batches are smaller, and here maybe I'll talk later about lean manufacturing, suddenly all of those phases become much less

painful. And that is why this advice seems counterintuitive, but actually it allows to feel less pain in performing all of those phases. That said, I believe that continuous deployment is the perfect example of applying this principle and putting it into practice. Because to me, there is absolutely nothing I can think of that is more painful than deploying to production for teams, right? That is so much more painful than resolving some git

conflicts. So during integration or spending a lot of time on testing, the deployment to production is when the website can go down, your app might stop working, you could scrap your database. So because it is much more scary and much more painful, that's why I recommend to do it as often as you can, and in this case automatically at every commit.

Yeah. So I think in the past, traditionally, I also used to experience those Big Bang release, right, where you kind of like have a certain window, maybe quarterly, whatever that is. And you kind of like build a ceremony around production deployment, right? Kind of like fearful whenever the new change going to be used by the real users and whether there's going to be a big impact or buck happening in the

production. So I think the spirit here, if it hurts right, do it more often so that your production deployment becomes like a more boring kind of event. I was going to say whole ceremony about deploying to production, the whole do not deploy on Fridays. That's not something that you should be prideful about. That's actually a smell that your process might not really be so great. If that deployment is that risky, it means maybe there's something going on that you need

to address. Perhaps do it a bit more frequently and split it in smaller smaller deployments.

Yeah. Speaking about the difference between continuous delivery and continuous deployment, I mean one big aspect that I see is the big differentiator is what you call the still kind of like the manual deployment production, right. So in the past, we used to have like this manual approval stage where you know somebody will decide when we actually deploy to production.

So tell us why it is very, very important to actually remove this last critical piece, because for some stakeholders, maybe they think having that is kind of like a guardrails to ensure things are still going to be safe. It's kind of like verified before things are deployed to production. So to explain why it is important to remove these, which might seem like a small piece and a very important guardly, I'm going to connect to the practice of lean manufacturing, right?

To me, really the most foundational practice of Lean is removing inventory from the value stream of software. So what we see as the founding principle of Lean is that instead of having big batches of inventory going in between workstations, we should try to reduce the flow of inventory that is going through the system to basically be only one piece

at a time. This allows to do some crucial things, which is to, for example, identify the immediately as they flow through the value stream, rather than waiting for an entire batch of things to be processed and going to the next workstation. But also, even more crucially, it allows to drastically reduce the cycle time, so the time it takes for any piece of work to flow through the entire stream and be in the hands of customers. Now, how does this translate to software?

This is a principle that was very successful when applied to manufacturing of physical goods. But of course it's very hard to define inventory and software. So what I think of inventory and software is pretty much every intellectual artefact that is included in software manufacturing. So this for example, could be user stories, it could be architecture diagrams, or it could be code. So I'm going to focus mostly about code here. How to define a batch of software?

For me, it needs to depend on how to identify a unit of software. Defining a batch of software basically has to depend on defining a unit of software. So what is really a unit of software like? When do we know that we have a big batch of inventory? A unit cannot be really defined as a single instruction in code. I don't feel like because that's not self consistent.

Instead, I define a unit of code as a commit because that is the smallest unit that we can have flow through that value stream. In this case, the value stream could be represented by our pipeline. So a commit is the smallest possible unit by which we can review software, test software, and eventually deploy software. So to me, a batch becomes when commits accumulate during this value stream, right?

So when commits accumulate in your pipeline and that is when you have some inventory that is stuck at a specific phase of your value stream of code and that is where the effects can accumulate. That is where your cycle time is slowing down and that is where all of the anti patterns of not

doing lean start to emerge. So bringing this back to continuous deployment, you can think of continuous integration and continuous delivery as the gradual transition from an old batch and queue processes such as Waterfall, for example, where there is a lot of inventory moving really, really slowly in between workstation. Continuous integration and continuous delivery represented a shift from that.

So with continuous integration, you're starting to have what looks like a one piece flow from the development phase at least to the integration phase because it says integrate frequently. So how frequently can you integrate at every commit of course, and how frequently can you test at every commit? So you have commit suddenly flowing one by one through at least some of the earlier phases of software manufacturing.

And so your inventory there is checked with the maximum granularity that it can be checked, of course, later than those phases. So say the deployment phase wasn't yet automated in continuous integration. So that's where commits or inventory could accumulate right before a deployment. And basically it would go through the rest of the process again in batches. Now with continuous delivery, the deployment phases following

that have been automated. So again, this one piece flow of inventory had been extended, but still with most implementations of continuous integration commits still accumulating preproduction or in staging. So that's where basically, again, your defects are showing up and your commits are accumulating right before production. Now this makes deployments to productions riskier because of

course you have a bigger batch. And so I mean the fact that has accumulated in that batch is going to show up upon deployment. So I think if you want to realize this one piece flow all the way end to end, you shouldn't have any manual points in the pipeline where these commits can accumulate and become stale. Wow. I think it's a very nice explanation why we should work towards this manner, right?

This practice, right? So one piece flow, I think that's a very important thing from lean principles, right?

So I think also maybe for some people who are still not yet in this kind of working mode, right? What do you see some challenges maybe from your consulting, maybe from, you know, having the chance working with enterprise mostly I believe they will have a lot of money shifts that need to happen. What will be some common challenges? Because deploying every commit straight into production is not something that all the team can

aspire to do, right? Some might have regulations, some might have compliance checks, some might have any gatekeeping inside the software delivery. So what do you think are some common challenges that people need to really start to move away? Yeah. And what you just said is exactly why I have at least two chapters of disclaimers around the when can you really do continuous deployment, When can

you? Not surprisingly, there's a lot of situations where you think you might not be able to do it, such as regulated industries, but in reality you're able to. It depends really on the framework and regulation.

For example, in my case studies, I have a new bank called N 26, which is subject to basically all core banking regulations, but they were able to still use a continuous deployment workflow because they have an automated pipeline that basically satisfies all of the auditability requirements of the process. And as they finally says in his excellent article, continuous compliance, an automated pipeline is an absolute gold mine for auditors.

Basically, it has so much information on which changes were applied to production, when, by who, for what reason, because it will be related to a certain ticket, what tests were they performed on them, how did these tests prove that they were successful, and so on and so forth. So that already really helps, and it's just a matter of figuring out what parts of your process can really apply to

satisfy this regulation. At the same time, it's very common in regulated environments to have authorized rules where nobody can really put a commit in production alone. That is something that also this bank was able to satisfy because they found a way by doing a combination of pair programming and micro branches with like immediate pull requests approved

by a pair. That was their way to prove that a commit was seen by at least two people, the person that authored the commits and the person that merged. It might sound counterintuitive, but there are ways to still use modern practices in regulated industries. Not because these practices are less safe, but because they are more safe than manual approvals and waterfall. And we just need to find a way to articulate that and really

explain that. Other situations where it might be difficult to perform continuous deployment are usually more by technical challenges rather than regulatory ones, in my experience. So for example, something that is really common that happens to teams that maybe want to move from continuous delivery to continuous deployment is when you start doing deployments really, really frequently, you might find out that your system is more sensitive to deployments than you might think.

And so this might affect things like your reliance on caches. Suddenly when you deploy many different versions all in the same day, your caches, if you have in memory ones, are going to be mostly cold most of the day. So that's when you need to find maybe architectural workarounds such as moving your caches externally and so on. Also, this applies to any kind of in memory state that you

might have in your application. So if your user flow depends on certain state being stored in memory for whatever reason, that is something that the frequent deployments are going to be disrupting really, really frequently. So you need to also fix those kinds of scenarios before you can really move on to this workflow.

Another really common scenario where people have trouble, and maybe this one is not so easily fixable, is when your production environment that you want to continuously deploy to doesn't belong to you. So after we think of production as a server that is in our hands and it is in our capability to deploy to, but what about, for example, mobile apps or IoT devices or on premises infrastructure that belongs to someone else, right? In that case, it's a bit more of

a grey area. What does it even mean to be done with the deployment to production? Are you done with your deployment when the first user device has been updated or when the last one has been updated? So there, that's when probably continuous deployment is not so easy to implement. But again, there are workarounds that you can use. So for example, you can use Pwas to move control back to the server web views. But yeah, that's when you may struggle a little bit.

Yeah. Thanks for the overview of some of the challenges which I assume some teams would have, right, whenever they want to try to implement this continuous deployment. But having said that, there's

also another team which probably claimed that they do continuous deployment, which I want you to kind of like explain because they may not have so many processes. They just have a commit and they maybe do something in the build pipeline, right? Maybe creating the artefact and just deployed production. I'm sure this is not something that you are advocating, right? Continuous deployment doesn't mean that you just deploy straight away, but there are

certain processes in place. So tell us what is the minimum thing that you should have in the continuous deployment? Yeah. And thank you so much for asking that question because I don't want anybody to take as a take away from this interview that whatever your the state of your automation now you can just get rid of any manual approval and go straight to prod and nothing will happen.

So basically there are a certain minimum amount of practices that you should definitely make sure that you have. So again, a pipeline that is fully automated, but that pipeline needs to contain tests and a lot of them. So that's what I go back to recommending, you know, the typical testing pyramid, which means you should have a pretty, pretty good unit test coverage, a pretty good coverage of automated tests both before and after deployments, if that is

possible for you. There are practices that can maximize this coverage. And the ones that I use most often are the ones that basically allow you to write your test first. So TDD is my prime example and outside in TDD, if you use this, you know that your coverage is growing together with your code in the same commit which is done going through the pipeline. So those can make you pretty

safe. Then alongside tests, which are great for functional changes, I always recommend some forms of static code analysis to be in your pipeline, such as security scanning, performance scanning. If you use a tool like Sonar Cube, that is going to be really effective at catching most regressions that maybe might be missed by your tests. And that's really important, right? Because if you're publishing every commit, then every single commit might affect the performance of production or

open a security vulnerability. So static scanning can really, really catch all of those regressions and that really more granularity. Then of course, another question that I get really often about what are the prerequisites is how do you hide your work in progress if you're just publishing every commit to production? Because you say you shouldn't have long Fisher branches where you hide your changes, but also everything on master needs to go

to prod immediately. So we need to also be comfortable with our team to use patterns like feature flags and the typical expand and contract pattern or parallel change or branch by extraction, whatever you call it. All of the those patterns can help you hide any new change that you don't want your users to see, even though that change is in production. So I highly recommend to be familiar with those. Then finally, of course, you need to have 0 downtime deployments as a minimum

practice. I think most teams do these days, but it's always best to check so you can achieve those with rolling updates. You can achieve it with blue-green deployments or red, black. You can also perform cannery deployments with tools like Spinacer, where basically the tool is going to be connected to your observability tool and observe for example, the error rate of your application and roll back immediately if that

error rate becomes worse. So observability and alerting is maybe the final absolute prerequisite that I recommend before turning on, you know, the fully automated pipeline, removing the last gate to prod. Because as employments get more frequent, there is also less oversight on them. As you said, they become just a boring event that happens multiple times a day. So we need to be proactively notified in the very rare event that something does go wrong.

And that's where I recommend having a really, really sturdy dashboard, keeping track of all your metrics. And most of all, take care of your alerting based on your metrics, because you definitely don't want your team to be used to seeing a lot of alerts and not having to react to them. So keeping those really tidy and only keeping alerts that are useful and require any sort of action that's going to really help. Thanks for the clarification.

I mean just to re emphasize one more time, continuous deployment doesn't mean that any kind of commit you just deploy to production straight away, right? So there are certain practices, certain minimum that you need to achieve, right. And Valentin, I just mentioned, you know several of them, right, which I believe are like some core practices that every engineering team should aspire to do.

I think 1 aspect of continuous delivery that is also very important is that everything that you prepare to go to production, right, must be verifiable and you have high degree of confidence that it will work in production as well. So it's not something that you kind of like Yolo, you know, you just wait for the users to report the bugs to you. And I think one other aspect is about backward compatibility,

right? So whenever you make so many changes, right, I'm sure some users, maybe some servers are not taking the changes straight away, right? So you need to make sure that some backward compatibility must be in place, which brings me to this thing that you mentioned,

right? Hiding work in progress, definitely. If every commit must go to production, we cannot finish a new feature just within one commit, right? So you mentioned about feature flag or maybe expand and contract. Tell us more a little bit about this practice because I'm sure some people might still think, oh, I cannot work this way simply because I have some work that I can't finish. So tell us how we can use those techniques to do this. Yeah, thank you for mentioning those.

They are the ones that I rely on most heavily throughout the book as well. So I think it's definitely worth to take the time and understand them. So let's start with the feature flags. So I recommend using feature flags whenever there are visible changes to the code base and the application that if deploy to production, will alter what the user experiencing and the observable behaviour of the system.

Most people are used to hiding such changes in version control branches and then the act of publishing them will be to merge the version control branch into main and finally let's main go to production. Using a feature flag instead still uses a kind of branch, but it is not a branch that belongs in the version control system, it is a branch that belongs to the tree of classes and functions within your

application. So I call it an execution branch for that reason, and that will do exactly the same job of Aversion control branch, except it's going to still allow you to integrate your code frequently and deploy your code to production frequently. So during development, you will basically put a feature flag as the very first step of beginning any new feature, and the feature flag will be off until that feature is released to the users in production.

I think this is really powerful because it also allows you to express the conditional logic of when you want your feature to be seen or not seen, for example, for a reason, in a way that is much more expressive than just the textual control that version control gives you. So I really rely on feature

flags a lot for that reason. Finally, for expand and contract, that is a pattern that I tend to use when I'm making not so visible changes, but changes that I still I'm not feeling super confident in, or that I want to split into smaller and smaller increments. An example of this could be refactoring a really big subsystem of your code base, or for example an entire endpoint of an API.

That's when I would basically duplicate the flow completely, making sure that the interface of the old flow and the interface of the new flow is respected equally. That allows you to basically commit at any point, because you can migrate the clients of the old flow to the new flow 1 by 1 and still completely respect the functionality of the system, which your regression tests should be able to catch if you broke that. So yeah, you can read more about

this in the book. I have a lot of concrete code examples. Yeah, I highly recommend reading this part because I think this is one key practice that is very important in continuous delivery and continuous deployment, right? So for those of you who also have not heard about expand and contract before, some people call it branch by abstraction. Maybe if you have like OOP, you have interface implementation, different abstract class and

different implementation. This is also one aspect that is kind of like similar in terms of implementation.

And I think this is also a good time to actually explain the difference between what we call deployment versus release, right? Because with things like feature flag, it allows us to kind of like control the time to release a particular feature. So tell us about this difference. Yeah, and this is the reason why I love feature flags, right? So with continuous delivery, it was still true that you could use feature flags and have a separation of deployment and release.

But with continuous deployment it becomes mandatory and the user feature flags becomes mandatory because your pipeline is not gonna stop anything visible from going to prod if the tests are green. Because those practices are mandatory, the distinction between deployment and release becomes much more well marked and much more important for teams. So a deployment in itself during continuous deployment is any roll out of new conversions into

production. But a deployment in and of itself should never alter the visible behavior of the system. At the same time, a release usually becomes the act of turning on a feature flag, which again, if you have a good feature flag framework, should never require a deployment because you can do so at runtime. So there is a very, very clear distinction in the timing and how these two things happen. A deployment becomes just an engineering concern and release becomes a product concern.

It's like, when is there a time to turn on this feature? Do we want? The entire user base to see it? Or do we want to run a 5050 test first? Or do we want to do a gradual ramp up? How does product feel like this would be best released? And this also brings us to a really great situation where a deployment being purely an engineering concern can be done

without the input of products. So we should never be in the situation, say, where I as a developer get told please do not deploy any new version because we are in the peak season for example, for our business and the product wants to wait. At the same time, I should never be in the situation where as a developer I have to tell product that feature cannot be released because as there is a pending deployment because all the code should have been in production in the 1st place.

So yeah, definitely these two things kind of take a life of their own with the use of feature flags. And that is why I consider them such a strong, strong prerequisite for any modern pipeline, be it continuous deployment or also continuous delivery. Yeah, I think that's a good distinction, right? Deployment should be the technical or engineering concern, while release should be the product or stakeholders

position, right. If the team can separate these kind of A2 important activities, I think that's a really great shape. And I think feature flag, so many tools these days, I'm sure having a platform that can allow you to do this behavior is something that is not impossible anymore. Another important thing that I

think what to call out here is about slicing the work, right? Because if let's say I want to build a new feature, sometimes a feature is not like very simple like change a color of something or create a button or something like that. But it's more like a maybe like workflow or something like that. How do you actually advise us to slice the work right?

Because making sure that the small batch committed up to production is something that maybe some of us are not used to. That's a great question. So there's always been that eternal battle between slicing work by code base and slicing work more vertically by feature. I'm going to say with this type of workflow, continuous deployment is even more important than it was before to start thinking of slicing work vertically by feature instead of

by code base. And it's for the very simple reason that as I mentioned, we have practices like feature flags and especially expand and contract that require multiple small changes to production on different systems a lot of the time to guarantee backwards compatibility until the feature is completed. So for example, in order to add the some new functionality to my system, I might need to alter my front end and back end code base and my DB.

Right now, because I'm doing continuous deployment, I want to maybe build it slowly and deploy multiple commits one at a time into production to guarantee for example backwards compatibility of my back end, say with my database schema. Maybe I need to create a new table, migrate something in my back end to go to this new table instead of the old one that's a new deployment on the back end. And then maybe I need to redeploy my database to clean up the old table, let me

synchronize some data. So if I slice these tasks horizontally, I have to create so many tasks just to jump around on which change do I need to make now on this system and on this system, And they're all gonna be blocking each other. So it's not a really efficient way to work with these ways of

working. Instead, if I slice vertically, I can split my task, my user story into multiple commits, but I have the freedom of relating them all to that ticket or that user story and to really plan out within the story what commits do I really need to do in which system and in which order to achieve that functionality at the very end. So yeah, it becomes much, much more important to slice

vertically. And it also allows you really to leverage the granularity of continuous deployment as a practice because the fact that everything is in production all the time, it can make emerge some situations where actually you're ready to release a feature in a state where you previously thought you wouldn't. So say after maybe only two or three user stories instead of after all 10 user stories, because perhaps product wants to run an IB test a bit earlier.

And so not only becomes important for technical reasons, but also it's like you really leverage the speed of continuous deployment. Yeah. So I think vertical slicing is the key here, right? It also goes back to how you slice your story, right? So a feature should be sliced into a different shape of stories. And if you can shape it more vertically, that brings value like any kind of value to the users, maybe some other third parties, right?

I think that will be the key. Talking about third parties, I

know these days people are working in a kind of like micro service or service oriented architecture, continuous deployment also have this challenge of having to coordinate your changes with other third parties APIs, whatever that is, right? So tell us how can we do this better? Because obviously you can't control how the other team is ready or taking changes from your new version, right? So how would you advise us to do this? It depends. There's third parties and third parties, right?

If you're working in a team that takes care of micro service within a larger ecosystem, another team within the same organization will be a third party to you. But also it could be that you have to communicate with a vendor system which is in a completely separate organization and they have a completely different change management

process. So I'm gonna say there is much more flexibility when you're dealing with other teams as third parties because you can coordinate much more frequently on what new version of what is going live at which point. And you might even be able to pair with someone from another team to always guarantee synchronisation between the

systems changing. On the other hand, if you're dealing with third party vendors, that's when you need some stronger practices that are much more formal to manage the contracts in between systems. And that's where I usually fall back to things like API versioning, which is something that I wouldn't do necessarily if I have an API just for

internal usage. I can very easily coordinate with other teams and say, hey, please change your usage like this and we'll tidy up without the need to maintain 1000 different versions of some API. That is going to be a lot of maintenance, but with an extended vendor where I have no control or what they're doing, that's when I would pay the price of versioning an API, for example. Same thing with contract testing.

I'm going to have much stronger contract testing and requirements if I am consuming or producing an API for a third party vendor than for a team within my same organization. So that's how I would handle it. And of course within the organization, that's what practices like expanding contract and feature flags get even more useful them within just a singular system because they not only allow to hide the work in progress within one system, but also across several

system. So they can also be used to guarantee backwards compatibility of systems that are independently deployed and where therefore you cannot bump everything to the next version exactly at the same time. So I encourage teams to look into these practices as well for working with micro services and testing integration with other micro services. Yeah.

And I would like to again emphasize backward compatibility, always ensure that you support a few versions behind, right, simply because you can't control the other teams whether they are taking a changes straight away or they will take some time before they actually apply those changes, right. So I think you mentioned also contract testing and things like that definitely is a good

practice as well. I just wanted to add as well that this is not just something that needs to be taken care of when you have a micro service environment. The concept of backwards compatibility applies to any system which is distributed, even if at a glance it looks like a monolithic system. In my experience any system that is interesting rarely is really monolithic right?

For example, even if you have just one simple website with one database, you already have 3 distributed components that you need to keep backwards compatible at all times. You have your front end which is executed on the user browser and it's not going to receive changes at the same time your server does. Then you have your server, and then your database is a completely separate executable and most often deployed on a different machine as well.

So the thing you really need to take into consideration is inter process communication. Whenever there is such a thing, which is almost always, you would probably want to rely on things like feature flags or expand and contract at a minimum. Continuous deployment just makes it much more evident because with that it happens all the time. You do deployments all the time and usually micro service oriented environments.

But this is not a new problem. This is a problem that we should have been solving already much before. And you know, when deployments are infrequent, sometimes it's easy to ignore if there's some deployment glitches. Whereas with continuous deployment you can't ignore them anymore because they just happen every single day. Yeah. And this goes back again to the principle. If it hurts, do it more often.

You should not have. If it hurts, then you stop doing it, or you kind of like reduce the frequency of doing it, right. So I think the more you do it, the more you figure it out how to make it safer, how to make it, you know, smoother. Another thing, apart from

technical practices, are there cultural aspects or maybe you know, like onset shift that people should know about before they can actually implement this continuous deployment? Yeah, for sure. And that's why I wrote a book about it as well. I think a lot of it has to be a mindset shift change, especially for developers. First and foremost, imagine that anytime you do git push origin main because you're working in a trunk based development fashion, of course you're not doing PRS right?

Every time you do that seven times a day, your users are on that live system that is going to receive that update 20 minutes later. Suddenly, if you were using continuous delivery before, it can be very paralyzing to think about what you're putting in the pipeline at any given time. But it doesn't have to be, right? So there needs to to be this mindset shift of putting production as a first class citizen every time you're writing code and really think through before you pick up a new

task or a new user story. What commits am I going to put in my pipeline that are going to make that happen? And which subsystem do I need to start from to not break production at any point? For example, if I have a database back in the front end, that is going to be probably the database system that I'm starting from if I'm adding something new. Otherwise, if I deploy the front end 1st and that's what I start developing, there's nothing underneath to make the feature work.

It's all going to be broken. Every single aspect of your work needs to be thought as a live change in a production environment rather than a plan for a change that might happen somewhere later and might be deployed by someone else at some specified point in time. So it feels much more like being an electrician working on an office building that is

currently in use, right? You have to always be careful of which diversions you're installing, how you're going to maintain compatibility with your system, to not disrupt this living beast that is production. And that also includes not only functional changes, but also security performance. There's going to be a lot that you you were used to thinking about maybe when your changes are in staging and you have to now anticipate all that thought during development, which to me

is actually really positive. It can be overwhelming, but once you're used to it, it means that you've fully shifted left. All of those quality aspects, all of those cross functional requirements for example, that maybe used to be relegated more towards the end of development. Yeah. So I like the concept that you mentioned. Always make sure that deploying to production is the first class

citizen, right? So ensure that you think whenever you make change, it's going to be straight away go to production, right? So sometimes this requires you to carefully think how you actually implement the changes and things like that, including the shift left. I think shift left still kind of like the correct mindset for implementing this, right? As much as possible automate as well those checks so that you know the pipeline can just automatically deploy the production once everything is

verified working. So 1 aspect that is very trendy

these days is about AI. Anything related to AI that can help continuous deployment practice. I don't know whether it can help, maybe it's more something to be aware of and slightly concerned about. So especially for teams that use continuous deployment today, I see that it is very common right now to just adopt Copilot. I'm one of the culprits of this by the way. I've recently installed it in my Intellij.

I just have it generate some code for me from time to time when I can't be bothered to look up some configuration or maybe I can't be bothered to do some test coverage. I ask it to give me a nice suite of tests for a certain function. But yeah, of course, if you're continuously deploying, it means you have to double check, triple check, quadruple check anything that the AI is giving you in order to be fairly sure that is

going through the pipeline. So when you re stick that you could use this, for example, perhaps the AI could generate the production code, but maybe you should be the one writing your tests so that at least worst case scenario your pipeline will fail. And in general, again, all the practices that I've recommended before as base prerequisites, such as test coverage, static code analysis, observability, those are also going to be a really good safety net.

And yeah, you you should really double triple check that your pipeline is really going to catch any and all types of regressions if your team is relying on AI really heavily. That said, it's a great tool. It can speed you up even more, but please be careful with it. Take it with a grain of salt whatever is spitting out on your screen because it is very easy to make mistakes if you're just get used to not double checking it and just trusting it blindly. Yeah, I think that's a really

great advice, right? Because AI, one thing for sure, it can help us turn out more code. But more code doesn't always mean good, right? Sometimes you have to check, especially if you don't have robust pipeline, robust automation tests and things like that, right? So always ensure that the principles actually the one that matters the most rather than churning out the code. So Valentina, it's been a

pleasure talking to you. As we reach the end of our conversation, I have one last question I'd like to ask you, which I call the three technical leadership wisdom. Just think of it like an advice you want to give to the listeners here. Maybe if you can share your version? Sure. So I thought a bit about it. I would say especially relevant to today's episode, we've been talking about like really modern practices, I think continuous deployment.

There's nothing faster and more modern than that that I know of in software development as like ways of working goes, one of the main pieces of advice that I like to give is many tech leads and developers tend to get sad if they can't use all the latest and greatest in terms of ways of working tech. And it's really sad because I believe it's not necessarily a must have, right? Having everything perfect all

the time. I think as long as the current practices of the organization are improving, that is a good thing in and of itself. So don't be worried so much about the status quo, be more worried about the trend. Is the trend an improvement or are we getting worse? And that's also true for people who are doing continuous deployment and really nice and modern things. You know, are you getting better at it every day or are you slowly getting worse even though you've enabled it some time ago?

So worry about the trend would be my first piece of advice. The second one is sometimes as developers and tech leads, we get emotionally attached to solutions. Again, I think of this is a bit of a pitfall and I believe our job is maybe more about informing. So informing of all different options and then let's take holders determine what is their preferred one. As long as they are aware of the pros and cons and they acknowledge any risks, let them do it.

Don't take it too personally. And finally, yeah, my main piece of advice is just if you're a tech leader like me, just trust the people that you're working with. Trust them to have thought about the edge cases, trust them to do their spikes, trust them that they're working and doing the best they can. So don't micromanage if you're a tech kid, would be my final piece of advice. Really beautiful advice, right? So I, I particularly like the first one, worry about the trend, right?

The direction that you're going, right? Sometimes you can take some more gradual steps rather than a big bangs approach, right? But still, if you're going to the right direction, I think that's the important thing. So Valentina, if people love this conversation, they want to check out more your resources. Is there a place where they can find you online? Yeah, so I'm mostly posting on LinkedIn, so you can follow me there. Valentina Servila, feel free to write.

I used to be on Twitter slash X, but now I've left, so I migrated to Blue Sky. You can find me there, but I don't post as much as on LinkedIn. So yeah, mainly LinkedIn, maybe Blue Sky or just drop me an e-mail. You'll find the address in the book. Thank you so much for your time. I'm sure people learn a lot about continuous deployment today, so hopefully people get to adopt these practices more and more. Thank you.