How to Master Software Engineering (From Junior to Architect)

Hi everyone, My name is Patrick O Kew and if you're interested in mastering software engineering as a craft, this episode is for you. Joining me today is Joris Gowkers, CTO and hands on Architect over at Trifork, and he has over 25 years of experience in this industry, more than yours and mine combined potentially.

He walks us through what to focus on when you're early in career or already senior with regards to how to master new topics, which environments to be in to thrive and much, much more. So enjoy. In essence, you've been in the industry for a lot longer than me, and I think for a lot longer than most listeners as well. What still gives you energy when you're creating software, when

you're delivering things? Yeah. What I one thing that I still like, and that's also why I like being very deep into a tech stack, is to actually have a feeling of mastery where if you do get a new problem, you don't feel overwhelmed. Or if I go, man, how we're going to do this, I've no idea. But you actually feel in control. You feel in a way that you can say, OK, I understand what you want, I can plot out some

alternatives. I can think about the different pros and cons of different alternatives, then make it work. And then when you do that and it actually works, that's still something that I find very rewarding. I always joke at work. That's my methodology is Kaiper's first time, right? I like to build something that just works rather than have to reiterate and fix and fix and fix. No, this is not the way.

And you can only do that when you, when you have a like a deep understanding upfront of what you're going to do, what you're going to encounter, what the pros and cons are. Because in the end, what I, what I like to do is on the one hand, I like to build custom software. That's what I've been doing for a very long time already. And I, I still enjoy doing that.

But I also like to do it in a, in a way that for customers sounds like a real benefit rather than something that just costs a lot of additional money because they couldn't buy something off the shelf. Right. That's very often the perception of, of enterprise custom software. It's, Oh yeah, it's just going to be expensive because it is expensive to do something that

is completely custom. So that's that feeling of mastery where you can both quickly show that you can deliver real business value and at the same time know that the way that you've done it is not like taking shortcuts or doing a hack, but to do something proper that you will be able to maintain for a longer time as well, for example. Yeah, that's still something I really enjoy. I feel like that, like the idea of mastering your, I love that.

But also in software engineering, there's so many new things that like come in and out. How do you still like have this confidence of mastery? Because even for me, I've done this for like 8 years now in the industry, I don't feel like I've mastered this craft.

I feel like new stuff always keeps. It's true, yeah. It's, it's what I like in general as to have to drive the direction of where I need to learn and where I need to evolve is to, to, to have real real work or real problems that I need to solve. So this has been a problem, for example, if for myself in my previous job when I was working for spring Source as a trainer and consultant, this would always be when it wasn't

training. The consultancy was typically short assignments, couple of days with the customer. They had some particular issue that they were struggling with or they wanted to learn about something that they didn't understand yet. And I would help them out. But when you do that and then you do some trainings and then every now and then you have like a week where there's downtime, you can just start studying stuff. There's nothing really to tell you.

OK, so out of the 5000 things that I could start to research now, what is it that I'm going to do? Because, well, you have your personal interests, but it doesn't really make sense to only have your personal interest drive your professional development. It should come from professional requirements. So what I really like about my current job is that most of the stuff that I research is actually driven by real

requirements. So if I'm, for example, I'm currently working for Nate Longslow today that does lotteries for a couple of years already where we are building a big integration platform. So the typical concerns there are stuff like making robust integrations to external systems where we have item potency and retries and circuit Breakers and observability and all of that stuff in place.

It's about orchestration, making sure that the one we have business processes that span multiple back ends and something goes wrong, that we can correct it in a way that everything is still consistent and that automatically that focus gives me a way of, OK, so this is what I need to learn. I need to dive into what tooling is available to make sure that we have stuff like circuit breaking in place.

I need to look into observability to make sure that not only do we have login, but we have metrics around all of this and we can do dashboarding and we can see what's happening because we have over 40 back ends that we're actually integrating into this platform. So it's, it's huge. And that's, yeah, that sort of automatically gives you a way, a way forward. We're running stuff on Kubernetes. So that means, OK, I need to learn about containerization, how to effectively do that with

my applications. The way that you package up a Spring Boot application, for example, in a, in a Docker image makes a big difference in how it actually performs and how it runs and how it builds. Also, you don't want to like have to rebuild the entire layer just because of a single code change. So yeah, that I usually try to let the the the the requirements of, of real projects of real customers drive what I'm actually getting into.

And that has been working out quite well for for the last 20 years. I mean, I've, I've been doing

something similar where if there's a certain problem at a customer with specific technologies, that for me is the opportunity to dive deep in those technologies specifically. I don't have this inkling of this interest to go deep by myself to just various technologies because I don't know if I'm going to apply that. Similar with me and sometimes I that could also be tricky, right? Because you're only learning the stuff that happens to be in the like. A chicken egg problem? No. Exactly.

So for example, recently with a colleague, I was invited to an event or a seminar around quantum computing. And this was one of those areas where it's like, oh, this is completely foreign to me. I need to, I don't know anything about this. And I know people also in the Java space that have been looking into what is quantum computing. They've tried to make it digestible and explainable to people. And I find it sort of interesting.

But it's indeed not the thing that I'm just by myself are going to be spending a lot of time on because I am very focused on professional engineering. That's always what I've been doing. Before I got into college, I I didn't want to study computer science because I thought that's just going to be sitting behind the computer. That's probably not for me. So I did AI. But I did AI in 1994 to 1999 so it was 20 years too early. But you're an AI expert.

I was most certainly I'm not anymore though. This was traditional AI mostly. It was at the free University in Amsterdam. It was expert systems, that sort of stuff. At the end they rebranded it to multi agent systems, but it was still expert systems. So that's what I did. And then my first job actually was the R&D department for software engineering geek.

And I thought that will be nice. I will learn some more server engineering than I learned in college and then I'll go back into AI. But I never went back into AI. But it means that my my drive to do software engineering to learn about software has always been professional. It's never been like, oh, I'm just really into computers and I want to learn all of the languages or I want to learn. It's always been, OK, we're going to solve this issue.

And I, I want to do it in the best possible way. And for that, I need to understand a lot. And when you start with enterprise Java in 1999, you have to learn a lot of stuff just to build a very simple system. Yeah, I think that that's also part of it, right? I've always been in this situation where you need, you needed to have like this super steep learning curve. And then only once you get past that, you can start to do something. So that's been becoming sort of a habit for me.

It doesn't matter if I need to learn a lot, I'll just dive in until I really understand. I feel like that that skill, that essence is what is going to make people successful Software engineer. Yeah, I think it has to. You have to. It's a combination of being curious but also being persistent. Yeah, right. Yeah. Piece of discipline and also kind of comfort in being

uncomfortable. Yeah, I like that you highlight that, like this position that you have of mastery that gives you the comfort, but it doesn't take away that there's a lot of technologies that you have to learn. Yeah, Yeah. And that's consistent. That's never going to go away. No. Exactly. Yeah. Yeah. What would you advise for people that are, like trying to master their own craft?

Because I feel like if I were to start out now and it's very hard to relate, but there's a lot of technologies, I do think that software engineering is a very good core to have. And if you can be a really good software engineer by essence of you being responsible for creating software, you're going to think of observability, you're going to think of infrastructure, you're going to touch on topics like data and AI

and cloud. Like I feel like software is the centre of it so it could be a good start to start with but

then how do you start and what do you master first or? Yeah, I think the start needs to be with the basics in terms of the language that you're using. You should just understand the language very well. I still see a lot of people programming in languages where it's clear that they don't really grasp a lot of the basics still. And that means not only that you're limited in what you can express in your own code, it also means it becomes a lot harder to understand other people's code.

So one thing that taught me a tremendous amount of stuff was reading other people's code. And not just anyone, but for example, the open source frameworks that you use that you know, have been authored by experienced people that have been sort of Spring in my case, it's not only a framework that I know very well because I use it. I've actually read a lot of the code to understand how it works. But then that that taught me a

lot of patterns also. Oh, this is the way that they approach is, hey, this seems to be the similar type of problem, but in a different domain. But I see that the approach that they take to solve it is similar. So you learn about these patterns, these use cases. I would start with that because it's going to be overwhelming if you're going to try to start as an architect that needs to think

about everything, right? You cannot think about, OK, this thing needs to be coded, but I also need to think about the build and I need to think about the CD because it needs to be deployed to an environment. And then I need to think about how will it actually run in that environment and how we're going to make sure that it is, that it is up and that we can observe it. And then I need to, it's just

too much. So I think a very natural way for most developers will be to start with just developing the functionality and leave all of that other stuff builds, run times, operations, leave that to other people.

And then over time, in my case, not even on purpose, but just how it goes, you will you will learn about that stuff because you need to pick it up because someone else is not there to do it for you or because you find that you are indeed interested in it. In my case, observability has been something that over the last seven years I spend a lot of time diving and just because I I really enjoy doing it.

It turned out that stuff like building dashboards based on metrics is something that once I learned that it could be done with because this is not something that was really available 10 years ago. But now with all of the tooling that's there, it's, it's quite straightforward. Yeah, I found it really enjoyable. So it was again a combination of, OK, I see that this stuff

becomes important. It it provides real value for customers who can also see things also business metrics, not just operational metrics, but it's also interesting to learn. And there is a lot that I don't know yet that I think I can get into. Yeah, just start, start with the basics, learn it properly and then take it from there, I would say. But learning it properly is, I think is, is indeed something

that I find really important. And I, I do see a lot of people skipping because they, a lot of people, I think are also asked to be not experts in one thing, but to do a little bit of everything. And when you start as a junior

and you don't learn anything deep, but your, your attention is just spread across everywhere. It's really hard to, to do this. And you'll see that with people, for example, who are working in two people teams at some startup, right? If you start there as a junior, you're, you're supposed to do everything. But so you do learn a lot, but you you learn only in a, in a very broad sense, and you don't ever go deep.

That's how I started. Like I I started in OPS and I got to touch on a lot of systems and then I moved more to software engineering. But it was also front and back. It was immediately kind of the whole stack and cloud. Yeah, exactly. Thinking of a lot of things. And that's, I think now if you want to do that nowadays, it's just you're, you cannot go deep down you're. Going to be overwhelmed. Yeah, you're going to be overwhelmed. Just look at.

I mean, the front end nowadays is even that in itself to keeping up with what's. Happening. It's quite a beast. It's huge, Yeah, it's huge and it's happening and it's happening so quickly. Also just catching up and making sure we had a a discussion yesterday someone told me that they were somewhere where they were still working on an Angular GS Angular one. Oh wow. For example. That's the sort of thing like from that is no different in

that sense from back end. You, you have legacy there as well and stuff moves really quickly and you need to understand why you're doing things. But again with AI, it's also easy to just say, well, you know, we're using some stack. It involves React and just give me something that seems to work and yeah. It's a quick win. It's a quick win for now, yeah, Yeah. And it depends a lot.

Also the one of the other things, apart from like mastery, one of the things I think that really defines how I work and what I find important is is also pragmatism. You need to understand always the context of which you're working in. And sometimes we work, or actually quite often we work on big systems that need to support mission critical processes that need to be developed and

maintained for many years. So then a lot of this stuff is important, but you also need to recognize that maybe now I'm building something that will be replaced in a year, but for now we just need a solution. You shouldn't apply the same principles then. I think that's also very important. How do you handle those conversations? Because I've been in conversations with regards to pre sales and customers and I have people where their discipline is like cloud

infrastructure. And I can see they look at the problem through their lens, right? If it's AWS, they go Dynamo DB, we can run it there And like those immediately those infrastructure components come in. And then I come in with regards to, OK, do we have metrics on which users use which components and what's the traffic? And then can we go just with this component and then live as fast as possible and then build from there? Like I have a very different

approach. And then we have the conversations, you'll get defensive and I call people out when they get defensive. I'm like, I'm not, I'm not trying to attack your judgement or whatever. I just think this is what we need. And then this is good enough. And they're like, well, but this is robust and scalable. And like, those conversations for me are always challenging. It is challenging, but I think it is the way to do it. You need to have that

conversation. I was not my previous job, but the job before that I started to work as an internal consultant for the the Natal onset bunk. So that's in English.

That would be like the Dutch bank, but they're not an actual commercial bank. They are actually a government organization that governs all of the banks and other financial institutions in the Netherlands. And this was one of those traditional government organization where there would be like, every other week someone would celebrate their 25th anniversary within the company. Right. People have been working there

unbelievably forever. Yeah. And that was true also for a lot of the developers. And I started to work there when I was around 25 years old and very fresh, yeah, very fresh. And basically my job was to tell those people how to work or how to start working in a different way. So that that felt, that felt exactly the same. Like, OK, so now I need to, to start to persuade people with 2025 years of experience in what they're doing, in listening to me.

And, and, and again, the way I think that that I managed to do that because it turned out that it, it was possible and actually quite successful to do that is to make sure indeed that you don't just tell people we're going to do it this way, we're going to do it that way. You need to convince them and they need to also be on board. But you do that with arguments. You always have to explain your reasoning. You always have to be able to say, I'm thinking about these

and these end goals. Therefore, I think that these and these things are important requirements to take into consideration, but these other things are not based on this. I think that we can go this or this way and also have a way to verify the results. So what what you're saying, for example, what I really like is get something out as quickly as

possible. If you have that, it's it's much easier to iterate on something that works than to work for a very long time on something where you don't really see the intermediate results, Right. This is why big projects fail very often. Still in IT, everyone just says, yeah, yeah, we're 80% there and for months, for months. Yeah, exactly. Incredible. It doesn't work. You need to have something out.

You need to have something working and you can almost always do that, I think, but you need to think about how to do it. So then you can show people that stuff works and then you can gain their confidence and they get on board and they become enthusiastic because they see, oh, actually this is a much nicer way of working than what I'm used to, but it's baby steps.

And yeah, you need to that that sort of consultancy aspects of getting people on board and open to change is tricky, but it can be very rewarding if it works, I think. I feel like it's it's a skill that is quite interesting because you need to be able to communicate properly to different audiences, upper management or your peers. Software engineers specifically, and maybe even more so people in tech have this bullshit detector.

So if you're selling them something that they don't believe in because you don't have that depth, then it's also a problem. You can't just be great at communicating, you need to have deep knowledge with. Absolutely. That's one of the reasons that I like my my current job is that I get to do all of these things at in a single position because I like developing, but I don't

want to be a full time code writer. Yeah, no, I I want to be hands on, but I don't want to do it all the time. I want to have also be involved with customer discussions. I want to have like an account manager like role with my customers. I want to be able to do this sort of mentoring and coaching. I like still delivering trainings as well. And Oh, my career job allows me to actually combine. All of that I do a lot, yeah. I don't get to choose and it's

also laziness. Maybe sometimes I, I, I don't really make very deliberate choices in my career and that's never been the case. It's always just happened sort of to me, but what I really like is that it, it does allow me to combine all of these different things and they sometimes they do actually interact with each other and, and enhance each

other in a very nice way, right? If you're, if you're used to delivering a lot of trainings, it means you are used to not only explaining things to people, but also just to speak in public. So it becomes easier to become a public speaker. When you do that, you'll learn that it's actually important that you know what you're talking about, but you don't need to be an expert at

everything, right? Very often people for a start with public speaking, they think like, oh, but I need to be a super expert in order to share something useful with people and otherwise they will just know I'm bullshitting. But this is not the case at all. But that's the sort of stuff I think that if you do a lot of that different things. Yeah, you, you build up a broad from that.

People talk about this deep profile and actually thought about it before going into this podcast, like because you had a question like what what sort of defines how you work? And I think it's that my not my goal because I don't think it's AT org. My ambition sort of is to say instead of like AT like profile, I actually want to have the the top of the T, But then then it needs to be brought. I want to be both broad and deep at everything.

And you can't. But that's why I think it's, it's so nice that you can get focus from the work that you do right. I don't need to know everything about everything. I need to know everything about the stuff I actually use on a day-to-day basis. And even then everything is probably a bit much. But now I'm still very often people will come to me before they Google, right? That's that's why that's. A good sign? Yeah, that's a good sign.

I think yeah, yeah, for me. So the, the depth is always something that I I think is interesting and I can go deep based on customer problems. And then I always have this kind of internal struggle because should I need to know this in advance or is this fine and can I go deep? And then what is time that is too long before I reach out to colleagues and like ask for help and stuff like that?

That's also something I like to explain people or when I coach developers, indeed, it's always spend some time first yourself because that's typically where you learn the most, right? If someone just provides you with an answer, do this and it works, then you're learning why or what could have tried.

But you also don't want to have someone, especially when they work remote, for example, who just struggles for a day at home. And it turns out that there was some small thing that that you could have solved with them like in 15 minutes. So I think it's always good to get feedback and ask questions, but always make sure that you, you, you know what you're asking, right? You don't just go to the first error or something doesn't work And you immediately say, can someone help me out?

That you first look into it and you think, oh, maybe I can understand it or I can't. And then once you have an understanding of the problem, but you don't know how to fix it, that's the time I think that you can ask for, for help. I've always tried to make sure that it becomes a learning experience where I ask for help instead of just someone saying, but it could be stupid things as well, right?

I had even my, my CEO who is also very technical, actually he just before his holiday, he wanted to get something fixed and he got to me on Slack. It was like literally his last day before he left. And he said something is off, but I don't see it. Maybe you see it and he needed to change a Git URL from SSH to HTTPS, and he forgot to replace a colon with a slash. Yeah, yeah. That's the sort of basic that can take half a day if you don't see it and you're just struggling.

And that's the that's why you need help. That's why you need to ask someone else for help. You shouldn't be working on that sort of stuff for for too long. I think that yeah, if it really comes to yeah, understanding the technology that you're using it, it's worth spending some time on. And that's what I find really important. So also when I have people on my team that are junior or meteor but not senior yet, I always

give them some time for this. I always tell them don't stress out right If if you don't get something to work or if there is an error in there, just spend some time on it. Just try to figure it out and then explain to me what you've done and especially if they're able to explain it and you can see if they learn anything. Yeah, Yeah. I like that you put a lot of importance on also people's own kind of personal development and growth within a team.

I feel like that's very important, not just to be part of a team that is kind of continuously learning, but also for there to be room and space. Yeah, but that last part, usually when there's high pressure, people are looking towards business outcomes faster. You might not be even optimal team size and then pressure gets high. Yeah, But there needs to be kind of a fine balance between learning, growing.

Yeah, it depends a lot indeed on on how your project is run, how it's being perceived to run also. Yeah, that's true. And I'm in the lucky position that the customers that I've I'm, I'm working with right now have a very deep trust in what we're doing.

And we're working more like a partnership relationship in there with them rather than a vendor's supplier sort of role. That's the best. And that's, yeah, that's ideal because you can have open discussions. You don't have to upfront explain everything that you're going to do just so that they can trust that you're doing the right thing. It also means that I think for six years ago already, maybe even more, we stopped doing estimations for for stories, for individual stories, of course we

estimate like, oh, we're. Going to have. Something I see how how big is this, but just high over indeed we had for this integration project. So we started traditional scrum. So we did the whole planning poker thing, and in the end, someone from a partner company actually said this doesn't make any sense. You're just spending a lot of time talking about stuff where you don't know what it's going to be anyways, because the integration is a really tricky domain.

You're always depending on other partners, on other companies that need to set stuff up. Can we even access this external system already? Do we have credentials for it? Do they have documentation? You're going to run into problems all the time. Everything gets blocked, and then you need to fix up something while you wait for that to become unblocked. And then in the end, every estimate becomes like, yeah, well, three points. It's just, it's, it's waste. It's waste.

And in the end what you want to do is you want to reduce that waste, of course. So we have a very well, I think Agile is sort of a dead work, but we have a very flexible way of working as more Kanban lifestyle where we do take some of the aspects of Scrum that we like. Like every three weeks we do three-week sprints or iterations. We still do a presentation and the demo just so that everyone is aware of what it is that

we're doing. It's also a nice point for us to show to everyone in the team what we have been doing because not everyone might be aware. We do daily standups as well, but we don't do stuff like indeed, the estimation we don't have burn downs. We also don't worry about, oh, we thought we would have finished this in three weeks, but well, turned out that it got blocked and now it's not finished. Oh, OK, then it's not finished that we'll pick it up when it's ready.

Yeah, we'll keep going. And I really like that way of working rather than the the real traditional way of doing Scrum because I've I've seen it go wrong also on our summer run projects so often that's I don't really want to do that anymore. I feel like it's, it's similar to what we discussed with regards to kind of being really good at communicating and then having depth knowing what you're talking about.

Because the communicating part, a lot of agile coaches that I've seen and perceived, they have that down. And then there's the depth missing on why are we doing story points in the 1st. Place and it's it's not just an IT problem. I think it's a problem that's happening everywhere. There's for example, person I really like in how what he writes and what he thinks about is bear to bear in the Netherlands. He also has an English social media accounts that he publishes

on a lot on various things. But one of the things that he's talking about, and I see this in IT, but I see this in other things within government, for example, there is no more expertise. Government used to be an expert at a lot of things. For example, in the, all of the

bridges that needed to be maintained in the dykes, they would have people themselves that knew how to do this. And over time, all of that has been outsourced, has been going away. And now they don't even have the expertise anymore to verify the results of the work that they outsource. That's a problem. And that is a real problem. You're but it's, I think something that has been seen in many different areas, right? It's in government, but you saw the same thing in healthcare.

My, my mother has been working in healthcare for decades, and she saw that same thing happening there, where managers were just thought to be generic process managers, right? You don't need to understand the process you're managing. You need to just understand management. And that happens in IT. It happens in healthcare, it

happens in the government. It happens everywhere where this, this notion of, yeah, you have to understand what you're doing and be good at it. It's just something that is perceived as, yeah, but we can just outsource that, right? We just need to manage contracts and then have regular goals about are we still on schedule and, and and that's the management part of it. And that doesn't really work in it. I think that's also why I think as a manager, I'm in the Mt of of our company.

Yeah, you don't necessarily need to be like the best expert at everything, but you definitely need to have the actual hands on experience of what it means to be in IT. Otherwise I don't think you can. Same thing with people who are in sales, right? It can be tempting to think that, well, if you're a good salesman, you can sell everything, but then indeed you're going to end up like you're saying with these people who have the bullshit detector.

They're just trying to sell me. They don't even know what they're selling. Especially if you're trying to sell custom software development, which is a really, as we found a really tricky service to sell because people are not looking for a company who can just build software. People are looking for, oh, I have a insurance problem or I have a problem with lotteries or I have a problem where I need to have a, a system that checks if you can answer all of your

driving license. Do you read exams? Correct, right. That's, that's what what customers see. But in the end, and I think that's happened quite, quite quickly in the last couple of years, in the end, companies tend to realize, but that is actually an IT problem. Yeah. I feel like slowly and especially with newer companies, the people that are in leadership teams or managers, they're not necessarily experts anymore. No, but they definitely do have expertise.

Like there's a lot of value in having managers and also expertise on what they're managing for the people that are doing the actual. Work. Yeah, the I see the same thing for example with my product owner from Das Lotteries. He's actually someone who can by himself analyse our production logs. If something is off, yeah, make an estimated guess on what is happening. And then after that comes to me and say, can you have a look at this?

Because I think we need to fix over there or maybe someone else needs to do something over there. But he's basically already done the analysis. And that's, that helps so much because it means that you can also get to focus a lot more on the work that you want to do, new development. That's rather than just putting out fires everywhere or analyzing things for other

people. And again, because this is the business I'm currently in integration, this, this can very quickly run out of hand where you're just spending half of your day explaining stuff to other clients or to downstream systems that you need to integrate with. And in the end, it turns out there was, it had nothing to do with the thing that you were doing. So that's another reason I think that it's important for people indeed to who are involved in these sort of things to, to know

what they are doing. Yeah. I feel like if I were to start my career and I wanted to do software engineering again, there's a lot of integration problems that need to be solved. And the part of that is kind of

resiliency and observability and security. But a lot of engineering work is going to go and I think also in the future, more into integration work. We have now a lot of software as a service companies, but also AI providers and everyone needs integration.

You also see that those companies that are for example, their core is an AI model with their core is chat bot AI, something they are hiring people on staff, they call them forward deployed engineers, which are basically integration engineers and they come into an organization, kind of ask consultants, they do the integration work and then they leave. I feel like a lot of people are, yeah, kind of looking for integration.

It's also a lot of work that we do tends to either be centered around integration, like my own project, or even if it's not an integration project, still a big component is always integration. Because, yeah, especially in enterprise, right, software is everywhere. So you're no longer working on isolated little islands that just run by themselves on a machine that is under someone's desk. Everything. Everything needs to be

integrated indeed. I feel like if you were like, if you get really good at the integration work then you've solved a lot of problems I feel like. And when it comes to learning, for example, I think this is after learning like basics of development and languages and that sort of stuff. Learning integration patterns is actually a great way of becoming more of a senior developer or or slash architect, right? Thinking about what it means to do stuff synchronously versus asynchronously.

What are the pros and cons there? What options do you have on a technical level for integration? Is it going to be based on Http://web socket? It's going to be based on message oriented middleware. What are the patterns for using message oriented middleware? What's the difference between point to point versus pops up? What technologies do exist there?

What is actually suitable for stuff like edge computing where you have a huge amount of clients sending in information versus just doing point to point between two big applications. All of that sort of stuff is is around integration knowledge basically and indeed it's it's quite important and it's also

much more available nowadays. One of the things I really like as a as an architect about the fact that so many clients, organizations are moving to cloud computing is that using something like Mises or in the middleware is just something you enable toggle. Yeah, it's just toggle. There's already credit card there.

You can just pay for it. Where I've been with a lot of clients in the past where they were just running stuff on their own data center and the small operations team responsible for things where you would go to them and you would say, you know, I have this use case now that we're building, for example, with the the Dutch Financial Times, we have once built their entire website what they were still using their old

content management system. So whenever the content management system would either publish a new article, news article or make an update to an existing article, we needed all of our application instances to know about that so that could drop their cash and refresh it. This is a classic pops up message use case, right? So we said, OK, what do you have for messaging system? No, we don't.

OK, can you just install something, maybe revving Q, active Q, whatever, because we would really like to have a little pops up system here set up so they can do this and say, well, no, we're not going to do that because well, we're three people. We're already struggling with the stuff that we have right now. We're not just going to introduce an entirely new piece of middleware because you asked us to, which makes sense. I mean, from their point of view, I can completely

understand that. But it means that well, I cannot use the best technology for the job and now I need to like an idiot manually do like individual requests to all of the endpoints that all of a sudden need to know. Whereas I I don't have to deal with anything like this.

And with cloud, this is not a problem anymore. Everyone has either something like if you're on AWS, you have SQSSNS or they're running some form of Kafka which is really popular nowadays, or there is Rabbit MQ, or there is something else with because it's not important what which thing there is. It's just important that there is such a technology in place and there's there's abstractions and adapters everywhere nowadays. So integrating something like this in your application is

peanuts. Yeah. I feel like I really like that story because it highlights, I think a really good thinking in the 1st place with regards to the total cost of ownership. And if we don't have, it should be really added. If there's a really good use case, then likely it's yes, because that burden is going to be somewhere in the 1st place anyway. And I feel like now it's a good thing that infrastructure is

becoming more accessible. And it also has that dark side where I feel like people can over engineer systems and resiliency where it's not necessarily needed. Like that conversation of cost of ownership, even on the cloud

should still be a conversation. And then yes, it can be easier to add to it. It's a. Similar thing where with micro service there's still conversation there like OK, how many do we need and how big do they need to be. I remember something like 3 years ago, not that long ago, we had a go to Amsterdam, that's conference that we used to work recognize where there was a speaker there who did a lot of micro service cloud native architecture with Go. But one thing I still remember

very well because it was very different from how we use, how we work, not even used to work how we still work is that to connect to a database, they would never just open up a database connection from their Go service. They would actually build like a dedicated adapter service in front of their database that they would expose over GRPC. And then over that GRPC you could actually make the calls to

the database. And it sounded completely weird to me, but then he started to explain and he said, yeah, well, actually we found that at that time at least the the the database supporting Go isn't actually that great when it comes to things like connection pools and validation. And we found that, well, some something like Java's actually really good at that. So just building a Java series for that put the GRPC front and and and everything else is Go.

So that I found that very interesting because it shows a very different way of thinking about granularity and how big should service be and what does that mean? And there is no right or wrong way, but they but you do need to think about it indeed and see what are the trade-offs when we start doing this. Yeah.

That's why I think it's super important that the people that build the software, they also are responsible for like running it and owning it and also in production. Yeah. And and I mean, I agree in the sense that I don't think that people need to be always like on call for their own software, for example.

But one thing for example, that the way I very much agree with is whenever we have someone interviewing with us for example, and they come from banking or insurance like traditional sort of enterprises, I always ask them, were you able to just even see the log files on production for the software

that you were building? And still so often I get, well, no, not directly, but if there was a problem, we could file a request and someone would make an export of a part of the logs and they would ship it to us. Like, fuck, if that's your, if that's the way that they, they do, they do trust you to build the software that runs your production, but they don't trust you to actually look at it running. That's strange, right? It's completely insane.

They actively break the feedback loop and you have no idea what what your stuff is doing. And that's another thing I try to cultivate in people indeed, is make sure that you, once you actually build it, that you look at it and make sure that you can see what it's doing. And if you find out that, well, you cannot see it properly, think about what you need to do to make sure that you can see it at logging, at metrics, whatever. Because, yeah, I think that is

super important for people. Once you move beyond junior is you need to, you need to build that feedback loop for yourself where you can understand what the stuff that you build is actually doing in a real environment. And if you don't have that, if that's missing, and there are a lot of companies where this is missing, people just don't see what their stuff is doing. Yeah, you're never going to move beyond a certain level. No, I agree.

You mentioned that you wouldn't necessarily put the people that create the software also in the role of being on call duty. No. Why? Why is that? Because I think a lot of a lot of problems that we typically see that people that are on call need to deal with are not about there is a bug in the software that we need to fix immediately and roll out. It's about other stuff in the integration business that I'm in right now. Typically it's not us, it's

downstream system that is struggling. But then that means the queues are filling up or maybe some threat pool is running out of threats because some system is just super slow. Now everything starts stalling and that causes certain things to happen. But it's not that often the case that that the developer will be able to efficiently deal with that.

And they might also not have like the complete overview of the entire platform rather than the system that they are responsible for to make these sort of decisions. So I think there is still a value in separating those roles. And I still see a couple of years ago when DevOps started to become a term, people made fun of companies who had a DevOps team because the whole point of DevOps was that you didn't have a DevOps team anymore, but that it was development and

operations doing stuff together. Rather than saying, Oh yeah, you're now, you're now the DevOps are. But I think that's changed DevOps. The meaning of DevOps itself has changed quite significantly actually. DevOps now means way more things like infrastructure as a software doing stuff with terraform, automatic provisioning of, of entire

environments. And, and again, that it makes sense to say, Oh yeah, that's not something that every developer can be expected to know about and to be an expert. And so we need people for that and that's dev OPS as at least as far as I'm concerned, a lot of that is what we now call dev OPS. And it makes a lot more sense to have those type of people on call.

And I don't need to also be experts at the system that they are operating that they just need to know like the basics of what is, what are the metrics that we need to watch in the health checks and what should we do if certain known issues arise. And only as a like a third line, you go to a developer and you say, well, I think this is actually something in the software that you need to fix.

Yeah, it's interesting because I followed this training from Gregor Hope. It was a couple weeks ago and he was in a company responsible for a team of architects and they had done something which needed to have kind of an on call duty for and their OPS team didn't want to do it. So then he was like, OK, then we'll we'll do it ourselves. And he did not, He had enough political credit to be able to do that in the 1st place. And then his team was like, well, do we really want to do it?

And his mindset was we are responsible for kind of architecting this thing and we also should have the confidence that it should run Florida State. No, that's true. Like build it Florida State exactly you have easy on. Call. No, that's why indeed why you should have the feedback loop. And this is an even stronger feedback loop. If you get called out of bed, of course, then if you could just see what's happening. But yeah, but that makes sense.

That is important, but it's also true, I think that a lot of things are not really about. It's indeed about what you expect from an engineer from certain year. If it's just about the software that they develop, that's a different thing than. The entire system that you run and maintain and you can do both. Amazon does this famously, right? You build it, you run it, but I don't think it's necessarily the

best model for everyone. And that's not saying that it's that you shouldn't do it. It's saying that I think for a lot of typical more, let's say, enterprise use cases, that is the business that I'm in. I think it still makes sense indeed to separate some of these responsibilities. Yeah, sometimes also for like regulatory or security reasons, right? Maybe you don't want your developers to be able to see and do everything that is possible in a production environment, right?

I get that. Yeah. And we've seen that a lot actually. We have had customers also in financial industries dealing with credit card payment gateway for example, where they needed to have very strict regulations around for ICE principal who can do what it needs to be audited. And then if you're going to say, well, every new developer on the team, the first thing that we're going to do is deploy to production, right, That that can be a really nice practice. But it's yeah, in in this sort

of situation, it can be tricky. I feel like in, in whatever you do, there's going to be risk. And then it's just how do we mitigate that risk? Yeah. Is it for a principle or is it something else where people are still tracked but they still have that flexibility of going to production? That is the conversation. And this is also, I think very much why it's so important that you that you know what you're doing, because then you can

properly estimate that risk. And I think it's a really good

point. I had AI did a keynote at Go to myself a while ago where I said I think a lot of people overvalue the return on investment that you get from testing on non production environments. Obviously you need to do tests, but a lot of people think that OK, if we do this, if you do this in the right way, we will never encounter some problem on production.

And if we do, then it will be very hard to fix because well, once something reaches production, it's always 100 times or 1000 times harder to fix than if we find it during development, for example. And in my experience, this is not necessarily true at all. Actually, just yesterday I did my final production deploy of this big integration platform before my holiday starts, and I was checking if everything

worked. It's expected. Now, I saw some unexpected errors there, and it turned out that I actually introduced an error myself somewhere. This was causing a problem in people during the day when they would try to see the result of a ticket. Like what did they win? So there's no nothing breaks in the sense that there's no data issue or anything. It's just that one small piece of functionality is no longer

there. I think from the time that I spotted the error in the production logs to getting a production fix out with the fix of 20 minutes, yeah, tops

perfect. And if if you know that you can do that, if you know that you have the experience, but also the CICD capabilities and you have the access rights and everything else in place that you can actually get these sort of fixes out that quickly, then, well, how much time would you actually want to invest in making sure that something like this can cannot happen by putting all sorts of guard rails and tests and everything in place? Yeah. It's just you have to look at

the return on investment. And sometimes it's just not worth it, No. It's interesting that like speeding things up might make things more resilient, like on the long term, which is a weird mindset, right? If we deploy to production faster, then what about all these risk kind of affordances that we have in place? Like are those going to be kind of sub converter or others not going to be there? But then when you're talking about this feedback loop, it's, it's so tangible, right?

You see an issue, you can immediately fix it. Sure, maybe if we deploy faster we create more issues, but we fix it faster and those issues will still be there if we deploy less fast. Yeah. And, and this is something that is also very hard to to explain to both I think customers, but also regulators. For example, within within those lotteries, we had an audit and this was actually a while ago already.

We have several audits. But one thing that they want is whenever we deploy a new version of a service, which is something that we we basically do all the time. They required us whenever this service was involved with, for example, the casino part of the of the lottery, that we would have a hash or something of this thing and that we would report what version was running in production. So we've done that and they didn't want it just as text,

they wanted screenshots. So that was one of our DevOps engineers has actually automated the process that whenever we go to production now we're going to send screenshots of all of the stuff that is in our Docker registry and show that it actually runs a reduction. But it means absolutely nothing. It it, it's, it's completely pointless. But it's just because these companies are used not to regulate custom builds, enterprise software that gets deployed on a very frequent basis.

They are used to regulating products. And these products are maybe going to have two major releases every year and three. Twice a year, that's. Yeah. And we need to be absolutely sure that the thing that they claim to have delivered is running in production. So you need to be able to prove that. But yeah, for us, it's, it's

just fewer overhead. And what we see with customers as well that sometimes they are already very scared to to put new versions of something in in production because, well, the thing that we have now actually works and there's a lot of people using it and who knows what happened.

But yeah, you need to really educate your customers on this, that the longer you wait and the longer you accumulate change, higher actually the risk is going to be of course, well, if you're doing it regularly, you become better at it because you do it very often. The amount of changes that are

inside of everything will become a lot smaller and people will still remember what was actually in there. And I think the last 1 is really important, right? If I'm going to deploy something and it it has changes from a colleague who's not even on my team anymore that they made like 6 months ago, and now it's finally reaches production, something is wrong with that. No one has a clue. No. So then we really have to dig. Yeah, we have to dig. We have to reverse engineer what

happened. Whereas if we're deploying every week, every day, and something is wrong, then Oh yeah, I still remember what I worked on. Now it must be that thing and then indeed my experience is that getting fixes out then is is way way way quicker. I agree, yeah, it's interesting that with requirements like, OK, we need to really have evidence of what we deploy. That's a requirement.

And then sure, you can think, OK, maybe that's strange and I don't really agree with it. But changing the system can also be hard and it requires time usually. And it's such an engineering mindset to then be like, but we really like fast deployments because of all these benefits, right? Quick changes. We know exactly what we deploy. We can kind of triangulate problems faster and get the fixes out faster. So we automate kind of this process of then giving you the

evidence. And maybe that's not what the people want because they expect kind of, oh, maybe 2 screenshots of like twice a year deployments and then get overwhelmed. But yeah, that helps. Going to change the system then. Yeah, that's really good. And fortunately, I think for me, I don't have a very, a very large amount of experience in

these regulatory environments. So this is the exception rather than the rule for me. What I've heard from a people, a lot of people who do more work and finance, for example, today, indeed they were, they're dealing with this all the time. And there have been some nice talks about it. Dave Farley, for example, is one of those people. He he invented the continuous deployment term.

I think I wrote the book on it together with Jess Humble, but Dave Farley has written also a lot about how can you actually work in a iterative, fast, often deployable fashion in a regulatory settings. And it basically means you have to work with the regulators to to say, OK, we understand what you're asking now, but that's not the real thing that you're after. That's just your current way of getting the information that you need for your ultimate goal.

So what is your ultimate goal? And once we establish that, we can find out how we can actually meet that goal in a different way and, and get it to work. But this is not, this is not my area of expertise because fortunately I haven't had to do this that often. But I think it's it is very interesting because, yeah, if you just say we cannot do this because of regulations. And it's. Not good enough? No, it's not good. You need that depth. Explain it to me.

Especially with engineers, for me it's really hard to do something without understanding. It's not good enough for your engineers, but in the end it's also not good enough for, well, your customers and your systems, of course. Yeah, you need to be able to work in the way that you want to work and still and meet these requirements. And it's possible, but it's it's, it's real work. Yeah, it really depends on the environment.

I think especially for your career, if you want to accelerate, you need to be in environments where people have the patience and also the respect to educate each other with regards to OK, this is the knowledge and try and create that shared understanding. There needs to be enough

interesting problems also for you to learn and grow and not just work on kind of surface level and not being able to see logs in production, for example, not owe what you build. And I think that's really going to make or break your career acceleration. You can coast. Coasting is fine. Not everyone needs to kind of aim for the stars. With regards to growth, it depends on your environment, but if you're in a really good environment, it can really help your growth.

That's mainly it. Yeah, yeah. And I think it's also, you can tell very often when, when you have people that have that drive and that it's, it's important also that they are able to do this because it, it becomes like a self accelerating effect. If, if people see the results of, of their efforts and then it, it sort of just bootstraps

itself from there. But you can see the same thing happening with people who indeed are, they're, they're, they're young and they're enthusiastic and they want to learn things, they want to do things. And then they end up in the situation where, Oh yeah, this is not really possible. Or by the time that you're working on something now, it it won't be running in a production environment for half a year. And then that maybe not even because someone changed their

mind, right. You see this sometimes in product development companies where you can be working on stuff that never makes, makes it into the light of day. Yeah, that's of course not very motivating. So, yeah, it's also important to so that people actually stay motivated to, to continuously improve themselves to the work. That you don't get motivated by doing the work, you get motivated by seeing the results of doing the work. And if you don't see those results, then yeah, it's really

hard to just keep going. Yeah, I mean, it makes me sad that people can get stuck in an organization like that and think, OK, this is it and this is it. And the same is on the other side and the same. I mentioned interviews before, like I always ask these questions to people from banks and insurance companies. I've had a couple of interviews also with people coming out of these sort of organizations where I told them at the end, well, you know, we're going to make you an offer.

You can accept it and not. But whatever you do, please leave your current job because the way that you're explaining what you're doing, this is just not a good position to be in. And well, we can be a company that can do better for you hopefully. But there are many other opportunities that are better as well. Because sometimes people just don't know, right? If if you're, if you're in your first job and this is what you're used to and this is what you do.

You think that, yeah, this is just the way that things work or. But but then you'll learn by talking to other people, like what is actually possible and how could it also be. Yeah. Yeah. Sometimes changing jobs is the easiest way of making things better. Of course, yeah. That's why I really like when people come on like you on podcasts like these, to share

their perspective. Yeah. Because otherwise it's just if no one shares their perspective, you only know kind of what's within your organizational boundaries. Yeah. And that's true. Yeah. Thank you so much for coming on, man. I really enjoyed this. Yeah. Good software engineering mastery. I feel like it's a long path. It's a hard path. There's continuous learning there as well, but once you have really solid fundamentals, you can really excel and deliver.

Yeah. And also, once you actually go beyond the basics, you start to learn the patterns and the the underlying abstractions, and then it actually becomes easier,

right? If I need to learn a new programming language, now I don't, it's a very different experience from someone who's just into computer engineering who needs to learn a language, right? I can just. Yeah, that's from there. That's from there. They probably have that you know what to expect. So it's it gets better, it compromise. It's like, yeah. But it's also the message to people struggling.

I think it gets better over. Time, yeah, and definitely you have to get through that hurdle if you're still here. Thank you so much for listening. Let us know in the comments section what you've thought of this episode. Leave a like if you liked the episode and otherwise we'll see you on the next one.