Hey Ben.
Hi Matt.
So normally we have a plan when we do these podcasts,
That is usually our plan to have a plan.
We have a plan to have a plan, and then this time we spent so long discussing what it is we should talk about that I said, why don't we just hit record and go with it? And that's what we're doing. Right. You know, we're doing it live We're just gonna...
So, what are we going to talk about?
It's a very good, very good question. Um,
We could go meta for a second, and talk about how we decide what we're going to talk about.
That's a brilliant idea.
Yeah.
Let's do that.
I like that
You and I both staring at a shared Google doc right now, where we have just as over the sort of the last few weeks, we've been gathering thoughts on interesting topics, often over a cup of tea that you and I will have, like on a, uh, an afternoon. And then we're like we get 20 or 30 minutes into our, our coffee date and then go, Damn, I really wish that we were recording this because this is gold. This is kind of in conversation is exactly the kind of thing we want to talk about in the podcast. And so we write it down and then when it comes to the cold light of day, we don't actually go, Oh, that doesn't it. It's not as good doing it when it's staged. Right. We had the really interesting conversation when we weren't recording and let's not try and pretend to make it something different. So that is our current process is a big Google doc and us scrawling notes all over it. And we've got, wow, what's that two whole pages of thoughts. And you'd highlight your something which says trunk based development as potential discussion today. So why don't we just start talking about what you meant when you wrote trunk based development and see if something comes out of that? Yeah.
Yeah. I think that's a great idea. Yeah. I mean, that's, it was sort of born out of, um, some of, some of our experiences at, uh, the company that we're currently employed at and contrasting that with some of the experiences that I know I've had at other companies, I'm sure you've had at other companies, um, where there is, there are lots of ways to skin a cat. Why are we so mean to cats? I keep using that phrase. You don't need to skin cats. How about zero ways? Uh, there are lots of ways to do things.
Cats get this all the time, right? There's cat gut, right? If you're going to make strings for musical theater and you make it out of cat gut, which I don't believe has ever been related to cats, but it sounds pretty horrible for the cat. Also, maybe after you skin the cat, you were also, you know, this is, this is bad moving on,
But yeah, there's lots of different ways to, um, to kind of do this, uh, where this is organize your code, right? Like there's, there's a lot of different ways and some of it can be kind of bike shedding, right. It's just sort of like, you know, Coke versus Pepsi, this or that decision that doesn't really have that much of an impact. So that can have very significant impacts on the success of your project or the success of your company even and sort of everywhere in between.
I mean, you have that, that expression about, uh, Conway's law.
Yeah. Yeah. I mean, you know, Conway's law is a feature or a bug kind of, depending on how you think about it and
What is Conway's law?
Yeah, yeah. So Conway's law is that the, uh, an observation that the structure of your software will match the structure of your organization, right? So you, if you have a software engineering department with three different teams, um, and you tell them to build something, you're probably gonna get a system in three parts. Right, right. Um,
Seems obviously right, without even evidence. Right. Which is a dangerous thing in general, but we've observed that it is generally true.
Yeah. I mean, it's an observation, right. It's sort of like looking up at the sky and saying like, I see that the sky is blue and that's the blue sky observation. Do you also see the blue sky observation? Ah we intersubjectively share the blue sky observation! Well, that's amazing. So it's something that people observe, right? Like when you, when you build software in an organization, the structure of the software tends to match the structure of your organization. Right. And the question is, is this a bad thing? I mean, I, I think of it as a thing, right? Whether it's good, bad, whatever labels you want to try to throw on, throw on it. This is a thing that happens. So you can think about it in whatever terms you want, but it's a thing that happens.
You should either work with it or have it work against you. There's really only two things, right. Either you go with it and say, okay, it, you said this before, it's either a feature or a bug, depending on which way you look at it. It probably makes sense to choose, to see it as a feature and say, how might I arrange my organization, such that I get the code arrangement that I want? Or how do I make sure the code arrangement appropriately mirrors the organization I would like to build.
Right. Right. And if you have some preconceived notions about how your organization should be structured or how your code should be structured, and those two things, aren't in alignment, you're going to have a bad time. Right. If.
That makes sense.
If you think, well, we should have four different teams and you have one giant code base, it's going to be real hard to have four different actual teams, right? Like you might have an org chart with four groups on it, but that doesn't mean you have four teams. So all of those things can have a pretty significant impact on your success. And so it's important to kind of think about them and some different ways that I know I've seen of just sort of structuring, I'm just going to say chunks of code. You can call them projects. You could call them right. Things. I don't know what you call them actually. But I almost always think of things these days. I tend to think of things more on repositories in terms of repositories, just cause of like GitHub and open source and all that. Right. Like that's just sort of, my brain has been warped to that after, after years and years of GitHub. But yeah,
Remember there was a time before it seems odd that, you know, you'd have to go and find some random CVS repository and pull from something or God help you SourceForge which still exists. I don't know how, but there are still projects that are hosted on SourceForge, which just seems like a terrible waste of your time to visit. But yeah, that was a, those were dark times, man. Right. Just tarballs occasionally...
I still can't roll up to a CVS and not think it's the pharmacy repository in CVS, they even have the slash there every time I see it I'm just like, that's the pharmacy repository,
There was, there was Perforce. Um, and then there. was some Subversion of course, which was kind of, and of course there was Visual Source Safe, which was none of those words.
Subversion Oh, I'm sorry. That's just my favorite expression.
Okay. So moving off of source control because whatever, we, I don't think we've had source control has not been a topic that we've discussed. Yeah. Nor should we.
I feel like these days, it's just kinda like use git right. And there's not really much debate about
Except, and git is like the least worst thing, but getting back to like the environment you say, you were saying about repositories being like the unit that you think of.
Right, right. Yeah. So it's like, like you can have a debate about mono repo versus multi repo, which is certainly a debate that I've been a part of from time to time. And, you know, do you
Never very rarely not been in that debate because you're either in one state or the other. And there are always the few vocal folks who have very reasonable reasons to move to the other approach. And in my experience, like it's either a tooling problem or an org problem, or if your org is big enough, you need a tool department to manage it. You know, very, very large web search companies that I've worked at before have had entire departments devoted to making sure that their giant mono repo is manageable given the way that they would like to administrate it. And then of course, if you have worked at one of those places, you think mono repos are the best because like, look, all these benefits you've got, you know, I can just put in code from some other part of the repository without really thinking about it is really trivial.
It's really easy. There's no build issues. I just, it's just all there. It's all there. I can git grep across the whole code base, how amazing. And then you don't realize that that doesn't necessarily work when you move to like a four person company and you don't have a giant team of people that are administrating the systems that make that feasible be it, the build systems themselves, be it the source control, um, code review staff be it like code owners. I mean, obviously github is starting to pick up all of these things and we're starting to have those, we're starting to have those kinds of features come. But, um, but yeah, it's, it's a thing. Um, and obviously then if you are bitten by that, you might decide it's just easier. Hey, I don't need to jump through all these hoops. I just want to make a little repository in my style, in my way away from the rest of the code base so that I haven't got thousands of other, um, commits, polluting my history. I don't have to fight with someone for a continuous integration server slot or, you know, merge conflict, blah, blah, blah. I've got my own little way. So they're both stable places to be. The problem is we live in this extraordinarily high dimensional space of trade-offs right. That is engineering. And any one set of issues is only one possible solution in a set of equally valid solutions to the problem of like, can we work effectively?
Yeah.
So what would you do? And I don't think I've ever asked you this. So let's, let's discover that we suddenly have a big difference of opinion or not. What would your natural tendency like if we were, if you and I, were to start up a company and start developing with say like 15 developers, just picking a sort of big enough to be multiple teams, but not so big as it's like an unwieldy set up, you know, we've got like three or four things to do. Would you go for the monitor repo or multi repo? What's your sort of default thought,
I think it entirely depends. I'm going to answer your question very specifically, but I'm going to start with what sounds like a wishy-washy answer.
Okay. Okay
I think it entirely depends on the people involved and the problem that you're trying to solve, because again, there's this deep relationship between the code and the organization. Right. And you have to keep those things in sync or you're going to have pain. And so you need to think about who are the people on my team, who are the people in my, in this group that want to work together, right? Either because they share like a common technology, like, Oh, these are the C++ programmers. And these are the JavaScript programmers or whatever. I don't know. Or, um, you know, these people have worked together before and they know that they work together well, and that was half the reason we hired them because we know they all work together well.
So it would be kind of a shame to split them up. Right. So think of the, the, cause the people, parts of the organization, I feel like are much harder to change. I kind of have this rule that people don't change until they do. Right. Which is to say, if your plans depend on someone changing, you need to get some new plans because they might change. You never know people change, but if your success depends on this person doing something that they've never done before, and have I shown no inkling before to do uh, you need to get some new plans.
You make a very interesting observation. That is kind of the, the, the good advice to give somebody who's planning on getting married to somebody else. Like if it, if your future happiness depends on changing that person to be someone that they aren't right now, there's probably not a good idea to get married to that person that they're not going to change. They might change. They might, they might change. You have to understand people do change and that's of course fine. And many people improve and all that kind of stuff. But like, if, if you're betting your future on it, probably better not to. And I guess to an extent, you know, there were a lot of similarities between getting married to someone and hiring somebody, right. You're bringing someone in, you're going to spend a bunch of time with them probably more time than you spend with your spouse. I mean, Covid not withstanding, but more waking time with them. You've got to work closely. There's going to be conflict from time to time. There's going to be differences of opinion. You need to be able to trust that you can get on with, with, with, with a sensible way of getting through those differences of opinion. And, um, yeah. And it's very expensive to get rid of them. Uh, if it turns out that it wasn't a good, good hire,
Which is not, not a good way to think of the thing. Right. Right. No one goes into either of those things expecting for, for, um, the, the worst outcome. Yeah. That makes sense. You don't expect folks to change. Um, so design again, as if that's a feature, not a bug, right? Hey, these people work well together. Let them work in the way that's most effective for them.
Exactly. Exactly. And in a lot of ways, this is just another kind of engineering you're just solving for constraints. Right? The constraints of this company are these four people work really well together. Uh, we can try to break that constraint just as you can try to force, you know, a, a type to be cast into something else and you can kind of squeeze it into the box and make sure all the bits align. And you're fine. Is that really what you should be doing? Well, probably not. Right. Yeah. So I mean, you can, if you need to, but probably not. It's like, um, you, you just needed to solve for this. Just like you'd solve any other constraints. And my argument is that the people are the hardest and most expensive thing to change. And so you should start with the people and then work back to the organization of the code, right?
So if you have 15 people that all worked at Google and all worked in the mono repo and really liked it, and they all sort of understand how it's supposed to work. And two of them are willing to be the tools people that build all the tools to make that work, take a monorepo. It sounds like will work great. If you have three people that worked really well together, five other people that hate those three people. And then the remainder of your people are just like, I don't really care, whatever, make sure that you have at least two teams and make sure that the boundaries between those, that codee is very well-defined.
Hate is very strong. And let's just say, don't work effectively together.
Don't enjoy working with right, right. I have worked at companies where there were people that just really, you know, for various reasons political or otherwise just didn't like each other, but you go to war with the army that you've got. So, you know, you say that's how it is. And you gotta think about this stuff, right. If you try to argue these things, I think entirely based on the technical merits, you have to think about the technical merits. But if you only look at those, I think that you're going to wind up in a really bad place. You have to, you have to consider more.
That makes sense. Um, I, I don't know if it was you, I was talking with about this earlier in the week, but I was definitely talking to somebody about a couple of people that I was working at a previous company where we'd reached a sort of Nirvana state where we didn't need, we went when really big on pull requests and all that kind of stuff, or code formal review. We'd obviously read each other's code as you went through it. And sometimes we'd just go through the log to see what the other people have been doing. But like we had reached this point where you, it was almost difficult to tell who had written what code. All of the code looks and fitted together beautifully. We'd kind of come to a gestalt mindset about how to approach a problem, such that you could, if there was three different ways of solving a problem, you could probably, you could bet good money.
That, in a vacuum, the four people that were working on this team would probably solve it in the same way we're using almost identical naming conventions, algorithms approaches. And that was a beautiful, it's almost like a flow state for a team, right. You're kind of interchangeable. It was, it was wonderful. And it meant that we never really had any conflict. We didn't think, think about it. We just got on with the job at hand. It was like a high point in my career to be working in a situation where I was just churning out code. And there was nothing really stopping me. And not even within the team, the team felt like it was moving forward and that could easily be a bubble. Now in that particular environment that we were in, we were in our own bubble. We had our own build system. We had our own CI set up.
We had our own deployment. That was slightly different from the rest of the team, because we felt strongly as a two or three of us that this was how it should work. And it overlapped like 80% with the rest of the team. It helped that we were in C++ everyone else was in Java so that there was an obvious like line that we could draw. But then we kind of widened the line a bit and said, no, no, we think it should be done this way. And so we are going to do it this way and we'll make sure that we work with you. We inter operate with you. So I guess what I'm trying to do is citing that as both, it was, I'm very lucky to have had that situation, but also that, that is a sort of natural occurrence. I think at least it appeared natural. Maybe my, my team lead was, uh, was actually maneuvering it that way because he knew that things were gonna work better if it was that way, but, you know. Uh, but you know, it was our, it sounds like we heard, we had stumbled on a nice low point or a high point in the, in the multi-dimensional space where that worked really well for us. And, and, uh, both the Conway's law version of like, we were our own little team in our own little world.
I mean, it's funny because it's sort of like you guys had that sort of independence right. Of, of being able to do things your own way, um, because you were in C++, and sometimes that's a convenient excuse for creating that, but honestly, maybe you should do that on purpose, right? Like if you have a group of people who are all C++ programmers, that doesn't mean that they should all share the same build system or share the same, makefile, or share the same project structure, just because they're all in the same language or even using the same technology, right? Like it's much more about what the people who are going to be tasked and responsible for solving the problem want and how they can work together effectively. Um, I've said this thing for awhile, uh, that a team is a group of people who succeed or fail together and succeed is sort of a, an interesting word in that phrase cause you, it can mean sort of different things.
And it's sort of a, the definition, there is a little fuzzy, a little bit on purpose, but success for most programmers looks something like being able to come into work every day and have the experience that you just talked about, right? Like, yes, there's financial success and that is a necessary, but not sufficient portion of being successful. I feel like as a programmer, but real sort of, you know, Maslow's hierarchy of needs stuff self-actualization comes from coming into work, being able to work effectively, do your best work, do work that you can proud of that you're proud of and solve a real problem. And if, you know, if that's what you're optimizing for, then I think you should be much more cognizant of what group of people can I assemble as a team that all sees success in the same way that all operate in the same way to create that success that will not create conflict between two people because they have different definitions of success. Right.
Interesting.
Because then you don't have a team anymore, right?
Yeah. That's a really, really good, um, yeah. I hadn't really thought of that. That's a great characterization of what makes a team, right? Because you can define a team as these, all people are working like notionally together, they have a tech lead and he's in charge of them and they have a manager and she's in charge of that, whatever, you know, like that set up. And then, you know, and that's defining it using Conway's law from the top down, like, Hey, your team is defined by the org structure, but then you've got the team of like, well, this person they want to do everything as beautiful, functional programming. And that's really their goal. Their goal is to turn the world into Lisp, or the world into whatever. And we've all worked with people that have various bents and, you know, we've all been guilty of similar things, right. Um, you've got, uh, and then other folks who are just like, no, um, it's gotta be just as fast as possible or other people that know I want to refactor it, or it's gonna be testable, you know?
And obviously there's a, there's a, there's a sort of core of things that are just required to be successful as you say, necessary, not necessarily sufficient, sufficient, necessary, not sufficient or whatever the right way. Um, but yeah, if you could get a, a group of people, like in the, the way I described before, where we all looked at the code and we're all like, no, the code isn't done, unless it looks like this where this was how we all wrote the code by that point. So that's, that's an interesting one. So can I ask a question about that though? Because the, it sounds great. It sounds amazing. And it's something that I've been thinking a lot about recently. And is there a problem that you end up with a group of folks who just think the same way is that, is that a problem? I mean, obviously from a diversity point of view, diversity of viewpoint and all the other axes we measure, you can, you can get into a really bad situation. If you just say, no only people that think like me and work like me can work together.
Yeah. I, I, so I have some very nuanced views on this. I mean, first of all, from like a cultural diversity standpoint, I think you need to be as diverse as possible, right. Like never do you want to give up on those kinds of constraints ever, like try to try to make your companies, your teams as, as culturally diverse as you possibly can, because you will just make you better problem solvers, right? People from different experiences will have different insights and you'll get different solutions that are more holistic.
While we sit here looking at each other two 40 something, white men, our backgrounds alone have already showed us that you are measured in that one tiny dimension. Seeing the world in two different ways is very, very valuable. And just to have more dimensions to that sound, right? The more the merrier, right? The more viewpoints the better.
Absolutely. I personally think that it is completely fine and maybe even more functional to have when it comes to sort of like technical diversity, right? Like different ways, ways of, of problem solving that are not necessarily cultural, but are technology-based right. Like the classical functional programmer versus object oriented programmer, for example, something like that. Right. I think it is very reasonable to have within a company, a diverse culture, but within a team, basically a monoculture. Right. I do not think that's harmful. And in fact, this
Is a mono techno-culture.
Mono techno-culture, right? Like all the functional programmers grouped together into a team, that's fine. You don't need to split it up. So that there's two functional programmers and two object oriented programmers on every team. Right. That's sort of like a little bit of a straw man, but that's kind of what I'm getting at.
Right. I see what you're saying. Okay.
I don't think it's harmful to say we're going to take all the Clojure Lisp folks and put them on a team and give them the tools that they need and build something in Clojure. And then that's a monoculture from that sense
On that very specific point, I've had the very first team I came into, uh, when I joined the company where you and I met was, uh, a desk where there were, there was some Clojure and I ended up pair programming with somebody in Clojure. And it opened my mind to the whole idea of functional programming. And I was a very, very dull, boring procedure, not procedural, what do they call it, imperative, programmer programming. Like, I mean, as you know, I've called my way up the stack from assembly. So, you know, that's how I think about the world. It's one statement after another. And to see this very implicit, very much more abstract and different way of thinking about it. I found it extraordinarily irritating for the first couple of weeks because I'm like, I just want to do my job. Why am I pratting about with this? This seems so much harder.
And then I had that moment of realization and my experience has opened up. And so I was exposed to something new and I learned a ton from that. And I resolved at that point to try and make sure that I look for those kinds of opportunities and take them every now and then obviously, you know, this is the exploration versus exploitation, right? If you've got a good team that gels well together, sometimes you want to just say, all right, you four folks, you go off and you do your thing and that's great, but every now and then you probably do want to kind of perturb it a little bit, a little bit of simulated annealing. Like, Hey, I got this, this, uh, this girl wants to come in and do this thing over here. And you know, she's her background is in, is in, uh, you know, I dunno genetics. And so maybe you can work with that and do something either that there's something else coming in where we might shake it up a bit and then everybody learns something. But I guess you have to be super, super careful that you don't put people in awkward positions, particularly if there's a person coming into an established team. And they're the outlier as it were
Yeah. Yeah. Well, like
In my case I was the outlier because I was just a weird C++ programmer being dumped into a Java and Clojure department, but I learned a ton and I was glad of it. But
I mean, I, going back to the metaphor of this was sort of just a form of engineering. There are always trade-offs right. And if you do some of the things that I'm talking about here, there are trade offs. And one of the trade offs is that you will have less, fewer, you'll have fewer opportunities to get the kind of exposure that you had working in Clojure. And, you know, my general theory on that is that you just need to be aware of that and you need to solve it in a very specific way, which is you need to move people around slowly. If you move people around quickly, that will be very disruptive, right? But you need to have a career path for the people in your organization. This is something they're going to want this, something that you need to give them. You're like, you're not just going to be on this one Clojure team for the rest of your time at this company.
Otherwise the time at this company is going to be pretty short, right, you're going to get sick of it, or you're going to want to grow, or you don't want to do something else, uh, or not. Right. Again, people don't change until they do, right. Maybe you do have people that just want to stay there forever and that's fine. Right. So you need to be able to handle those cases. And so one of the things that's really important is understanding that within the team, there is a little bit of a culture. Like there's, there's usually a little bit of a esprit de corps. It's this thing that you're kind of talk about where you had these four people that you worked with and everybody just sort of like had mine melded. Right. And like they, all, everyone knew what to do for any particular situation. Right. I would say that's a team that is probably ripe for either moving somebody out or moving somebody in assuming that there is somebody that wants to do that. Right. Because what you have is a very, it's almost like a sourdough starter, right? Like you have a very healthy strong sourdough starter. Now it's time to make some bread. Right? Like you're going to pull somebody out, you're going to put them somewhere else. And you're going to see what happens.
I mean, that team was two, to start with, that was me and, uh, the team lead and we, it did take us a while to find our, our level. Right. Um, I think I've alluded to people before who would just sit down and think a lot and then type out the right answer. That was my, my lead had he, he was very much, the think about a lot and less on the actual tests because code didn't ever really go wrong. And there was only one of him right. When I joined. And so then I joined, I'm like, well, I don't know what I'm doing here. I need, I need the safety net of more testing. And then I think we found our level, right. And then we became more and more productive. And then we wanted to grow the team and we had a fresh out of university person join us.
And that was really interesting for us, both because we got to bring someone up to speed, who was a blank slate. And that was a really exciting experience for me personally, to actually sort of be able to help someone on their journey from pretty much say nothing. Obviously if you've been to it, you've got a degree in computer science. You've, you've, you know, a lot about it. But like, we also know that like, you know, it's like when you learned to drive a car, you pass the test and then you learn how to drive. Right. That's the second step of actually doing it for a bit. Um, but yeah, as you say, the sourdough starter, I feel that we had got it into a nice sit situation where we could bring them along. And that really, really helped them. Um, I believe anyway, uh, and then, then when we added another person to the team, it was just a natural way, but that was not the kind of thing where we were throwing in a curve ball that wasn't like the Clojure person came in and we worked and were learning back and forth.
That was definitely a very much a, here's a blank template in the first case. Or here's an industry veteran who came in as like the fourth member of the team. Can we just make sure that they, they fit in the role that we, if it's three to three to one, then it's a good chance that you're even those edge edges that you might have where you don't fit in exactly. They get knocked off pretty quickly, or else the whole team adapts a little bit, which I think is a natural process. So yeah. Yeah. So yeah, sourdough starter. I liked that. I like that as an example for a decent little team that can grow,
But yeah, I think that metaphor works well, but I mean, so, so the, all of those things then inform the technology. So if we're going back to the whole debate of like mono repo versus multi repo, like you might have a situation where there's like a one to many mapping from teams to repositories, right? Where like each team manages a set of repositories. You might have a one-to-one mapping where like each team basically has what is effectively a mono repo. And even though that team might be building lots of different services, uh, maybe even unrelated services they all work out of the same mono repo, because the tooling is there and the, the flow is there
The utils dot. Everyone's grab bag, yeah, we just need the case insensitive string comparison routine, the same one that we copy paste into or whatever it is. Yeah.
So, so you can, if you think of things that way, then I think that the, the tech, the technology choices become constrained in a good way in that good way that you sort of have, like, you know, you were saying this sort of multidimensional problem that you need to try to solve. You take a few of the dimensions out and all of a sudden the possible solutions become a lot more easy to reason about.
Right. Interesting. Yeah.
And so, you know, a lot of those things can kind, kind of solve themselves in that way. Friend of mine once kind of mentioned. Had an observation I thought was really insightful once he said, teams are immutable. And what he meant by that is every time you add or remove someone from a team, you don't have a different team. You have a new team because you have to kind of incorporate all of the needs and all of the abilities of that new person into the holistic team and reconsider all the things that you considered before. And some of us have very concrete situations where someone joins the team and all of a sudden, there's a four hour meeting about why we do things a certain way, right? Like,
Well, yes. And then some of us may have been the cause of those meetings.
Yep, exactly, exactly. Um, and so it's, it's like, it is both an opportunity, a little bit of a burden perhaps, but an opportunity to sort of reevaluate why it is that you do things the way that you do and reassess them in the context of what is essentially a new team. And my point here is that the technology is sometimes the thing that changes after that, right? Like you have a new team. Sometimes the result of having a new team is that you need new technology. So it's sort of like, well, previously we used make, and now we're gonna use CMake or previously we used CMake. Now we're going to use Basil. I don't know, uh, or whatever it might be. Right. Um,
On the subject of like the repo monorepo stuff, there are a lot of, in my experience of the, the arguments for, and against those two things is, again, the tooling we talked about, like having teams, if someone says, well, there's a tool that can make it easy to extract, publish a library. Maybe we can now use Conda or Conan or VS code or insert pack. Maven. Everyone loves that one I heard. To extract out some code. And then, but now you get an interesting sort of side effect of that is that now you've changed the way you develop and maybe that's a positive. Maybe that's a negative, you know, we've all been in situations where versioning some chunk of code that used to be inside your repo, where you could make a change and then test how it works across a bigger code base in situ is a convenient thing to be able to do.
And so you're like, well, that's an argument for a mono repo, right? If I have to refactor and change the name of this particular interface, it just works like the refactoring tool takes a while, but it can refactor across the entire code base. Fantastic. But also we've also been in situations where like, what I really want to break the API, but I don't want to do go through the pain of changing everyone who's using it right now. I'm going to do it in my repo. And then everyone else can pin to the oldest version that does what they need. And I can move into a new version of that. And so there's a, there are trade-offs even along that dimension there. So yeah, I derailed you from, do you talking about the new member to a team being.
Like creating a new team?
I've derailed us completely. Yeah. Creating a new team. Yeah. And we've moved into something else, which I wanted to talk about actually, anyway, which is these kinds of things. These kind of trade-offs because what I just did, if you, if you, I mean, unless you want to finish off from that point,
My point was that that can change the technology of the team. And so I think it's actually kind of a natural thing to say, okay, what are some of the technology changes that can come across from that.
I don't want to jump the gun there.
No, I think you were maybe like 30 seconds ahead of me. It was perfect.
Perfect. Well, so yeah, that's one thing. So that, that can be a feature that you can extract a library and then you like pull in version 1.0 of the library and all that kind of stuff. But that sort of begs the question, asks the question, I should say, cause people will complain if I use the word beg to make a question, nothing ever is actually begging a question is should you be at trunk all of the time? Because when you have a mono repo, you are kind of by definition, you're at trunk. Um, at least for those kinds of changes, right? If I do a, uh, a pull, then if somebody edited, edited some library library, I'm going to use air quotes for this library. Cause it's just a subdirectory of my back. My, my mono repo, then, um, I got the, I've got the change immediately, right?
That's, that's something I get straight away. It's more, it's more of a, um, there's more decoupling. There's a sort of clutch plate between, um, parts of the code. If you use a sort of CI server to build and deploy a sort of compiled package, then you pull back down again into another project. You've kind of got this, uh, separation, which can be a feature sometimes. But if you don't live at the trunk all of the time, if you, if you allow yourself to float away, there's a danger that you can float too far. And then one day you're like saying, Oh yeah, yeah. That bug that you fixed in version 2.7, could you back port it to version 1.8? Cause you know, we never quite got round to it now it's a huge big deal.
Yeah. Those are, those are painful.
What kind of trade-offs other trade-offs? What am I, what else am I missing in that? I mean, sure. There are tons of other things. What are your thoughts?
Yeah. I mean, there's, there's a lot of, there's a lot of trade-offs there for sure. I tend not to use libraries as a form of dependency management to solve that problem if I can avoid it. Or it's maybe a better thing to say that I use them very intentionally. I use them when the difficulty to refactor across that library is again, sort of like a feature and not a bug, right? Like I want it to be difficult because of reasons. Right. And the reasons might be, I don't want inexperienced programmers mucking with something that is very core to the business or very proprietary or very difficult to understand or whatever it might be. And so we're going to encapsulate that in a library or I don't want, non-programmers having to solve very trying to solve very difficult software engineering problems like multithreading, uh, or parallel computing. So I'm going to build a library that solves that, those problems on their behalf. And if I've done a good job of that, they will hopefully never have to reach into the guts of it.
Tuck it away almost as a, just a, you know, putting the panel something saying, you know, like Warranty Void If Removed
Yeah, exactly. Right, right.
Yeah. That's that sounds, uh, it sounds pretty bad because obviously anyone can look at it through the code anyway, you could go find the source of that code, but, but it's just an, a, an extra sort of insulation layer that you're using instructionally within an organization to say like, you don't need to go past there. Obviously there will be people who, will
Yeah. We want to encourage the cross team communication that will be necessary if changing this code requires. So going to somebody else and being like, Hey, can you change this function for me? It'd be like, actually I can't. And here's why. Right. And have you considered maybe trying this instead, all of those kinds of nice conversations that might never happen if you don't put those sort of speed bumps. Now again, they are kind of speed bumps. And I generally don't like it because it sort of creates a slowness. You just want to do that intentionally. You only want to do that on the block with the kids playing sign. You don't want to do it literally on every street in the city. Right. Um, so, so I use it intentionally. The other thing that I tend to do is I personally tend to prefer breaking things out, not as libraries but services, uh, not microservices, but I had this phrase that I used a while ago called uh one second services to sort of like define this like approximate size of a service and how much behavior it should have. It's, it's this is the first time I've said testing in this podcast. It's one second worth of unit tests.
Oh my gosh.
So about a thousand.
We're about 30 minutes in, and you said testing only now That's impressive.
Yeah, I know. And I'm not going to talk about it any more other than to say...
I didn't have that on my bingo square, I'm not going to win this. I'm so mean to you.
I deserve it. I deserve every minute. Um, but no, it's sorta like if you have one, if you have a thousand tests, that's about the size of a service that is nice to work with. It's not hundreds of thousands of lines of code. Obviously the tests only take a second to run. So they run super fast. It's not a microservice. You don't have this thing where it's like, okay, I'm bundling up my single Lambda function. That's eight lines of.
pad left!
pad left, right. As a, as a service that you call, right. That's insanity in my opinion. Um, but it's also not a giant, you know, monolith of a single service. It's got like, you know, a huge interface to it and a thousand different things and connects to, you know, a database and a Redis queue and a Kafka queue and a, you know, dynamo DB instance and everything else in the world. Right.
But a second's worth of tests is just as good as anything of licking your finger and putting in the wind like, yeah, this is about the size that, I mean, I just know instinctively the kind of level of complexity that comes with that amount of code. And obviously you said a thousand tests there because the other sort of aspect of this is that the test would be fast, which we've talked about a lot. So we won't cover again here.
I'm not going to talk about it again.
Right. So you prefer services and I mean, all of these are APIs, they're different ways of making APIs, right? This is what I the layering between software be it like you call into a library and you can't actually see the library because it's, there's no code. You just got like a header or a couple of like, uh, bits of, uh, framework-y stuff that you talk to or you make it a service. And now you've got some kind of RPC mechanism that again, defines a very strong API and enforces it as well.
Kind of. Yeah, I do see those things is a little different though. The library it's usually an all or nothing upgrade, right? You can't have four different versions of a library active in your application at the same time. Yeah.
Not without pulling heroic creeks. Yeah. Right, right.
Whereas like, if you have an API, right. You can have backward compatible. Like if you're making a breaking change, your API, that's not strictly additive. You can have both versions of that API live at the same time. Right.
Oh, interesting.
Clients, clients can even use both versions at the same time. Right. Like they don't have to make a choice of we're moving to version four. I think we'd be like, well, for this section of the code, we use version three in this section, of the code we use version four. Right? Yeah.
Very interesting. I'd never considered that. I mean, obviously there is, there's got to be a cost to writing a service that seamlessly supports some number of older versions of it. But I can see the benefit that comes from it too. Right? Yeah. As you say, this allows us sort of more transitioning or transitionary. That's not even a word and more, an easier way to transition between older and new code without it being the code where the code is like, well, you're either calling, you know, Bob, uh, the Bob function with two arguments or the Bob function that takes, uh, uh, some other completely different structure. And obviously some languages will let you do some amount of overloading and other trickery like that. And of course you can have versioned bits of code and you can probably do that with libraries as well. And kind of pull in bits of this library and like enable there are various tricks to do it, but it's very clear and clean in a microservice environment.
Yeah. Potentially you have, you have at least the opportunity to do that. And then if you can align those with the structure of your teams, such that, you know, the services that interact with other teams have very stable APIs and the services that interact within a team have more flexible dynamic APIs because maybe the team has a single mono repo with its 12 services in it. And all of the external facing APIs have, you know, hard worn integration tests, but all the internal APIs can all be deployed at once. Cause it's all a big one mono repo. And if you want to change the signature between three services, that's fine. Cause they all get deployed at the same time anyway, then you can do a lot of cool stuff with that. And it's just, it's all about being cognizant of the boundaries between the people and how that reflects in the boundaries in the code.
Got it. So you talked about deploying there in terms of like deploying a whole bunch of services and obviously that fits together. If you, uh, you have, uh, uh, exactly as you described, uh, as a team, that's got all these microservices in one kind of gob that they can deploy, but in general deployment, especially when you have got tiered things like V1 V2, V3, V4, there's usually a lot of machinery to make that work. There's some kind of dispatching system somewhere higher up that you kind of, well, I want this microservice to appear here in some logical, uh, space. How does one deal with that kind of stuff with regards to, you know, experimental stuff, maybe can I make it do what, what do you think? Like maybe I've got like some, um, I'm rewriting a server and I want to like deploy it, uh, somewhere. Should I be doing that in trunk? Should I be making a branch and then deploying that in some other namespace thing? How, how do things like that work in what, talking about that? That seems,
I will say that I have never built a system. I mean, there are a lot of people that are going to be way more experts on this than me. Um, in terms of building systems with like hundreds or thousands of running services, you know, deployed on AWS with all different kinds of versions and stuff like that. And I'm not gonna, if you're, that's what you're doing, you should not listen to me cause I've never done that before. And I have maybe even intentionally avoided finding myself in that kind of situation because there's a lot of, there's a lot of stuff going on there. The systems that I have built where I think that this model works well, are systems where you have sort of dozens of services, all of which are large ish, there are these kind of like one second size services, right?
Where they're, it's not, you know, millions of lines of code. It's also not eight lines of code,
but it has a non-trivial amount of responsibility.
Right, right. And so, and so, and some subset of those services are dependent on each other in potentially breaking ways. And most of the time what I've done has been able to deploy the things that are hardened against each other as a unit. Right? So it's like if you've got three services that all need to change together, then they all get deployed together. There's not even an option to deploy them one at a time. But for the things that do have sort of those harder interfaces, you can deploy them independently and that's what you plan to do. Right. So you're going to change an API. You're like, all right, we're going to introduce the new API in service cluster A and then we're going to update service cluster B to use the new API. And then we're going to go back after that's done and everybody's happy and remove the old API from service cluster A right. And you do that as sort of a three-step process. Right. And that's kind of just the way that I have done it for these types, this type of structure. But you know...
I've just been recently doing a whole bunch of work off on my own little, um, branch of a big, big code base, essentially a mono repo for all intents and purposes in its own domain. Right? There's a lot of different people in different ways of writing code. They're all in that same repository. It doesn't represent the whole company's code at all, or even like the whole domains code. But if it has a mono repo feel to it. And so I've been in the world of making changes on the side, uh, in my branch and then obviously suffering the pain of merging in all the time to try and keep myself up to up to pace. But there is another world in which I could have done all of this in, in, in trunk. And I know like at again, big web search engine company, feature flags all the way forward for that.
And I, and I've done, I've done those things. I've used feature flags for like turning stuff on and off, but it can be very difficult to do that well, especially when you're making structural changes, it's very hard to say what I redid, the class hierarchy behind a feature flag. You're like, well now you've just got two entire copies of the code base, right. And somewhere you have to make that switch. And, and so, um, I'm thinking, I've been thinking about it. I realize it's sort of, this is slightly tangential to the things that we've been talking about, but it sort of fits into the workflows and how teams work. There's a, there is a world in which everybody lives in their own branch and they kind of occasionally merge back into a master or a main branch, a main trunk of development. And I guess just channeling what I've learned from you, this, in this conversation, right. Again, it's let the teams find the right way to do that. If, if it works well for having people maybe breaking off one or two at a time and going off and doing something and then bringing it back to the center, then that then go with it. But yeah,
I had a conversation today with two people that work in that very same repository that you're talking about which shall remain unnamed. And, um, we were talking about them doing some work in another repository that is just the three of us, it's me and the two of them. And I was telling them, you guys don't need to send me PRs. You can just put your, your work right into master. Right. Um, because the there is the benefit of the sooner you get it in the master of the sooner I can start coding against it, right. If you've changed some method signature or some class hierarchy, the sooner I see that change integrated into my own work, the less painful it's going to be for me. So I want you to do that as soon as possible. And if that means that you need to check in code, that's not running yet because it's been disabled by what is effectively a feature flag. I would prefer that to you sending me, uh, you know, sending me a PR and then doing all your work in a branch and all the other things that kind of go along with that.
Well. I mean, there's definitely, yeah, there's a benefit to that. There's definitely a thing on like, I, I have a personal extreme and we haven't got enough time left for me to go on and waxing lyrical about.
next time, next time.
How to do a good pull request, but certainly a way to do a bad pull request is to drop a hundred file PR on someone because you've been working away in your, in isolation for, for six weeks. So we'll come back to that another time. But it does that not encourage people to check in like slightly not ready code.
Well, I think you can balance that is with continuous deployment. So if, if your policy is code, that's pushed to the trunk automatically gets deployed, assuming that it goes through the continuous deployment pipeline, then that's sort of setting the standard for, for code that gets shared with you. I mean, I think it was Michael feathers. I keep quoting him a code is a way we treat our coworkers. It's just like, you know, not filling up the coffee machine, right. Or, you know, leaving your three week old lunch in the office refrigerator. And so for me, a reasonable approximation for is this code good enough to inflict on my coworkers? Is, is it good enough to go to production? Or is it good enough to get deployed basically? And so coupling those two things together to where whenever you push to the trunk, it's going to go to prod. So you better make sure that it's not completely terrible, right? It's not perfect. There are certainly ways in which you can write code that, you know, isn't going to negatively affect the production environment, but will negatively affect your coworkers. Um, but it's, it's a, it's a, it's a sort of strong enough correlation where I personally, I'm, I'm a big fan of that
It's more measurable than just dissatisfaction of your colleagues. Right. Which is harder to measure maybe once a year during the, you know, peer review that it might come up, but it might be a bit late.
Right. Right, right, right. So yeah, I think that's a reasonable way to do it. And then of course, that implies given all the things I was saying earlier, the organization of your code is such that that is also the unit of deployment. Right. Because otherwise, if you can't do that, right, if you, if you have to deploy a bunch of things together and it deploys automatically, then practically what's probably gonna happen is that they're all gonna be in the same repository. And now you have what is essentially a mono repo, but just for those things, right? Otherwise you're building very complicated tooling to make all that work.
I was just to make that point. Yeah. The tooling is the other solution to that problem, but it amounts to the same thing. Isn't it like if you can carve off an area of the problem space and make it work in this way, through being in its own repo through tooling, to make it, uh, to, to isolate a unit of deployment testing and deployment, and then have this kind of, uh, thought process where you say, uh, check into the trunk and then, and then like, you don't need to do pull requests or anything like that because you do the, the, the standard that, which you hold yourself to to say, is this going to, this will go to production as soon as like, as like, as I commit, it kind of is the barrier. Now I, I can get behind tfhat, but I think
Pull requests have other benefits, which we can talk about in maybe a pull request related episode.
I think we should do a whole topic on pull requests,
On pull requests and code review, or trade-offs
I'll add to the list! Since we have a list of those things.
Add it to that list we were talking about fabulous. There you go, everybody. This is how we do things. And I can actually, I can see his cursor moving around in the document. Now he's not even lying ladies and gentlemen. Oh my God. Okay.
Pull Requests
Well, you type that. And I will, as I say, I have very strong opinions and I have a number of open source projects and I get to be on the receiving end of a lot of reviews for those things. And I've developed strong opinions on those. So we can talk about that some other time. Yeah. Cool. All right. My friend, well, we've, we've, we've definitely covered a whole bunch of stuff. And, uh, we made up as we went along, which is kind of how we do it always, right.
Yeah, it is.
There's no pretense here that there's a heavily scripted aspect to this, uh, this extremely hi high production value. Yeah. But one thing we've never been really good at is ending podcasts because, you know, we're just chatting and then they were like, Oh, I guess we should finish now. So I guess we should finish now.
and cut!