[Best of 2023] #125 - Patterns for API Design - Daniel Luebke

Hey, a quick message for those of you who are listening to this episode on Spotify. I have a small favor to ask. Spotify now allows mobile users to rate podcasts. I would really appreciate it if you can take a quick pause to go to the Technically Journal podcast page and leave your favorite show your best rating on Spotify. It will help me a lot to get this podcast to reach more people on the platform. Thanks a lot. So let's start. To your book.

So you have had a lot of experience in the industry. You have written thesis work in large financial company. Why specifically? Writing an API design book because API has been around for many years, especially when you said about SOA, right SOA. And there are so many books already published as well. So is there anything different that you try to convey or try to teach us from your book? Yeah, definitely. Thanks. So the story behind the book is a little bit longer.

I need to go back some years. All I have to my money was the first author and the team lead of this pattern approach. I knew him for quite some years. I said OK I want to have this idea. We are building all these services we have and then back in the day microservices were really the hype rebuilding all these and FI need to teach those two. And I have corporations. We only need to build API. So I want something to teach.

I at the time was new at a very large scale project where they are connecting land, registries, banks, notaries and some other parties across whole Switzerland. So it's a system where you have like 1000 different organizations connected and automating all these business processes. As you can imagine there are many services, many APIs involved for doing that. A bank places a request, the platform generates some documents, send them back to the bank so that they can digitally

sign this. It goes forward to the notary and then to the lender Just ran whatever you can do there. There were some people very new to service design and we had many discussions about things which I would have thought are basic, but obviously they weren't. So my motivation was more for my daily work life to just say, here's a pointer, please read this there and then we can have this discussion again. Yeah. So that was essentially how it started.

And then other people joined like Merkel and Sizar and Uber, I can't recall now exactly in which order, but quickly became this team of five people. Uber and Sizar both were very experienced as patent authors. Already Olaf and I and Mirko as well had some practical, more practical experience, although Olaf and was a professor and Merkel actually now as well. I'm still in industry. I think that's the motivation. And then the question is, how do you teach these things?

So I mean, design, architecture, always about decisions and decisions have these nasty aspects of that. You get something, but you lose something. And if you go through the right door, you can't go through the left door. That sounds easy, but people always assume that there's this perfect solution. You just do something and then it's perfect. And you do this every time because it works all the time. The silver bullet thing. And we all know it doesn't work.

I think even if you're fresh from university, at least after three years, it doesn't work that way. And patterns make that explicit. I think that's so great because firstly you have a name and I think we'll talk about names as well. But you have the section bound forces. So what are the influences on your design? And some things are better if you use a certain pattern and some things are worse, so you

pay for your decision. And patterns make this explicit, and that's why it's so great as both a teaching tool and also reference tool. So even nowadays if I'm in projects, it's just nice to go through the pattern list and check whether there's something I should have considered or not. Because it's always busy when you're doing things, it's nice to have checklist. I always love reading pattern books because it is practical. That's the first thing, right?

And also it kind of like lists down the decisions or the trade-offs some people say on why you choose a certain pattern. So for example if you opt for scalability, then maybe this pattern might be better for example microservices, right? But obviously we know that microservices is not silver bullet. In some cases it might not work as well. So Fred Brooks have this write up last time, no silver bullet.

So I guess it has been mentioned many times in any kind of architecture kind of discussion I had in this Technician episode. Your subtitle of the book also mentions specifically about Loosely coupled. Message Exchanges. So tell us more why you? Choose specifically loosely. Coupled message exchanges as your subtitle. Well, actually the subtitle was one of the last things we decided. And if we look at the book, what's special about this?

It closes the gap between high level books like Enterprise Integration Patterns, which are not high level or very abstract, but they're certainly on a high level of abstraction. And then you have these books about technology. So how do I use REST or GRPC or graph, QR or whatever, and what verbs to I use and which circumstances, how do I build good UIS and all these things. And in between there is

something. So you have a conceptual API design, something where you decide, OK, I need these kinds of operations, I need to have these quality attributes and because of this I perhaps to pagination. I chunk things and not transfer a whole message and that's independent of the integration technology of the protocols you use. Back in my first project was a banking system or a banking platform and they used CORBA all the way. It was all CORBA already there.

There were many patterns which we still use today and more REST http://apis because these are really long living and even now whatever comes up next, you will have these problems and you will have these patterns hopefully addressing those problems. So it's not tied to any specific technology. First of all, so you can make your API design on a conceptual level using these patterns and then shoots your technology.

So that's one aspect. The patterns themselves are not coupled to any technology themselves, but obviously it's probably more about the coupling between API clients and API providers. I think a good API and that's why nowadays finally we all agree that API first design is the way to go is you don't want to expose your internal data models or your internal logic too much on your APIs. And the more your clients are not under your control, the less

you want to do that. I think most patents here will allow you to deal with this problem. So how do I build APIs which well if I do them by the pattern book so to say, although it's not your goal shouldn't be to use as many patterns, but if you just follow these forces and things then you will hopefully have a design which allows you to have these loosely coupled exchanges which are not dependent on your internal models. So you mentioned something. API first design.

I think this is mentioned many times. In fact, I think I have one episode James Higginbottom also talking about that and in the book you also. Mentioned a couple more. Good practices for API design. So I think these days almost all back end developers will create API especially REST API or maybe GRPC or Graph QL. Are there any other things that you would advocate for all these engineers? What kind of good practices they should do or they should consider when building API?

Well, first of all you should think about how to express that if you start with any software development, it's just not only APIs, but it's in general. You need to understand your domain thing, that's important. The problem with APIs is let's go back to this project. You have a team of people who if you want to build a good solution or a very good solution, you need to understand what banks are doing, what notaries are doing, what land registries are doing.

And the interesting part is in the last couple of 100 years, I mean this domain is very old. Houses and government management of houses have a long time. So land registries are around like forever and banks are very old concept and no trees I think as well. And over these couple of 100 years, these three parties still haven't learned what they are doing. So that was very interesting. A bank doesn't know data and model of land register, which is

very interesting and surprising. So if you want to build APIs for domain or for clients where you don't know exactly their requirements, it's the first thing where you need to reach out. To clarify this and it's easier said than done. There are a couple of approaches. So most of all you're doing workshops, building your first API version, release it, try to evolve it, but then you need to deal with life cycle issues.

So one part of this loosely coupled is also you need to decouple the life cycle of your client and your provider, because you won't deploy new versions simultaneously, so you will have points in time, many points in time where they're different versions. So client normally supports an older version and the provider rolls out a new API version which should be backward compatible so client doesn't break. So that's kind of the ideal

scenario. Well, many people say OK, just build backwards compatible APIs, yeah, but it's not always possible. Sometimes you have breaking changes. If you say OK, I don't know my clients well enough, I want to iterate and I want to improve things. It's more likely that you will have breaking changes. For instance, banks usually had one to many relationships where the Land Registry internally manages many to many relationships. These kinds of things if you know them later on.

So you have your API defined with one to many and now you need to support many to many. It's likely going to be a breaking change. Also in your environment there might be breaking changes. So in Europe we have the Euro as a currency, which before were multiple national currencies and if you had online banking bank then to money transfer, there's no way of doing that now as it was a breaking change that there's a euro and not all these local currencies.

The same goes for account numbers only have been local national account numbers and now we have the Ibens International Banking account number. So there will be changes in your environment which will break where you can't do anything. And there will be changes where you learn your domain better and where you can say, OK, in theory I should have known this already, but you didn't. The more you iterate to get to your requirements, the more you need to think about your life

cycle. Some patterns we have there is, for example we have an experimental preview pattern.

So you say, OK, let's say we roll this out and currently I don't give you any guarantees, you can check this out, you can give us feedback, but it will break without us telling you so That's not nice for the client, but the client also has some benefits because he gets a preview, so a better version where they can use the API, preview it, use it, give feedback, and perhaps hopefully getting a better API down the road due to this feedback, even if it has breaking changes.

And obviously some organizations or some developers will say, OK, no, under these circumstances, having only an experimental preview, I don't integrate, I don't use this API, which is fair because it's known. What's unfair is if you say, OK, here's an API and then you just say don't support it anymore, I break it in some way or another. So being clear about your evolution process, I think that's important.

So have your requirements, know how good you know them and then plan ahead because in the beginning you can. So if you just had guarantee like we support this for three years and then you just try to alter this later on, people just come to your office and hit you. Probably These are decisions which you need to do as early as possible, and then obviously you need to flesh out all the domain knowledge in your API payloads.

So yeah, it's always wise to start with the domain understanding first, and in fact things like DDD or Domain Driven Design is also something that advocates you to actually understand the problem. 1st. Rather than coming up with just the technology or the solutions, people these days whenever they build API, the first question that they have in mind is definitely which technology like HTTP, Ras, GRPC, Graph, QL and things like. That, but actually.

Specifically in your book you also covered the other important parts, which is the payload itself. You mentioned the messages. Maybe for us engineers here, can you give an overview what aspects that we should consider from the messages point of view, the payload point of view, in order for us to design our API better? Well yeah, I think the most important part is how many requests do you have and how large your message payload is.

So if we look at that, obviously you could make an API operation which might be an HTTP resource where you say get me all customers and then you get back all customers and everyone's happy. You can deal with the data whenever you like. This approach obviously has some drawbacks, So much load on the API provider because it needs to fetch all these customers. You have much load on the network because you need to transmit all these customers.

So we don't do this design. What we usually do is get a certain customer or search for customers, but it's that everything we need to consider, No, usually not. So if we look at the search, even a search might return too many records. That depends a little bit on your domain and how many objects you have, what domain objects you have. So if we have a service which would return list of currencies, we know how many there are and

it's quite manageable. If we are fetching customers from any large organization at the Microsoft on the whole Salesforce database, whatever you like, that's too much. So we need to have some mechanisms if we are running into this problem to limit our response sizes. So you can either say OK I only returned the 1st 200 hits I have might be a good strategy. We could have pagination. So I give you the first hundred and if it isn't sufficient you

can query the next hundred. We could also say, OK, perhaps you need not all customers, but query all sessions in the conference. And that's a very huge conference, like 600 sessions. To each session you have some information like the title. OK, just a string who's giving the presentation? That's just a string, the image of the presenter. So that's obviously some large payload part. When I put this in my message, it depends on the use case, but most likely not.

I would just put a link to this image and whenever the client probably pops into the detail view then it can fetch this data as required. So we have this decision on whether we need to include data or whether we reference data. Depending on your use case, you often have things which you don't need. Coming back to the Swiss land registries, you have hundreds of years of historical data and for most use cases you don't need historical data. So it makes sense in one way or another.

And there are multiple options. Address what data the client needs and you could do many operations. You could do some parameters which there's a pattern called wish list. So I say I want this historic

data or not. So it's just a boolean flag and if you set it you just get back 20 times the data what have If you have no historical data and if it gets more complex then you might have not only wish list but a wish template where you can define in your whole data structure and all the nested things what data you want to have or not. Which then closely or easily gets you to graph QL. And all-purpose is just to specify a request with exactly

the data you need. And so there are many things you can do to adjust or trade off the number of requests and request size on the response size especially. And I think that's the point where we need to make more conscious decisions. On average, I think most people just dump out everything which they have and then at a certain point they just see OK, well it doesn't scale, the database load gets to have me and things are getting so slow.

I hope you enjoyed this short clip from Tech Lead Journal Favorite Playlist. If you find this episode useful, please help share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you want to listen more from this conversation, please go back and listen to the original full conversation with my guest. Stay tuned for the next Tech Lead Journal episode, and until then, goodbye.