The Design of Web APIs, Second Edition

Welcome to the deep dive. Today, we're jumping into something that powers so much of our digital world but often stays hidden, Web API design. Our main guide for this is the second edition of the Design of Web APIs by Arnoul Loray. It's described as taking you from well novice to pro in building APIs that work well, are easy to use, and importantly aren't a pain to deal with? So why should you care about this? Simple? Really a well designed API, it's like a secret startcut to making

That's exactly right, and the challenge we're really going to unpack today is figuring out what goes into making an API that one does the right job, two is actually user friendly and works well with other systems. Three considers all the real world constraints like security and performance, and four is built using a process that keeps getting better. It's a skill that applies everywhere, whether it's an internal tool or a huge public platform.

Okay, let's back up a bit. What exactly is a web API?

Fundamentally, fundamentally, it's a software interface. Think of it as a way for different applications to talk to each other over a network, mostly using HTTP, the language of the web.

So when my phone app gets new updates, that's an API talking to the back end precisely.

Or when different server applications need to coordinate complex tasks behind the scenes, APIs make that communication possible, and they're not just single channels. One API can offer many different operations and many different apps can use that same API. We should probably also mention the types private for internal use, partner for specific collaborators, and public APIs for well everyone and a key thing. Even private APIs might still be

Right, which brings us square to why good design is so critical. Often the best way to see that is to look at the pain caused by bad design. The book mentions things like cryptic or inconsistent naming, poor data organization, maybe missing data or operations we've all been there, right, I definitely remember hours spent trying to figure out why a call failed, only getting back a totally unhelpful bad request error. Just maddening.

Oh. Absolutely, it's a classic developer nightmare. Laurie uses this brilliant analogy in the book The Kitchen Radar. Imagine a microwave, but instead of normal controls, it just has one button bag and you have to hold it down to heat things, figuring out the timing yourself.

Huh, so you have to become a magnetron expert just to warm up coffee exactly.

It sounds absurd, but it perfectly mirrors how some poorly designed APIs force developers to understand obscure internal details, and the impact is huge. Developers write fragile, complex code, end users get crashes, confusing errors, maybe workflows that make.

No sense, which leads to frustration, maybe canceling subscriptions or just switching to a competitor.

Right, and for the company providing the API, it means rising technical debt, wasted engineer time, production problems, security risks, even lost revenue. It's bad for everyone.

Okay, So if the kitchen radar is what we want to avoid, how do we actually build something good? How does this design process fit into actually building software?

Well? API design sits in that crucial space between figuring out what's needed and actually starting to write the code. It's a process. You move through stages like identify API capabilities, then you design the programming interface. Then you need to describe it clearly, and finally enrich those design documents.

Okay, and the book breaks this down conceptually into layers.

Yeah, four conceptual layers, each tackling a different aspect of the problem. The helpful way to think about it. First, a versatile API design that does the right job. This is about meeting the real user needs, hiding the messy internal stuff, and making it usable in ways you might not even anticipate yet. Think about a so network API.

Interesting future proofing, almost what's next.

Second layer, an API design that is user friendly and interoperable, making it easy to understand, easy to use, and ensuring it plays nicely with other systems. Common sense, but crucial makes sense. Third third, an API design that considers constraints. This is where you really integrate things like security performance, the specific context it operates in, and how you'll handle future changes. And the final layer, finally, a reasoned and

Okay, let's dive into that first layer. Then identifying capabilities. Where do you even start?

You start by analyzing the user needs. This often begins pretty vague, like we need a social network, you.

Know, right, very high level exactly.

But you need to quickly get more specific. What does that actually mean for users? Maybe it's allow users to tag friends in a photo. The key is expressing these needs in plane language first, something everyone involved can understand.

So how do you break those big needs down systematically?

Decomposition is key breaking down those use cases into small, manageable steps. The book introduces a tool called the API Capabilities Canvas for this. You work through it twice. Essentially, First, you map out the happy path, the way things should work ideally, the normal flow right. Then you go back

Okay, so you've identified these capabilities, how does that translate into the actual interface a developer interact with the programming interface?

Those capabilities generally map onto two core concepts, resources and actions. A resource is like a now a high level concept like product, user, order and actions are the verbs what you can do to those resources get product, create, order, update, user search products.

And how are those represented in a typical web API using HTTP.

That's where the standard AGTP methods come in. Get to retrieve, post to create, put to replace, patch to partially update, and delete to remove. You combine these methods with resource paths, which are like addresses for your resources products for the collection products one, two, three for a specific one.

So the path structure shows relationships like users, four, five, six, orders exactly.

It describes hierarchy. But, and this is crucial, the paths should not directly mirror how your data is stored internally. You want to hide the provider's perspective and focus on the consumer's needs.

That keeps it flexible, right decoupling. Now, when a developer makes a request, where does the input data actually go?

Good question? There are standard places within an HTTPU request. Path parameters are usually for identifiers like that product's one twenty three. The one to twenty three is a path parameter.

Okay, it makes sense.

Query parameters live after the question mark in the URL. They're great for things that modify the request like filter, status, published, sorting, or pagination in any ear, you don't want to force these into the path as they're often optional.

Got it? What else?

Headers carry metadata about the request itself, things like authentication tokens, authorization bearer, or specifying the content type being sent, content type, dot, application, adjacent.

And the main data for creating something new.

That usually goes in the request body, especially for post put and patthch requests, where you're sending a representation of the resource.

Okay, so that's input. What about the output? How does the API tell the consumer what happened?

Primarily through HTDP status codes, those familiar three digit.

Numbers like two hundred to ok or four or four not found exactly.

They fall into classes two XX means success. Four x means the client messed up, bad input, not authorized, et cetera. Five xx means something went wrong on the server side. Common success codes include two oh one created when you successfully make a new resource, or two oho four to no content after a successful deletion where there's nothing to return, and errors four hundred to bout amount. Bad request is common for invalid input, four to one, unauthorized, four oh

Makes sense, Let's shift to the data itself. How do you actually define the structure of the information being passed back and forth.

That's where data modeling comes in, usually using Jason's schema within an open API specification. It gives you a formal way to describe the data structure, types, constraints, everything.

And the book suggests a particular approach.

Yeah. It emphasizes the idea of starting with a theore, medical or complete resource model. Think of your product resource to find everything it could possibly contain all the details the maximal view. Right then that complete model becomes the source for deriving other more specific models you'll actually use in the API, like summarize models for when you return a list. You don't need every detail to summary minimal models that might just contain the ID maybe for linking

And PCCh for updates.

Exactly, modification models for PTH, where typically all fields are optional because you're only changing some things. This approach brings a lot of consistency. You're not defining similar structures over and over again.

That leads nicely into data user friendliness. It's not just about structure, but the content of the data, right.

Absolutely, it's about selecting data that's genuinely relevant and helpful to the consumer, not just dumping everything from your database. The idea here is providing supporting data or even process data.

What's an example of that.

The book gives a great banking API example. Instead of just giving the raw count balance and the overdraft limit and forcing the consumer to calculate if a purchase is possible.

You could provide a calculated safe to spend amount exactly that directly answers the user's likely question, reduces the chance of errors on their side, and makes the API much easier to use.

It's about anticipating needs.

That's smart. What else contributes to user friendly data?

Using well known or standard identifiers whenever possible, Like use an IBAND for a bank account if that's the standard. Don't just expose your internal database ID.

Makes sense less look up needed right, and use.

Human readable codes instead of account type one. Use account type checking. It's immediately understandable, and of course, be super consistent with your atomic data types and formats. Always use ISO eighty six to ZHO one for dates and times for example, no surprises.

Okay, And how about organizing the data within a response? Structuring it well?

Good structure is key. Use objects to group related feels together, like instead of overdraft limit and overdraft, use floating around group them in an overdraft facility.

Object keeps things tidy.

Yeah, and use a raise for lists of similar items, like a balance's array if an account can have multiple currency balances. But there's a big caution here. Avoid embedding large lists directly within a main resource response, especially if those lists might need filtering or pagination.

Like putting all of a user's transaction history inside the main user object exactly.

That's usually a bad idea. If you need to get transactions, provide a dedicated operation like users one, two, three transactions that supports pagination and filtering properly. Don't overload the primary resource.

That makes a lot of sense for performance too.

Absolutely, and the golden rule overarting all of this data design is consistency, naming, conventions, data types. How you structure objects in a raise keep it consistent across the entire API. That predictability makes the API dramatically easier and faster for developers to learn and use.

Right, Okay, let's pivot to constraints. Starting with a big one security, How does API designed influence security massively?

Security isn't something you bolt on at the end. It needs to be baked into the design choices you make from the start. Your design is part of your security posture.

So how do you design for security?

A fundamental principle is minimizing the API surface area. Only expose the operations and the data that are absolutely necessary. Really challenge requests for sensitive data. Do we really need to expose the user's exec GPS coordinates or their phone number?

If so, why and if you do need it.

Maybe can transform it, perhaps return a city or region instead of precise coordinates, or maybe split very sensitive operations off into a separate API that has much tighter access controls. Less exposure means less risk.

Speaking of access control, how does that work in API design? You mentioned scopes earlier.

Yes, scopes are a common mechanism for authorization. They define permissions. You might have a red profile scope and a separate update profile scope, or maybe a transfer money scope.

So the API consumer requests certain scopes and the API checks if they have them before allowing an operation.

Exactly, and scopes can be defined that different levels of granularity per operation, per resource type, maybe even based on the specific user role or the application making the call. Depends on your security requirements.

Okay, what about protecting the data itself? Data integrity?

Several design aspects contribute here. First, using HTTP methods correctly get should be safe, no side effects. Put in delete should be idempatant. Running them multiple times as the same effect as running them.

Once prevents accidental double charges or double deletions. If a request is retried precisely.

You can also prevent malicious or accidental request replays by requiring an adempotency key header. For non adempitent operations like PFOST, the server uses this key to ensure it only processes the request once AH and for updates. Condition updates are vital. Imagine two users trying to update the same customer record at the same time.

One might overwrite the other's changes without realizing it right.

Using E tag hitters, which represent the version of the resource and if match hitters, which say only perform this update if the E tag matches what I last saw prevents this lost update problem.

Got it? That seems really important for data consistency. What about handling errors securely?

Yes, secure air handling is crucial. You need to give feedback, but not too much information that could help an attacker. Use standard machine readable error formats like problem details for httpapis. This provides structure type, title, status, detail, instant It.

Avoid generic unhelpful messages like just error definitely.

But also return all validation errors at once if possible. Don't make the consumer fix one thing, resubmit find another error. Fix that. It's incredibly frustrating.

Yeah, that back and forth is terrible. Any other security pitfalls and.

Design one big one. Sensitive data in URLs. Even with HTTPS URLs, paths and query parameters can leak. They might end up in browser history, server logs, referra headers, So.

Putting a password or credit card number in a query parameter is a definite no.

No absolutely for anything sensitive, especially if it's being used for searching or filtering. Strongly, consider using a post request and putting these sensitive data securely in the request body instead.

Good tip. Okay, let's switch from security to another key constraint. Efficiency. Why is this so important?

Efficiency benefits everyone? For consumers, it means faster apps, less battery drain on mobile, lower data usage. For the API provider, it means lower server costs, less strain on infrastructure, better scalability. Small efficiencies can save a lot of money at.

Scale, So how do you design for efficiency? Caching seems like an obvious one.

Cashing is huge. Using standard HTTP caching headers like cash control tells downstream caches, browsers, CDNs proxies how long they can use a response without.

Asking again, saves bandwidth and reduces load massively.

And even better is using conditional requests with ETag and if none match, the client sends the ETag it has for a resource. If the resource hasn't changed on the server, the server just replies with a tiny three to four not modified response without sending the data. Again super efficient.

Nice. What about reducing the end of data sent in the first place?

Several techniques we mentioned Resource model selection earlier let consumers ask for a summarized view if they don't need all the details. Field selection is another, allowing consumers to explicitly request only the fields they need, like dot field's name. Balance, Why send data they're just going to ignore?

Good point.

Also look for redundant data. If the same piece of information appears in many different responses, maybe centralize it in its own resource or operation, and definitely prefer partial updates using PTCH over full replacements with pet when possible, Sending only the changed fields is much more efficient than sending the entire resource representation.

Again right, the book mentioned two types of PTCH.

Yes, Jason merge patch is simpler, basically merging the provided JSON into the existing resource. Jason patch is more powerful, using explicit operations like ad remove, replace targeted at specific parts of the Jason document. It's better for complex updates, especially involving a raise.

Okay, what about handling large lists of items efficiently?

Pagination is essential, don't try to return thousands of items in one go. Break it into pages.

Index based or cursor based.

Both have pros and cons Cursor based is generally more robust against data changing while you're paginating. But the critical design decision here is a default page size. The book warns against huge defaults like ten thousand that can hammer your database and the network. Choose a sensible default, maybe twenty fifty or one hundred, and let consumers request more if they really need it.

Sensible advice any other efficiency boosters.

Bulk operations can be great. Instead of making one hundred separate API calls to create one hundred acas ounce, allow a consumer to send a single request with all one hundred much less network overhead.

But how do you handle errors in bulk operations? If one fails?

That's the key design choice. Is it all or nothing one failure fails the whole batch, or do you process each one individually and return a mixed result. The each TTP two O seven multi status response code is often used for mixed results indicating success or failure for each item in the batch.

Okay, and finally managing usage.

Right rate limiting. You need to control how frequently consumers can call your API to protect your infrastructure from overload, whether accidental or malicious, and crucially be transparent about it. Use standard headers like rate limit limit, rate limit, remaining, rate limit reset, or the older X rate limit versions so consumers know the rules and can adjust their behavior.

Makes sense. Protect yourself, but let users know the limits exactly.

It's about fairness and stability. Now, beyond security and efficiency, good API design has a consider the broader context that operates in What are the specific needs or constraints of this particular situation.

Can you give some examples of these contextual constraints?

Sure? Think about file handling. If you need to upload a file along with some metadata, multipart form data is a standard way or maybe Base sixty four encoding for small files embedded in JSON, though that increases size.

What about really big files, uploading videos or something.

Yeah, for large files, hitting your main API gateway might not be efficient or even possible due to size limits. A common pattern is to use sign URLs your API generates a temporary, secure URL, maybe pointing to cloud storage like S three gives it to the client, and the client uploads or downloads directly from storage by passing your main API infrastructure for the bulk transfer.

Ah, that's clever, offloading the heavy lifting. What else?

Event notification Sometimes the consumer needs to know when something happens immediately without constantly asking the API anything new.

Yet that's pulling right can be inefficient.

Very The alternative is web hoooks. The consumer provides a URL, and your API calls their URL proactively whenever is specific event occurs, like a new order, place to transaction completed. Standard formats like cloud vents help make these web hoook payloads consistent and understandable.

So the information flows the other way when needed. What about tasks that just take a long time.

Long running operations, Yeah, like starting a complex data analysis job or a simulation. The initial API call might return immediately, maybe for the status urrol. Then the client has two options, polling checking that status URL periodically until the job is done, or, if supported, providing a callback URL similar to a webhook, where your API will notify them once the long operation is complete.

Choosing the right pattern depends on the use case, I guess absolutely.

And this idea of choosing the right pattern extends to the type of API itself. Rest Over HTTP is the default for many webapis, and it's great for request response interactions with resources, but it's not the only tool.

When might you choose something else.

If you need to stream data continuously from server to client, maybe e Server sent Events SSE is better for real time two way communication like a chat app. Web sockets might be the answer. If you have mobile apps needing very specific, flexible data combinations to avoid overfetching. Maybe graphic cool makes senses a back end for front.

End layer and for internal micro services.

gRPC is often chosen there for its high performance and strong typing, especially in polyglot environments. The point is understand the need first, then choose the appropriate API style or technology. Don't just default to rest for everything.

That leads to a really important question. APIs aren't static. They need to change. How do you manage modifications and versioning?

This is critical for maintaining stability for your consumers. The first step is understanding the difference between backward compatible changes and breaking changes.

What's the difference?

Backward compatible changes don't break existing clients. Adding a new optional field to a response, for example, or adding a new value to an existing enumeration, Consumers who don't know about the new stuff.

Just ignore it, okay, safe changes right.

Breaking changes, on the other hand, will break existing clients if they're not updated. Renaming a required field, changing a field's data type like from a string to a number, removing an operation entirely, making an optional field required. These all require consumers to.

Adapt, So how do you manage those breaking changes? That's where versioning comes in exactly.

You need a versioning strategy. Semantic versioning like one point oh one point one two point zero is common, where the major version number the two point zero increments only when you introduce braking changes. Minor increments are for backward compatible additions. Some use deep based versioning like twenty twenty five oh eight twenty, which can be simpler if you release very frequently, but makes it harder to know what change without reading talks.

And how is the version usually specified in the API call.

The most common way is path level versioning. You stick the major version right in the URL path like V one products or VW two products. It's simple, explicit, and easy for routing and casing. Other methods exist head query prams, but path is very popular.

To minimize the need for breaking changes, can you design for extensibility from the start?

Absolutely, that's key. Make choices that leave room for future evolution. For instance, always return chase on objects as the top level in your response bodies. Even if you only have one field right now, it's easy to add more fields later without breaking things. Good tip use a raise of objects, not just a rays of simple strings or numbers. If you think those items might need more associated data later. Prefer numerations like status active over simple booleans is active true.

Okay, we've covered the what and how of design? What about the final layer, the design process itself, making it reasoned and improving.

Right, This is about ensuring your design decisions aren't just random or based on gut feeling. You need confident decision making. That confidence comes from relying on trusted sources. Maybe it's your own well defined internal API design guidelines. Maybe it's established external standards like the HTTPIE speck itself, or things like problem details, cloud events, maybe industry specific standards like ISO twenty twenty two in.

Finance, So standing on the shoulders of giants essentially.

Kind of yeah, and importantly document significant decisions, especially when you deviate from a standard or make a tricky trade off. Architectural decision records ADRs are a great way to capture the y behind a choice.

You mentioned API design guidelines. How valuable are those?

Immensely valuable. Think of them as your teams or organization's cookbook for APIs. They provide agreed upon recipes for common patterns, how we do CRI, how we handle searching and filtering, standard file upload methods, how we structure errors. Having these guidelines saves countless hours arguing about the same things over and over again on different projects. It promotes consistency.

Can you automate the enforcement of these guidelines?

Yes, that's where automated guidelines or API line come in. Tools like Spectral are designed for this. You can figure the linter with your rules. Properties must be CamelCase, pagination must use these specific parameters, air responses must follow the problem detail structure, etc. Then the linter analyzes your open API definition file and points out any violations.

Giving feedback directly in the design phase exactly.

It gives clear feedback, errors, warnings, info hints, helping designers stick to the standards and catch potential issues way before coding even starts. It's a huge boost for quality and consistency.

Finally, the open API document itself the description of the API. Any tips for optimizing that definitely.

Make it easy to read and maintain. Use ref Jason pointers heavily to reuse common components like schemas, parameters, standard responses. Don't repeat yourself.

Define it once, reuse it everywhere right, and.

Enrich it with details. Use JSON schema keywords precisely specify min length, max length, pattern regular expressions for validation enum allowed value, use default values. The more precise the definition, the better, and examples Examples are crucial. Include realistic examples for request bodies, response bodies, parameters. They make the API much easier to understand and test for consumers. Don't skimp on examples, and as a final point on process, consider

File so you can interact with it before it's built.

Yeah. Hey, let's you and potential consumers try out the design, see if it feels right, catch awka parts, and iterate much faster than if you wait until the code is written. Wow.

Okay, so we've really journeyed through a lot there. We've taken you, the listener, through these layers, from figuring out what the API needs to do, designing that interface, modeling the beta, carefully building insecurity and efficiency, handling context, managing change with versioning, and finally refining the whole process with guidelines and tools.

Yeah, it really underscores the good API design isn't just a single task. It's an ongoing craft. It's this constant balancing act between what users need, what's technically feasible and constrained, and how things might evolve. It's less about just banging out code and much more about deliberately crafting these interfaces that enable smooth, effective digital experiences for everyone involved.

So here's a final thought to leave you with as you go about your day thinking about the systems you build or use. What seemingly small API design choice being made today might turn out to be a major roadblock, or perhaps a brilliant enabler for your users and systems down the road,