Writing a Proper REST API

(The past two weeks have been frantic on the project front, leaving me no time to write — my apologies to all of you.)

Two weeks ago a round of layoffs at the company dropped several people’s work onto my plate, and a good chunk of it was a series of REST APIs. Their author claimed they were essentially complete. I tested them, and found that while from a functional standpoint these APIs met the explicit requirements of the spec, from the standpoint of actual use they fell far short — all manner of implicit requirements were missing. A REST API is the channel through which a system’s backend talks to its frontend (or third parties); it connects what comes before with what follows, and it carries a great many implicit requirements — consistency with the RFCs in the interface contract, both internal and external security, and so on. It is not as simple as exposing a few endpoints and returning the appropriate JSON data. When I studied the original author’s code carefully, I found far too much missing, and each API was essentially fighting its own private war. Rather than patch them, it was better to rewrite (and not out of any contempt of programmer for programmer). So I spent an entire week rewriting every API. I’ve jotted down a few lessons, and in this article I’ll talk about how to write a “proper” REST API.

RFC Conformance

A REST API is generally used to expose some kind of resource, and the operations permitted on that resource, to the outside world, so that callers can operate on the resource in the correct manner. Here, both the handling of input and output must comply with the HTTP/1.1 RFCs (and, before long, HTTP/2.0), guaranteeing the consistency of the interface. What follows focuses on the input method/headers and the output status code.

Methods

HTTP provides a great many methods for manipulating data:

• GET: Retrieve a resource. GET should be idempotent, and free of side effects.
• POST: Create a new resource.
• PUT: Replace an existing resource. PUT has side effects, but it should be idempotent.
• PATCH (RFC5789): Modify an existing resource.
• DELETE: Delete a resource. DELETE has side effects, but is likewise idempotent.

Idempotence is defined in HTTP/1.1 as follows:

Methods can also have the property of “idempotence” in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.

Nowadays few people writing REST APIs…

Put simply, if an operation is idempotent, then given the same data and parameters, executing it once or many times produces the same effect (side effects).

Most REST frameworks today support the HTTP methods correctly; some older frameworks may not support PATCH, so take care. If you’re hand-writing a REST API, be sure to distinguish the scenarios appropriate to POST/PUT/PATCH/DELETE.

Headers

One fairly serious mistake many REST APIs make is paying little attention to the request headers. For a REST API, certain HTTP headers matter a great deal:

• Accept: what content the server should return. If the client asks for "application/xml" and the server can only return "application/json", it’s best to return status code 406 not acceptable (RFC2616) — though returning application/json does not strictly violate the RFC either. A proper REST API must flexibly return the appropriate data according to the Accept header.
• If-Modified-Since/If-None-Match: if the client supplies a condition, only return data when that condition is met; otherwise return 304 not modified. For instance, if a client has already cached some data and merely wants to check for something new, it will use one of these two headers. If the server ignores them and goes through the full routine, returning 200 ok, that is neither professional nor efficient.
• If-Match: when performing PUT/PATCH/DELETE on a resource, the server should require the client to supply an If-Match header, and only perform the operation if the Etag supplied by the client matches the Etag of the corresponding resource on the server; otherwise return 412 precondition failed. This header is extremely important, as explained in detail below.

Status Code

Another mistake many REST APIs make is this: when returning data they don’t follow the status codes defined in the RFC, but instead return a blanket 200 ok plus an error message. When both client and API belong to the same company, that’s barely workable — but the moment the API is exposed to third parties, it not only invites ridicule but leaves behind a great many interoperability hazards.

The above are only the most basic considerations. To comply fully with the RFCs, beyond consulting the RFCs themselves, Erlang’s webmachine or Clojure’s liberator are both fine implementations — currently among the few libraries/frameworks that do REST APIs right.

Security

As I said earlier, a REST API sits between what comes before and what follows; it is the interface the system exposes to the outside world, so its security matters enormously. Security does not simply mean encryption and decryption — it means integrity, confidentiality, and availability.

Request Data Validation

Let’s start from the first step of data flowing into the REST API — validation of the request data — to guarantee security. You can think of request validation as a giant funnel, filtering out all unnecessary access at the very front line:

• Whether the request headers are legal: if some header that shouldn’t be there appears, or a required header is missing or has illegal content, return 4xx according to the type of error. For example, if your API requires some special private header (e.g. X-Request-ID), then reject any request lacking that header. This can fend off all manner of aimless webots and crawlers and save the server overhead.
• Whether the request URI and request body are legal: if the request carries data that shouldn’t be there, or required data is missing or has illegal content, return 4xx across the board. For example, if the API only permits query in the querystring, then a request like "?sort=desc" should be rejected outright. Quite a few attacks make trouble inside the querystring and the request body; the best counter-strategy is to filter out every request containing data that shouldn’t appear.

Data Integrity Verification

REST APIs often need to modify backend data. Modification is a fearsome operation: we must ensure that legitimate service requests are handled correctly, and also guard against every potential attack, such as replay. The bottom line of data-integrity verification is this: guarantee that the data to be modified is identical to the data on the server — and this is accomplished through Etag.

An Etag can be thought of as the unique version number of a resource. When a client requests a resource, that resource’s Etag is returned along with it; when the client needs to modify that resource, it must supply this Etag via the "If-Match" header. The server checks whether the Etag supplied by the client matches the Etag of the same resource on the server, and only modifies if they match; otherwise it returns 412 precondition failed.

Using Etag prevents erroneous updates. For example, A obtains Etag X1 of Resource X, and B also obtains Etag X1 of Resource X. B modifies X, and the system generates a new Etag, X2. Now A wants to update X too; because A holds the old Etag, the server refuses the update until A re-fetches X and can update normally.

Etag is like a lock — the single most important safeguard of data integrity. Etag can strangle the vast majority of integrity problems in the cradle; of course, race conditions still exist: if B’s modification has not yet reached the database and A’s modification request happens to pass the Etag check, consistency problems remain. This calls for a pre-write consistency check at the database write layer.

Access Control

A REST API needs a clear definition of which operations can be accessed publicly and which require authorization. Generally speaking, if the security requirements for a REST API are high, then every operation of every API requires authorization.

There are many ways to handle authorization on top of HTTP — HTTP BASIC Auth, OAuth, HMAC Auth, and so on — whose core idea is all the same: verify that a given request was initiated by a legitimate requester. Basic Auth exposes the user’s password over the network, so it is not the most secure solution; the core of OAuth is much like HMAC Auth, except it adds a great deal of token-distribution machinery. Here I’ll focus on the idea behind HMAC Auth.

Back to the three attributes of security: integrity, confidentiality, and availability. HMAC Auth guarantees integrity: the requested data has not been altered in transit, and therefore can be safely used to verify the legitimacy of the request.

HMAC mainly uses two fields in the request headers: Authorization and Date (or X-Auth-Timestamp). The content of the Authorization field is split into two parts by a “:” — before the “:” is the access-key, after it is the HMAC value of the HTTP request. At API authorization time, an access-key and access-secret are typically generated for the caller; the former may be exposed on the network, the latter must be kept safely. When the client calls the API, it uses its own access-secret to compute the HMAC over the request’s headers/body as required, then fills its access-key and HMAC into the Authorization header. The server takes this header, retrieves the secret corresponding to the access-key from the database (or cache), computes the HMAC in the same way, and if it matches the one in the Authorization header, the request is legitimate and has not been tampered with; otherwise it is illegitimate.

GET /photos/puppy.jpg HTTP/1.1
Host: johnsmith.s3.amazonaws.com
Date: Mon, 26 Mar 2007 19:37:58 +0000

Authorization: AWS AKIAIOSFODNN7EXAMPLE:frJIUN8DYpKDtOLCwo//yllqDzg=

When computing the HMAC, headers from the request such as the request method, request URI, and Date/X-Auth-Timestamp are included in the HMAC. Including the timestamp in the HMAC has the benefit of preventing replay attacks. Normally the UTC time difference between client and server is very small, so if the timestamp carried by a request and the server’s timestamp when the request arrives differ too much — exceeding some threshold (say 120s) — it can be deemed a replay, and the server proactively discards the request.

Using HMAC can largely prevent DOS attacks — invalid requests are discarded at the HMAC verification stage, protecting the server’s compute resources to the greatest extent.

HTTPS

Although HMAC Auth is very secure in guaranteeing request integrity and can be used to authenticate whether a request was initiated by a legitimate requester, the request data and the server’s response are still transmitted in plaintext, which is not a sufficient level of security for certain demanding APIs. In that case, HTTPS must be deployed — adding another layer of protection on top.

Other Considerations

Once you have achieved interface consistency (RFC conformance) and security, a REST API can be considered proper. Of course, a well-implemented REST API should also have the following:

• rate limiting: restrictions on access.
• metrics: the server should collect the access time, arrival time, processing time, and latency of every request, so as to understand API performance and the distribution of client access, and thereby better optimize performance and respond to sudden bursts of requests.
• docs: rich interface documentation — the API’s callers need detailed documentation to call it correctly; this can be achieved with swagger.
• hooks/event propagation: other systems should be able to integrate with the API conveniently. For example, after some resource is added, expose a message to the outside world via kafka or rabbitMQ, so that the corresponding subscribers can perform whatever processing is necessary. Note, however, that hooks/event propagation can break the idempotence of the REST API, so use them with care.

Some relatively mature REST API frameworks/libraries in various communities:

• Python: django-rest-framework (django), eve (flask). Each has its strengths. A pity that Python has no good implementation comparable to webmachine.
• Erlang/Elixir: webmachine/ewebmachine.
• Ruby: webmachine-ruby.
• Clojure: liberator.

I haven’t had much contact with other languages, so I won’t introduce them. You can visit the corresponding awesome repo for a language on github (google “awesome XXX”, e.g. awesome python) and look at the REST API section.