The Foolproofing of the Backend?

Two weeks ago, RethinkDB finally officially released Horizon, a backend application built on RethinkDB: it lets you operate on RethinkDB directly from the frontend after you’ve done just some basic security configuration and validation.

Does that sound familiar? It’s also, more or less, like the minimongo that Meteor uses, only implemented somewhat differently: minimongo is a subset of the MongoDB API; whereas Horizon’s database API is not only a subset of ReQL (RethinkDB’s query language, analogous to SQL), but the API interface itself has been completely rewritten to give better control over how the frontend operates on the database.

There’s quite a lot packed into that paragraph, so before reading on I’d suggest:

• If you’ve never heard of Meteor, go take a look at its website, generate a few of the sample projects, and run them to get a feel for it;
• If you’ve never heard of RethinkDB, head to its official site to learn how it differs from MongoDB, then brew install rethinkdb, log into the web admin, and get a taste of ReQL.

A few more words on RethinkDB. One of RethinkDB’s standout features is the changefeed. It can publish changes to the result set of a given query in the database for others to subscribe to. This feature is extremely useful for realtime collaboration apps. Take an online project management system: if user A modifies some project x, and we want the UI of every user associated with project x to update in real time immediately, how do we do it?

With a traditional solution, we need the client and server to maintain a websocket connection; once A’s modification succeeds on the server, we publish a message to a message queue and route it to an appropriate worker for processing; the worker takes the message from the queue, broadcasts it to all relevant websockets, and then the client’s websocket, upon receiving the data, dispatches it to the appropriate function to update the UI.

RethinkDB, at the database level, packages all the backend parts of that scheme into the changefeed. You simply define which query’s changefeed you’re interested in, and when data matching that query is modified (created/deleted), the changefeed keeps pushing out the changes for your use.

With changefeed, the backend logic for realtime features suddenly becomes much simpler and can cut out many intermediate steps — don’t underestimate just adding a message queue and a corresponding worker; turning that into a product means a substantial amount of effort and investment. Although using changefeed can’t replace the traditional message-queue-based solution — especially in high-concurrency scenarios at scale (like Slack), where changefeed’s performance hits serious bottlenecks — for an MVP, or a small product in rapid growth, it’s an elegant solution that’s efficient for engineers.

Back to Horizon itself. On top of RethinkDB, Horizon further encapsulates all CRUD operations on data, whether basic or complex, so that a product can run with almost no backend intervention and support its first hundreds of thousands of users. Horizon isn’t the first to do this — Parse, acquired by Facebook, and Firebase, acquired by Google, took the same approach; and among open source projects there’s also postgREST, a tool that, through clever design, exposes the database as an API for the frontend to call.

This trend that has gradually taken shape over the past two years is: the foolproofing of the backend.

To understand why everyone is so keen on “foolproofing the backend,” let’s look at what functions a product mainly needs in its early stages:

• authentication: authenticating and verifying user identity (not strictly necessary)
• Storing, retrieving, and processing user data
• Displaying content
• Storing, retrieving, and processing content
• It would be ideal if everything above could happen in real time (not strictly necessary)

“Content” here means all the data related to the product that needs to be shown to users. For example, pieces of text, images, videos, or individual SKUs — all fall within the scope of content.

Displaying content is user-facing — the part of the product exposed to users; while storing, retrieving, and processing content is content-team-facing — the part beneath the waterline of the iceberg. A CMS (Content Management System), for example.

Setting aside whether the product is realtime, just implementing these functions requires the backend to have a set of APIs handling everything data-related, including login, while also taking care of server maintenance; and the frontend needs at least a user-facing app (like a mobile app) plus a CMS UI for the content team. If realtime is required, the difficulty increases further.

If you’ve read my earlier Rest API series, you know that building a good API is no easy task. And yet the API, as the author of postgREST put it, involves far too much repetitive work — you’re endlessly reinventing the wheel, or reassembling the car: data validation, user authentication, user authorization, data persistence, and the API’s own CRUD functionality.

For a startup in the MVP stage, it’s very hard to do all of these well at the same time. Most people’s choice is: cobble together a so-so backend, then concentrate your firepower on the user-visible frontend product. It’s an unavoidable compromise cursed by the Mythical Man-Month — it lets the MVP reach users as quickly as possible and lets the market test its mettle — but at the same time it plants a serious hidden danger for future development: the parts you cobbled together earlier will later cost several times the effort to refactor or rewrite.

This is the significance of foolproof-backend products: they help you build a usable, secure, sufficiently scalable backend system at nearly zero labor cost. A product only needs to integrate its SDK, make the right settings, and call a few ready-made APIs, and then it can largely ignore backend concerns.

This is a trend, and I believe more and more tools will emerge. It’s a pity that an excellent product like Firebase can’t be used in China (or can be used but nobody dares to — what if one day it just can’t connect?), so we can only rely on tools like Horizon, Meteor, and the now-open-sourced Parse.

Applied to building an early-stage technical team, this trend means a CTO who knows a bit of everything, paired with excellent frontend engineers for the platforms the product covers.

Yeah, that’s about it. Of course, backend engineers are still important, but their bigger stage is in somewhat larger teams that have found product-market fit. At that point, the product gradually needs: 1) more complex APIs and backend processing capabilities (not just simple database handling); 2) data tracking and analysis; 3) A/B testing; 4) personalization — so you need to start stockpiling more backend engineers.

The product team can now focus single-mindedly on polishing the product itself,

foldr visitor pattern

Let’s look at the main tasks a backend engineer handles day to day: 1) server maintenance (various application servers, database servers, etc.); 2) providing API support for the frontend’s various operations; 3) data statistics and analysis (including log analysis). Among these, 1) and 3) have all kinds of ready-made tools to handle them and don’t require engineers to write much code; while 2) is where the backend engineer

If we use Meteor, we need the client to subscribe to updates of the project’s collection. Everything else, the Meteor framework will take care of for you.

It’s somewhat like Parse, which Facebook has already killed off, and somewhat like Firebase, which shone brightly at Google IO. If cloud-based closed-source products (Parse was open-sourced after being killed off) aren’t comparable, then:

awesomeQuery.changefeed().subscribe(
  data => socket.send(transform(data)), // process the data, then publish it to the client
  err => console.log(err)
)