Overseer: A Supervisor for Distributed Systems

Recently I found something interesting in otherwise tedious work, and I’ve been pouring energy into it both at work and in my spare time. I had planned to wait until the final results were ready last weekend before writing this post, but too many things intervened and the code still isn’t finished. The first ten days of February have already passed; if I wait any longer, Spring Festival will be over. So this time, Mr. Programmer is getting on the train first and buying the ticket later.

Requirements

Here’s the story: two weeks ago a colleague urged me to upgrade a wheel I’d built earlier, merlin—see last year’s post, “Stop, Take a Breath, and Build a Wheel.”

In that post I explained why we needed such an internal release build tool. Since then, merlin has been shepherding releases for several internal Elixir services for months, and on the whole it has performed well. However, some flaws in its original requirements and design have led to these problems:

1. It depends too heavily on GitHub releases—if no new release is created, no build can be triggered automatically. This is unfriendly to services that rely on staging for integration testing. Users have to bump the version in a pull request (just to produce a build). As a result, multiple developers must stagger their versions across pull requests, which is very inconvenient. During busy development periods, our patch version (the third digit in SemVer) would jump by five or ten in a single day, more terrifying than the meter in a San Francisco taxi.
2. The two machines merlin ran on were both t2.medium, and a single release build (including a release upgrade build) took ten-odd minutes. When builds were busy, requests at the back of the queue waited a long time (bad latency).

So in the next version I want to fix these problems. My initial idea: when a build request arrives, launch a powerful spot instance, let it handle the build, and after the build finishes and uploads to S3, the spot instance terminates itself. Build requests can come from a GitHub release message (backward compatible with the previous version) or from an API—which further lets us build a CLI tool so users can trigger a build for any git commit from the shell. With this rough idea in mind, let’s sort out the requirements:

1. Users (including GitHub) can trigger builds through an API.
2. After a build is triggered, launch a spot instance (or ECS Fargate—but spot instances are simply too cheap, e.g. a C5.large at $0.03/hour, so ECS has no price advantage).
3. The spot instance starts from a prebuilt AMI (or a Docker image, in the ECS case) that contains the build-processing software. Once it boots, it can automatically handle the build task.
4. The metadata describing the build task is placed in the spot instance’s user data at launch, and the build software accesses it via http://169.254.169.254/latest/user-data.
5. During the build, some telemetry may be sent to merlin.
6. When the build completes, send the status and build information (e.g. where the tarball is) back to merlin, then terminate itself.

These requirements are fairly clear, and implementation isn’t especially difficult—it’s just a matter of time. For a startup like ours, this is the kind of work you can roll up your sleeves and start immediately, cutting a road through every mountain and building a bridge over every river.

merlin’s earlier pit was one I’d dug myself, and this business is neither sexy nor urgent. My backend teammates are all heads-down on high-visibility projects whose names alone make your blood boil, so they can’t spare the time—and I’m not willing to waste their time on this anyway. So I have to eat my own dogshit. I’m busy fooling around during the day and lazy at night—unless some idea hits my G-spot and makes me go without food, drink, or sleep, a project like merlin, which you can see through at a glance from head to toe and which offers little challenge, can’t fire up my little universe. So the requirements were settled; since there was no rush anyway, I lazily, intermittently mulled it over in my head.

It turned out that this laziness—rather than going all out—led to more and deeper thinking on my part. When I have time, I’ll write up the whole thought process; I believe it may offer you a little inspiration.

Implementation

Under the requirements above, merlin is split from a single service into two parts: a control plane and a data plane (please forgive a former network engineer’s deep-seated obsession with separating paths). Simply put, the control plane assigns work and monitors it; it’s a scheduler, like a madam. The data plane does the work; it’s a pool of resources, like Su Xiaoxiao, Liu Rushi, and Li Shishi. And each build task is a task to be completed—an unexpected visit from Zhao Ji, Liu Yong, Ruan Yu, and the like.

Let’s generalize merlin’s requirements a little:

1. A caller can trigger a task through an API.
2. After receiving the task, the control plane assigns it to a resource on the data plane for execution.
3. The data plane aggregates telemetry to the control plane.
4. After completing the task, the data plane reports the result to the control plane and enters an idle state to await the next scheduling.

To align with core socialist values, let’s switch metaphors: the control plane is like a Supervisor in Erlang/OTP, while the data plane is like a GenServer. Those less familiar with Erlang can read my post, “And God Said: Let There Be a Language for the Future, and There Was Erlang.” You don’t need to understand the code, but you do need to understand the idea.

However, a Supervisor in Erlang/OTP is only responsible for starting and monitoring processes. If you want to start and monitor nodes, there are many problems:

1. How do you dynamically launch a node in the cloud?
2. How do you make this node automatically join the cluster?
3. How do you give this node the software necessary to run the task?
4. How can the control plane communicate conveniently with the data plane?
5. How do you hide all the details above so that starting and monitoring a node is as simple as a Supervisor starting and monitoring a GenServer—and friendly to programmers?

If 1, 2, and 3 are solved, 4 can be solved directly by wrapping RPC.

We mentioned 2 above—we can solve it by providing UserData to the newly launched instance. In AWS, when you launch a new instance, you can preload some JSON data that can be obtained locally at http://169.254.169.254/latest/user-data. Thus we can put the cluster cookie and the node name of the control plane node in there, so the new node can join the cluster by itself.

Let’s look at 1 and 3. The simplest way to solve 1/3 is to use a prebuilt AMI—bake all the relevant data-plane software into the AMI, and create nodes with the request-spot-instance AWS API. But this means every time the data plane code changes, we have to rebake the AMI. Even if the AMI-baking is automated by CI, the control plane still needs to ensure it launches the data plane with the correct AMI each time. That’s a bit of a hassle.

What programmers hate most is hassle. Humility makes one progress; hassle makes programmers innovate. What to do? Can we build a loader that dynamically loads a compiled module—or even a release—onto a remote node?

Bingo! This is a good question, and a good question is worth far more than a good answer. So on a weekend about two weeks ago, I wrote a few hundred lines of code and produced an initial version of ex_loader. See GitHub: tubitv/ex_loader. The code is open source under the MIT license.

ex_loader lets you do things like this very simply:

{:ok, module} = ExLoader.load_module("hello.beam", :"awesome-node@awesome.io")
:ok = ExLoader.load_release("https://awesome.io/example_complex_app.tar.gz", :"awesome-node@awesome.io")

Even if you don’t understand Elixir code, you can probably guess that the first line loads a local module onto a node in the same cluster named awesome-node@awesome.io; the second line loads an Erlang release hosted on some website onto the same node.

Joe Armstrong once happily recounted at a conference that he would run many empty processes on an Erlang node—processes that did nothing and didn’t know what to do—but when he needed them, he would have those processes load new modules and, presto, turn them into processes with a specific capability. ex_loader takes this a step further: you can spin up empty Erlang nodes, and when needed, have those nodes load the release you want them to run, turning them into servers with a specific function.

ex_loader simplifies the control plane’s job of publishing software to the data plane, and we now have a better solution for 1 and 3. However, we haven’t yet touched on point 5 mentioned above.

This is where Overseer comes in—a new OTP behavior analogous to Supervisor. Let’s first look at how to use Overseer:

local_adapter = {Overseer.Adapters.Local, [prefix: "test_local_"]}

opts = [
  strategy: :simple_one_for_one,
  max_nodes: 10
]

release = {:release, OverseerTest.Utils.get_fixture_path("apps/tarball/example_app.tar.gz")}

MyOverseer.start_link({local_adapter, release, opts}, name: MyOverseer)
MyOverseer.start_child()

Defining an Overseer is simple:

defmodule MyOverseer do
  use Overseer
  require Logger

  def start_link(spec, options) do
    Overseer.start_link(__MODULE__, spec, options)
  end

  def init(_) do
    {:ok, %{}}
  end

  def handle_connected(node, state) do
    Logger.info("node #{node} up: state #{inspect(state)}")
    {:ok, state}
  end

  def handle_disconnected(node, state) do
    Logger.info("node #{node} down: state #{inspect(state)}")
    {:ok, state}
  end

  def handle_telemetry(_data, state) do
    {:ok, state}
  end

  def handle_terminated(_node, state) do
    {:ok, state}
  end

  def handle_event(_event, _node, state) do
    {:ok, state}
  end
end

Let’s roughly explain what Overseer does:

1. start_link: on startup, it accepts some parameters describing the spec of the node we want to launch. Nodes currently support two adapters, local and ec2. I made it an adapter so that more node types (such as ECS) can be supported later. The strategy currently supports only simple_one_for_one, meaning all nodes use the same spec and are created by Overseer when needed.
2. start_child: Overseer can launch a node according to the predefined spec—for example, an EC2 spot instance. After the node boots successfully, its initial code connects to Overseer using the node_name and cookie from UserData. When Overseer sees a node_up message, it creates an internal Labor data structure and sends the release defined in the spec to this labor node to load and run.
3. pairing: Unlike a Supervisor, Overseer must not only listen for node up/down events and make corresponding decisions (such as restarting a new node), but also receive telemetry sent from the node. So the process running Overseer must establish a relationship with some process on the labor node. I call this process pairing, analogous to pairing Bluetooth devices. After start_child succeeds, Overseer sends its own pid to a designated interface on the labor node; the labor node then explicitly sends a pair request to Overseer through that interface, after which the two processes are linked.
4. As a GenServer similar to Supervisor, Overseer hides the details and state machines of labor-node monitoring, exposing only higher-level events that the upper-layer software cares about, such as connected / disconnected / telemetry.

The figure below is a sequence diagram I drew by hand about a week ago. At the time it wasn’t called Overseer yet but GenConnector, though the basic idea is the same:

Overseer’s source code will be released after it’s finished in the next few days. Stay tuned.

With ex_loader and Overseer, what’s left for merlin becomes much simpler: split the codebase into a control plane and a data plane, use Overseer in the control plane, reuse the previous code for the data plane, and after some minor modifications we have a distributed build system that can scale at will.

The best part is that although ex_loader and Overseer were born for merlin, their decent level of abstraction makes them applicable to almost any distributed task-processing architecture with a control plane plus a dynamic data plane. In my earlier thinking I actually went a step further and designed this system as a distributed system with a Fleet / Carrier / Fighter structure: a Carrier is a labor node of the Fleet, and a Fighter is a labor node of the Carrier, analogous to the Imperial fleet in Star Wars. In this blueprint, merlin is just one Carrier in the Fleet (though I probably won’t have time to implement that anytime soon):

All right, enough chatter—I’d better get back to writing code. In advance, happy Spring Festival to all the uncles and aunts, older brothers and sisters, younger brothers and sisters! And to all my friends also in their zodiac year with the “Fuyin” pattern, I wish you a booming Year of the Dog and no offense to Tai Sui! :)