Abstraction Revisited

In 2016 I wrote an essay titled ‘The Ability to Abstract’; those who haven’t read it may want to take a look. That essay served as an introduction and offered quite a few real-world examples of abstracting code and requirements. Pay particular attention to the last example—the abstraction of the feed parser. A feed is the list of short videos that Hollywood studios provide for us to play. Every studio has its own feed: broadly similar in structure, but with no small differences in detail. In format, some are XML and some JSON. In content, a title may be under the key title or name; a video URL may sit at the top level or be nested inside. When I took over, the project consisted of dozens of PHP files, each handling one feed. Given all the differences in feed details, I can well understand my predecessor’s approach—such differences are not easy to abstract at the code level, and besides, the requirements for handling feeds arrived one by one at different times. Under pressure, engineers often can only focus on the immediate. Unlike the young masters and ladies who come from product backgrounds, we are already doing our utmost just to stay alive and finish the requirements.

Recently I have been upgrading a not-so-urgent project (Merlin; see last year’s essay, ‘Stop, Take a Breath, and Build a Wheel’). Thinking about its requirements brought that essay back to mind. More and more I feel that while the relentless pursuit of abstraction in code—abstraction at the level of implementation details—is certainly very important, abstraction of requirements is the real king. The former is technique; the latter is the Way. For the relationship between the two, you might draw an analogy to the Agile Manifesto.

With the same requirement, some people see a single star; others see the vast Milky Way behind it. The same programmer’s implementation of these two views will be worlds apart—not only is the code clearer and applicable to a larger problem space, but writing it is also far more delightful. Take feed handling: grinding out one parser after another is mind-numbingly repetitive work, whereas abstracting it into a ‘language’ that describes how to extract the needed information from a feed—and then implementing the interpretation and execution of that ‘language’—is the kind of requirement that, just thinking about it, feels like a spring breeze and fills you with fighting spirit.

Today we continue this discussion of abstraction and explore how to better abstract problems at the requirements level. I will first walk through my thought process over the past week regarding the Merlin upgrade, then abstract these mental activities and summarize them.

The Thinking Process Behind Upgrading Merlin

Merlin is an internal release build tool at Tubi TV. In the ‘Building a Wheel’ essay I already explained why this requirement arose. Since then, Merlin has escorted the releases of several internal Elixir services for a few months and has generally performed well. However, some flaws in the original requirements and design have led to these problems:

1. It relies too heavily on GitHub releases—if no new release is generated, no automatic build can happen. This is unfriendly to services that depend on staging for integration testing. Users have to bump the version in a pull request (just to generate a build). As a result, multiple developers have to stagger their versions across pull requests, which is very inconvenient. During periods of frequent development, our patch version (the third digit in SemVer) would jump by five or ten in a day—more terrifying than the meter on a San Francisco taxi.
2. The two machines Merlin uses are both t2.medium, and a single release build (which also includes a release upgrade build) takes about ten minutes. When builds are busy, requests at the back of the queue wait a long time before being reached—bad for latency.

So I want to solve these problems in the next version. My initial idea was: when a build request arrives, launch a powerful spot instance, process the build task, and after the build completes and uploads to S3, the spot instance kills itself. Build requests could come from a GitHub release message (backward-compatible with the previous version) or from an API—and thus we could make a CLI tool that lets users trigger a build for any git commit from the shell. With this rough idea in hand, let us sort out the requirements:

1. Users (including GitHub) can trigger builds via API
2. After a build is triggered, launch a spot instance (or ECS Fargate—but spot instances are just too cheap, e.g., C5.large at $0.03/hour, so ECS has no price advantage)
3. The spot instance starts from a prebuilt AMI (or Docker if ECS) containing the build-handling software. Once started, it can automatically process the build task
4. The metadata describing the build task is placed in the spot instance’s user data at launch, and the build software accesses it via http://169.254.169.254/latest/user-data
5. During the build, some telemetry may be sent to Merlin
6. When the build completes, send the status and build information (such as where the tarball is) back to Merlin, then kill itself

These requirements are fairly clear, and there are no real implementation difficulties—it is merely a matter of time. For a startup like ours, this is the kind of work where you can immediately roll up your sleeves, cut through mountains to open a road, and build bridges when you reach water.

The hole that Merlin fell into earlier was one I dug myself. This business is neither sexy nor urgent; the backend teammates are all throwing themselves at high-visibility projects whose very names set your blood racing, so they have no spare hands—and I wouldn’t have the heart to waste their time anyway. So I could only eat my own dogshit. I fritter away my days and hide in laziness at night—unless there is some creative idea that hits my G-spot and makes me go without food, drink, or sleep, a project like Merlin, which can be seen from head to toe at a glance and offers little challenge, cannot fire up my little universe. So once the requirements were set, since there was no rush anyway, I lazily, off and on, turned them over in my mind.

Facts proved that this laziness—rather than wholehearted effort—led to more and deeper thinking and kept shifting the problem that the Merlin project needed to solve first.

In the above requirements, Merlin is split from a single service into two parts: a control plane and a data plane (forgive a former network engineer’s bone-deep obsession with separating paths). Simply put, the control plane assigns work and is a scheduler, rather like a madam; the data plane does the work and is a pile of resources, like Su Xiaoxiao, Liu Rushi, and Li Shishi. Each build task to be completed is a client—Zhao Ji, Liu Yong, Ruan Yu, and the like—arriving unannounced.

Let us generalize Merlin’s requirements a little:

1. A caller can trigger a task via API
2. After receiving the task, the control plane assigns it to a resource on the data plane for execution
3. The data plane aggregates telemetry to the control plane
4. After the data plane completes the task, it reports the result to the control plane and enters an idle state, waiting for the next scheduling

Thus Merlin becomes one particular implementation of this generalized requirement. In the essay ‘Naming in Code: A Monk Knocks at the Moonlit Gate,’ I discussed the brilliance of names such as pipe, stream, and flow, which draw analogies from everyday life. So let us also give concrete images to the parts of this generalized requirement. The metaphor above is apt but hard to elaborate; let us switch to another:

In this design, I call the scheduler a Carrier (an aircraft carrier), a resource a Fighter (a carrier-based fighter jet), and a task a Mission; several Carriers form a Fleet. Below are some casual thoughts about Fleet that came to me while riding a train:

and further thoughts around this idea:

The requirements began with Merlin and finally arrived at Fleet. Merlin turns from a standalone system into an application under a grand architecture called Fleet. This architecture provides the foundational components for running control plane and data plane code, so applications with requirements like Merlin’s need only focus on business logic and need not worry about how the control plane and data plane are organized. As for Fleet’s concrete implementation, I will describe it in a separate essay.

Now that we have a rather rough set of Fleet requirements, let us plug in Merlin’s day-to-day operations and, following the sequence diagram above, dry-run in our minds the concrete process of how the system runs:

1. On the Fleet UI, load the carrier Merlin. Merlin needs two sets of code: one is the release tarball for the carrier, the other the release tarball for the worker.
2. Fleet starts an Erlang node on the local machine (or remotely) and deploys the Merlin carrier tarball onto it. The local Fleet node and the Merlin carrier node form a cluster.
3. The Fleet API automatically supports the new endpoint: htts://fleet.tubi.tv/carriers/merlin/missions. Requests sent to this endpoint are RPC’d to the Merlin carrier node.
4. A new mission is sent to this endpoint. Upon receiving the request, the Merlin carrier launches a new spot instance, eagle-1, using the fighter tarball and instance type supplied at creation time.
5. The AMI of eagle-1 contains the simplest bootstrap code, which starts an Erlang node and then, using information such as the cookie and carrier node in the user data, connects to the Merlin carrier.
6. The Merlin carrier deploys the fighter tarball to eagle-1 and then starts eagle-1’s mission.

(Fleet UI)

(Fleet UI)

Everything seems plausible.

Next we notice that the processing flows of Fleet, Carrier, and Fighter share some common requirements:

1. Fleet needs to start a Carrier, and a Carrier also needs to start a Fighter.
2. Fleet needs to dynamically load code onto a Carrier, and a Carrier also needs to dynamically load code onto a Fighter.

These two requirements already had rough sketches in the ‘Fleet basic requirement’ diagram, corresponding to two libraries that need to be split out and handled separately: fleet-connector and fleet-loader.

As the names suggest, fleet-connector is responsible for starting a new node, letting the new node join the cluster, and having it perform some work; fleet-loader is responsible for dynamically loading the execution code we need into a node already connected to the cluster.

Thinking further, fleet-loader itself has no direct relationship with or dependency on Fleet. In a broad sense, it can be redefined as this requirement: node A loads some local BEAM, app, or even release into a remote node B in the same cluster, ensuring correct execution. Fleet merely uses fleet-loader’s functionality. Therefore, we can decouple fleet-loader into an independent package unrelated to Fleet, called ex_loader.

fleet-connector has no dependency on Fleet either. Based on the description above, we can compare its functionality to that of a Supervisor:

• A Supervisor starts a new process according to a preset MFA (Module, Function, Args); fleet-connector starts a new node according to a preset ITA (Image, Task, Args).
• A Supervisor links the supervisor process and the worker process; if the worker process dies, the supervisor process restarts it with the same MFA. fleet-connector links the carrier node and the fighter node; if the fighter node dies, the carrier node restarts it with the same ITA.
• A Supervisor supports several restart strategies—one-for-one, rest-for-one, one-for-all, and simple-one-for-one; fleet-connector currently considers only simple-one-for-one, but it is not hard to extend.

So we ought to extract fleet-connector into a standalone package that benefits other applications. I call it ex_connector.

ex_loader’s functionality is straightforward; we can even define its interface:

• ExLoader.load_beam(beam_file, node): load a BEAM file into a remote node.
• ExLoader.load_apps(tarball, apps, node): load one (or several) apps from a release into a remote node.
• ExLoader.load_release(tarball, node): load an entire release into a remote node.

ex_connector is not so simple. It involves both the carrier (master) and fighter (slave) sides, and many events are asynchronous, so it is hard to describe with a single library. We know that in Erlang, to encapsulate such behavior, one generally uses a member of OTP’s GenServer family; therefore it should not be called ex_connector but gen_connector—or perhaps gen_executor would be even more precise. As a gen server, its interaction with the user module is as follows:

OK—the requirements have been generalized, reorganized, decomposed, and finally abstracted to this degree, which is about enough. At least we have obtained ex_loader and gen_connector, entirely unrelated to the project’s original intent. No matter how Merlin’s requirements change or how much further Fleet is generalized, work on these two components can already begin.

Let us summarize a little: what did a week of laziness and procrastination produce? We originally wanted to upgrade Merlin to make builds faster (reduce latency) and fix some limitations; in the end, without writing a single line of code, we turned a system into a library, a gen server, a framework, and an app:

• lib: ex_loader
• gen server: gen_connector
• framework: fleet
• app: merlin

Setting aside how much long-term value this whole stack will yield, from the perspective of abstraction alone we have already gained a great deal. Originally, these abstractions should have emerged during the work on Merlin and other systems, practiced step by step. For example, when building another task system similar to Merlin, we might discover the same requirement and, out of DRY (Don’t Repeat Yourself) considerations, split Merlin’s code and extract the functionality related to gen_connector into a separate organization. Then perhaps Fleet would be abstracted. Whether ex_loader would be extracted is entirely a matter of luck. I originally considered putting the entire data plane release in the AMI, which would not be improper either—it is just that whenever the data plane changes, the AMI needs to be rebuilt. With that design, ex_loader would be completely unnecessary and would never be created. However, because they were conceived in advance, many possible detours in between are avoided.

At the same time, generalizing, reorganizing, and decomposing requirements at the requirements stage—turning the abstraction process from bottom-up into top-down—can greatly improve architecture and design and make the system better adapt to future requirement changes. Splitting ex_loader / gen_connector / fleet out of Merlin and making them the foundation for similar projects is, in the short term, a negative return for Merlin itself; but for the project’s future, not necessarily. Moreover, these three have additional benefits—if implemented properly, as expected, they can greatly reduce the effort engineers spend developing and deploying applications that separate control plane and data plane in a distributed environment and whose data plane is highly scalable. It is like Ethereum opening Pandora’s box of ICOs: when something complex becomes simple enough, it can shatter the existing ecosystem.

Abstracting the Abstraction Process

I have rambled on at such length above in order to record my entire thought process exactly as it was. I believe that the way we think and the process of thinking are far more important than the conclusions reached after thinking.

To summarize, here is what we need to do when abstracting at the requirements level:

1. Keep refining and deepening your understanding of the requirements; understand the why, and ask why not—rather than rolling up your sleeves to design and code once things are roughly clear. At this stage, thinking in your head and dragging your feet in action is beneficial and harmless. This is roughly what Rich Hickey means by hammock-driven development (I suggest you bring your own ladder (VPN) and go watch it carefully on YouTube).
2. Generalize the requirements and see how many common parts you can find. Look for similar scenarios in life and work, map them in, and see whether they hold together. Analogy is indispensable here; it helps us transfer experience. The ability to draw analogies is something that must be trained continuously in daily life. You will notice that my essays always like to put similar things together for comparison. Once this becomes a habit, you will unconsciously connect new things with things you know well, looking for similarities and differences.
3. Resist the urge to roll up your sleeves and start coding. Before you begin, imagine yourself as an event (or trigger) in the requirement scenario and dry-run the entire processing flow in your mind, reasoning through it step by step. If your brain is not enough, draw it out (make sure paper and pen, or a pad and pencil, are always at hand)—yes, draw by hand, not with tools, unless the tool itself is so simple that it will not interfere with your brain’s flow. See how I reason and present evidence: I have contributed most of my manuscripts from this week.
4. Think of several alternative scenarios.
5. Don’t use large blocks of time; don’t think during serious working hours—the more earnest you are, the less likely you are to hit the point. Think while running; while in the bathroom, after reading Programmer’s Life, you can also think; when work tires you, think; spend half an hour before bed thinking. Yes, before bed. Don’t fear insomnia—if you can’t sleep, get up and think; when sleepy, go back to sleep. We must believe that if you never forget something, there will eventually be an echo—the whole point is to activate those idle asynchronous threads in the brain so that, besides smoking, drinking, and getting perms, they can do something useful.
6. The span of the whole thinking process should be long rather than short. Too short, and it is not conducive to laying down scattered dots, let alone what Steve Jobs called connecting the dots; too long, and it does not really matter—after all, the virtuous and multi-talented Mr. Gao (Donald Knuth) spent ten years writing TeX—provided, of course, that you (or your boss) can be philosophical about it. One or two weeks is generally enough.
7. Document your thoughts, ideas, and drawings, removing the chaff and keeping the essence in the process.

Thank you for your patience in reading this far.