Stop, Catch Your Breath, and Reinvent the Wheel

Since last Thursday, I’ve spent roughly 50–70% of my time building a wheel — a tool called merlin.

Here’s how it started: one of our important internal services needed to be upgraded to Elixir 1.5. The service’s existing Ansible deployment code went something like this — clone the code on the target machine, compile, generate a systemd-compatible release, update the systemd config, restart the service. Now you might say: if a cluster has dozens of machines, building on every single one is slow and laborious, and the odds of something going wrong along the way go up considerably — why not just do the build in a CI tool like Travis, produce a tarball, and have the target machines fetch that tarball and deploy it?

There are historical reasons. Our service does a lot of work at compile time and needs access to resources like the database; since we use a third-party CI tool, we’re not comfortable exposing even read-only DB credentials to it, so we were left with this deployment scheme that really doesn’t suit Erlang/Elixir.

Our production Erlang/Elixir runs on 19.x / 1.4. A rash upgrade would hit live services hard; provisioning a new cluster, deploying, switching traffic, and then killing off the old cluster works, but having to go through that whole ritual for every version bump isn’t great. On top of that, the current build-on-the-target-machine approach is getting heavier and heavier — a single deploy takes several minutes, which is far too inefficient. So I got the itch to build a simple build system.

The idea is straightforward: we’re heavy GitHub users, so tie the whole scenario to GitHub releases. Provide a webhook that handles the new release event — when a new release is created on some repo, the webhook receives the event (containing the repo name, tag, and so on); we do a bit of processing and shove it into AWS SQS. Then a periodic job pulls messages out of SQS, checks out the code according to the repo and tag in the message, builds it, and uploads the resulting tarball as an asset on that release. When everything is done, we delete the message from SQS. That’s the entire build flow. To deploy, you simply tell the deploy script which release you want; we grab the tarball from the release assets, unpack, and deploy.

The benefits: the build system lives in our own VPC, so it can fetch database credentials from vault; and we only need the right Erlang/Elixir version on the build system, then build with ERTS included, so the target machines needn’t care at all about the Erlang VM version.

You might ask why not insert the message into an Erlang-internal queue inside the webhook, instead of bringing in an external SQS. Because the build takes a long time and touches many parts of the system and toolchain, with all sorts of things that can go wrong along the way — if the process handling a message dies, the message goes with the wind. That would cost us the chance to finish that build, which isn’t good. But handling all those edge cases and doing persistence properly would take a fair amount of extra code. SQS guarantees messages aren’t lost: after dequeue, a message is merely hidden, invisible to others for the duration of the visibility timeout, so a failed processing attempt is no big deal — once the visibility timeout expires, it can be processed again; only when everything is finished and we explicitly delete the message does it truly leave the queue. Besides, for this kind of message volume, SQS is practically free. Why not?

This task took nearly a week, produced 650 lines of code and a dozen-odd releases, and it’s already serving three internal Elixir repos in production, running well so far.

Below is the design sketch of the whole system:

Below is a GitHub screenshot of a release right after publishing, before it has been compiled:

And here is a GitHub screenshot after a successful build:

The system is so simple that we only need to write a few APIs with plug, and then have a periodically running GenServer process messages and spawn a process to do the build. No need for much explanation — I’m sure you could write it quickly too.

What I really want to say in this post is this: every so often, you should stop, catch your breath, and build a wheel from scratch. You’ll be surprised by the rewards.

Most of a programmer’s working hours are spent — to put it nicely — making stories, or — to put it bluntly — doing chores. We need those chores to earn a living and support our families, but those day-after-day chores don’t deserve our growth. If you’re forever “doing odd jobs,” even at a 996 pace, you’re still running blindfolded — or, as the saying goes, running on a hamster wheel.

Over the past few months, in Beijing and San Francisco, I’ve interviewed quite a few programmers. Every time I pose a problem and ask the candidate to write a solution in the language they know best, almost everyone — before I’ve even finished speaking — can’t wait to open their editor, or generate skeleton code, or start Googling how some function is used, or write the first line of code. Rarely does anyone take a moment at the whiteboard or on scratch paper to sort out their thinking, clear their head, and sketch out an approach. This is the chore mindset we’ve been passively trained into — every task is like piecework on an assembly line; no one needs to think much, not even ask many questions, just follow the existing behavioral patterns and habits of thought and you’ll get by.

Working at a company, the longer a piece of code has been maintained and the longer it has survived in production, the more cautious everyone becomes. Many reasons intertwine: newcomers don’t quite follow their predecessors’ logic and dare not change anything lightly; a framework that’s been running for years has features piled on features until it resembles Ptolemy’s geocentric model, buckling under its own weight, and nobody has the nerve to tear it down or start over, because in this world you survive on one word — “stable”; your skill stack is locked in place, and you want a breakthrough but have no idea where to start; worse, you’ve been domesticated by the company’s overly luxurious dev environment and toolchain until, like a pampered nobleman of the Eight Banners suddenly dumped into the wilderness, you can no longer ride a horse or run a mile.

Think about it: when was the last time you took a brand-new project from zero to one — from a blueprint, laying down line after line of code, until it was finally deployed to production or into users’ hands? If it’s been more than half a year, then you probably need to stop and catch your breath, and seriously build a wheel from scratch.

You’ll find that where filling in features inside a big project once had you writing code with ease and momentum, now, in a blank project, even choosing and defining your data structures has you scratching your head raw.

Once the basic functionality is in place and it’s time to deploy, you discover that writing an Ansible deployment script from scratch is worlds apart from your usual copy-paste. What should be abstracted into roles, what hooks to set up — reading other people’s scripts always seemed effortless, but when you honestly sit down to write it yourself, you realize that someone else’s code is, after all, someone else’s code.

Setting up CI/CD costs you half your skin, and service monitoring costs the other half. So many things you never had to worry about suddenly become important. “The existing system” is like a prison, walling you off from the wild, free world.

Through the zero-to-one process you tremble and sweat like a fountain. The “Little Prince of Programming at the Center of the Universe” your colleagues crowned you turns out to be wearing nothing but the emperor’s new clothes — one poke and it’s gone.

After all that agonizing, the wheel is finally built; and having learned your lesson the hard way, you pick up the code you had once nearly memorized. Reading it after having written it yourself, those cold, lifeless lines become plump and rosy; the countless question marks in your head start clicking into place one by one. Yes, your job has been deceiving you: the daily chores give you a vague sense of satisfaction, but you simply don’t know what you don’t know.

So, every so often, stop, catch your breath, and corner yourself into building a wheel.

Break, and then build.