How to Move with Ease Across Dozens of Repos

Sindre Sorhus is a programmer I’ve followed and admired for a long time. His day-to-day on GitHub looks like this:

For a long time I’ve wondered: what kind of productivity allows one person to work across dozens of repos in a single month and produce hundreds of commits? A fanatical passion for programming? A mastery of code that makes everything effortless? Or an extreme pursuit of automation?

Over the past three months I’ve been swimming through a crowd of repos built from elixir, ansible, terraform, javascript, a little python, a little erlang, and I’ve gradually come to grasp this way of working efficiently: build a working model for yourself (and your team), then gradually crystallize it into habit. Once the habit takes hold, everything is like the old oil seller who could pour oil through the square hole of a coin — “nothing but practice.” Of course, practice making perfect is only one link in the feedback chain; we also need to distill that honed skill, through reflection, into wisdom, and feed it back into the working model, so that “learning” and “asking” truly fuse into “scholarship.”

What follows are a few humble lessons of my own, offered to draw out better ones.

Push automation to the extreme

I’ve already given plenty of examples in Code is Law of using tools to automate process, so I won’t repeat them here. Automation is not a slogan, not a mere process, and shouldn’t even be just a way of doing things — it should be a conviction deep in the bone: all our effort exists to reduce our future effort. The internet revolution that has surged over the past twenty or thirty years, tracing back to the IT revolution that has thrived for fifty or sixty, and even back to the first industrial revolution — most of the effort in humanity’s development has gone into automating everything that can be automated.

Software automation has been evolving for a long time, yet in most organizations it’s still in its infancy. We can examine an organization’s automation capability from these angles:

• TTT (Time to commiT): How long does a feature take from the first commit to completion?
• TTDP (Time to DePendency): How long does a feature take from dependency resolve to completion?
• TTC (Time to Compile): How long does a feature take from compile to completion?
• TTCI (Time to CI): How long does a feature take from the start of CI to its end?
• TTPR (Time to PR): How long does a feature take from commit to a created PR?
• TTR (Time to Release): How long does a feature take from the end of review to a released build?
• TTD (Time to Deploy): How long does a feature take from the start of deploy to its end?

At ArcBlock, taking elixir — currently the language with the largest share of our projects — the numbers are:

• TTT: Since we enforce a pre-commit hook, every commit runs the full compile/lint/format/test cycle — about 10s on average.
• TTDP: Starting from zero, resolving / compiling dependencies takes about 5 minutes on average.
• TTC: Same order of magnitude as TTT, about 15s on average.
• TTCI: From zero, doing deps / compile / lint / format / test — about 6 minutes.
• TTPR: Same order of magnitude as TTT, about 15s (version and changelog are updated automatically).
• TTR: From zero, doing deps / compile / lint / format / test / build staging & prod release / upload github release — about 10 minutes.
• TTD: Using ansible to fetch the asset from the github release, then rolling upgrade to production — about 45s on average.

We track these numbers because they are intimately tied to our efficiency. If this automated pipeline isn’t ruthlessly efficient, every minute wasted on it gets multiplied by the number of engineers and fed right back to us. Worse, enough quantity turns into a change of quality, setting off a chain of bad consequences.

For example, from the numbers above we found that TTDP / TTCI / TTR were all too long, and the root cause in each case was a slow dependency resolve. So we put all shared dependencies into a single mix-deps repo and used CI to resolve and compile them for every environment; other projects then, once their CI starts, first pull down the pre-compiled results and only then run their normal pipeline. As a result, TTDP / TTCI / TTR dropped to 2 minutes / 3 minutes / 6 minutes on average. That means if we did releases non-stop, a single Travis concurrent job that could once do 144 releases a day can now do 240 — a 60% improvement.

What good does it do to shorten each step? It drastically cuts a programmer’s time to wait, and thereby indirectly lays the foundation for getting more done in a day.

Note that throughout this pipeline we want problems to surface as early as possible — for instance, introducing a commit hook at local git commit time that does most of what CI does. On the surface it slows you down; in substance it greatly improves everything downstream, and thus indirectly raises overall throughput.

Automating the workflow, and endlessly optimizing it, is an important part of automation — but not all of it. We must never overlook: code automation.

When you define an API, or a family of APIs, the API schema has to be written once; the corresponding DB schema (if the API reads all or part of its data from the DB) has to be written again — the two are maybe 70% identical; then the API docs have to be written yet again, essentially a repeat-plus-extension of the earlier schema. In unit tests, tests have to be written, which to some degree are a rehash of the examples in the API docs; then, when defining end-to-end tests (with runscope, say), almost the same thing gets defined once more, only with a different testing approach than unit tests (this time no mocking — the play is performed for real). As the API keeps iterating, maintaining all these scattered copies of code and documentation is a desecration of a programmer’s time. Hence: code automation.

Code automation is really a bridging process. Instead of directly writing the final form of the code, Bob, we write a more abstract (more descriptive) code, Alice, and then use a parser to turn Alice into Bob. Ansible is a fine example — you only describe the desired state in YAML, and as for how to reach that state, ansible figures it out. Code automation is simply writing, one after another, our own ansible-like tools. At ArcBlock, the tool we use to define GraphQL APIs is goldorin (to be open-sourced soon); it has saved us an enormous amount of time going from absinthe to ecto and then to slate, letting us focus on the logic layer instead of wasting time on presentation- and data-layer schemas (see: Thinking, Problems, and Methods).

Another direction for code automation is the template repo and the utility belt.

Template repos are extremely useful for frameworks with complex project structures. If we treat the framework as a product and engineers as its users, then a template repo is the product’s onboarding. Think about it: where would the world be if React had no create-react-app, Phoenix no phx.new, Rails no rails new?

However, the official template is only a minimal template to help people onboard better. An organization needs its own templates — for instance, in our elixir repo, travis, makefiles, deps, version, release are all pre-configured in the template; in theory, a repo created with arcli create:repo only needs its corresponding keys travis encrypt-ed and the whole build pipeline is ready to go. Moreover, building a subscription-capable GraphQL API with phoenix + absinthe + ecto requires a lot of upfront work — you need to write a great deal of foundational configuration and scaffolding before the system is usable — which is exactly where template repos shine. In places like these, it pays to spend the time: one generation plants the trees, the next enjoys the shade.

The utility belt is another direction for automation: it takes all those functions you wrote ad hoc, tossed at random into some utils / helpers directory, or even a functional private function you casually scribbled in the current file, and extracts them, organizes them together — they can then be tested on their own, released on their own, designed more deeply, and abstracted more appropriately when the need arises. The only downside is that you have to add it to your project’s dependencies, and when you’re modifying the utility repo and the feature repo locally at the same time, it gets a bit tedious. But that tedium buys you efficiency when you’re working across more projects.

Unify the working interface

If you’re constantly switching among dozens of projects, a unified working interface is essential — it saves you untold time. For each repo, for example, we define these targets:

• make init: project initialization — install submodules, install the software the project depends on, set up the env (e.g., create a virtualenv for python), and so on.
• make dep: install dependencies — npm install for a nodejs project, mix deps.get for elixir; python is more of a pain: first workon into the project’s virtualenv, then pip install.
• make build: compile the code — webpack for nodejs, mix compile for elixir, etc.
• make run: run it locally.
• make test: run unit tests on the code.
• make build-release: package the compiled output for different environments into a tarball (ready for upload to a github release).
• make travis: used for Travis builds.
• make deploy: used by Travis to deploy the release.
• make create-pr: automatically bump the version, generate a changelog, and create a github PR.

See: Code is Law.

Once interfaces like these are unified, we don’t care what the repo’s directory structure is, which language it uses, or what the language’s various CLIs are — everything can be handled through a standard interface, very much in the spirit of don’t make me think. Work then flows like water: you just bang on the keyboard without thinking. If the backend has changed an API and you want to see whether the frontend behaves correctly, just clone the frontend repo and run make init; make dep; make run.

Make CI/CD faster — and faster still

I mentioned earlier that we use mix-deps to improve CI performance, but that’s only the tip of the iceberg. CI/CD is the most important link in the whole development chain — the final destination of written code is production. If the non-tooling code you write isn’t for some business running online, what’s the point of writing it? So if I had to pick one metric to measure a research and development team’s ability to iterate, I’d choose Time to Prod (TTP) — how long does it actually take for a feature / bug from the end of development to deployment in production? (Here I count the start of a canary release as the endpoint, ignoring the full canary duration, because for different projects at different scales, a canary going from 1% to 100% can stretch out considerably.)

Note that this number shouldn’t be compared with others — it’s only meaningful against yourself. A mature internet company’s TTP might be a few hours or even measured in days, but that may be because its business is complex, its CI pipeline long, and after CI there may still be manual intervention before deployment can begin. So if your own TTP is 30 minutes and you learn that some big company takes 7 hours, by all means don’t get smug.

Comparing against yourself means asking whether these 30 minutes have room for improvement — never underestimate the qualitative change that comes with collapsing a dimension of time. Here, if we can compress 30 minutes down to 3, we approach our work from an entirely different angle. At the 30-minute scale, our thinking goes: let me batch up the few small bugs I’ve fixed and deploy them together, because each deploy costs half an hour of feedback time, and a couple of rounds eats half a day. At the 3-minute scale, deploys can be sliced into very small units — even a tiny bug fix, we’ll just deploy on the spot, since it costs almost nothing.

When the engineers in an organization can deploy on the spot, the whole world is different. Engineers naturally chop features into pieces, iterating and shipping a little at a time. This is entirely in keeping with human nature — the smaller the release, the more controllable it is, the lower the chance of trouble; and if trouble does come, tracking it down is easier and rolling back is simpler. So who in their right mind would sit on one big move just to make trouble for themselves? But when an organization lacks this capability, people are forced to accumulate code, delay releases, and then confront complex bugs in complex scenarios. At Juniper, our release cycle was 3 months — thousands of engineers’ three months of accumulated code shipped all at once; never mind the release itself, just the various conflict merges were enough to skin you alive.

At ArcBlock, many of our repos have released more than 30 versions by now; the most prolific, the ocap service, has released 55 versions — on average a release less than every two days:

All of this is thanks to our reasonably good TTP. At the moment our elixir projects’ TTP is around ten minutes, which is decent. Of course, there’s still plenty of room to optimize CI — for example, we use slate to generate our API docs, and slate’s dependencies are ruby gems that install slowly, taking about a minute. We could actually build a ruby-deps to sweep up all ruby dependencies in one go and save that time.

Treat unit tests as an exploration of product capability

Unit tests are what developers least want to face — writing them is repetitive, they seem to do nothing for your coding skill, you have to labor over all kinds of mocks, and every so often you can’t even tell where the boundary of a unit test lies. That gives us the first attitude toward unit tests: they’re an unavoidable part of the process, so just go through the motions.

The other attitude toward unit tests is TDD — don’t write code first, write tests first. I’ve seen companies mandate TDD without truly grasping what’s good about it — which is awkward — everyone does TDD for TDD’s sake, going through the motions, writing a pile of test cases that end up useless or even deprecated.

I’ve never seen testing as a shackle on development; I see it as development’s umbrella. In an earlier article, How to Open TDD the Right Way?, I explored how to use TDD to explore product design — if you’re building an API that other engineers will ultimately use, then the API’s design process should be woven into how others will call that API, and how others will call it is something TDD can design and rehearse. So testing is the exploration of product capability, and also its verification. From this angle, tests are precious — my tests represent my product capability — and within this framework, as long as the product capability doesn’t change, I can tinker with the code however I like. That’s why good code is easy to refactor: not only because its structure is clear and its threads easy to follow, but because its tests guarantee that if product behavior changes during a refactor, it will be caught red-handed.

Refactor code anytime, anywhere

Uncle Lei says: refactoring your code once is not hard; what’s hard is being able to refactor it anytime, anywhere, for the rest of your life.

Refactoring anytime, anywhere is less an attitude than a capability. Generally speaking, that capability depends on:

1. Unit test coverage
2. Code complexity
3. Code coupling
4. Whether the language offers support for refactoring

As I said earlier, with good unit test coverage you can tinker with the code at will, and once CI flashes its green check, your heart doesn’t flutter. Without the corresponding tests, anything slightly complex turns into a fog of confusion the moment you tinker — Murphy’s Law is waiting right there. So code that can be refactored anytime, anywhere needs enough unit tests underneath it.

Programmer Jun has said: complexity is the natural enemy of code. Our best strategy is not to confront complexity but to dissolve it in every day’s commits. To that end, make good use of lint tools — turn every lint warning into a lint error, and hook that into git. Then, on every commit, the pre-commit hook can block unqualified code and force you to think again, to write it in a more reasonable way — to write it from another angle, or to split the complex logic apart. Day by day this keeps lowering the code’s complexity, holding entropy within a controllable range — and in the end, believe me, time will reward that diligence. I once saw a single C function over 5000 lines long, riddled with the scars of the years — like the little pits on the moon’s chubby round face — every pit an ugly fix left behind by a customer issue. Ugliness is contagious; fastidious as I am, face to face with code like that, all I could do was quietly add another switch case, planting one more festering sore on that already rotting face.

The process of controlling complexity is precisely the process of continuous refactoring.

In the C era, we cared a great deal about whether a function was re-entrant, and if not, at least thread-safe — this was really about decoupling a function from its context. A function whose execution is tightly bound to its context is a terrible thing — it’s hard to test (you have to reproduce the resources it depends on — some state, some lock, etc.), hard to refactor (without a deep understanding of its dependencies, you’re likely to dig a new pit), and hard to reuse. Decoupling means decoupling from context. Fully decoupled code is a pure function, akin to a mathematical function, a mapping from domain to range; fully coupled code is a non-re-entrant function. If human society is a spindle-shaped structure, then in the dimension of coupling, code is roughly spindle-shaped too: small at both ends, large in the middle. Across that vast middle ground (pure / impure), every time we write a piece of code, we must carefully choose whether to step left or right.

Here the language’s own support helps a great deal — in a functional language, for instance, immutability means any piece of code you write can run in any context (see: Eternal Charm). The question then becomes easy: how do we build layers that are sufficient and good enough, maneuvering things so that the right function appears in the right place?

In fact, a good system, no matter how complex, should be born out of simplicity: look at human society — beneath the precision of the social machine lie the simple, independent behaviors of individuals, and yet those independent behaviors, interwoven, carry incomparable complexity (think of the Gujing roast goose you had for lunch — from fertilized egg to your mouth, how many processes it went through!). Likewise, a living organism: the body is made of organs, organs of tissues, tissues of cells. The Daodejing says: the Dao gives birth to the one, the one to the two, the two to the three, the three to all things. That’s exactly the principle.

Some languages, like erlang and elixir, naturally model the behavior of the natural world, letting simplicity carry complexity through bottom-up decoupling. In the erlang universe, a process is like an independent, decoupled individual; individuals form teams (supervision trees), teams form companies (applications), and companies organize into a complex society (a release). This language-level support for decoupling makes refactoring a breeze, and more than that, accidents caused by refactoring (say, the code jumping the rails) are borne only locally — if an individual dies, the blast radius is no wider than the team that individual radiated into.

With all of this in place, facing dozens of repos, making a few commits every day in different repos, letting PRs made of several commits merge and produce a release, and having that release quickly deployed to production for feedback — you can face a thousand loose ends and still be busy without confusion — I fear no floating cloud that veils my gaze, for I stand upon the highest peak.

A moment of Zen

Huizi said to Zhuangzi, “Your words are useless.” Zhuangzi replied, “Only one who knows uselessness can be spoken to about usefulness. Heaven and earth are vast and broad, yet of all that expanse, what a person actually uses is the patch of ground beneath their feet. But suppose you cut away the ground around those feet down to the Yellow Springs — would there still be anything of use?” Huizi said, “It would be useless.” Zhuangzi said, “Then it is clear: the usefulness of the useless.”