Gone with the Wind (in the Cloud)

The biggest event of 2018 so far has finally come to rest: in the early hours of Thursday morning, we (Tubi TV) migrated all of our services and data from one region to another. Inside the engineering team, the project got a resounding name: Move to Oregon. For a technology company, this kind of migration is a one-way door — succeed, or die trying. If we couldn’t resolve every serious issue surfaced during the migration and had to roll back, it wouldn’t just mean that more than half a year of engineering effort had been wasted; the company’s capacity to deliver service would be cast into a very dim future indeed.

The groundwork was laid before the New Year — indeed, a lot of the earliest work reaches back to the final quarter of last year. We reshaped our infrastructure with Terraform (infrastructure as code) to build a code-review-driven production and staging environment, hoping thereby to manage environment changes — previously undocumented and capricious — in a more disciplined and transparent way. Our backend team (yes, we have no dedicated DevOps) spent a great deal of time writing scripts and building out the staging environment, and then the production environment. Once everything was in place, all that remained was to press the button, switch the traffic, and wait quietly to see what good or ill would befall. Like Xie An at the Battle of Feishui: heart pounding (Murphy’s Law — please, let nothing go wrong), outwardly serene (we had, after all, prepared for every contingency).

Many historical events, seen from the vantage point of later generations, can be recounted with a light and breezy air; only those who lived through them know the hardship of the process. Several kinds of database services, dozens of services, hundreds of machines, and a long, seemingly endless action list — migrated over by roughly ten backend engineers within a few low-traffic hours, all while minimizing impact on users and on ad revenue. It was nothing short of performing open-heart surgery to swap out an engine on a plane in mid-flight. Months of effort, all riding on a single battle; the pressure each of us bore was extraordinary.

From watching my former boss AK rise and fall at Juniper, I learned three things: 1) No one is perfect, and the world is not black and white — whether or not you like someone, if there is something about them worth learning, learn it, even if you can’t stand them; 2) In a lifetime, one rarely encounters a project of true importance — perhaps never a second time — and when such an opportunity comes, one should step forward bravely and seize it rather than shrink from it; 3) One must look forward. Past glories and humiliations are past; don’t keep rehashing them and let them hobble your continued growth. Keep an even heart — “the whole world praises him and he is not spurred on; the whole world blames him and he is not cast down.”

For me and my team, this region-to-region migration is exactly that kind of once-in-a-lifetime project — a gift from the heavens.

So I want to write something down while the memory is still warm, to persist it — as a way of settling accounts with myself.

The Doctrine of the Mean in the Book of Rites says: In all things, preparation leads to success; without preparation, there is failure. When words are settled beforehand, there is no stumbling; when affairs are settled beforehand, there is no distress; when conduct is settled beforehand, there is no regret; when the way is settled beforehand, there is no exhaustion. If we plan and prepare thoroughly before acting, the odds of stumbling into something we bitterly regret, of painting ourselves into a corner, drop dramatically. A data center migration is hard, thankless work, and what it tests is whether every last detail has been accounted for. Which services do we have, which are external, which internal, which third-party, which homegrown, who owns each one, what is its current state, what is the ETA — such a responsibility matrix must be built properly. The migration itself proceeds in stages: what to do in the hours before, the order in which DNS is switched during, what to verify afterward, and what the contingency plan is if something goes wrong. It’s like playing chess — think as many moves ahead as you possibly can, so you don’t end up frantic, harried, and making mistakes in the heat of the moment.

Since we have no dedicated DevOps, all the Terraform and Ansible scripts were written by the backend team. This has its good and bad sides. Two advantages:

• The backend team genuinely has end-to-end ownership of the services it builds, and the whole feedback loop runs efficiently. If a service is developed by one group, deployed by a second, and maintained by a third, there is bound to be trouble: buck-passing is unavoidable, and the feedback loop is torn and inefficient.
• High-performance services arise from the interplay of the resources that host them, third-party tools, and the service code itself. The developer is best positioned to weave those three together into an optimal whole.

So our basic principle is: whoever develops it, deploys it, and bears full responsibility for it.

The downside is that with a large DevOps project like this, developer time is hard to guarantee. The backend team each has their own projects in hand, and it’s hard for anyone to go all in.

As a result, the project’s deadline slipped again and again, and it seemed there would never be an end to it — even once most of our scripting work was done and staging was running stably, we still couldn’t settle on a consistent timetable for the cutover. Watching the first quarter of 2018 about to slip away like this, in early March, with the support of the project’s core members, I set a deadline: switch the traffic for real at two or three in the morning on 3/22.

What followed was a fight for resources — or rather, a fight for priorities. Although staging was ready, we still had to stand up every service in a shadow production environment and get them all passing tests. Standing them up wasn’t hard — just run the prepared scripts — but a few services still went sideways, and it took considerable effort before the problems were resolved.

Even as shadow production stabilized and the major issues were resolved, I had a worry: the new data center used a new domain, abc.com, while all clients use tubitv.com. Even if every service on abc.com was flawless, we could hardly guarantee that nothing would break once tubitv.com pointed at the corresponding services on abc.com. The trouble with Tubi TV’s business as a whole is that there are far too many clients — some legacy clients no longer maintained by engineering are, as far as we’re concerned, black boxes. So we came up with a thoroughly hacky way to test: we switched the company HQ’s DNS server over to a DNS server we built ourselves with bind9. That server hijacked every subdomain of tubitv.com, pointing them at the new data center, while forwarding all other DNS requests. This approach helped us flush out a number of serious bugs we had not anticipated at all.

On the day of reckoning, most of the US engineering team stood by in the middle of the night, while the engineers in China were sharpening their blades on the other side of the planet. External services were taken down as a whole, services were switched over to the new region one by one, incremental database backups were restored, and so on — the details of which I’m afraid are trade secrets and cannot be disclosed. Although we had prepared as thoroughly as we could, we still ran into three or four serious problems; after resolving them, we wrote up a post mortem, walking through each issue from root cause to solution to lessons learned. By the time everything was done, it was past 7 a.m. on 3/22 — roughly the hour I usually arrive at the office.

(This post was not published on the official account; declassified in January 2020, with all details redacted.)