Snowmobile Rhapsody

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. — Tanenbaum, Andrew S. (1989). Computer Networks. p. 57.

An old friend to every CS graduate (we all read his Operating Systems: Design and Implementation back in our OS course, didn’t we?) — Professor Tanenbaum’s precise claim from nearly thirty years ago. Snowball/Snowmobile proves, in full, that it still holds today. There was also a two-page RFC1149 back in 1990 — A Standard for the Transmission of IP Datagrams on Avian Carriers — that nailed it:

Avian carriers can provide high delay, low throughput, and low altitude service. The connection topology is limited to a single point-to-point path for each carrier, used with standard carriers, but many carriers can be used without significant interference with each other, outside of early spring. This is because of the 3D ether space available to the carriers, in contrast to the 1D ether used by IEEE802.3.

So the concept behind Snowball/Snowmobile was not invented by Amazon. At tubitv.com, for years the primary way Hollywood studios delivered films to us was also by FedEx-ing us hard drives. But Amazon is the first company to take the data-transfer problem seriously, attempt to standardize it, and turn it into a service.

Wikipedia (search for “Zettabyte” yourself) says some estimate that humanity produced 4ZB of data in 2013 — that is, 4,000 EB, or four million PB. The programmer who ponders his sovereign, his state, and the commonweal sank into deep contemplation. All that data would take 40,000 Snowmobiles, each hauling 100PB, to move.

Which raises the question:

Which school teaches the best excavator operating skills?

At the rate humanity generates data, more and more companies will reach the EB scale, and hauling it around a dozen or so trucks at a time just won’t cut it. Is there a better way to transport it?

My prediction: in 2018, under intense customer pressure, AWS will announce the Snow Train:

A single train could carry several EB of data. But since North America’s rail system isn’t robust enough, the mainstay of Snow Train operations ultimately ended up in my beloved Song Dynasty.

By 2020, Snow Train could no longer resolve the contradiction between the people’s ever-growing data production and the backward state of data-carrying capacity, and so the Snow Container was born:

(Hold your laughter — I photoshopped it with Skitch.)

A 200,000-ton freighter carrying 10,000 containers, each holding 1EB (a container holds 100PB today; a 10x increase in three years is hardly unreasonable) — 10ZB of data in total.

With that, not only is transport capacity assured, but data can cross continents — thanks to the enormous electricity bills for cooling brought on by climate change, the big data centers have all migrated to the Russian Far East, Alaska, and Iceland. Data flows out of the data centers by the truckload, converges at rail yards, is hauled by train to the ports, and finally the great ten-thousand-ton ships set sail.

In 2030, humanity successfully turns the Moon into a tourist destination. The Lunar Development Corporation, however, discovers that the programmers who make up 60% of all tourists complain loudest about the overall travel experience: the network is too slow, and the connection keeps dropping. Pulling code from GitHub, looking things up on Stack Overflow, and running npm install (mix deps.get / brew install) to install dependencies takes an unbearable amount of time, which sours their mood for fixing bugs and building features, and interferes even more seriously with the travel experience. Giants like Faceless and SneakChat go further, threatening to remove the Moon from their employee vacation plans if the network problem isn’t solved. After much painful reflection, the Lunar Development Corporation partners with Amazon to build the Snow Rocket — its first shipments being Wikipedia, the npm registry (front-end power!), apt mirror sites, and a full mirror of Stack Overflow, by popular demand. Oh, and of course, to accommodate China’s vast population of programmers, they also joined forces with WeChat to mirror every last byte of its official accounts’ content.

Okay, I can’t keep making this up. I do think the means of storing and transmitting data will change drastically over the next 10 years; perhaps some black magic from biology will solve it (a single gram of DNA holds hundreds of EB of data, after all). Otherwise, at the current rate of data creation, the Earth will be drowned in hard drives by then.

Back to Snowmobile. According to Amazon, 1EB of data can be migrated within 6 months. At the announced Snowmobile pricing of $0.005/GB per month, the storage cost of migrating 1EB comes to 30M USD (5 x 1000 x 1000 x 6). For a company that owns data at that scale, considering that storing 1EB on S3 costs 27M USD / month, that price is still acceptable.

Amazon hasn’t disclosed the details of the truck’s interior, but one can imagine it contains:

• 1000+ 100TB Snowballs? There’s no reason not to reuse the existing Snowball. Besides, all hardware has an MTBF (Mean Time Between Failures); with thousands of Snowballs, the odds of one or two failing rise sharply, so you’d need a few redundant Snowballs on standby.
• A high-availability power cluster + several diesel generators. Supplying power to that many disks and related equipment is no small feat.
• Several 10GbE ports + several 1GbE ports. Assume a month is allotted for transport, so the one-way data copy must complete within 75 days. Copying 1EB in 75 days means at least 13 x 10GbE ports copying in parallel — 1EB / (10Gbps / 8) / 86400 / 75. Factoring in other overheads, 20 x 10GbE ports plus 50 x 1GbE ports should meet the needs for data ingestion and egress.
• High-capacity switches.

Suppose a company at the EB scale wants to move data (whether to AWS or from AWS to a self-built data center) and plans to use Snowmobile — what threats might arise?

One or more trucks get into a traffic accident

The probability that one of 10 trucks has an accident is not small. If that happens, data delivery is delayed enormously, so beyond redundancy within each truck, you also need a RAID5-like redundancy scheme across trucks. That way, delivery is guaranteed to be unaffected. As a result, the final cost above might rise by a third, to 40M USD.

The convoy gets hijacked

For an internet company, data is life itself. When data sits in a data center, it is at its safest — like cash in a bank vault; but when data is physically in motion, it is at its least safe, much like an armored cash truck. A Snowmobile rolling down the highway is, to a data bandit, a lamb waiting to be slaughtered. Hijacking data used to require brains — a cat-and-mouse game of wits; now all you need is a gun. Whoever is nastier wins.

So for extremely important data, a private security escort is indispensable — you might even need helicopters or satellite tracking. (By the Snow Container era, it’ll be warships, maybe even aircraft carriers, providing escort. What a beautiful picture.)

disclaimer: This article is not a serious technical analysis — merely some wild ravings based on nothing more than the official Snowmobile information Amazon has released.

Today’s keynote by Werner Vogels was so dense with information that I haven’t yet had time to organize my thoughts. So I’m publishing these ramblings, scribbled down after a bout of insomnia this morning, to toss out a brick and see if it draws forth jade.