When the 99.9% Cloud Falls: AWS S3 Outage Leaves North America Reeling

Incident Recap

At 10 a.m. Pacific (2 a.m. Beijing time), AWS S3 began to malfunction. Many startup engineers found that their services could not upload or download files normally. Someone on Hacker News asked: Is S3 down? and quickly received confirmation from everyone.

However, AWS’s own status page (https://status.aws.amazon.com) lagged behind; looking at it, everything was delightfully green. Just as everyone began to think they were being paranoid and that it was all groundless speculation, AWS engineers discovered to their horror that because this page depended on S3, it had actually gone down too. They hurriedly put up a banner explaining the situation and telling the world that the green was an illusion:

At 11:35 a.m., after some effort, the page finally showed a normal status:

As you can see, the hardest-hit area was S3 in Northern Virginia. Because S3 was not working, services that depend heavily on it—such as Elastic Map Reduce (which needs S3 to store intermediate steps and results) and Athena, just released at re:Invent last year (the data it queries must be stored on S3)—also went down completely. Services that depend on S3 less heavily were not in great shape either.

S3 is the earliest cloud service AWS released, Simple Storage Service, solving the storage problem. Storage is the cornerstone of any Internet service—whenever there are large data objects, whether images, videos, or text, we need a suitable storage solution to keep them. Before the cloud, this content was stored either in prohibitively expensive SANs (Storage Area Networks) or in disk arrays across large numbers of PC servers, accessed through special filesystems such as HDFS. To maintain the durability and availability of such data, Internet companies had to spend enormous manpower and resources on this infrastructure and could not concentrate all their engineering capacity on the business. When S3 and similar services were born, for many startup Internet companies it was like rain after a long drought: 99.99999% durability and 99.9% availability could not have been more delightful, so they rushed to build their own storage architectures on S3. Today, the number of websites using S3 has reached 148,213 (data from TechCrunch).

So when S3 (mainly in Northern Virginia) went down this morning, the entire North American Internet became a landscape of wailing.

Slack could not upload files; the progress bar ran forever:

Trello declared, ‘I’ve already been acquired; resting, just resting a while, won’t hurt’:

Atlassian, which acquired Trello, was no better; the copy was comically upbeat (I even wonder whether their engineers noticed the problem):

Giphy, recently a darling of VC, looked completely normal on the surface (the CDN was carrying GIF downloads), but if you wanted to upload a GIF: sorry, we have no idea what on earth happened:

As for aloof Quora, it did not even bother to provide a warm-hearted page; it just said, ‘We’re done’:

…

In principle, a service like Quora should not have its reader-facing side depend so heavily on S3; if something failed, the whole site should not go down—at most the part where users write answers. I don’t know why it collapsed so completely.

Let’s look at what an ordinary S3 GET returned when the problem occurred:

AWS tells you outright: Internal Error. From an error-handling perspective, when we write code we should catch this exception and handle it appropriately. Unfortunately, a service like S3 is so foundational—like water and electricity for the Internet—that everyone assumes it will never fail. As a result, many engineers simply don’t handle errors: like Slack, they let the progress bar run on foolishly; or they let the error bubble all the way up until the entire service dies, like Quora and Trello; the better approach is to catch the exception and confine the error to a specific page, as Atlassian and Giphy do.

How Can S3 Users Protect Themselves?

A similar incident is unfortunate for any company, especially SaaS companies that guarantee SLAs to customers. What you can do is:

• When a cloud provider outage occurs, contain its blast radius as much as possible. Letting even the landing page go down, as Trello did, is really unacceptable; a service like Quora could prepare a static mirror so that, when trouble hits, readers at least have something to read.
• Cache dynamic content as much as possible, or even make it static. Redis cache, nginx cache, and CDN are all means of caching and even statically serving content. Maintaining multi-tier caches is a pain, but when the underlying service fails, they become a rare strategic buffer. The half hour to several hours a cache buys you may carry you through the most critical moment (this S3 outage lasted roughly four hours), giving you room to search for solutions with relative composure, urgently publish a new page, or migrate services and minimize losses. Otherwise, you can only do what many companies did in this incident: leave it to fate, press your palms together, and pray that AWS engineers are capable enough to fix the problem.
• Use the Simian Army to exercise your system’s fault tolerance on ordinary days. This suits larger companies whose engineering teams have spare capacity. Netflix uses AWS heavily yet has emerged unscathed from repeated AWS outages because it too had been deeply stung by the cloud’s ‘instability.’ Their Chaos Monkey service (later developed into the Simian Army) simulates various outage conditions at any time and place, disrupting production. For example, to rehearse this incident, you could configure: simianarmy.chaos.fails3.enabled = true. Then these annoying monkeys will, without your knowledge, randomly modify your servers’ /etc/hosts, making all S3 APIs unavailable, so you can experience scenarios you rarely encounter and then find corresponding countermeasures (note: try it in staging the first time; otherwise, if your boss fires you, don’t blame the Programmer).

If AWS really has a large-scale outage and you have not taken any measures, the sky won’t necessarily fall. At this very moment, your investors, your customers, and your partners may all be busy solving their own outage problems. On Hacker News (https://news.ycombinator.com/item?id=13755673), there’s a joke that goes like this:

Why do we host on AWS?

Because if it goes down then our customers are so busy worried about themselves being down that they don’t even notice that we’re down!

See, this is the cunning of CIOs (if you build it yourself, when something goes wrong you have to clean up your own mess).

How Can You Make Use of Such an Outage?

Google engineers hurried over to twist the knife and educate users:

You see, society is just wolves circling. Don’t even mention not trying; you are trying, but if you stumble once, wolves come to gnaw at the meat. For clients, the more wolves, the more choices—they can hardly contain their joy; as vendors, you want to cry but have no tears. But every company, even every person, plays different roles in different contexts—sometimes the client, sometimes the vendor.

What? You ask whether Tubi TV went down? We have our own troubles, but thanks to the CDN, we’ve been just fine for the past few hours.