2020: Living Long Enough to See Everything — Mainstream Western Apps Trip Their "Circuit Breakers"

In this world, nothing is certain except death and taxes. — Benjamin Franklin

If you build mobile apps overseas, there is one company you can hardly avoid: Facebook (FB hereafter). It has built — and nearly monopolized — an end-to-end service running from user acquisition to monetization (advertising), a temptation so enormous that you cannot resist it — provided, that is, you bundle FB’s SDK into your app.

Bundling third-party SDKs into an app to enjoy a wealth of “free” service integrations is nothing new in this industry. Developers — small and mid-sized ones especially — feel a natural trust toward SDKs that come from a giant like FB, that are open source, and that they simply cannot avoid: everyone else is using it, so what could possibly go wrong?

Yet today (5/6/2020), around 15:30 US West Coast time (the earliest problems trace back to 14:44, since this was a rolling deployment), every app using the FB SDK crashed en masse. The spectacle was impressive:

The casualties include Douyin’s overseas incarnation, the currently unstoppable TikTok, plus apps everyone uses such as Spotify and Pinterest:

An epic volume of crashes! Truly, live long enough and you’ll see everything! Perhaps some engineer felt that only the finance and manufacturing crowd had suffered, while most internet companies were still singing and dancing in peace, and — overflowing with a sense of justice — decided to take a blade to themselves? (Purely in jest.)

If charts like these still aren’t vivid enough for you, this developer’s lament [4] should hit hard enough:

Our crash is happening to 40% of new app starts. It has not dropped yet.

Users of big-company apps will curse a few choice words and move on, because they still have to use them. But when a small team’s app gives new users this kind of first impression, those users are probably lost for good (one-star review plus uninstall).

For a while Twitter exploded, and the pans of blame flung back and forth finally landed on FB’s head:

(Translation: Quit your whining — if your app is crashing all over the place right now, the FB SDK is to blame.)

On Hacker News, programmers took a professional tone to explain to the uninitiated why, even as you eat hot pot and sing songs alongside TikTok and friends, it’s FB that has to carry the can for your app crashing — while also debating a question that cuts to the heart: for apps that don’t use FB ads and only use FB Login (Spotify, say), should they fall back to plain old OAuth?

To be or not to be — that is always the question. OAuth’s user experience is an order of magnitude worse than the SDK’s; retreating to it abruptly would hurt UX, and that is simply unacceptable. Yet using the SDK always leaves you feeling throttled by the throat — as today’s black swan event demonstrates. What’s more, the SDK quietly hands your users’ information to FB in some form or another, so your user data fills out FB’s social graph on your dime. For app developers, every SDK you install in your app is like practicing the Star-Absorbing Technique: you ingest an outside practitioner’s inner power, but you never know when — or in what form — it will turn on you. Hence the ironclad rule at my company:

Using third-party SDKs is extremely dangerous — tread with the utmost caution!

The curious among you may ask: what exactly did Facebook change that took everyone down with it? Let’s look at the analysis in this GitHub ticket [4]:

The corresponding client-side crash is:

Fatal Exception: NSInvalidArgumentException
-[__NSCFBoolean objectForKeyedSubscript:]: unrecognized selector sent to instance 0x200729d00
-[NSOrderedSet initWithSet:copyItems:]

I haven’t traced this crash in detail myself, but if this fellow’s analysis is right, it means an FB backend engineer shipped an update that sent a boolean where a dictionary structure should have been returned to the client:

"restrictive_data_filter_params" = "{"is_health_app":false}"

Here, is_health_app should have carried a dictionary but instead carried false. If the cause really is that bizarre, then I’m intensely curious how such code got through code review, and then through CI. So I left a comment:

My own feeling is that the programmer bears responsibility for a problem like this, but shouldn’t be the main scapegoat. The deeper problem lies in the whole process:

1. Naming: why give a dictionary structure a name like is_health_app that so obviously reads as a boolean?
2. Type safety: why is there no type checking on the data the server API returns (a typechecker would have caught this instantly)?
3. Unit testing: why didn’t unit tests and CI catch it? Is the test case missing? And how many more gaps like it are there?
4. Code review: why did no one spot it in review? It seems that even a seasoned hand at the gate wouldn’t have saved this bug.

I look forward to FB publishing a full postmortem in a couple of days.

What lessons does this incident teach us? Two things, I think, are crucial:

1. Be extraordinarily careful with third-party SDKs: either you’d die without it, or having it delivers essential help to your revenue or user growth. Otherwise, if you can do without it, do without it.
2. Always type-check what your server APIs return! Where possible, prefer strong typing — GraphQL, gRPC, and the like.

One more thing

The incident had a side effect nobody anticipated: Google’s Crashlytics got dragged down with it. My guess at the rough cause is that the Google Firebase team had never seen such a tsunami of crashes either (epic-level “live long enough and you’ll see everything” again), so the entire crash-report processing pipeline got severely backed up — possibly hitting its designed ceiling — beyond even the scalability that cloud services are forever boasting about. As a result, a dozen-plus hours later, Crashlytics still hasn’t recovered.

References

1. downdetector: https://downdetector.com/
2. twitter: https://twitter.com/reneritchie/status/1258177289436770305
3. hacker news: https://news.ycombinator.com/item?id=23097459
4. github ticket: https://github.com/facebook/facebook-ios-sdk/issues/1373