Let It Crash: Misunderstood, Therefore Misused

Strictly speaking, I am also a half-baked Erlang programmer. To date, I have not written any Erlang code that actually runs in production. I have, however, written two thousand lines of Elixir that run in production, with several thousand more about to be deployed, so I flatter myself that I understand Elixir reasonably well. Since Elixir and Erlang share the same lineage at the VM and language-core levels, I also have a rough grasp of Let It Crash. Today, let’s talk about Let It Crash.

What Is Let It Crash?

In Erlang, Let It Crash means the programmer need not worry excessively about unknown errors and write exhaustive defensive code. Instead, you do very little handling, and when such an error arrives, you let the context in which the error occurred—usually some process—crash and exit. When the process exits, it reports this condition to a monitor process, which decides how to handle the error further.

A somewhat imperfect example: you write a gen_server that implements a payment state machine, and it relies on a third-party payment service. The third-party service has a few return statuses: timeout, success, or fail. One day the third-party service has a problem and returns a status it should never return: busy. Your server has no idea how to handle this status, so the whole process crashes and exits. Then this process’s monitor process—usually a supervisor process—is notified and, according to a predefined strategy, restarts the server process. The whole affair happens in an extremely short interval (within microseconds), so the outside world never even perceives that the service crashed.

Is Let It Crash Erlang’s Exclusive Patent?

Sorry, but this is where we have to slap down certain self-assured answers: yes, definitively, absolutely. The idea may not have originated with Erlang, but only the Erlang VM truly lets programmers practice Let It Crash with peace of mind. Other languages and frameworks—sorry—do not have this capability. Akka on the JVM is the closest imitation, but because of the JVM’s inherent disadvantage in this scenario (it was not designed around Let It Crash, after all), Akka is merely close.

Many people mistakenly believe that Let It Crash means letting the program crash and then relying on tools such as systemd / supervisord / monit to recover after the crash—this is a desecration of the Let It Crash idea. We see some answers confidently spouting nonsense: look, my project uses multiple processes + shared memory / memcached, and it can Let It Crash too. Utter rubbish! Try writing code that way—if your boss doesn’t fire you, you can count yourself lucky. Process-level monitoring and restart often takes hundreds of milliseconds or even seconds to complete. In high-concurrency scenarios, one second of downtime can mean hundreds or thousands of requests are delayed and left unhandled. Delayed requests are the least of it: a hacker can try a few input patterns and discover that your service can be made to delay its responses with a specific input (because it crashed and restarted). Then DOS / DDOS becomes easy: send you a thousand crafted requests per second and keep your service restarting forever, so it never gets any real work done.

Why Is Let It Crash So Difficult?

To support Let It Crash, the Erlang VM may not have sweated blood, but it certainly racked its brains. Let’s see what must be done if you want code to skip defensive coding and still have the system work well (without crashing):

• Errors must be isolated—an error in one processing context must not affect or propagate to other places, and the affected context should be as small as possible
• Errors must be handled by someone—previously programmers handled them through defensive coding; now a suitable system must handle them
• Error recovery must be fast enough—if you isolate and handle errors, but handling is as slow as described above, there is still no practical point

Let’s look at each.

Error Isolation

The Erlang VM provides the following mechanisms that together accomplish error isolation:

• Use the actor model; every code execution context is a separate process. A process is a very fine-grained runtime unit—think of it as a cell making up a system. Among several concurrency models (threading, CSP, actor model), the actor model offers the best isolation. On the actor model, see my article:
• Introduce immutable data. If data cannot change, the possibility of its being shared by multiple contexts is cut off. Thus, if data read by process A is corrupted, it will not affect process B (setting aside ets tables and databases for the moment), and error isolation is further guaranteed. This point is very important. In the multi-process + shared memory / memcached answer mentioned earlier, if the shared data goes bad, you can Let It Crash a million times and it will still read bad data; without defensive coding, the problem simply cannot be solved.

These two are indispensable. One answer says the key is immutable data; but with immutable data alone, when a problem occurs and a thread—or even a process—crashes, that is still unacceptable.

Error Handling

Do not naively think that Let It Crash means errors do not need to be handled. It is just that the handling of errors is carried by “someone” for you.

Before you let “someone else” carry the error for you, you first need a mechanism to notify them of the error.

In Erlang, this is done through link / monitor. In short, when a process starts, it links to an existing process; then if one of them crashes, the other is notified and can take necessary action.

Erlang’s OTP framework further encapsulates this pattern as supervisor—that is, some processes do nothing but manage other processes—just like a people manager in your company, hence the vivid name supervisor. Because people managers do not write code, they do not make mistakes; still less do they write code out of curiosity to grab mooncakes, so they will not be terminated. They only do one thing: when someone is terminated, quickly hire a replacement to fill the hole. Ah, I seem to have digressed. A supervisor process does no concrete work; the work is done by worker processes. Worker processes do more, so they have a higher chance of making mistakes. In some situations, if something is not handled well, biu—they are gone. The supervisor is notified and immediately starts a new worker process to take over.

So Let It Crash is not about not handling errors; it is just that the way it handles them is by monitoring for crashed processes and then handling them according to certain strategies.

There are several handling strategies: one_for_one. One person dies, hire another to fill the slot; one_for_all, one person dies, I no longer trust the whole team, lay off the entire team, then hire a full new set to fill every slot; and rest_for_one, a team has five people, one two three four five, employee number three dies, so I lay off numbers four and five too, then hire three people back.

Through this kind of organization, we can arrange a system into a strict hierarchy, forming a supervision tree. This tree is very much like a company’s org chart. Several workers are led by a manager, several managers are managed by a director, several directors report to a VP, VPs report to EVPs, each EVP runs a BU and all obey a fellow called CEO. The coming and going of people at the bottom does not affect the company’s survival (nobody even notices when a lowly engineer leaves). Even if a whole BU has a major problem, you cut the BU, the EVP leaves, everyone below disperses, resources are reallocated, and a new BU is assembled; for the company this is a bigger impact, but not fatal.

So a system built with Erlang, if organized properly, is very hard to crash. The error isolation described earlier, plus the tree-shaped error handling here, minimizes the impact of errors.

Error Recovery

Restarting a process takes time. In Erlang, that time is small enough that most of the time you need not care. An Erlang process that stores no state has a memory footprint of only a few hundred bytes after startup: the process’s execution function and the information registered with the scheduler (for scheduling). In Erlang, spawning a process is almost on the same order of magnitude as executing a slightly complex function, which is why you can afford Let It Crash—because restart completes in an instant.

Moreover, for stateless processes, the cost of restart is nearly zero; more often, a process has state, and restart means reinitializing the state (fetching it back from persistent storage), and sometimes part of the unpersisted data may be lost.

Another example. You have a user service. For each active user, you use a process to track and handle their real-time state (games often do this), persisting the state (or writing an oplog) at specific intervals. If a user’s process dies, restarting it involves rereading the user’s state; data that was not persisted may be lost.

We said earlier that the entire Erlang language and VM are designed around Let It Crash, so this scenario is also considered: it has an in-memory store like ets to hold data, so that even if a process crashes, real-time data need not be lost. You can use a process to “manage” an ets table (setting this process as its heir). When a worker starts, the table is handed over to the worker; when the worker crashes, ownership of the table is returned to that process.

See—under Let It Crash’s rough exterior there are such exquisite details! This is something that was gradually perfected over the long evolution of the entire Erlang / OTP system; it is functionality derived from taking Let It Crash truly as a philosophy and letting various problems and requirements catalyze it. Akka has no ets, nor similar ownership-management capability, which is why I said above that Akka is merely close.

The ets table’s heir and give away mechanisms ensure the uniqueness of the data accessor (again for isolating problems). This is something memcached / redis and similar solutions cannot match. In redis, no matter how ingeniously you design isolation, a single software bug—writing a key that should not be written—can make others read corrupted data and thereby crash the entire system.

Does heir / give away look familiar? Yes! Rust’s ownership / borrow mechanism in memory lifecycle is remarkably similar.

Do Not Abuse Let It Crash

Although Let It Crash has many benefits, we should not use it without restraint. Confucius said: Excess is as bad as deficiency. Even a Manchu-Han imperial feast every day will make you sick; no matter how good an idea is, using it when it should not be used will still lead to “the masts and oars vanishing in flying ash and smoke.”

Why? Let It Crash’s goal is fault tolerance and auto healing—a system that can heal itself and undergo metabolism. Let It Crash is only the means to that end.

Furthermore, we must clearly recognize that although the cost is low, Let It Crash does have a cost—just let 2 million processes crash and restart and see what happens.

So in which situations should Let It Crash not be used?

• Input from the outside world that does not match your expectations. A software system needs to isolate the external world from the internal world. This isolation can be achieved through data transformation and cleansing, not by crashing at the slightest disagreement. Think about it: if you are writing an API and the caller uses the wrong parameters, is it better to return 400 Bad Request with a helpful message, or to crash and return 500 Internal Server Error?
• Predictable errors. A function that should use guards does not; pattern matching that should provide a catch-all does not; reading a file without first checking whether it exists—these are things you should think through when writing code. Failing to do them and instead calling it Let It Crash in a pretty name would infuriate the venerable Joe Armstrong.
• Contexts that cannot be recovered should, as far as possible, not use Let It Crash. A typical unrecoverable context is a TCP connection. If it is not a serious problem, and you inexplicably crash a process that is handling the current connection, the user behind that connection may never come back. Think about it: would you use a voice chat app that automatically drops your call at any moment while you are talking?