And God Said: Let There Be a Language for the Future — And So Erlang Was Born

Today let’s talk about Erlang. I’ve written about a fair number of languages before — Elixir, JavaScript, Golang — but those pieces skimmed the surface; they were mostly simple introductions. This article is different. Erlang is not a new language, and plenty of people have already introduced it at a basic level; I want this one to distinguish itself in depth. Writing is hard when you want to please the layman and the expert alike: go too deep and the beginner or the outsider loses interest; stay too shallow and the expert feels his time is being wasted. So I’ll do my best to put profound things into plain words.

Back to business. Erlang is a gift God handed down to the world some thirty years ago. It is so singular, with so many ingenious ideas kneaded into its construction, that thirty years ago it set out to solve the problems we would only meet at scale thirty years later — in the history of programming languages it stands as a benchmark unto itself. To explain Erlang properly within a single article is difficult: it starts from a complete and utterly unique worldview, provides a VM as intricate as an operating system, and then abstracts the industry’s best practices into a framework (OTP) that solves many of the complex foundational problems of distributed concurrent systems. No other language or system cares as Erlang does — not only about how software is written but about how it runs; not only about how software runs today but about how it will be upgraded tomorrow. So this article is a survey — like the general outline of the Dugu Nine Swords — touching on every aspect of Erlang, discussing interesting things, but never descending too far into detail.

Let’s start with old man Joe’s worldview — that is, his worldview.

The Worldview

A programming language carries its own distinctive thinking. Python has the Zen of Python; Golang says, Do not communicate by sharing memory; instead, share memory by communicating. Yet neither has been elevated to the level of a worldview. Old man Joe, in his doctoral thesis, explicitly laid out the worldview he had in mind for a new language:

• everything is a process.
• process are strongly isolated.
• process creation and destruction is a lightweight operation.
• message passing is the only way for processes to interact.
• processes have unique names.
• if you know the name of a process you can send it a message.
• processes share no resources.
• error handling is non-local.
• processes do what they are supposed to do or fail.

These unadorned sentences, which look like nothing much at all, form the cornerstone of the Erlang world.

Let me unpack them in four parts.

process

In the Erlang world, everything is a process. Processes are like atoms: they compose molecules, cells, tissues, organs, and finally a whole organism (see my essay: boundary). So what exactly is a process?

It is, in fact, almost identical to the OS notion of a process — a PCB (Process Control Block) plus the memory space around it: heap, stack, mailbox, and so on. The difference is that an Erlang process has a tiny memory footprint: a freshly initialized process occupies only a few hundred words (on a 32-bit CPU, 1 word = 4 bytes; on a 64-bit CPU, 1 word = 8 bytes). So in theory the number of processes the Erlang VM can create is bounded only by memory — we can easily create millions, even tens of millions, of processes (a 32-bit CPU uses 28 bits for the pid, a 64-bit CPU uses 60 bits; you can work out the upper bound yourself). Erlang’s process comes from the idea of the actor model, one of several common concurrency models (for the other models, refer to my essay: ). Processes handle all manner of business by sending and receiving messages with one another.

(From: erlang garbage collector and why it matters)

A process is actually a lot like an object in OOP: processes are isolated from one another — one process cannot access another’s internal state, which is the notion of encapsulation — and by using parent/child processes one can implement inheritance and polymorphism perfectly. In fact, Erlang can be counted as the orthodox heir of OOP’s progenitor Smalltalk (the other heir, ObjC, is gradually fading from view).

routing

Once a system contains countless independent processes, a pressing question arises: how do we find the process we need to interoperate with? That is a routing problem. In Erlang, we can locate a target process by its unique pid, like dialing a phone number directly; we can locate it by a name that maps one-to-one to a pid, like calling the number from your phone’s address book because you can’t memorize Mr. Program’s number; and we can reach it through a call-forwarding service, via a third party, when my number isn’t stored on your phone at all — in Erlang, that third party is called a name registry.

A brief digression.

Routing is a weighty subject in system design, casting its shadow everywhere. We know that everything a system does is built on event triggering, and the origin of event triggering is routing. When you flick your finger up and down while reading this article, you trigger tap/swipe events, and those events must be routed precisely to their handlers to produce the right behavior. What is DynamoDB at its essence? You give me a key, I route it to a value under some node under some cluster, and whether you read or write, kill or spare, is your call. The gossip and consistent hashing it uses are merely means and algorithms in service of routing. In C++, when you invoke a pure virtual function, underneath a vtable routes your call to the right subclass implementation. Virtual memory is a “routing table” from virtual to real addresses; VMA and friends are merely the tiramisu of this routing feast. So when we study a system, we begin with its routing; when we design a system, routing is a core piece as well. As the Master once said: Master the routing table, and you need fear no journey under heaven.

error handling

Erlang’s error handling is very distinctive — errors bubble up through links between processes. A link is a mechanism whereby two parties watch over each other, somewhat like the baojia collective-punishment system common in ancient China from Shang Yang’s reforms onward: one man breaks the law, and those linked to him are punished too. Take two linked processes — let’s call them Xiaoming and Xiaohong. If Xiaoming dies, Xiaohong receives an EXIT signal and, if she doesn’t handle it, will normally die as well; and Xiaohong’s death in turn triggers the other processes linked to her, sending tremors through the entire chain of interests connected to Xiaoming. This is the famous let it crash philosophy. This peculiar mechanism makes error handling genuinely non-local: everything related is affected, as with cancer cells — unless there is a targeted remedy, all related cells are killed without mercy. By contrast, error handling in other languages is less handling than hiding — like a border alarm with beacon fires blazing, which the Ministry of War suppresses, reporting to His Majesty that all is peaceful under heaven, after which the horses keep running and the dancing goes on.

A link is a two-way collective-punishment mechanism. Sometimes, for problems with limited blast radius, we don’t need anything so heavy-handed — hence the monitor. A monitor is like a scout on the battlefield, watching a process for any movement and reporting back the moment something changes. The process that receives the report may choose to stand pat or to adapt on the fly.

Erlang/OTP’s supervision trees are built atop these very simple link/monitor mechanisms. And one big reason Erlang can let it crash so wantonly is that processes are extremely lightweight: creating and destroying one costs no more than creating and reclaiming a handful of C structs.

Golang’s error handling has no comparable link/monitor mechanism, which I consider a flaw of moderate size (though it may well be a compromise made for performance — after all, there are no ideal languages, only languages compromised for a particular purpose). As for whether Golang should adopt such a mechanism — whether goroutine and channel, goroutine and goroutine, channel and channel would need link/monitor, and how it would work — that is beyond the scope of this article.

concurrency

As mentioned, Erlang’s concurrency model uses the actor model. The actor model says that the one and only way for an actor to communicate with other actors is message passing. Each actor therefore has its own mailbox for receiving messages. In Erlang, the process is the actor. To schedule vast numbers of processes, the Erlang VM has its own scheduler — in an SMP environment (roughly, multi-core), each CPU core gets one scheduler. Like the OS scheduler, it has run queues and wait queues; the scheduler takes the next process off the run queue and executes its code. If a process does not explicitly block itself and get scheduled out through something like receive, the scheduler will schedule it out after a fixed cycle (called a reduction in Erlang). So from the Erlang programmer’s point of view, Erlang has preemptive scheduling, and moreover each process’s CPU time is nearly fixed, so events get scheduled within a predictable time — this is called soft realtime. Few garbage-collected languages achieve soft realtime; Java can’t, Golang can’t, because their global GC has the notorious STW (stop the world) problem (when GC works, the world goes quiet, and how long GC will work, heaven only knows).

Since we’ve brought up soft realtime, what is hard realtime? A CPU timer interrupt is hard realtime. It fires at a fixed interval, absolutely precisely, rain or shine.

OK. With these fundamentals in hand, we can move to the next stage — message passing, scheduling, and OTP.

Message Passing

Erlang’s message passing is treated in many articles as a given, not worth prying into — like water or electricity: who cares where it comes from? While studying Erlang, Mr. Program had quite a few questions about it:

• How does a message travel from one process to another? Is data copied along the way? Are there locks? At what granularity?
• Can a message be lost in transit? If it is lost, what then?
• Those enormous messages received from the TCP/IP stack — do they get copied when passed from Xiaoming to Xiaohong? If so, how inefficient must that be?
• What exactly is the relationship between signals like exit, link, and monitor, and messages?
• …

I haven’t fully figured these out myself (especially link — to this day I don’t even know where its source code lives), but learning with questions in hand is always better.

Start with message delivery. Set distributed Erlang aside for now and consider only a single node. In the Erlang VM, Xiaoming sending a message to Xiaohong means copying the message body from Xiaoming’s heap into Xiaohong’s heap (or a heap fragment — no need to care about the distinction for now), then appending to the tail of Xiaohong’s mailbox a message control block pointing at the copied message body. If Xiaohong is an influencer, Xiaogang might be messaging her at the same moment, so Xiaoming has to compete with Xiaogang — and that’s where a lock comes in. The lock is per process, so it’s very light.

Having understood this process, we’ve answered the first two questions. Messages are never lost in transit, and this is an exactly-once message passing scenario (in my earlier article on zeroMQ I noted that exactly-once is impossible in distributed systems — think about why it works here). But what if Xiaohong ignores Xiaoming while Xiaoming, with unwavering devotion, keeps sending her messages forever? Xiaohong’s mailbox will grow and grow until the seas dry up and the rocks rot — in other words, until memory is exhausted. Then the VM crashes and every process goes down together.

Xiaoming may find it dull to endlessly text Xiaohong “I love you,” so now and then he writes her a love letter, the way Xu Zhimo wrote to Lin Huiyin. Once a lengthy letter is finished, Xiaoming would have to transcribe another copy into Xiaohong’s heap — a waste of time and a bad deal; and what if he wants to send that letter to Xiaohua, word for word, some other day? Xiaoming is cleverer than that: he sends a link to the letter, and she clicks it to read.

This is precisely how Erlang’s message passing handles binaries. Large binaries (allocated by the binary allocator) live in a shared pool, and sending a message only sends a reference. On receipt, the other side can find the data directly through the reference, avoiding pointless copying. We know references have a cost — the binary must track a refcnt, incrementing and decrementing as readers “subscribe” and “finish,” ensuring the binary is not freed before a consumer has used it, and equally ensuring it is not left unreclaimed after all consumers are done.

Scheduling

Scheduling is a centerpiece of the Erlang VM. How do you schedule so many processes sensibly to guarantee soft realtime? When we talk about scheduling, we inevitably talk about cooperative multitasking and preemptive multitasking — the two main scheduling approaches used by operating systems.

Cooperative multitasking hands control of scheduling to the processes themselves, believing in the inherent goodness of each one: every process is kind and will voluntarily do a context switch after using some CPU. This approach was used in prehistoric operating systems: MacOS, Windows 3.X, and various companies’ proprietary OSes such as NetScreen’s ScreenOS. The timing of cooperative multitasking is predictable (because the process’s own code does the yielding) and it’s simple to implement, but it’s very hard to guarantee that every process gets the timing exactly right. A single process with an unreasonable implementation — a never-ending loop hogging CPU — will starve all the others.

Preemptive multitasking improves on this: control of scheduling is firmly in the OS’s hands. The system sets an upper bound on each process’s CPU time and triggers scheduling via the CPU timer interrupt; a process that hits the limit gets scheduled out. The timing is unpredictable, and can occur anywhere in the process’s code, which makes the implementation — especially the context switch — more complex. But for the application author it’s a blessing: he needn’t worry about when he should yield so his application gets scheduled.

From the Erlang application’s point of view, the Erlang VM’s scheduling is preemptive, because each Erlang process (your Erlang or Elixir code) neither knows nor cares when it will be scheduled. From the perspective of the BEAM scheduler’s implementation, it is cooperative: the main function of each process (C code) — process_main, in beam_emu.c — explicitly calls erts_schedule (in erts_process.c) to see whether its reduction count is exhausted and, if so, do a context switch.

A reduction is Erlang’s instrument for gauging process throughput, akin to Linux’s nice value. One reduction can be thought of simply as one function call. Thus a process, after CONTEXT_REDS function calls (see erl_vm.h; 4000 in 19.x), will “voluntarily” schedule itself out.

I don’t know whether Erlang’s choice to make itself a functional programming language was pure authorial preference or an accident. But that choice has an enormous influence on its soft realtime. Suppose Erlang saw itself, as Golang does, as the heir to C — then accounting for reductions would be a headache. Imperative languages are full of nondeterminism — loops above all. If the function call remained the unit of reduction, then a function containing thousands of iterations internally and a function with no loops at all would differ enormously and asymmetrically: some processes might run 50μs before yielding, others 10ms — you couldn’t call that realtime. In fact, long ago while fixing ScreenOS bugs at Juniper, I was very much scratching my head over when a task under cooperative multitasking ought to yield. In C, choosing the right point at which to give up the CPU is quite hard, and as soon as someone adds code before or after it — especially code containing loops — that point may become a bad one. So when we hit a problem where some task occupied the CPU too long and got crashed out, all we could do was sprinkle yield calls into the relevant loops to dodge it — thoroughly powerless.

A functional language, by contrast, has no loops, only tail recursion. That makes it simple: every recursive step is a function call, and gets counted as one reduction.

What’s more, functional languages make you write functions as concise as possible. In code written in most functional languages, functions over 50 lines are rare, whereas in the beam source or in any open-source project at all, 500-line C functions are everywhere. On top of the functional basis, Erlang unflinchingly copied over almost all of Prolog’s “awkward” syntax — statements separated by commas, functions terminated by periods. That awkwardness reminds you at every moment: hey, we’re writing functions here, not doing Yang Peng’s GMAT long-sentence drills; be like Mi Meng — write short, punchy sentences!

From the scheduler’s perspective, some of Elixir’s syntax may be slightly ill-considered — Ruby-like syntax makes it easy for people to write long functions the Ruby way; in the plug codebase, for instance, functions over 50 lines are not rare. In OTP code, apart from the receive functions that handle messages, 50 lines is a big number. Elixir also offers Lisp-like macros, so it can easily spawn more Elixir code — and all of it, once compiled to Erlang, may become one fairly large function, which isn’t great for soft realtime.

I’ve wandered a bit far. Back on track.

The Erlang VM scheduler is a standard scheduler — one logical scheduler per core, with multiple queues separated by process priority. max is a priority reserved for the system internals; below it come high / normal / low, scheduled in that order. Although normal and low share a queue, a low-priority process is only scheduled after N visits. The scheduler also handles other work, such as ports. Ports are Erlang’s interface to the outside world (files, sockets, drivers, etc.); we’ll set them aside for now.

For better load balancing, when a scheduler has nothing to do it can go “steal” processes from another scheduler’s run queue; likewise, when its run queue is too long, a scheduler will actively try to migrate its own processes to other schedulers. We’ll set these details aside too.

You may ask: how are events like timers and timeouts handled? The answer is standard — the beloved, universally admired timer wheel.

The VM maintains a per-scheduler timer wheel, handling timer events for one scheduler. Erlang’s send_after and OTP’s call both hang an event on the timer wheel, and then, when erts_bump_timers runs inside scheduler_wait, timers that have expired in the current tick are processed.

On the subject of the timer wheel, a bit of bragging: I once filed a US patent, ager ring optimization (on how to file a patent, see my earlier article: how to apply for a patent).

A digression. From my experience, Elixir’s (that is, Erlang’s) regular expression performance is noticeably worse than JavaScript’s. Recently, leafing through the otp/erts code, I happened to notice that the Erlang VM also uses PCRE for regex handling. Same C code — by rights everyone’s efficiency should be about the same. On StackOverflow someone’s answer is: under Erlang, the PCRE regex implementation, out of consideration for reductions, may yield the CPU during processing and resume when next scheduled. It’s a compromise made for the sake of soft realtime scheduling.

Erlang’s design embodies its authors’ concern for latency. Golang only context-switches on communication; GHC, I’m told, only on memory allocation. Those scheduling granularities are coarse and very uneven, making stable latency hard to guarantee. Only Erlang measures each process’s throughput (reductions) at the VM level, as painstakingly as an artisan, so that every process gets its share of the rain and dew. Of course, this is achieved at the cost of local efficiency. Laozi said: That which you would take, you must first give. Compromise is the price a programming language pays to reach its goal.

OTP

OTP stands for Open Telecom Platform; the name no longer really matters. Basically, when we talk about Erlang we talk about OTP — it has become part of Erlang.

If the Erlang VM is the kernel of an OS, then OTP is the rest of that kernel’s components: stdlib, initd, syslog, OAM, etc. Beyond that, OTP provides Erlang with a number of far-reaching design patterns: gen_server, gen_event, gen_statem, supervisor, application, and release, among others.

These foundational design patterns have seeped into the marrow of every Erlang application, so much so that outside of teaching code, you will rarely see an Erlang app written by hand without OTP patterns.

Let’s take gen_server as an example. The process of handling tasks via message passing between processes can be abstracted into the classic Client/Server model (or RPC). A client requests some service from a server; the server handles it and returns a result (or doesn’t). This seemingly simple process is in fact full of problems:

• What if the server is already dead (or not yet started) when the client sends its request?
• What if the node the server lives on (thinking distributed here) is unreachable when the client sends its request?
• What if the server is too busy (too many unprocessed messages in its mailbox) to get to it?
• What if the server crashes midway through handling the request?
• Once the server replies, how does the client know this reply belongs to that earlier request?

These are problems everyone building distributed messaging systems will meet, and they are hard to handle well. In OTP, gen_server takes care of every detail for you, leaving you to care only about when to call and how the corresponding handle_call should process the incoming message. On top of that, it also does these things:

• It supports process-level ISSU (In-Service Software Upgrade), that is, hot code upgrade, and provides interfaces so you can migrate data during the upgrade.
• It supports tracing, so you can conveniently debug production issues online.

No other language or framework looks after you so thoroughly on these two points. The details of gen_server won’t be elaborated here; perhaps I’ll write a dedicated article on OTP explaining this pattern in depth.

gen_event is an implementation of gen_server, handling the observer pattern.

gen_statem is also an implementation of gen_server, for finite state machines.

supervisor was touched on above and in an earlier, not entirely rigorous, article on let it crash.

application was mentioned in the boundary essay. In Erlang, an application is a very self-contained, plug-and-play thing, rather like a component.

We’ll skip these patterns for now and introduce them properly when the chance comes.

What I want to spend some space on is the often-overlooked release. Erlang/OTP is one of the rare languages that cares both about how you write code and about how your code runs. Reading code in other languages, you feel like you’re conversing with a series of corpses; it’s hard to infer from them what they looked like laughing and talking when alive. Erlang is different. Code is clearly divided into distinct applications; applications have start/stop, and each has a supervision tree consistent with its runtime shape: you know clearly what your code looks like when running, which process/server is controlled by which process/supervisor.

Beyond that, Erlang/OTP also spares you the worry of how your system gets released. A release bundles several applications, each application’s version, their dependencies, the startup order, and the modules to be loaded — all spelled out in the release file. Even better, if you want to upgrade, you can generate an appup file describing which applications and which modules changed. When the upgrade happens, the relevant processes are temporarily suspended, the new modules are loaded, and after the code_changed callback completes and data has migrated from the old version to the new, they are resumed.

Other languages have similar packaging mechanisms — in Python, for example, you can use setuptools to produce a software package. But those tools only care about how your service runs in the foreground; they don’t care about things like:

• how to run as a daemon (start, stop, restart);
• if running as a daemon, how to attach for debugging;
• how to connect your software across machines into a cluster;
• how to upgrade/downgrade between releases.

While the first two can be substituted with third-party tools in other languages, the last two are almost the exclusive province of systems built with Erlang.

Erlang’s Shortcomings

Finally, let’s talk about Erlang’s shortcomings.

The things that make Erlang unpleasant are:

• The learning curve is steep.
• Like JavaScript and Python, it does runtime type checking rather than compile-time type checking.
• The ubiquitous memory copying introduced by immutability drags down the whole system’s efficiency. Doing the same sequential work, Erlang’s performance can even fall short of JavaScript’s.

As for the learning curve, there’s no help for it — who asked you to learn four things at once: Erlang syntax; the functional programming mindset; a VM nearly equivalent to an operating system; and a set of practical design patterns. Yet the price is worth paying: it raises your system-level design ability and your ability to develop distributed software systems up a notch. Even if you never use Erlang, the price is worth it — you can carry many of its ideas into the language you’re familiar with to solve problems.

Erlang is strongly typed, but checking types only at runtime draws plenty of criticism. It’s not that Erlang’s authors didn’t want compile-time checking, but that they had no choice: the dynamism and uncertainty of hot code reload make compile-time type checking impossible. Still, as a language that compiles to Erlang, Elixir’s failure to introduce a type system — the way TypeScript did for JavaScript — is a pity.

As for performance, Erlang sacrifices a great, great deal for soft realtime concurrency and hot code reload. We shouldn’t expect Erlang to take on compute-intensive work with fingers fast as wind and force swift as lightning, the way C does.