Face to Face with Joe Armstrong

Joe, the grand old man, is a totem in the Erlang world.

His sentences from books, his public talks, and his private conversations are quoted everywhere by different people—at this year’s Code Beam, several speakers used Joe’s exact words to support their own views. Joe is regarded as a genius, a hero, a legend because Erlang was born. But Joe himself attributes it to Ericsson’s “magnanimity”: Do you know why Ericsson open-sourced Erlang? Because one day senior management suddenly announced that all projects could only use C++ or Java, and no other language. So we went to the bosses and said, since even we ourselves no longer care for Erlang, let’s just open-source it. The boss said, whatever, I don’t care. That is how Erlang escaped Ericsson’s control and gained a new life.

Thank Ericsson for its stupid decision; otherwise the world would have one fewer such strange and elegant language, and I would not be able to worship an old man half a world away.

If you know nothing about Erlang, you can read my earlier article: erlang.

In that article, I introduced Joe’s worldview:

• everything is a process.
• process are strongly isolated.
• process creation and destruction is a lightweight operation.
• message passing is the only way for processes to interact.
• processes have unique names.
• if you know the name of a process you can send it a message.
• processes share no resources.
• error handling is non-local.
• processes do what they are supposed to do or fail.

All his ideas about Erlang are more or less related to this worldview.

While chatting with Joe, I asked a somewhat adolescent question: “Does Erlang plan to introduce a type system in the future?” I knew that a type system would make hot code reload hard to handle; I also knew it would make the language more complex, but I asked it anyway, for some inexplicable reason. Joe did not answer me directly:

John Hughes (one of the inventors of Haskell) always says when talking about Haskell: think type first. And I say: think concurrency first. This is the essential difference between the two languages. I don’t care about type systems—programming is not about code, it’s about understanding. A good type design reflects the designer’s understanding of the system being designed, but a type system cannot help you better understand the system you are designing. When a person misunderstands the problem they are dealing with, they can write code that compiles normally and that passes tests—because the author’s understanding of the problem is wrong, the test cases they write are wrong too. Everything is so beautiful, yet the software does not solve the problem.

So agonizing over the type system and hoping that it will keep you from making mistakes (or keep an engineering team from making mistakes) is looking for a silver bullet.

think concurrency first. Our world is inherently concurrent/parallel. OOP tries to describe the world in a wrong way, and therefore produces systems that are complex and hard to understand. Take an elevator scheduling system—three elevators stop at their respective floors, waiting for instructions to decide how to operate. If you try to describe the elevators with OOP, setting properties for them and providing methods, you are actually complicating the problem. Three elevators are just three independent processes, each with its own state (the floor it is on). They pass their state to the scheduling system via messages; the scheduling system, according to messages from user input, sends a message to the most suitable process, and then the process decides how to act. If you think concurrency first, then the problem is easier for you to understand and easier to model.

Then Joe delivered the verdict:

if you want type system, use a different language.

Throughout the conversation, Joe spoke fast. Often, while I was still thinking about the previous conclusion he had thrown out, he had already tossed out the next question—making it hard to imagine that he was already 67 years “old.” I asked him whether he was still writing code, and he gave me a very definite answer—he said he was trying to solve some very fundamental problems. When I asked what problems, he smiled slyly: You’ll know after my keynote tomorrow.

My good friend Sun Bo, when talking about daydream travel, often uses two words: imagination and curiosity. Imagination and curiosity are the most precious qualities of the human species. Joe’s voice was soft and searching. He kept throwing out questions, full of curiosity like a child, and like a philosopher he used questions to inspire you to explore the root of things. His angle on problems often surprised me—wild and unconstrained, yet hitting the key point. After talking for a while, he would smile, approachable like the old man next door, with no airs and no worldliness, as if the person sitting across from him were not a kneeling admirer but a long-lost friend across the generations.

As we talked, I gradually relaxed and my questions became casual. I asked him what he thought of this Code Beam. He candidly said he had listened to several talks, and this year’s speakers were no good. I smiled knowingly. He continued—people often ignore the problem when they give a talk and jump straight to the answer. people didn’t really distinguish problem & solution. what’s your problem? why’re you doing this? Before your problem is clearly explained, we cannot resonate; so why should I care about your answer? If you don’t solve the problem, or don’t solve a problem I care about (you need to find a way to make me care), then no matter how glossy the solution is, it is meaningless and worthless. I trembled and broke into a sweat—because my own talk also started by discussing how tubi’s Elixir service is built, deployed, upgraded, and monitored, without talking about what problem we encountered and why we did this; especially when explaining why we made a separate piece of software for releases, it was very abrupt. Fortunately, thanks to this conversation, in my actual talk I spent six or seven minutes making the problem and the why clear.

Following this thread, he said that doing research requires looking at cracks on a marble floor:

We have too many programming languages now—Python and Ruby are so similar, so why do we need to invent two? If we have Java, why do we still need C#? Everyone puts far too much energy into building a better language, but very few languages truly deserve to be called languages with their own ideas. Most languages can be grouped under a few categories. If you view each language as a blackbox, then everyone only tends their own little plot; if you view each machine as a blackbox, then everyone is crowded onto a tiny patch of land fighting, one rising as another falls. We should build things outside the boxes more. Between blackboxes lies a vast territory that no one asks about. How blackboxes are connected, how they communicate and collaborate—there is still a great deal of work to do.

Warming to the topic, he continued to advise me: looking at cracks also depends on the occasion—prioritize the important cracks. He told me a story about Altair—a footnote: Altair was a pioneer of the PC era; the Apple I was invented under its inspiration, and Mr. Gates’s main business before DOS, Basic, also began with Altair Basic. He said that in the late 1970s Altair was already hot among geeks. A guy from the synth scene thought that building a synth tool for Altair would surely be a hit. But after the software was painstakingly built, only a handful of people were interested. So you can make a mark in a small, as-yet-unrecognized niche, but don’t try to make one at the intersection of two niches. The market is too small; you won’t survive the darkness before dawn.

Next I asked Joe what he thought of blockchain.

Joe’s voice instantly rose an octave. what are you talking about? are you referring cryptocurrency? or open ledger? or smart contract? or something else? people usually mixed these things up with “blockchain”

I hurriedly said, let’s first talk about cryptocurrency, or bitcoin. What do you think…

Joe interrupted me: bitcoin is morally wrong. Isn’t it?

For the next ten minutes or so, Joe interrogated bitcoin as if interrogating a war criminal—or, more precisely, interrogated PoW, which meaninglessly wastes resources and warms the planet. Several times I wanted to interject and steer the topic toward technical matters, but Joe flicked me away. Joe was passionate; I was afraid that if I kept insisting, the one roasted over the fire would no longer be PoW but me. After all, the charge of “morally wrong” was an ironclad conclusion, impossible to refute, so I tactfully dropped the subject and quietly became a good listener.

Bitcoin created a world of mess… As a software engineer, our responsibility is to reduce complexity, or to reduce entropy….

I strongly agree with this. Complexity is the natural enemy of software. All the principles, methodologies, paradigms, and patterns we devise are aimed at reducing complexity. I once wrote an article proposing an idea: make deletable code a design goal. That is also a way to reduce complexity and reduce entropy in software.

After the attack on bitcoin’s moral indefensibility, Joe seemed a bit tired. I quickly took the chance to pull the topic back to Erlang—I still had a pile of questions from my friends. I picked a simple one: What do you like most about Erlang?

Joe immediately perked up, eyes bright:

Simplicity. Look at the way we handle concurrency—it is the only solution that completely and yet simply and clearly covers security, isolation, and fault tolerance. The problems we solved in the 1980s are still being exhaustively worked on by everyone today.

I am an old man almost 70. If I fall off a stool, can I recover myself? Obviously not. I can only send a signal (message) to you, and then you help me up—or even better, I don’t have to send it actively. There is a link between us (a link is a kind of connection between Erlang processes), which makes you watch me closely, so that the moment I fall, you immediately receive a visual notification (just like {'EXIT', Pid, Reason}), and then you quickly help me up. This is fault tolerance: simple and clear, no need for esoteric computer knowledge to justify it (yes, I’m talking about you, exception handling); even a child can understand. The reason this solution is so easy to understand is that it is a very natural solution—we live this way every day. Look again: processes can communicate only through messages, so process and process are completely independent (isolation), like individuals. Thus a problem in one process can hardly spread to the whole system, and security is guaranteed as well.

“Absolutely agreed!” I echoed. “And message passing, as a way of doing concurrency, not only works on a single machine but also applies seamlessly to distributed systems. With other approaches, you use one model inside a single machine, but between machines you still need message passing, so you have to switch between two models, which makes reasoning very complex.”

Joe nodded. “If something is too complex, then something must be wrong somewhere—most of the time it is a deviation in understanding the problem.” Then he changed tack:

People are always creating solutions for problems they haven’t thought through. This is a serious problem. Take JSON. what’s the reasoning? I don’t understand why we needed to invent JSON. Maybe the way we exchanged information before JSON was too complex, or too unusable, so it appeared? For a serialization scheme like JSON, human readable is absolutely a false premise. Think about what is transmitted over the network—they are just signals, transmitting bits one by one—so why should we use text rather than binary to save bandwidth and reduce consumption? Human readable makes sense on a screen; on the network, in memory, it is complete nonsense.

After we had talked for over an hour, lunchtime was approaching. Joe gave me a sly smile: You’ve prepared a pile of questions, haven’t you? What else do you want to ask? I want to go out for a walk and stretch my legs.

I looked at the questions I had left:

• What’s the future direction of erlang/OTP?
• How do you see current distributed erlang? Any plan to improve it? for example, will we have gen_raft, or gen_pbft? or replace the full mesh with gossip for distributed erlang?
• How do you compare erlang and elixir?
• How do you do trade off between performance and debugability, giving that OTP put so many efforts to make things traceable?

There were quite a few more, so I picked several to ask him.

Regarding adding more distributed consensus behaviors to Erlang/OTP, such as gen_raft, Joe was indifferent. He said it didn’t matter. Although he had the ability to influence the OTP team, it was their choice. About Elixir, Joe was full of praise. He said he had three children, Erlang was the third, and Elixir was his beloved grandson. Another language on the Erlang VM, LFE (Lisp Flavored Erlang) by his old friend Robert Virding, was less lucky—Joe called it a distant nephew. On performance, Joe’s view has never changed: correctness is 1st, performance is the least thing I considered. This reminded me of something he once said:

Make it work, then make it beautiful, then if you really, really have to, make it fast. 90% of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful!

At the end, I asked Joe for his impressions of China. He said he liked China very much, liked Beijing very much, and even showed me a photo of him and his wife in front of Tiananmen. I said, then when do you plan to visit China again? He laughed: That depends on when you invite me to China for an Erlang conference.

Postscript

Joe passed away on April 20, 2019. I have always wanted to write something to commemorate him, but opening these articles one by one, reviewing his worldview and his imperishable ideas, and passing them on—I feel this is the best way to remember him.