Thinking in Tools

Paul Graham observes in Hackers & Painters that programming is a perfect profession because it combines 1) measurability and 2) leverage. We often say that one “excellent” programmer can be worth three to five, or even dozens of “mediocre” programmers. That statement is both right and wrong. “Excellent” and “mediocre” are relative; as the saying goes, one man’s meat is another man’s poison. Put Forrest Gump in the right place, and even he can be called excellent.

To be precise, an excellent programmer should be able to carry a “mediocre” team and make it three to five times, or even dozens of times, more efficient than before; or an excellent programmer should be able to seek out sensible solutions and carry themselves, doing a given task as efficiently as three to five, or even dozens of, former selves.

Here, thinking in tools plays a decisive role. Below are some principles of thinking in tools that I have gradually distilled over more than a decade of work, for your reference.

R1: For code not on the critical path, as long as the cost-performance ratio is reasonable and a third-party (paid) tool can do the job, never build it yourself

This principle does not deny the value of reinventing wheels. As an engineer, I myself quite enjoy building wheels. In our world, constantly replaced wheels are the cornerstone that drives the software industry forward. Without PHP, we would still be using CGI; without Rails, developing complex web apps would be the exclusive preserve of large teams (Java/.NET); and without Chrome and Firefox, IE would be the nightmare of all humanity.

In fact, everything we do is building a wheel—a better wheel on the path of your core competitiveness. Thus at Tubi, our wheels are the transcoding engine and the ad server; at ArcBlock, our wheels are the open chain access protocol, Blocklet, and the ArcBlock utility chain. Wheels not on the critical path should not be built lightly unless necessary—this is especially important for small teams. The reason is simple: “Unlike you young masters, just staying alive takes everything we have.”

To follow this principle, you first need to clarify what is the critical path and what is not, then find the non-critical paths inside the critical path, and keep refining all the way down. This analytical process is sometimes obvious and sometimes quite headache-inducing. Moreover, the critical path changes dynamically: what is unimportant today, what can be traded off today, may in the future lie squarely across the critical path.

For example, for ArcBlock, developing a programming language is clearly not on the critical path, and developing a monitoring system is certainly not on the critical path either. What about building a scalable / robust / secure infrastructure? Building an ETL pipeline? Indexing data from existing main chains? Writing an API gateway? All of these need careful thought: what can be done, and what cannot.

After identifying those services that are not on the critical path but are still very important, the next step is choosing which tools to use. Whether to use an open-source solution or a commercial tool, the focus comes down to one phrase: cost-performance ratio. Note: cost-performance ratio, not price. Many people are scared off by the price of commercial tools and lose their ability to judge value. This is especially true when the decision-maker is a programmer: the moment they see the price, they cannot help thinking, “Damn it, I could build this in a bit of time, and they want that much? Are they out of their minds?”

So do not do business selling to programmers. If you must sell to programmers, try to put the purchasing decision in the hands of the non-coding boss or boss’s wife above the programmer.

The formula I use to determine cost-performance ratio is:

CPR = (development cost + deployment cost + maintenance cost + training cost + usage cost) / benefit

Here, the benefit of a particular tool (such as a monitoring system) can be treated almost as a constant, so what needs to be reduced are the various costs. If you want to build a system roughly equivalent to Datadog, you need to set up Grafana / InfluxDB yourself (including Terraform / Ansible deployment scripts and the learning time to set them up—at least one engineer-week of work), pay monthly costs for servers on AWS, pay for data storage and backup, and then pay DevOps maintenance costs (assuming six hours a month spent on it). If it integrates with other tools, there may be additional usage costs (such as building various dashboards), and so on.

If we calculate based on a DevOps engineer with a $150k annual salary (multiply all company costs by a 1.2 coefficient, which makes $180k), the initial development cost of 15k / 4 = 3.8k, amortized over the first year, is $300 per month; the AWS bill is initially about $600 per month (c5.2xlarge × 2 + disk); the monthly DevOps maintenance cost is 15k / 160 × 6 = $560. All told, the first-year TCO is close to $1,500 per month. What does $1,500 mean? Datadog monitoring costs $15/host/month, which can handle monitoring for 100 servers. When you have a large volume of metrics from 100 servers to monitor, your on-premises Grafana / InfluxDB deployment may no longer be enough, requiring more compute and storage resources and more DevOps time, which in turn corresponds to more cost.

So, roughly calculated, if you manage fewer than 150 servers—or even fewer than 200—building your own monitoring platform may not have a favorable CPR.

R2: If you can afford the time to lay a foundation, you must lay the foundation first

Internet software development is interlocking. The part exposed to users is only the tip of the iceberg above the water. To reveal that tip, you can throw an ice cube into the sea, or you can build the iceberg bit by bit from the bottom up. The user sees roughly the same thing, but the ice cube will slowly melt, while the iceberg is extremely solid. Often, after we quickly throw together a service and enjoy the flowers and applause, we need to ask ourselves: Is this a castle in the air? Can I (and the team) continue building more features on top of it, or will we soon have to tear it down and start over?

Laozi, when speaking of “those skilled in the Way,” has a wonderful rhetorical question: Who can make the turbid, through stillness, slowly clear? Who can make the settled, through stillness, slowly come to life? Most of the time our work is like a cup of muddy water mixed with silt; it requires our hearts to be “level” and “still,” patiently sorting out the threads from the complex and murky, “reaching the utmost emptiness, holding fast to stillness.”

Foundation-laying work seems to drag down the whole process, but in reality it is an enabler that accumulates strength for a breakthrough. You can spend ten minutes clicking out a set of resources in the AWS console (VPC, EC2, security groups, EBS, ALB, …) to provide a service; however, you may need to repeat the same thing again and again. If you spend the time building an automation tool for it, it may take a highly skilled engineer proficient in Terraform a week. To do it or not: that is the question.

My answer is that unless your eyebrows are on fire, you must do this basic automation work. Improved efficiency is secondary; more importantly, it greatly reduces the team’s chances of making mistakes. Human beings are the worst species in the universe at mechanical repetition: copy the same character ten times and you get ten different shapes; repeat a wizard with a dozen steps ten times (which is why Taylor emphasized standardized “scientific management” on the assembly line), and the probability of error increases greatly. So we must produce and use tools, and through this foundational work reduce the probability of error. Of course, before building, use the principle of R1 to guide yourself and choose the most suitable tools to build these foundational tools.

During my first month at ArcBlock, I put 70% of my coding time into building infrastructure, writing several thousand lines of Terraform, Ansible, and Packer code. That is the reason.

For example. We all know that “security” and “usability” are often in conflict. When we pursue extreme security, we inevitably bring great inconvenience. For example, everyone knows passwords should be updated regularly, but think about the password for your production database—how long has it been since you changed it? Everyone knows MFA (two-step verification) can maximize the security of important functions, but how many people apply it to AWS access keys? It is not that we do not want to use it; it is that the extreme inconvenience makes us choose to ignore security best practices.

If such a “foundation” cannot be built well from the start, it will bring endless trouble later. So we must use tools to compensate as much as possible for the usability lost in the pursuit of security:

This command simplifies the pain brought by MFA. The engineer only needs to run it once, and their shell environment stores the temporary access key and session token generated after two-step verification, so they can use AWS API capabilities without hindrance for a day.

R3: Less is more

Let me give another example. When processing Ethereum data, we need to perform various transformations and map between source data and target data. We can define this mapping by writing code line by line, but a better approach is

R4: Data Is Value—Always

Rich Hickey said (damn, I have quoted this line countless times): You have to have at least 2 solutions to make a trade-off.

Imagine an F1 race—a driver’s pit-stop strategy

Less is more; more is confusion.

The greatest vessels are completed last.

Not knowing the constant, one acts recklessly and meets misfortune.

Reach the utmost emptiness; hold fast to stillness. Who can make the turbid, through stillness, slowly clear? Who can make the settled, through movement, slowly come to life?

Recently I have been reading Nan Huaijin’s Laozi Ta Shuo (Laozi, as I Say), and what impressed me most was his explanation of these two lines. I personally think these two lines can be used as a mantra, like a Buddhist gatha, to be recited when the mind is troubled.

“Reach the utmost emptiness; hold fast to stillness.” This is a highest state advocated by Daoism. Daoism holds that a person should be “empty,” that is, free of preconceptions about anything, like an empty vessel: only by constantly remaining in an empty, void state can one be used to hold things and play a great role. But how can ordinary people do this? We always habitually use existing knowledge to reject or ignore other viewpoints. Only those who have cultivated themselves to the utmost—those who believe from the bottom of their hearts that they are lacking—can possibly absorb more knowledge. Many great scholars appear extremely humble; this is probably a state of reaching the utmost emptiness. The next profound state is “stillness”: treating everything without impatience or arrogance, without self-abasement or self-importance. Perhaps when a person is “empty” to the extreme, they can naturally calm down and handle anything: in scholarship, quietly listening to others’ viewpoints with a humble attitude; calmly viewing honor and disgrace, gain and loss, and so on. When we are restless, when we are depressed, ask ourselves: Am I full or empty? Am I already an expert in some professional field? Am I really more remarkable than so-and-so? Try reciting this mantra and savor the state it points to; perhaps it will help.

“Who can make the turbid, through stillness, slowly clear? Who can make the settled, through movement, slowly come to life?” This line is full of positive, uplifting meaning and at the same time has strong dialectical power. Laozi’s preferred attitude toward handling affairs is “turbid.” He does not advocate the attitude of “when the whole world is turbid, I alone am clear”; rather, he advocates “muddled, as if turbid,” “harmonize with its light, merge with its dust”—that is, do not think oneself extraordinary. Everyone’s behavior has its reasons; we should understand and accept them, and be like them, appearing a little muddled on the surface, but “still” inside, like a cup of Yellow River water containing sand: only in a quiet state can it slowly become clear—that is, the mind sorts out the threads from the complex “turbidity” and becomes clear-headed. The latter sentence is very positive: it is not enough merely to have quietly figured things out; one must also move, give back to society and others, and play an active role. But do not move recklessly. First, be “settled,” that is, have a steady state of mind. If you act in a panic, you will become truly turbid and unable to understand the situation. Second, “slowly come to life,” that is, take it slowly; if you are too anxious, you often fail.

Laozi’s “stillness” and “emptiness” have a wonderful resonance with Buddhism’s “emptiness,” and are worth savoring carefully.