In the blink of an eye it’s already July. It’s been more than a month since I last updated my WeChat public account. It’s also been two months since I joined Arcblock. If we think of a person as running software, then in these two months I’ve already gone through several iterations—like Goku in Dragon Ball, relentlessly training aboard the spaceship bound for Namek.

Growth has its triggers, and they come down to two cases: being mentored by others, or achieving self-enlightenment. Lü Meng was originally a rough military man; after Sun Quan urged him to study, he applied himself with resolve. Once, when Lu Su discussed affairs with him, he was so astonished that the saying arose: “After three days apart, one must look at a person with new eyes.”—that is being guided by another. In my case, because I moved to a new environment and built anew after tearing down the old, my past accumulation and thinking found space and objects to apply themselves to. On top of that, alone in Seattle with little else and plenty to do, I threw all my time into work—thinking more, and doing even more—so my abilities rose accordingly—that is self-enlightenment.

Confucius said that at thirty one stands firm. Stands on what? On the three immortals—establishing virtue, establishing merit, establishing words? Or the more modest trio: establishing oneself, establishing a family, establishing a career? Everyone on the road toward forty has their own interpretation and reflection. But what is beyond doubt is that past thirty, one must gradually form one’s own thinking and methodology. My previous article Code is Law, a set of conventions I defined for Arcblock’s GitHub repos, is the product of my own thinking and methodology—you won’t find anything like it in any existing public documentation. Is it perfect? No, certainly not; after using it for a while we’ve already gained some new insights. Is it unique? One of a kind, and valuable.

In God Said: Let There Be a Language for the Future, and There Was Erlang, I quoted the worldview of Erlang that old man Joe mentioned in his doctoral thesis:

• everything is a process.
• process are strongly isolated.
• process creation and destruction is a lightweight operation.
• message passing is the only way for processes to interact.
• processes have unique names.
• if you know the name of a process you can send it a message.
• processes share no resources.
• error handling is non-local.
• processes do what they are supposed to do or fail.

In Joe’s eyes, Erlang really isn’t mysterious at all—just six functions capture its entirety: spawn, send, receive, register, whereis, self.

(Thanks to Xiao Shan for contributing the old man’s handwritten explanation.)

Think about it carefully: it’s terrifyingly simple, like the grand unified theory in physics, attempting to leap out of complexity and return to first principles. More terrifying still, these six functions don’t just cover Erlang; they seem able to explain many systems in the software domain—they’re everywhere, and within a system they carry the same significance as the Force does in Star Wars.

• spawn: create a resource. For Erlang, this resource is a process; for a service, it’s the service itself.
• send / receive: send instructions to a resource and receive instructions. For Erlang, the instruction is a message, packaged as an Erlang term, traveling over IPC/RPC; for an HTTP service, the instruction is a request, packaged as JSON / msgpack / protobuf, traveling over HTTP / HTTP2.
• register / whereis: how a resource is registered and discovered. For Erlang, this is a process registering with and being discovered via the name register; for a service, it could be Consul / DNS.
• self: returns one’s own identity. In Erlang, this is how a process finds itself; in the microservice scenario, every service implicitly has its own identity.

I love Joe the same way I love Rich Hickey: while spreading knowledge, they also spread their own unique understanding of and thinking about things.

Back to my own reflections on the processes and methods of getting things done. Beneath those surface-level methods, there actually lies an important consideration: how to let the team communicate and collaborate at low cost. My approach is: convention by configuration. Once you understand the structure of a repo and know what make init, make build, make run, make create-pr do, you naturally understand the operating rules of every repo at Arcblock—whether it’s Elixir, Node.js, or Python. And regardless of whether Elixir uses asdf, Node.js uses nvm, or Python uses virtualenv, a single make init builds the entire environment for you, after which you can safely make build and make run. A newly onboarded engineer doesn’t need to come ask me (or anyone else) how to initialize this repo or get it running—which saves us a great deal of time; nor does he have to labor through a long “Installation Notes” chapter in README.md just to get it running—which saves him a great deal of time too.

This is just the tip of the iceberg. We’ve done a lot of other work to reduce the cost of team communication and collaboration. The time and energy we save everyone can be spent on more important things. Accumulated bit by bit, this ultimately has the chance to translate into one-fold or several-fold productivity, forming a competitive advantage.

Our Open Chain Access Protocol (OCAP) chose GraphQL rather than REST for its API layer for similar reasons. For developers, there’s some learning curve at first, but once they’ve adapted, whether we provide 1 API or 100, whether we support one chain or several, the cost to the user is roughly the same. With REST APIs, on the other hand, learning 100 differently shaped endpoints would be a nightmare for developers. For us, laying the foundation in the early days takes a lot of time, but afterward, behind all the varied queries, it’s just combinations of a few basic resolvers.

“How to let the team communicate and collaborate at low cost” has been an important line of thinking for me over the past two months, and it’s the sudden release of years of accumulated knowledge and experience.

Another line of thinking I’m still groping through these two months is: “How do we build our services and their ecosystem in a more advanced and efficient way?”

Arcblock is currently a small team; even if the R&D team grows to dozens of people, it’s still small. Given the scope of what we want to do, if we can’t find a more effective way to develop, we’ll be exhausted and progress will be slow. Take OCAP: building a set of APIs for developers is not as simple as just the API and the service behind it. An API needs documentation, an SDK, a definition of the API interface, and the service supporting that interface. Much of this is repetitive work: the API doc, the API interface, and the SDK all keep repeating similar content and code. When we change an API’s definition, moving one hair affects the whole body—several places need to be modified, and these are all very mechanical actions. So we need to find a way to strike from a higher dimension.

At Tubi, the UAPI system I built integrated the API and its documentation, so it could be defined once and take effect in two places, saving everyone time. For OCAP, we went a step further and tried to define the problem like this:

1. Define a “language” to describe our APIs.
2. Write parsers (code generators) for different directions, converting them into code for specific scenarios.
3. Build the parsers into the build pipeline, so a single build can generate all kinds of results.
4. The generated results must be easy to extend and integrate with other parts of the system.

We tentatively call the language we defined AADL (Arcblock API Description Language). To make it easy for anyone to write and understand (e.g., a product manager can define it easily too), we used the YAML format. For example, the RichestAccounts query is defined as:

- name: RichestAccounts
  doc: Returns richest accounts, order by balance.
  return: PagedAccounts
  params:
    - name: paging
      type: PagingInput
      doc: Describes which page of data to return.

And the data structures used here are defined like this:

- name: PagedAccounts
  doc: Represents paginated Accounts.
  properties:
    - name: page
      type: PageInfo
      doc: The pagination information.
    - name: data
      type: [Account]
      doc: The paged Account data.

Through this definition, we generate:

• Slate-style API documentation (github.com/lord/slate)
• Absinthe’s GraphQL query schema definitions (Absinthe is a GraphQL library for Elixir)
• Absinthe’s GraphQL type notation definitions
• Ecto’s DB repo definitions
• Ecto’s DB schema definitions
• Ecto’s DB migration definitions
• SDKs in various languages (e.g., Node.js, Python, Go, etc.—still in planning, not yet started)

Then, in a build pipeline, all the code is generated. For example, the generated Absinthe query looks like this:

(
  @desc "Returns richest accounts, order by balance."
  field(:richest_accounts, :paged_bitcoin_accounts) do
    [
      (
        @desc "Describes which page of data to return."
        arg(:paging, :paging_input)
      )
    ]

    resolve(fn parent, args, resolution ->
      apply(Resolver, :paged_bitcoin_accounts, [parent, args, resolution])
    end)
  end
)

The code generated above satisfies the constraint mentioned earlier: “the generated results must be easy to extend and integrate with other parts of the system.” In this example, the programmer only needs to further write the Resolver.paged_bitcoin_accounts function.

At present this flow is still experimental; our online service, the OCAP playground, runs on generated code. We wrote about 3,500 lines of Elixir and 1,000 lines of YAML ourselves, and it generated 1,500 lines of Elixir code (Elixir supports Macros, so we generate these source files just to make it easier to troubleshoot).

Although there are still many problems, the biggest benefit of this system is that during development we can freely adjust the API structure without having to painstakingly modify code in many places every time. When we don’t yet have a good definition of many APIs’ behavior, this is a tremendous blessing. And later, when we need to add new APIs at scale, we’ll be able to support them very quickly.

This is currently our answer to “How do we build our services and their ecosystem in a more advanced and efficient way?” It’s still a hundred thousand miles from perfect, but it’s ready to use. Often, asking the right question is more meaningful than finding the right answer. A good question is like groping for the exit in a dark tunnel when your hand suddenly finds a flashlight that instantly lights up the whole journey.

I’ll stop here for now—I hope it sparks your thinking and your questions.

Building services with Elixir is actually another answer. The benefit of Elixir is that you can at any time pull out some piece of code and organize it into a standalone GenServer, much like a microservice. The Richest Accounts query above is actually very well suited to being pulled out into its own GenServer that periodically fetches data from the database and caches it, greatly reducing the pressure on the DB.

unfair advantage

mission

Often, time slips away quietly while we stand still: doing the same thing year after year…

Either we receive guidance from an elder, like Linghu Chong and Feng Qingyang, or Zhang Wuji and the white ape; or we suddenly become enlightened on our own.

My previous article Code is Law, expounded some of my reflections on methods and principles for getting things done. In fact, what method is…

Haven’t updated for a month and a half.

I think that even the most steadfast reader…

Dear readers and friends, I’ve missed you so much~

A couple of days ago, a design veteran—Rokey, founder of EICO—came to Arcblock and gave us a fantastic BBL (brown-bag lunch) session. It was a wake-up call that made me rediscover design. For example, he said:

The starting point of business is assets, which then become models, then form brands, reach users, and finally convert into new assets.

A product is a container: it satisfies experience, carries business, supports philosophy, and transmits perception.

What struck me most was when he said: a team’s ability to collaborate at low cost is a company’s greatest competitive advantage. I couldn’t agree more.

Scenarios and business

Growth

A product is a container that carries user value

Redefining engineers

Architecture: people and environment, customization Industry: people and goods, mass production, assembly lines Digital: people and information, infinite reproduction, zero cost + viral spread, cross-industry collaboration

Separate principle from structure; design and product can be decomposed

Competitors may come from the other side of the mountain, not necessarily from your industry chain

Solve the problem of effectiveness amid information overload

What is a digital product? A starting point and a process. But not about endpoints and results. The endpoint is death. Products have no boundaries. You need to find dynamic stability, a foothold. Pivot.

Designers need a definition of a sense of achievement—a living product

At a new point in time, how can I help him

Low-cost team collaboration is a company’s greatest competitive advantage

Throw in a requirement, pick a few people, and you get an 80-point product

      Users
 Cognition |  Experience
  Brand   -+-   Assets
 Philosophy |  Business
      Model

The starting point is assets, which become models, then form brands, reach users, and finally convert into new assets.

The core is the product, not the brand.

The big eye model

Product: is a container

Satisfies experience Carries business Supports philosophy Transmits perception

What does business want?

Digitizing the business: healthcare

Making digital a business: blockchain

Business is a burger: the top layer is digital, the bottom is physical, and workflows string the top and bottom together in between.

Design is what strings together business, humanities, and technology.

Business design Technology design

Design is a means.

Youjian (a barbershop chain)—simple at the touchpoint, complex behind it.

Conflict of interest,

Foundation design: aesthetics management and usability management Content design: quality management—information operations and process standards Decision design: direction management—cognitive positioning, principles, mechanisms

Thinking in Tools

In Hackers & Painters, Paul Graham says: the profession of the programmer is a perfect profession because it combines 1) measurability and 2) scalability. We often say that one “excellent” programmer can be worth three to five, or even dozens of “mediocre” programmers. That’s both right and wrong. “Excellent” and “mediocre” are both relative; as the saying goes, “one man’s honey is another man’s poison.” Put Forrest Gump in the right place and he’d hardly be un-excellent.

More precisely, an excellent programmer should be able to carry a “mediocre” team, raising its efficiency to three to five, or even dozens of times what it was; or an excellent programmer should be able to reasonably seek solutions and carry himself, so that his efficiency at a given task equals three to five, or even dozens, of his former selves.

Here, thinking in tools plays a decisive role. The following are some principles of thinking in tools that I’ve gradually distilled over more than a decade of work, for your reference.

R1: For code not on the critical path, as long as the cost-benefit is reasonable, if you can use a third-party (paid) tool, never build it yourself.

This principle is not a denial of the value of reinventing the wheel. As an engineer, I myself love building wheels. In our world, ever-evolving wheels are the bedrock that pushes the software industry forward. Without PHP, we’d still be using CGI; without Rails, developing complex web apps would have remained the exclusive domain of large teams (Java/.NET); and without Chrome and Firefox, IE would be the nightmare of all humanity.

In fact, everything we do is building wheels—building a better wheel on the path of your core competitiveness. So at Tubi, our wheels were the transcoding engine and the ad server; at Arcblock, our wheels are the Open Chain Access Protocol, the blocklet, and the Arcblock utility chain. Wheels not on the critical path should not be built casually unless necessary—this is especially important for small teams. The reason is simple: “Unlike you young masters, just staying alive takes everything we’ve got.”

To follow this principle, you first need to clarify what is the critical path and what is not, then find the non-critical paths within the critical path, and keep drilling down. This analysis is sometimes obvious and sometimes quite headache-inducing. Moreover, the critical path changes dynamically: something not so important right now, something you can trade off, may in the future lie right across the critical path.

For example, for Arcblock, developing a programming language is clearly not on the critical path, and developing a monitoring system surely isn’t either. What about building a scalable / robust / secure infrastructure? Building an ETL pipeline? Indexing the data of existing main chains? Writing an API gateway? You have to think carefully about all of these—which are worth doing and which are not.

Once you’ve identified the services that aren’t on the critical path but are still very important, the next step is choosing what tool to use. Whether to use an open-source solution or a commercial tool, the focus comes down to one word: cost-benefit. Note it’s cost-benefit, not price. Many people are scared off by the price of commercial tools and lose their judgment about cost-benefit. Especially when the decision-maker is a programmer—the moment he sees the price he can’t help but think: damn it, I could build this in a little time, and they want this much for it! Are they out of their minds!

So don’t do business with programmers. Even if you must, try to put the purchasing decision in the hands of the non-coding boss—or the boss’s wife—above the programmer.

The formula I use to determine cost-benefit is:

CPR = (development cost + deployment cost + maintenance cost + training cost + usage cost) / benefit

Here, the benefit of a particular tool (like a monitoring system) can be treated almost as a constant, so what you need to lower are the various costs. If you want to build a system roughly like Datadog, you’d need to set up Grafana / InfluxDB yourself (factoring in Terraform / Ansible deployment scripts and the learning time to set it up, that’s at least a week of one engineer’s work), servers on AWS have a monthly cost, data storage and backup have costs, plus there’s the DevOps maintenance cost (say 6 hours a month spent on it), and if you integrate with other tools there may be extra usage costs (like building all those dashboards), and so on.

If we figure a DevOps engineer at an annual salary of $150k (multiply the total outlay the company pays by a factor of 1.2, so $180k), the initial development cost is 15k / 4 = $3.8k, which amortized over the first year is $300 a month; the monthly AWS bill is roughly $600 at the start (c5.2xlarge * 2 + disk); the monthly DevOps maintenance cost is 15k / 160 * 6 = $560. The first-year TCO comes out to nearly $1,500 a month.

What does $1,500 mean? Datadog monitoring is $15/host/month, which can handle 100 servers’ worth of monitoring. When you have 100 servers with a large volume of metrics to monitor, your on-premise Grafana / InfluxDB may again be insufficient, requiring more compute and storage resources and more DevOps time to handle—which maps to more cost.

So, roughly calculated, if you’re managing fewer than 150 servers, or even fewer than 200, the CPR of a self-built monitoring platform probably doesn’t pay off.

R2: If you can afford the time to lay the foundation, by all means lay the foundation first.

Internet software development is interlocking. The part exposed to users is only the tip of the iceberg above the water. To expose that tip, you can toss a block of ice into the sea, or you can build the iceberg from the bottom up, bit by bit. Though what users see is roughly the same, the block of ice will slowly melt while the iceberg stands firm. Often, after we quickly throw together a service and enjoy the flowers and applause, we should ask ourselves: is this a castle in the air? Can I (and the team) keep building more features on top of it, or will we soon have to tear it down and start over?

When Laozi talks about “those skilled in the way,” he poses a wonderful rhetorical question: Who can, from turbidity, through stillness, gradually become clear? Who can, from stability, through stillness, gradually give rise to life? Most of the time our work is like a cup of muddy water mixed with sand, and it requires our hearts to be “calm” and “still,” patiently untangling the threads from the complexity and muddiness—“reach the utmost emptiness, hold fast to stillness.”

The work of laying the foundation looks like it drags down the whole process, but in fact it’s an enabler that pays off later. You can spend ten minutes clicking together a set of resources in the AWS console (VPC, EC2, security groups, EBS, ALB, …) to provide some service—but the same thing may need to be repeated over and over. Yet spending time building an automation tool for it might take a week of a very skilled, Terraform-savvy engineer. To do it or not—that is the question.

My answer is that unless the house is on fire, you must do this foundational automation work. The gradual efficiency gains are secondary; the important thing is that it greatly reduces the team’s chances of making mistakes. Humans are the species in the universe least suited to mechanical actions—the same character copied ten times comes out in ten different shapes, and repeating a wizard with a dozen steps ten times (which is why Taylor emphasized standardized “scientific management” on the assembly line) multiplies the probability of error. So we must produce and use tools, using this foundational work to reduce the probability of mistakes. Of course, before building, let yourself be guided by the principle of R1 and pick the most suitable tools to build these foundational tools.

In my month at Arcblock, 70% of my coding time went into building infrastructure—thousands of lines of Terraform, Ansible, and Packer code—for exactly this reason.

For example. We all know that “security” and “ease of use” are often at odds. When we pursue maximum security, it inevitably brings great inconvenience. Everyone knows passwords should be updated regularly, but think about your production database password—how long has it been since you changed it? Everyone knows MFA (two-factor authentication) maximizes the security of important functions, but how many people apply it to AWS access keys? It’s not that we don’t want to, but that extreme inconvenience leads us to ignore security best practices.

If such “foundations” aren’t built well from the start, they’ll bring endless trouble down the road. So we use tools to make up, as much as possible, for the usability lost in the pursuit of security:

This command eases the pain MFA brings: the engineer runs it once, and his shell environment holds the temporary access key and session token generated after two-factor authentication, so he can use the full power of the AWS API freely for a whole day.

? Build an arcli ssh:ec2 to automatically find … online

We now have our own VPN server, letting employees dial in to work inside the protected cloud (e.g., to access private services); we have a bastion server and related tools (in the future there will be an arcli ssh:ec2 to automatically discover and log into machines in the private network), balancing the security and convenience of SSH; we have a series of Ansible + Packer scripts that bake common services like nginx, elixir-service, consul, etc. into ready-to-go AMIs.

Recently I spent two days building an Ethereum indexer, and a day and a half of it went into writing a DSL that reads configuration from YAML and generates the corresponding code,

When we do things, we should be careful not to get flustered and lose our sense of proportion.

Confucius also once said: Do not seek speed; do not covet small gains.

R3: Less is more.

Another example. When processing Ethereum data, we need to do various transformations, mapping between source data and target data. We can define this mapping by writing code line by line, but a better approach is…

R4: Data is value, always.

Rich Hickey said (damn it, I’ve quoted this line countless times): You have to have at least 2 solutions to make a trade-off.

Imagine an F1 race—the driver’s pit-stop strategy.

Less brings more; more brings confusion.

The greatest vessel is the last to be completed.

Not knowing the constant, one acts recklessly and meets misfortune.

“Reach the utmost emptiness, hold fast to stillness. Who can, from turbidity, through stillness, gradually become clear? Who can, from stability, through motion, gradually give rise to life?”

Recently I’ve been reading Nan Huaijin’s “Laozi Ta Shuo” (Laozi, As He Said), and what left the deepest impression on me was his explanation of these two lines. I personally think these two lines can be used as a mantra—like a Buddhist gatha—to recite when you’re feeling agitated.

“Reach the utmost emptiness, hold fast to stillness.” This is a highest state advocated by Daoism. Daoism holds that a person should be “empty”—that is, harbor no preconceptions about anything, like an empty vessel; only by keeping oneself in an empty, void state at all times can one be used to hold things and play a great role. But how can ordinary people manage this? We habitually use existing knowledge to reject or ignore other viewpoints; only those who’ve cultivated to the extreme—those who, from the bottom of their hearts, feel they are lacking—can take in more knowledge. Many great scholars appear extremely humble, and that, I suppose, is a state of reaching the utmost emptiness. The next profound state is “stillness”—neither impatient nor rash, neither servile nor overbearing toward anything. Perhaps when a person is “empty” to the extreme, he can naturally calm down and handle anything: in matters of learning, humbly and quietly listening to others’ views, calmly viewing honor and disgrace, gain and loss. When we’re restless, when we’re dejected, ask yourself: am I full or empty? Am I already an expert in some professional field? Am I really more remarkable than so-and-so? Try reciting this mantra and savor the state it points to—it might help.

“Who can, from turbidity, through stillness, gradually become clear? Who can, from stability, through motion, gradually give rise to life?” This line is full of positive, upward meaning and carries strong dialectical power. Laozi’s attitude toward handling affairs is “turbidity.” He doesn’t advocate the attitude of “the whole world is muddy, I alone am clear,” but rather “turbid, oh, like muddy water” and “soften your glare, be one with the dust”—that is, don’t think you’re so remarkable; everyone’s behavior has its reasons, and we should understand and accept them, being like them: seemingly a bit muddled on the surface, but “still” inside—like a cup of Yellow River water full of sand, which can only slowly become clear in a quiet state. In other words, clarify your thoughts from the complex “turbidity” in your mind and become lucid. The latter half is very positive: it’s not enough to just quietly figure things out; you also need to get moving, give back to society and others, and play your positive role. But don’t move recklessly—first you need stability, that is, a steady mindset; if you rush about in a panic, you’ll become truly turbid and lose track of the situation. Second, “gradually give rise to life”—that is, take it slowly; doing things too hastily usually fails.

Laozi’s “stillness” and “emptiness” share the same wonderful effect as Buddhism’s “void,” and are worth savoring carefully.

Redefining Software Engineering

I once mentioned in an article that you shouldn’t let the language you use limit your abilities—programmers always call themselves Java programmers or Elixir programmers, but have you ever heard of a doctor calling himself a tetracycline doctor?

I’ve also said on many occasions that a programmer should have end-to-end development ability—from an idea all the way through to release and deployment, plus online monitoring, alerting, and post-hoc analysis. If knowing how to write code is like graduating from elementary school, then once you’ve built end-to-end development ability you can call yourself a graduate student—you don’t have to rely on others to pave the road and build the bridges for you, and you don’t even need others to tell you what to do, because you can already find problems yourself, not just solve them.

When I talk about this, my starting points are: 1) build a matrix of development abilities rather than being confined to one point; 2) shorten the feedback loop of the whole process as much as possible.

On building a matrix of development abilities, I believe the great majority will agree; but shortening the feedback loop of the whole process is not so obvious. For knowledge workers, if we think of each mind as a thread, the greatest waste of their time and ability lies in the process of synchronization. Synchronization kills efficiency, and it also kills creativity. When you’ve finished developing a feature, if you then have to hand deployment over to DevOps, there must be communication between you—a classic “stop-and-wait” process.

However, I haven’t clearly described the why—why building a matrix of development abilities matters? Why use end-to-end building ability to shorten the feedback loop? And why a shorter feedback loop matters.

The “stop-and-wait” process, or rather…

—If we think of a person as a versioned product: if my feedback loop is a week and yours is three months, then I evolve 52 versions in a year while you have only 4, and the difference in growth speed is immediately apparent.