The ABT Network Deployment Chronicle

Preface

Two days ago—March 28—the ABT network was officially released. By connecting and weaving multiple blockchains into a network in a fully decentralized manner, and by using cloud nodes and woven chains as its mesh, ABT network redefines the next-generation blockchain infrastructure. This article recounts some of the interesting experiences that occurred before ABT network came into being.

For ease of reading, let me first briefly introduce a few concepts discussed in this article:

• ABT network: a network formed by multiple blockchains.
• ABT chain node: blockchain node software built by ArcBlock using the Forge framework.
• Forge framework: a framework ArcBlock built for blockchain development; think of it as Ruby on Rails for the blockchain world.

Making Nodes Deployable by Anyone

While developing the Forge framework and ABT chain node, we held a deep conviction: a blockchain project running on Forge could be highbrow or lowbrow, a large application handling millions of transactions a day or a solo developer entertaining themselves and a small circle of friends. Therefore, as long as a node has decent compute, it can run Forge—so anyone willing can deploy a node they can afford.

So what counts as ‘decent compute’? Considering app developers’ financial capacity during development, we set the target at a monthly cost of under $15 per node. On Digital Ocean, that corresponds to:

That is, bargain-basement cloud hosts ranging from 1GB / 1 CPU / 25GB disk all the way up to 2GB / 2 CPU / 60GB disk.

During most of our development time in January and February this year, we were using the $5 supreme bare-bones host—and we deployed four nodes in one go across the US West (SF), US East (NY), Western Europe (London), and Southeast Asia (Singapore), forming a p2p network for Forge development. We believe that an extremely harsh environment can forge software that is as robust as possible, exposing every kind of problem early.

With the environment in place, we needed enough traffic to simulate real application scenarios. To that end, we built a simulator and a simple description language for specifying how simulations are launched (excerpt):

pools:
  account_migrate: 5
  create_asset: 5
  declare: 5
  exchange: 5
  transfer: 10
  update_asset: 5
  consume_asset: 5
  poke: 5
meta:
  tick: 500
simulations:
  - name: exchange token and assets
    interval: 2
    num: 2
    type: exchange
    settings:
      value: "1000..20000"

  - name: transfer token and assets
    interval: 5
    num: 2
    type: transfer
    settings:
      value: "1000..5000"
      after:
        - interval: 1
          action: consume_asset

By changing the pool size, we can adjust the degree of concurrency; by controlling the tick, we control the traffic rate; and by adding more simulations, we vary the diversity of the traffic.

Under the simulator, for a long stretch our development network crashed every other day—sometimes out of memory, sometimes too many open files, sometimes gen_server timeout, sometimes TCP send/receive buffer full. On a host with 4GB memory / 4 CPU / 100GB disk, these problems would have only a small chance of surfacing; we deliberately made them happen in the development environment, allowing most of them to be properly addressed. For example, we found that the consensus engine we were using was unstable and crashed from time to time; after a crash, it easily corrupted the state db, causing the node to collapse completely and be unrecoverable. Our approach was: whenever the consensus engine crashed, we let Forge crash automatically (sacrificing the Erlang VM’s powerful crash-recovery mechanism), and then our forge starter would restart Forge. After restart, we rolled back to the data from the previous block and reapplied it. If the consensus engine could recover, we continued forward; otherwise, we kept crashing and kept rolling back.

In this harsh environment, Forge gradually matured. The network of supreme bare-bones nodes kept dying and being reborn, like Tom Cruise in Edge of Tomorrow, growing from a rookie into an indestructible survivor and welcoming its one-millionth transaction.

The good times did not last long. At around 1.5M txs, the network crashed again:

This crash was thorough: all nodes were wiped out, and I could not even SSH in. Digital Ocean’s monitoring showed CPU at essentially zero. Just as I was wondering whether the disk was full, one machine had a brief deathbed rally and gave me a chance to log in and run du. Sure enough, the 25GB disk had been eaten up completely. Took a snapshot, swapped in a larger disk, done.

In early March, we finally abandoned the $5 machines and upgraded to $15 ‘large’ nodes. On Digital Ocean’s cloud, we ran several networks at once to do rolling upgrades. Previously we had a milestone every week, releasing one major version and several minor ones; starting the second week of March, we released a version every day. As a result, we had too many versions and not enough networks…

Soon, the 1 million txs milestone was replaced by 5 million:

Then by 6M, 7M, and so on. Later, because we had too many breaking changes, we stopped accumulating that number.

Being able to deploy blockchain nodes stably on $15 machines is an innovation for us at ArcBlock. We have run nodes for other public chains—the recommended configuration they provided cost more than a thousand dollars per node per month. If an app developer wanted to deploy their own chain and initially serve their users through their own nodes, assuming nodes deployed across four global regions with two nodes per region, that single expense would exceed ten thousand dollars per month—small players without ample cash cannot burn that kind of money. So we wanted to bring that number down to a few hundred.

However, Digital Ocean ultimately serves small customers. For a serious dApp, using DO during development is entirely reasonable; in production—once the chain goes live—a more capable cloud service is the better choice. Our own ABT network, for example, is deployed on AWS.

A Production Environment That Is Simple but Not Simplistic

Because ABT network emphasizes weaving chains into a network, we launched with three element chains named after chemical elements—Argon, Bromine, and Titanium (Bromine being a test chain dedicated to running the latest nightly build). We therefore needed to prepare a secure and trustworthy production environment for these three chains.

This is how we thought about the online production environment:

1. Each chain is deployed across four regions in Asia-Pacific, Europe, and the Americas.
2. Argon and Titanium each have sixteen nodes; Bromine has four.
3. All nodes expose only the p2p port externally.
4. A node’s GraphQL RPC and built-in block explorer are accessible externally through an ELB, while gRPC is restricted to local access.
5. For each region, the domain name of each chain’s ELB is load-balanced by Route 53 based on latency.

Most importantly, it had to be automated and sufficiently cost-effective.

Automation goes without saying; we already had deep experience with Ansible and Terraform.

Saving money is a craft.

Under the above configuration, even using only the cheap and cheerful c4.large / c5.large, with 110GB EBS per node and one ELB per chain per region, fixed costs alone would come to $3,721 per month.

Calculation formula: 0.11 (c4.large price) x 36 x 24 x 31 + 36 x 110 x 0.12 (EBS price) + 25 (ELB price) x 12

Of this, EC2 accounted for the lion’s share—close to $3,000.

Our goal was to reduce this cost as much as possible.

So we turned our attention to spot instances. The charts below show spot instance price trends in us-east-2 and ap-southeast-1:

Prices were basically stable at 20% of on-demand instances, meaning that for EC2 we could bring the cost down to $600, for a total of only $1,300 per month.

However, with spot instances, there is an unavoidable hurdle: if an instance is killed, how do we restore service as quickly as possible—especially for validator nodes?

Our approach was to separate the root disk from the data disk. All data stored by Forge goes on the data disk, while Forge’s configuration, node private key, and validator private key go on the root disk; after initialization, these are backed up to an AES-encrypted, write-once S3 bucket. Subsequently, while nodes are running, we periodically back up the data disk of a healthy node for each chain in each region. Thus, when a validator node is killed, we can restore its data disk from the most recent backup and then retrieve that validator node’s private key and configuration from S3.

This idea sounds simple and intuitive when described, but implementing it required considerable thought. In the end, we smoothed out the path and proved it feasible. For dApp developers and even peers in other blockchain projects, this approach of running blockchain nodes on spot instances has reference value.

In the end, our deployment scripts, forge-deploy, were divided into four parts:

1. Scripts that only need to run once: for example, creating a security group for each VPC in each region.
2. Scripts for building the Forge AMI: every time we release a new version, we create a new AMI.
3. Scripts for creating the resources needed by a new chain: for example, creating spot requests, EBS volumes, an ELB, and a target group; setting up listeners (and listener rules); and creating domain names and domain-resolution policies.
4. Scripts for managing an existing chain: for example, initializing the chain, restarting nodes, upgrading nodes, repairing damaged nodes, and adding new nodes.

In the last two weeks of March, forge-deploy was developed and tested on top of the existing ad-hoc scripts (the scripts for deploying DO machines)—we built chains, tore them down, and built them again. In two weeks, we traveled a road that many blockchain teams might not traverse in a year: at our peak we had six chains running in parallel, and counting those ephemeral, mayfly-like chains such as abtchain, origin, bigbang, test, and abc, we created and destroyed more than thirty chains in all—note that this refers to multi-region, multi-node chains; single-node chains are not included.

Thanks to the confidence we had accumulated, on the day ABT network went live, we arrogantly tore down the three chains we had already created for the release—Argon, Bromine, and Titanium—less than half an hour before the launch countdown ended, and redeployed them, so that the entire team and community members who cared about us could witness blocks jumping from zero to one. There were some hiccups—the deployment scripts ran slower than expected—so we had not finished deploying when the countdown ended. But in the end, after a delay of about twenty minutes, all three chains went live as desired. Deploying each chain required only two commands:

Among them, create_fleet performs the following tasks in all four regions:

1. Retrieve the default VPC ID for the current region.
2. Retrieve the VPC’s subnet ID.
3. Retrieve the IDs of several pre-created security groups.
4. Request a spot fleet for validator nodes using the preset configuration.
5. Request a spot fleet for sentry nodes using the preset configuration.
6. Wait for all requested instances to become fully operational.
7. Create an ELB.
8. Create a target group and add all instances to it.
9. Retrieve the ID of a pre-uploaded certificate.
10. Create two ELB listeners: port 80 issues a direct 301 redirect to 443, while port 443 forwards traffic to the target group.
11. Create DNS records and configure a latency-based policy.

After all four regions are complete, create an Ansible inventory for all instances of this chain for subsequent operations.

Next, init_forge_network does the following:

1. Mount the data disk onto the corresponding instance and format the filesystem as XFS.
2. Start Forge with a temporary configuration file to generate the node key and validator key.
3. Back up the generated keys to S3.
4. Based on the inventory file, identify the validator nodes and write their validator addresses into the genesis configuration.
5. Start Forge.

Once all nodes are up, after a brief moment, a chain is born in all its perfection!

---

[1] Anyone who has worked in security should be aware of this: generally, a node as important and attack-prone as a supernode should not be exposed to the public internet. Instead, several front-end sentry nodes with public IPs should absorb traffic and attacks, while the supernode should focus solely on producing blocks and then hand them to the sentry nodes for broadcast across the network.