Blockchain and Databases: Attain the Utmost Emptiness, Hold Fast to Stillness

If Newtonian classical physics is a manifestation of Einstein’s theory of relativity at low speeds, then the database technology we know so well can be regarded as a special case of blockchain technology in a weakly distributed environment.

“Weakly distributed” is a term I made up. You can understand it as a distributed environment with an extremely limited number of nodes and a highly controllable runtime environment. A database cluster running in the same data center, or across different data centers, is still a controllable runtime environment as long as it has the same administrator. In a controllable environment, by default there are no “malicious” nodes, so BFT (Byzantine Fault Tolerance) is beside the point. We therefore do not need a complex consensus algorithm; generally, two-phase commit, or paxos / raft, can converge on consensus and meet the need. Thus the database’s consensus algorithm is a special case of the blockchain’s consensus algorithm.

In the blockchain world, transactions and the state produced by transactions are strictly separated. A set of transactions is validated and packaged into a block by miners selected through the consensus algorithm, and then broadcast. Other participants in the network then validate the legality of every transaction in the block and write it into their own state db. In Bitcoin, the state db is the UTXO set; in Ethereum, it is the world state.

The database world seems to have no transaction records analogous to those in blockchain (a transaction in a database is another concept), but on closer thought, its transaction history is precisely the WAL (Write-Ahead Log). When a request arrives from the outside, the database first writes it to the WAL, ensuring it reaches persistent storage, and only then writes it into its own “states.” From this angle, we can regard every record in the WAL as corresponding to a transaction in blockchain; it is a special case of a blockchain transaction.

If you think about it carefully, WAL, blockchain, and CQRS (Command Query Responsibility Segregation), which Martin Fowler advocated long ago, are at this level all “one idea, separately expressed”: all emphasize the separation of “events” and “state,” and from the previous state plus the current event one can derive the current state. In this way, as long as we have an initial “state” and then record all the “events” that have occurred in the system, we can reconstruct the “state” at any moment. (If CQRS interests you, don’t miss our ArcBlock Tech Learning Series tomorrow morning: CQRS & Commanded — hack.arcblock.io/learning.)

Let us return to transactions and the “blocks” that contain them. You will notice that “block” is a curious thing. Why does a database not need a concept like “block” as a container for “transactions,” while blockchain does? We know that in the blockchain world, uncertainty and certainty are like twins: the rules are certain, while the executors of the rules are uncertain. As the saying goes, miners take turns—next round it’s my turn—so how do we define a “round”? To answer this question, we need some mechanism that clearly marks the beginning and end of a miner’s status within a round—and that beginning and end is a “block.” Moreover, in a distributed environment where physical clocks are not synchronized, a “block” also carries the function of a global clock, ticking the entire network forward. The concept of a “block” is so important that it naturally becomes the foundation of the consensus algorithm: everyone must first agree on the sequence number of the next block to be produced; otherwise the game cannot proceed. By contrast, in a database system, within a database cluster the master (equivalent to the miner) is fixed. The master waves its command flag, and the slaves follow immediately, striking wherever they are pointed. There is no rotation of the chair, hence no rounds, so in fact every “transaction” is a “block.” Thus in the database world, logically every transaction—or every WAL record—constitutes an implicit “block” of its own.

Let us explore this conclusion from another angle. Another important role of the “block” is crash recovery. In a blockchain network, whether a node is disconnected or crashes, its state must differ from the state on which the network has reached consensus. How, then, does it recover from this inconsistent state to a synchronized state? The answer is the “block,” because it is the only unambiguous product of consensus. A node can always find the nearest committed block height that is consistent with the network, then synchronize block by block from that height onward, running in order all transactions contained in each block and updating its local state, until it finally reaches consistency with the network state. Here, the “block” is the smallest unit for detecting and achieving state consistency. In a database system, after a crash, the system synchronizes the latest WAL from other nodes and executes commands record by record from the position of the last committed WAL onward, until all records have been run; at that point the database state is restored to the cluster’s current state. Here, the WAL record is the smallest unit for detecting and achieving consistency, so calling it an implicit “block” is not wrong.

In the blockchain world, a transaction needs to be validated. Validation here has two meanings: 1) identity validation—the transaction is correctly signed by its originator; 2) integrity validation—the transaction’s change to state is legal. Identity validation is easy to understand: when you sign with the private key of your own wallet a transaction transferring 1 ABT to me, the system verifies that you really are you. Integrity validation means that in the states db, your account must indeed hold more than 1 ABT token before you can initiate the transaction. In the database world, identity validation is handled nakedly and directly by an access control system such as RBAC (Role-Based Access Control), while integrity validation is similar to that in blockchain.

Next let us look at determinism. By determinism, we mean that under the same state Sn-1, everyone takes the same transaction, executes it independently without relying on any third-party information, and obtains exactly the same result. Purely with respect to the processing from transaction to states db, blockchain and databases are completely identical here; both can guarantee determinism. However, if a blockchain is to support transactions carrying extra information, and that information triggers the execution of some code previously deployed on-chain (for example, a smart contract), then we must pay attention to the code itself needing to be deterministic. Determinism means nothing more than:

• Do not use a nondeterministic random number generator in code—for example, using the computer’s clock as a seed to generate random numbers is nondeterministic, because at the moment the transaction is executed, we cannot guarantee that all participants’ clocks are precisely synchronized.
• Avoid multithreading in code. Race conditions caused by multithreading are nondeterministic.
• Do not use the system clock. No explanation needed.
• Do not use uninitialized memory. Who knows what is in there.
• Do not use floating-point numbers—this one is bizarre, because different CPU architectures, compilers, and even different CPU models support different floating-point instruction sets, which can lead to different results.
• Do not use data structures in the programming language that may have random behavior. For example, iterating over a map.

Basically, once these are avoided, the code is deterministic and can execute on a blockchain. Then why can stored procedures in a database allow the execution of code without determinism? For example, a stored procedure can insert a record using the current time. If we return to first principles and look at the problem from the angle of “transactions,” we can see that a stored procedure is like “off-chain” executed code. Although it is rooted in the database, it is actually the source of “transactions.” The execution of a stored procedure produces new transactions, that is, WAL records, which are then synchronized to other nodes. So a stored procedure can be non-deterministic, because the WAL records it produces are already deterministic: when the master executes the act of adding a record with the current time, it has already completed the action of taking “the current time” and obtained a definite value, which is carried in the WAL. This differs fundamentally from the concept of a smart contract in blockchain, and it is why a stored procedure need not be deterministic while a smart contract executed “on-chain” must be. From this angle, a database system is also a weakened blockchain system.

Since both blockchain and databases store data, having discussed data integrity and determinism, we come next to data consistency. Blockchain is clearly a paragon of eventual consistency: the larger the network and the more participating nodes, the slower a block spreads; at any moment, reading state on different nodes has a high probability of producing inconsistent results. Yet as long as nodes can synchronize to the latest block, the state of the entire network converges, and eventually everyone can obtain a consistent set of state data. In fact, by this reasoning, all distributed systems that use WAL and CQRS ideas are eventually consistent in their data state—which seems not to match our impression of the strong consistency of classical databases. However, if we pull our perspective down to the micro level, we can see that strong consistency is merely eventual consistency with some additional conditions attached; it is a special case. Suppose we require a blockchain to satisfy the following conditions:

1. Whenever any node receives a new block, after the transactions have finished executing and been written to the states db, it must send an acknowledgement to the miner node (which is, in fact, the current master).
2. After the miner node receives all acknowledgements, it broadcasts to all nodes in the network that this block has been successfully committed.
3. Before a block receives the miner node’s aforementioned broadcast (that is, before writing is complete), queries sent by clients enter a queue and wait (in fact, a read-write lock).

Then, from the outside, it too appears strongly consistent. Of course, the third point is a bit too strict. Ordinary database implementations use MVCC (Multiversion Concurrency Control), letting each client see a snapshot of the current state, so there is a very small window in which the data seen by different parties is inconsistent. If you want to split hairs, MVCC does not count as strong consistency; of course, no one would insist on that.

Through the above rules, a database can sacrifice some performance to create outward strong consistency. But sometimes, for the sake of some lofty—ahem, sordid—ideals, a database system can also break these rules and claim higher performance. MongoDB, in a cluster environment, can return write operations without requiring node acknowledgement, and thus we have the euphemistically named “weak consistency.”

A blockchain network could in theory use the above rules to cultivate the outward feeling of strong consistency, but this is not actually practical. The larger the network, the greater the latency, so practically operable strong consistency can occur only in an environment with very few nodes, all in the same data center. From the angle of data consistency, the database is also a special case of blockchain in a special scenario.

Finally, let us talk about performance. Performance is inversely proportional to network scale. There are two main reasons: 1) the more nodes there are, the harder it is to reach consensus; 2) the more nodes there are, the longer it takes for a “transaction” to propagate through the network. So how do you achieve universe-conquering TPS (Transactions Per Second)? It is actually not hard—since a database is a special case of blockchain in a weakly distributed environment, we simply degrade blockchain in the direction of a database. PoW says, “Are kings, lords, generals, and ministers really born to their station?” and lets the entire network compete for the Iron Throne. PoS lets “a small number of people get rich first.” DPoS goes one step further: “let the leaders go first.” Perhaps in the not-too-distant future, someone will unleash the ultimate move: the entire network has a single supreme sovereign, all the tricks available in databases—replicaSet (we still need HA, after all), sharding, and so on—are brought into play, and then one uses the art of war to subdue the enemy without fighting: “I have now trained eight hundred thousand sailors and would like to join you in a hunt in Wu”… And thus, with full justification, one can snatch the laurel of performance.