Design Patterns for System Development

Separating the Control Plane and the Data Plane

These two concepts are practically introductory material in any university networking course. One important reason Juniper rose to prominence at the end of the last century was that it rigorously separated and delineated the control plane and the data plane, then implemented the data plane with ASICs. The data plane refers to the module in a network device that forwards packets; its efficiency determines the efficiency of the entire device and is generally implemented in hardware. The control plane refers to the protocol-related part of a device and need not be as efficient as data forwarding.

When you open a browser and visit Google, network devices across the Internet spring into action with a single goal: forward your request to Google’s servers. Anyone who has taken a networking course knows that the network devices involved are routers. A router must forward fast enough, with latency as low as possible—this tests the efficiency of the data plane. The routing information on which data-plane forwarding decisions are based is produced by protocol processing in the control plane.

In an internet system, there seems to be no clear boundary between the control plane and the data plane. We may crudely treat the paths users access as the data plane and the paths related to management as the control plane. For work on the data plane, we can carve out a separate cluster to handle it, striving to process every request as efficiently as possible; work on the control plane can be completed with comparatively few resources. The most important principle here is to separate the paths of the data plane and the control plane, so that the large volume of requests on the data plane does not interfere with the normal operation of the control plane, and slow tasks on the control plane do not drag down access speed on the data plane.

First Path vs. Fast Path

When building firewalls, one inevitably encounters the concepts of the first path and the fast path. A firewall handles bidirectional data flows and must record state, hence the concept of a session. On the first path, a complete full path is traversed and a session is created; on the fast path, various pieces of information in the session can be used to process data packets quickly.

In internet systems, a similar mapping is easy to establish. In a system that requires user login, the entire login process can be regarded as the first path, and subsequent access as the fast path.

User login is a complex process, and not merely a matter of validating the user’s legitimacy. While presenting the post-login page to the user as quickly as possible in the foreground (response speed matters), the backend must load a series of user-related data into cache (e.g., Redis) so that the user can retrieve it quickly on subsequent visits. The loaded data might include the user’s friends, hot data the user is likely to access, various counters, and so on.

Of course, the first-path/fast-path concept applies not only to login and post-login access; there are many other scenarios. For example, in a rules system, the first access extracts user-relevant rules from the rules engine, compiles and caches them; subsequent accesses read efficiently from the compiled rule cache.

Note that the first-path/fast-path distinction is relative. Like fractal geometry, within the first path one can further distinguish first path/fast path, and within the fast path one can again distinguish first path/fast path, iterating continuously. The purpose is to continuously optimize the 80% of paths most frequently used in the system, maximizing their efficiency.

Slow Path vs. Fast Path

Slow path/fast path is very similar to first path/fast path, but not exactly the same. For user login, we assume (or have actual data showing) that 80% of users log in via username/password, so username/password login should be placed on the fast path, while other login methods such as LDAP, OpenID, and XAuth are placed on the slow path.

The benefit of distinguishing fast path/slow path is that, when needed, we can implement the corresponding code in a more efficient way. For example, if the entire system is implemented in Python and some fast paths lie in hot user-access areas, we can consider implementing them in Go.

Queue-Based Design

In network devices, queues are everywhere, almost becoming the most basic operation. After a packet arrives from the hardware, it is placed on the driver’s queue, and then continuously enqueued/dequeued at each layer of system processing. Queues have many benefits, such as deferred processing, priority, and traffic shaping.

A complex internet system also often needs queues to control the pace of task processing. For example, something like email verification need not be completed within the current request; it can be placed on a message queue and handled by a backend process. In addition, queues can have different priorities: sending email and converting images to different sizes can obviously be scheduled in different priority queues.

For internet projects, there are many mature message queue systems, such as RabbitMQ and ZeroMQ.

Pipelines

In network systems, if a task is complex and requires a lot of CPU time, it can be decomposed into multiple smaller tasks for execution; otherwise, the task occupies CPU time for too long and prevents other tasks from running. Once a task is decomposed into smaller tasks, each small task is connected by queues: after one finishes processing, it is placed into the next queue. This makes task scheduling more balanced.

In internet projects, pipelines have many applications. For example, a state-machine transition in a workflow may execute a series of operations and finally migrate to a new state. If this series of operations is executed in one large function instead of being decomposed into several small operations connected by queues, then slow operations in the whole process will affect the system’s throughput. Moreover, this is very unfavorable for concurrency.

In a large system, the degree of pipelining determines the degree of concurrency. And the extent to which pipelining is applied affects the throughput of the entire system architecture. Some programming languages, such as Golang, naturally turn your thinking toward pipelines (through go/chan).

Finite State Machines

Since I’ve mentioned state machines, let me discuss them. A state machine consists of two elements: states and state transitions. State transitions are triggered by actions, so a state machine can be expressed as state machine = {state, event} -> (action, new state). As long as you draw a two-dimensional table, you can analyze all possible paths of the system, and omissions are unlikely. In network devices, most protocols are described by state machines, such as OSPF, IGMP, TCP, and so on.

In internet projects, state machines are everywhere. Take order processing. An order’s processing flow is perfectly expressed as a state machine. Below is a sample piece of code I once wrote (Python):

ORDER_EVENTS = {
  (const.ORDER_EVENT_PAYED, const.ORDER_STATE_CREATED): {
    'new_state': const.ORDER_STATE_PAYED,
    'callback': on_order_event_payed,
  },
  (const.ORDER_EVENT_PAY_EXPIRED, const.ORDER_STATE_CREATED): {
    'new_state': const.ORDER_STATE_CANCELLED,
    'callback': on_order_event_cancelled,
  },
  (const.ORDER_EVENT_CONFIRMED, const.ORDER_STATE_PAYED): {
    'new_state': const.ORDER_STATE_CLOSED,
    'callback': on_order_event_confirmed,
  },
  (const.ORDER_EVENT_CONFIRM_EXPIRED, const.ORDER_STATE_PAYED): {
    'new_state': const.ORDER_STATE_CLOSED,
    'callback': on_order_event_confirm_expired,
  },
  ...
}

Watchdog

Finally, a brief mention of the watchdog. Generally speaking, network systems such as routers and firewalls are real-time systems: any task should finish within a prescribed time, otherwise it is a system error. So we need watchdogs to monitor tasks (there are hardware watchdogs and software ones). A watchdog can also help developers discover problems such as deadlocks, overly long loops, and unreasonable task allocation in the system. If a task runs for too long, it will block other tasks; if all CPUs are occupied by such tasks, the system cannot respond to events and may also be unable to schedule these tasks away.

In internet projects, request handling, asynchronous task processing, and the like all run on a limited number of processes. If a process deadlocks, or runs for too long (possibly due to an abnormal condition) and enters a “false death” state, we can use a watchdog process to kill these “false-dead” processes and restore the system’s throughput to a normal level. If we don’t, the “false-dead” processes will accumulate and may eventually make the entire system unable to work.