The World of ZeroMQ

This past National Day holiday, the most heartbreaking news for programmers was the assisted suicide of Pieter Hintjens, the author of ZeroMQ. No doubt that moving essay, A Protocol for Dying, has already circulated through your social feeds. If you haven’t read it yet, scroll through your feed, read it carefully, and then bookmark it — come back to it in a year or two. The estimable Pieter, even in his last words, couldn’t resist a nod to his own trade of messaging, borrowing Alice and Bob (https://en.wikipedia.org/wiki/Alice_and_Bob) for a little playful banter.

I don’t actually know much about Pieter; the only bridge between us is ZeroMQ. To be fair, ZeroMQ is an outstanding broker-less messaging lib (as opposed to brokers like RabbitMQ and Kafka — note it’s a messaging lib, not a message queue; the name ZeroMQ is a bit misleading), with a friendly API (sockets), high efficiency (Pull/Push of 100-byte messages at ~2M/s, see https://github.com/zeromq/jeromq/wiki/Performance), and strong scalability (a Pub/Sub setup can have 10k subscribers). Yet it also has conspicuous flaws: the much-maligned zmq_ctx, and a concurrency model that is flawed by design (http://zeromq.org/whitepapers:architecture). Of course, I’m not writing this to praise or disparage ZeroMQ, nor to popularize it or offer a beginner’s guide (getting started with ZeroMQ is fairly simple). Rather, I want to talk about the valuable ideas — the patterns — of communication system design embedded in ZeroMQ. These patterns have had no small influence on me.

Fundamentals

Before discussing ideas, we need to understand a few concepts.

The most important thing about a messaging system is its delivery guarantee: at least once or at most once. At least once means the same message may be transmitted between 1 and n times; at most once means it may be transmitted between 0 and 1 times. This is easy to grasp: if the messaging system has built-in retransmission and persists messages to disk so that they’re delivered even if the process crashes, then it’s at least once. Conversely, if none of these mechanisms exist and messages are sent and forgotten, it’s at most once. Unfortunately, ZeroMQ is not strictly at least once nor at most once. Take its Pub/Sub mode: ZeroMQ builds in message acknowledgement and retransmission, but does not persist messages, so running out of memory or a process crash will cause message loss, while retransmission may cause a message to be delivered 1 to n times. This is why I don’t consider ZeroMQ a true Message Queue — though of course it can be used to build one. Note that in a networked environment, message delivery can only be one of the two cases above; exactly once is impossible. Anyone who claims otherwise is misleading you.

The attentive reader may wonder: TCP has built-in retransmission, so why does ZeroMQ bother adding it at the message layer? Because TCP’s retransmission only guarantees retransmission of packets at the network layer, whereas ZeroMQ’s message-layer retransmission guarantees that once a message is delivered, it is delivered whole.

The use case for at least once is easy to understand: when we send a message, we naturally want the receiver to guarantee receipt. As for the side effect of guaranteed receipt — duplicate retransmissions — as long as message processing is idempotent, there’s no problem. The use case for at most once is more puzzling: when would we send a message and not mind if it’s lost? For example:

• heartbeats under various network topologies (in most cases heartbeats can just use IP/UDP, no need for messaging); losing a few messages here and there doesn’t hurt.
• dense status report messages or tracking events. Lost messages pose no threat to the overall picture.

The last concept is back pressure. In any messaging system, the two ends — producer and consumer — will have mismatched processing speeds. If the producer sends messages too fast for the consumer to keep up, messages become congested, and this pressure is propagated back upstream, layer by layer, until it reaches the original producer and forces it to stop generating more content.

Messaging Patterns

With the fundamentals out of the way, let’s look at some of the messaging patterns ZeroMQ involves.

REQ/REP

REQ/REP is the most basic pattern. The client sends a request and the server responds.

PUB/SUB

Pub/Sub is a very common and very useful pattern for message transmission. It shares its lineage with the observer pattern, decoupling publishers of data from subscribers — the publisher only produces data and needn’t care who the subscribers are or how many there are. For instance, if you’re building a chat room, everyone is both a publisher and a subscriber. The publisher needn’t care about subscribers joining or leaving; messages fan out to each subscriber in a 1:N fashion.

PUB/SUB (forward proxy)

Pub/Sub composed with itself can solve many practical problems. For example, you have a lot of data to publish to internal and external applications, and the data accessible to external applications is a subset of that for internal ones. By composing Pub/Sub, one (or more) subscriber receives the data, filters out the topics (or channels) intended for external publication, and republishes them for external applications to subscribe to.

Push/Pull (map reduce)

Push/Pull is another important messaging pattern. Its characteristic is that either the Push end or the Pull end can act as the server, binding to an address and waiting for the other side to connect. If we bind the address on the Push end, it’s a Push server, and corresponding Pull clients can connect to it to pull data out. Conversely, if we set up a Pull server, corresponding Push clients can connect to it to push data in. With this, we can easily build a map-reduce framework for tasks. As shown above, the workers in the middle can be scaled up or down as needed.

Taking it a step further: when all tasks have been aggregated (reduced), we want to terminate all workers. We can add a Pub/Sub mechanism to the system: the sink publishes control messages, all workers subscribe to that channel, and upon receiving SIGTERM they terminate their processes.

Push/Pull (fair queue)

Another application of the Push/Pull pattern is the fair queue — Push clients take turns writing data to the Pull server.

Router/Dealer

The Router/Dealer pattern is a classic broker pattern. In a many-to-many network, the broker mediates transactions between two parties who don’t know each other. A supermarket is a typical broker: customers needn’t deal with every supplier individually, and each supplier needn’t know every customer to close a sale — the whole transaction completes under the supermarket’s mediation, with neither side even aware of the other’s existence.

In many-to-many networks, the Router/Dealer pattern is very useful. Suppose we have N Reply servers and M Request clients. To guarantee high availability, normally both sides would need an M x N full-mesh network so that any client can connect to any server. By inserting a layer of Router/Dealer in the middle, the M x N connections are simplified to M + N. The complexity of the network is greatly reduced.

Putting It All Together

By combining Pub/Sub, Router/Dealer, Pull/Push and other patterns, we can easily support very complex network applications, as shown above.