Exploring the Possibilities of Bitcoin Quantitative Arbitrage

Articles about Bitcoin arbitrage trading are a dime a dozen online—especially in 2014 and 2015. Back then, spreads between exchanges were substantial, arbitrage opportunities were plentiful, and making money while lying down was no difficult feat. Today, arbitrage windows have narrowed; those left standing on the battlefield are probably only the big players, and the chance to win effortlessly is slim. Recently I helped a friend connect with opportunities in quantitative crypto trading. I had intended to be a soy-sauce-passing middleman—brush off my sleeves once the deed was done and quietly serve as the chubby-division champion of Sanhe City—but my fingers started itching, and before I knew it I had sunk six or seven hours into it last night.

On how to do arbitrage trading, I won’t belabor the point. You can read this article; it has substance and story: https://daily.zhihu.com/story/4831821.

This article has no story—only technical and product analysis.

Since we’re doing a feasibility analysis, the first step is to observe trading data. Although domestic exchanges have been cleared out under policy requirements, overseas markets are still bustling. From Bitfinex and Kraken all the way to Bithumb, Korbit, and others, they all provide REST APIs, with a few offering WebSocket interfaces. Documentation is sparse, but the APIs are simple and not hard to get started with.

After a quick trial run in Node.js, I decided to rewrite everything in Elixir.

There are several reasons for this:

1. For data scraping alone, we need some internal microservices. For example, Bithumb and Korbit quote in Korean won, while the others quote in U.S. dollars. If we want to unify on dollars, we inevitably need a stable internal service to provide that conversion. Erlang/OTP’s structure is naturally cute and dopey—no, naturally loosely coupled. Each GenServer is its own self-contained entity, which suits this requirement well.
2. I want to separate data scraping from data processing. After periodically fetching data from each exchange, we can use a pubsub structure to publish the data and let each subsystem subscribe—the opportunity finders calculate trading opportunities, the persistence handlers write to databases, and whenever we think of a new way to consume the data, we can add a new subscriber at any time. With another language, this would inevitably drag in dependencies such as ZeroMQ (introducing complexity and other problems). Erlang/OTP has process groups, which suit pubsub, and phoenix_pubsub has already wrapped this elegantly—out of the box.
3. Erlang/OTP’s fault tolerance and concurrency/parallelism can greatly increase efficiency.
4. because I can - Insider Man (2006).

Once the data is scraped, we can compute theoretical arbitrage spreads over time (actually executing them is another matter), but the data itself is too abstract to move me—or anyone else. For example, over the past hour there were ten arbitrage opportunities above 0.5% (after deducting fees on both sides, less than 0.1% remains). How feasible are such opportunities, and what do they actually look like? There’s no way to know.

Guangdong Top Ten Outstanding Youth Fang Shiyu’s mother, Miao Cuihua, said: When traveling the jianghu, the most important thing is a single word: ‘looks.’ Dry numbers and tedious theories are like so many skeletons; even when they hit the mark, it’s hard to find resonance in the hearts of product puppies and business kitties. As the saying goes, no picture, no hero; without some UI and visualization, even a gold mine will be buried under yellow earth. In Struggle, Lu Tao’s father feared his son had no concept of a number like 300,000 yuan, so he had cash brought out and packed into a large paper bag for him. That is visualization.

But building an extensible visualization UI is hardly the work of a day. You finally get a chart working with line / bar / area and multiple series, and then while you’re at it you may need arithmetic between two series, and afterward even more complex operations. Putting on makeup to pass as a refined young lady was supposed to be a stopgap, but once you enter frontend, the sea is deep; from then on, change requests cling to you forever. So unless absolutely necessary, don’t build visualization yourself. I’ve tried several open-source Bitcoin arbitrage trading projects; their UIs are quite ordinary and their extensibility poor—a lot of effort for little thanks.

If we don’t build it ourselves, who will?

In my day job, I work with Datadog a lot. Datadog does performance monitoring: we just emit metrics, and storage and visualization are left to Datadog. Although it has nothing to do with quantitative trading, we can try handing the role of visual appeal to Datadog.

Emitting metrics is simple: write a subscriber that listens to the scraped data and writes to Datadog like this:

arb.xchg.btc.bid:4182|h|#bitfinex
arb.xchg.dsh.bid:344.38|h|#bitfinex
arb.xchg.eth.bid:301.93|h|#bitfinex
arb.xchg.xmr.bid:97.762|h|#bitfinex
arb.xchg.xrp.bid:0.2023|h|#bitfinex
arb.xchg.zec.bid:299.22|h|#bitfinex
arb.xchg.btc.bid:4009.071877180739|h|#bithumb
arb.xchg.dsh.bid:330.42568039078856|h|#bithumb
arb.xchg.eth.bid:288.8171667829728|h|#bithumb
arb.xchg.xmr.bid:93.35310537334263|h|#bithumb
arb.xchg.xrp.bid:0.1936496859734822|h|#bithumb
...

We provide one metric per cryptocurrency, tag it with the exchange name, and push data acquired at the same time to Datadog in a single batch. This lets us visualize results like this:

Clearly, for Bitcoin, there are ample arbitrage opportunities between Bitfinex and Bithumb:

This chart is obtained by subtracting one series from the other and converting the difference into a percentage, so the vertical axis is in percent. Datadog has been deeply engaged in the APM market for years and already supports this in its UI—we can edit and manipulate it directly. Isn’t that heartwarming?

In Let the Bullets Fly, Zhang Mazi asks: what the hell is ‘convincing people with charts’? Fang Shiyu’s mother’s co-father-in-law replies: this is what I call convincing people with charts. The arbitrage spread is now clear at a glance. Let’s look at opportunities across all cryptocurrencies:

Opportunities exist, but whether they can be realized is another matter and beyond this article’s scope. Moreover, we currently have only a small amount of data, which is not yet sufficient.

At this point, let’s sort through the system architecture:

• The system has four Applications. Exchanges is responsible for scraping and publishing data; Catcher is responsible for persisting data (writing to Datadog or InfluxDB); Simulator is responsible for simulated trading (currently only real-time data simulation; historical data simulation should be considered later); and Trader actually turns each suitable opportunity into a trade.
• Exchanges and Catcher are basically implemented; Simulator is under consideration; Trader can wait.
• Simulator can dynamically generate traders based on configured conditions (a bit like NPCs in a game), have them receive real-time data and seek their own trading opportunities (using real trading fees and simulating various latencies). Whenever a trade occurs, the trade data and post-trade total market value are pushed to Datadog, where we can finally generate a real-time ‘garden of flowers’ leaderboard to see who among Ada, Bella, Clare, Daisy, and company is actually the best.
• The system has good fault tolerance. In theory, any part crashing will self-heal without fatal impact (for example, if the publisher dies, data that cannot be sent out during that moment is lost, potentially missing some trading opportunities).

The following figure is the skeleton of exchanges_app:

In Catcher, besides writing data to Datadog, we also plan to write to InfluxDB. Why? Although I like Datadog very much, its specialty is APM, and sending data has a long delay (20s by default). That is, after you write a batch of data, it first goes locally over UDP to the Datadog agent; the agent leisurely lets it sit for a while, decides it’s about ready, and then sends it to datadoghq. This delay is harmless for after-the-fact analysis, but if you’re watching the market, it’s not so comfortable. InfluxDB, by contrast, is entirely under our control and has no such problem. That’s reason one.

The last time I used InfluxDB / Grafana was a year ago. Recently I heard that Grafana has introduced alerting, and this project is a good excuse to pick them up and study them properly. InfluxDB / Grafana have better real-time characteristics and allow some operations to be done directly in Grafana—for example, using query statements to find trading opportunities and then trigger an alert. The alert endpoint is a webhook, and trading happens in the webhook. In this way, as long as the formula for discovering trading opportunities can be written as a query, we don’t need to write code, greatly enhancing the system’s flexibility and extensibility. That’s reason two.

So far I’ve written nearly 700 lines of code, taking about five or six hours. Some reflections:

• Use tools well. Some tools—like Datadog in this article—can yield unexpected rewards when used from a different angle.
• Our time is remarkably elastic. Rather than saying ‘I have no time,’ it’s more accurate to say ‘I’m not interested’ or ‘it’s not important to me.’ Counting from 7:30 last night until I went to bed a little after 1 a.m., I had a half-hour one-on-one and more than an hour of new-hire training in between. On an ordinary day, if I spend two hours in meetings or discussing problems with colleagues, I’ll be so tired that night that I don’t want to do anything and go to bed early. Yet last night I forced out the time to write code with intense focus.
• The right challenge can spark a person’s interest, and then everything flows naturally.