On the Beauty of Unix
Unix’s philosophy favors small, composable tools over monolithic commands. A simple pipeline like `docker ps -a -q | xargs docker rm` shows how four independent commands can combine elegantly to solve a problem.
SYSTEMS & ARCHITECTURE · 55 ESSAYS
← All Columns & SeriesSystems & Architecture
Distributed systems, OS kernels and schedulers, networking and cryptographic protocols, serverless infrastructure, cloud-native systems, and blockchain internals.
Why You Need to Understand Information Security
Data breaches, leaked emails, stolen cards, exposed employee directories—these failures keep coming, and nobody is immune. Even outside security work, you need to grasp availability, integrity, confidentiality, and the human weaknesses attackers exploit.
The Black Box
A black box on an aircraft records the critical data needed to reconstruct a crash. Software systems need the same capability: a flight data recorder that captures the right runtime evidence before failure, so root-cause analysis remains possible.
Touch and Feedback Systems
Today I sat in on a Stanford lecture titled "EngX: The Digital Sensory System," fulfilling a long-held wish to take a proper class there. If you're ever in Silicon Valley on business, check event.stanford.edu for a lecture worth attending.
Measurement
In the Lean Startup loop, measurement is the crucial link between build and learn: once some or all of a system exists, we must find the right instruments to extract data worth reflecting on. A software feature you cannot measure is like a doctor's prescription for the common cold — who knows whether it works.
Why SOA Matters So Much
SOA is not a stack of fading protocols but a discipline of strict loose coupling through service contracts. By turning capabilities into independent services, teams can decouple languages, scale, measure, and optimize without dragging old code into every new feature.
Writing a Proper REST API
Handed a batch of supposedly finished REST APIs after a round of layoffs, the author finds them technically to spec but practically unusable — and rewrites the lot. What follows is a distilled account of what actually makes a REST API qualify: RFC conformance, integrity, and security.
Network Security in Application Development
A friend recently asked me to help diagnose a problem in his system. He handed me a username and password and told me to SSH into one of his servers—a Redis instance caching database queries and running other business logic. The issue took considerable effort to pin down, and along the way, at my request, he gave me the credentials for two more servers.
Software Performance Tuning: Follow the Data, or the Buzzwords?
Everyone invokes zero-copy when performance comes up, but real tuning begins with measurement. A simple benchmark shows that memcpy is often cheaper than the syscalls and network round trips we overlook.
Beats: Monitoring and Log Processing
A look at Elastic's Beats collectors—especially Filebeat and Packetbeat—and how they plug into the ELK stack to ship logs and network packets for monitoring and analysis.
On System Design Interviews
An API on Rails (or Django, or Express) and MySQL jumps from 6,000 to 20,000 RPM and buckles under the load. Design an application-layer cache to relieve it — without caching at the web/proxy layer or using any built-in or third-party cache module.
On Writing APIs, Revisited
Last year I rewrote our team's amateurish REST APIs—but the resulting system still lacked docs, tests, and monitoring. Here's how I rebuilt the API system at adRise from scratch: pipelines, validation, and contracts.
The Foolproofing of the Backend?
Two weeks ago, RethinkDB officially released Horizon, a backend app that lets the frontend manipulate RethinkDB directly after just basic security config and validation. It resembles Meteor's minimongo, but differs in how it's implemented…
The World of ZeroMQ
A farewell to Pieter Hintjens, and a tour of the messaging patterns that made ZeroMQ great — from at-least-once versus at-most-once delivery and back pressure, to REQ/REP, PUB/SUB, Push/Pull and Router/Dealer, and how they compose into real systems.
AWS re:Invent 2016: New Products and Punchlines
At AWS re:Invent 2016, Las Vegas hosted 32,000 engineers and a wave of new services. This dispatch covers F1 FPGA instances, AI as a service (Rekognition, Polly, Lex), and Snowball—plus some programmer jokes along the way.
Snowmobile Rhapsody
Tanenbaum warned us never to underestimate the bandwidth of a station wagon full of tapes. Amazon's Snowmobile proves him right — and from there my imagination runs wild: snow trains, container ships, and rockets to the Moon.
AWS re:Invent 2016: Primitives, Not Frameworks
Transformation was the theme of Werner Vogels' keynote this year, and he even took the stage in an Autobot T-shirt to match it. On the surface, the talk was about how AWS is helping customers become transformers of the internet-scale era through innovations in development, data, and compute — but look closer, and it's really a retrospective and roadmap of the AWS architecture blueprint.
Pay As You Go: When Programmers Set Their Sights on Car Insurance
Car insurance may be one of the most irrational products in the world: drive a few hundred miles a year and you still pay nearly the same premium as someone who drives tens of thousands. Programmers decided to fix that — with OBD, GPS, and a pay-as-you-go model.
The Life and Times of a Policy Engine
As a video streaming service, TubiTV must honor contracts pinning down when, where, and on which platforms each title may air. This is the story of its policy engine—rewritten from a naive rule engine, to a JISON-based BNF evaluator, to Elixir-compiled pattern matching.
When the 99.9% Cloud Falls: AWS S3 Outage Leaves North America Reeling
At 10 a.m. Pacific (2 a.m. Beijing time), AWS S3 began malfunctioning. Startup engineers found they could not upload or download files; a Hacker News post asking 'Is S3 down?' was quickly confirmed—yet AWS's own status page lagged behind.
Service Performance 101
A practical introduction to service performance: when startups should care, how to measure latency from CPU caches to cross-continental round trips, and how to analyze, profile, and optimize bottlenecks with the right tools.
Wasting Memory? What's the Big Deal?
We no longer pinch bytes out of memory: bits have vanished from mainstream languages or been locked away, and string-based protocols like HTTP have ceded once-elegant data structures to crude JSON. Wasting a little memory? No big deal — or so you'd think.
On Processes
Erlang and Elixir treat every process as an isolated, autonomous entity — closer to Smalltalk's objects than to OS processes. Through a simple key-value store, this essay compares Python objects with Elixir GenServers and asks what it means for code to have a life of its own.
The Evolution of My Passwords
Around 2008 or 2009, a website was breached and millions of weak passwords were cracked. At Juniper, IT circulated guidance recommending a sentence plus a rule to generate passwords. I adopted it, and it became a habit that has stayed with me ever since.
Exploring the Possibilities of Bitcoin Quantitative Arbitrage
Bitcoin arbitrage seemed easy in 2014–2015, when wide exchange spreads rewarded anyone who looked. Today the windows have narrowed and big players dominate; this six-hour detour explores quantitative opportunities through exchange APIs, OTP pipelines, trader simulation, and Datadog/InfluxDB visualization.
The Sound of Silence: The Obscure Black Magic of Network Transmission
A job interview with a Russian Android engineer leads to a playful reverse-engineering of the AliveCor EKG sensor, which secretly broadcasts heart-rate data over inaudible ultrasound — and a meditation on sound as a niche transport layer for IoT.
Pixel 2 XL: Software Is King
After six years on iPhone, I had grown weary of iOS, and the new iPhone X disappointed on features. Enter the Pixel 2 XL: rough-edged hardware, but software so good it changed how I capture, organize, and rediscover my life.
Don't Let Angry Customers Be the Ones to Tell You About Your Bugs
After a Sunday hike, I tried to unlock a shared bike, but the app froze at 0 percent and then claimed my previous ride was still unlocked, so I could not use another bike. It became a lesson in why systems must detect and resolve such failures before angry customers report them.
What I Talk About When I Talk About Scheduling
Scheduling is a function that maps a set of tasks onto a set of resources. Drawing on Linux, Erlang, and Go, this essay examines what schedulers cost, how they trade off, and the patterns behind cluster scheduling.
Scheduling Algorithms: Linux O(1)
Nearly fifteen years ago, as I began my career, Linux shipped its landmark 2.6 kernel. The most exciting feature was the O(1) scheduler. This essay explains the algorithm, after revisiting the much-criticized 2.4 scheduler to understand its lineage.
Overseer: A Supervisor for Distributed Systems
A dull task yielded an unexpectedly interesting idea: ex_loader, which hot-loads releases onto remote Erlang nodes, and Overseer, an OTP behavior that starts, pairs with, and supervises dynamic cloud workers much as a Supervisor does GenServers.
On Distributed Systems
Blockchains did not appear out of nowhere; they grow from roots in distributed systems. This essay traces those roots—logical clocks, message passing, coordination, CAP—and shows how uncertainty shapes every distributed design.
Sincerity Is the Way of Heaven
Inspired by Mencius on sincerity, this essay asks how we can trust data without trusting its custodians, and sketches a decentralized, publicly verifiable database built on signatures, Merkle trees, and broadcast roots.
Bitcoin in Brief
A chance lunchtime conversation about Bitcoin in 2013 leads the author, years later, down the blockchain rabbit hole. Rereading Satoshi's whitepaper and tracing its design choices, he discovers why network and systems engineers may finally get their due.
The Quantum Computing Threat to Bitcoin
Before the main course on consensus mechanisms, a little dessert: how quantum computing threatens Bitcoin. Google's 72-qubit machine revived the question for seemingly unbreakable cryptocurrencies, and this piece examines what Shor's and Grover's algorithms can actually break.
AWS Best Practices
I'm a heavy AWS user — so heavy that I could skip a meal but never a day of AWS. Drawing on years at Juniper and Tubi and my first month at Arcblock, here are the AWS best practices I've worked out — and though they target AWS, they may hold lessons for any cloud.
A 1024 Tribute: The Simple Chain
Two weeks ago we launched CyberMiles' supernode; since its mainnet runs on Tendermint, I spent last week studying it. Tendermint's ABCI lets developers focus on state rather than consensus — so I built a Simple Chain in under 400 lines of Elixir to show what really matters.
Data: A Free and Easy Wandering
For programmers, data is the stuff our code works on: now a file sleeping in the filesystem, now a variable floating in memory, now a byte stream soaring across the network. Zhuangzi asked whether the sky's blue is its true color—or just the look of something infinitely far away.
Blockchain and Databases: Attain the Utmost Emptiness, Hold Fast to Stillness
If Newtonian mechanics is relativity at low speeds, then the database is merely a special case of blockchain in a weakly distributed environment. Tracing transactions, WAL, blocks, and consistency shows the two are expressions of one underlying idea.
Time Travel in Programs
“Nor Time, nor Place, nor Chance, nor Death can bow / My least desires unto the least remove.” Three years after first falling for functional programming, I revisit immutability—and how genesis states, events, and persistent data structures let programs travel through time.
The Beauty of Mathematics
Passing the sheer wonder of mathematics to a six-year-old is no easy feat. But when dry numbers become patterns, curiosity ignites — and those same patterns reveal the deep laws of idempotence, commutativity, and associativity that govern distributed systems.
The ABT Network Deployment Chronicle
Two days ago, on March 28, the ABT network officially launched: a fully decentralized web of blockchains built from cloud nodes and woven chains, redefining next-generation blockchain infrastructure. Here is the story of the curious experiences before its birth.
A Practical Look at Bitmaps
With launch imminent, a colleague and I investigated a slow GraphQL API that returns non-empty blocks while skipping N and returning M. We show how an extensible bitmap, MSB lookup, cursor pagination, popcount, and benchmarks turn an O(n) filter into fast bit operations.
The Age of Sandboxes
On December 4, 1995, Brendan Eich's team bolted a patchwork language — Scheme's ideas, Self's prototype model, Java's syntax — onto Netscape Navigator as JavaScript. Nobody foresaw that the age of software sandboxes had quietly begun.
WireGuard: The Beauty of Simplicity
This article runs about 4,300 characters — 10 minutes for a casual read, 30 for a careful one. Recently, at an internal BBL session, I shared WireGuard: how this new-generation VPN achieves breathtaking simplicity through its interface concept, tunnel negotiation, and packet handling.
The Noise Framework: A Blueprint for Building Secure Protocols
The Noise Protocol Framework is less a finished protocol than a blueprint—a meta-protocol for engineers who want to design their own secure channels. Freed from the weight of PKI, it offers a lean, flexible foundation for building encrypted protocols over any transport.
Encrypting Files Peer-to-Peer with the Noise Protocol?
What if the Noise Protocol could encrypt files for peer-to-peer sharing? With only your public key, I can seal a file via a one-way handshake and leave it on untrusted storage—cloud drives, FTP, IPFS, even a blockchain—so only your private key can open it.
How to Store Passwords Safely?
Modern life gives us countless passwords, forcing us to reuse weak ones or trust managers. Drawing on Signal's key-derivation design, this essay shows how to derive every app password from one master secret—and introduces cellar, a small Rust tool built to do exactly that.
2020: Living Long Enough to See Everything — Mainstream Western Apps Trip Their "Circuit Breakers"
A single backend change at Facebook took down a huge swath of Western apps at once — TikTok, Spotify, Pinterest and more — in one afternoon. A postmortem on the fragility of third-party SDKs, type safety, and the hidden price of convenience.
The Linux Kernel Memory Model: Understanding the Rules of a Multicore World
Starting from a baffling concurrency bug, we dive deep into the elegant design of the Linux kernel memory model. When multicore processors make memory access unpredictable, how do we strike a balance between performance and correctness? This is a philosophical meditation on events, relations, and causality.
Ten Years of Microservices: From the Ideal of Small and Beautiful to the Return of Complexity
A deep dive into James Lewis's GOTO 2024 talk: a decade of microservice architecture, its nine defining characteristics and design philosophy, and the fresh challenges and second thoughts that cloud-native complexity has forced on all of us.
The Magic of the CPU: How Processors Work, and Why Every Programmer Should Understand It
A deep dive into how modern CPUs actually work — pipelining, branch prediction, out-of-order execution — using real experiments and profiling tools to show how clever hardware design delivers performance, and how to write code that exploits it.
Abandoning Multi-Environment Entirely: Reshaping the Next-Generation Serverless Architecture with Rust and Pingora
When we return to first principles and reconsider what "compute" really is, we find that multi-environment setups are an illusion, Kubernetes is an over-abstraction, and Serverless cold starts and connection-pool storms are self-inflicted. This essay dissects an environment-less architecture built on Baggage Header routing, a Pingora smart gateway, Firecracker microVMs, and a Rust Data Bus — an attempt to answer: what should compute look like in the post-Kubernetes era?
More Terrifying Than Forgery: A Phishing Email Signed by Robinhood Itself
SPF, DKIM, DMARC and ARC all pass on this email — because Robinhood really sent it. Nothing was forged. An unescaped Device field let an attacker ride Robinhood's own compliance-mail pipeline, then a Google open redirect laundered the link.
The Future of Logs: Rectifying Names
For a decade we built observability into a beautiful tower on a foundation of drifting strings—field names by convention, cardinality by luck, bills by endurance. This essay proposes a radical fix: use protobuf to give every observed event a proper name, unify logs, metrics, and audit into one versioned, owner-tagged, retention-aware event catalog, and keep an escape hatch called ForensicEvent for the unknown unknowns.