Wasting Memory? What's the Big Deal?

Think back to Gates in his glory days, when MS was riding high: between jokes and laughter, 640k of memory was all a man needed. — Programmer Jun

We’re no longer in an era of pinching memory out from between our fingertips. In mainstream interpreted languages the bit has either vanished without a trace or been locked away in the attic as a “high-(idi-)level” feature — like managed code in .NET, repressed into the plight of a neglected wife. String-based protocols like HTTP have only fanned the flames. The dignified, elegant, faintly book-scented data structures the internet world once painstakingly built have been ceded to crude, lumpen JSON.

In yesterday’s slides, at the part about how to improve memory, I put up a screenshot of the RFC 793 TCP header. A sharp-eyed reader spotted it, and we had the following exchange (read: grumble):

Slide 33 is titled “memory” — so why is the diagram a TCP header? :)

Sharp eyes and a careful mind! I couldn’t be bothered to write text for that slide, so I took a jab at the declining standards of the day. Web devs reach for JSON at the drop of a hat; their command of data has regressed to integer / string / array / map. No more enums, no more bits; anything that ought to be an atom/symbol is just a string…

I suspect many people will shrug this off. We’re no longer in an era measured in megabytes, let alone kilobytes. A gigabyte of RAM costs about the same as a bowl of oden at the 7-Eleven. We’re not short of money: 32G isn’t enough? Upgrade to 128G. Still not enough? As long as the system supports it, we can scale out by the terabyte — what’s the big deal? Squeezing a few bytes of memory at the cost of our company’s precious programmer time — that’s hundreds of thousands of dollars a second we’re talking about!

All right, then, let’s talk about it.

Start with the easy-to-grasp part. A data structure that fits comfortably in 256 bytes — what happens if you let it balloon to 2k?

• CPU sequential read time inflates 8x
• Network transmission time grows 8x

Fine — those are small numbers, nanoseconds and microseconds, nothing to worry about!

Let’s keep going:

To be fair, let’s say neither side does gzip (if you insist on being pedantic, make the premise 1k and 8k instead). Since 2k > the standard MTU (1514), the data can’t go out in a single packet. So the sender must fragment it and the receiver must reassemble it. Fragmentation is the root of all evil. As I mentioned in yesterday’s article, a single round trip from the US to Europe is 150ms; split one packet into two and you need two round trips before the complete data structure arrives and downstream processing can begin — that’s 300ms minimum, right out of the gate. IP is unordered, so when fragments reach the receiver the rear may become the vanguard and the second fragment may land first. That means reassembly — and reassembly is the least of it: one extra packet on the wire means a markedly higher chance of packet loss. And packet loss, as everyone knows, is the mortal enemy of network performance.

Bigger data means more read/write time, more transmission time, possible fragmentation and hence reassembly, plus a higher packet-loss rate. Each of these multiplies the others — the blunt-spoken programmer’s term for the result is a cluster of f**k.

So don’t underestimate a few extra bytes of memory you never should have used. Like the peasants in “A Few Extra Bushels of Rice,” the hidden butterfly effect may flap its wings straight into your peace and quiet.

OK — even that isn’t such a big deal. What used to finish in 150ms now takes 500ms in the worst case. No biggie.

Now let’s come at it from another angle: capacity.

Say a server has 16G of RAM, 12G of which belongs entirely to your app. With a 256-byte data structure, setting aside other memory overheads for the moment (mm fragmentation, control blocks, and so on), you can support a capacity of 48M. With a 2k data structure, that shrinks to 6M. Again, an 8x gap. Suppose your system must hold 40M of these as hot data. In the first case, a cluster with one primary and two replicas — three machines in a full mesh — is plenty. In the second, you need at least 3 × 7 = 21 machines, and you’ll have to think a little harder: will full-mesh synchronization cost too much? Should we switch to a gossip protocol for the cluster?

If numbers don’t speak to you, let’s convert to money. An M4.xlarge with 16G of RAM runs about twenty cents an hour; at 750 hours a month that’s $150. Three of them cost $450; twenty-one cost $3,150. Duang — just like that, the difference is the salary of a junior engineer plus a programmer-encourager (yes, that’s a real job title in China).

And managing more machines takes more time — not linearly more, but more nonetheless. Your side takes −1 morale; their side gets +1 to its army, plus a free Archangel, so everyone’s morale goes +1. Back and forth, the efficiency gap widens considerably.

Of course, you can say we have plenty of seed capital and money is no object. Fine — let’s keep going.

Next, let’s look at rigor in program design. For reading convenience, here’s the TCP header from the previous article pasted back in:

As we can see, a TCP header without options is 20 bytes, 5 words.

Now suppose we described a TCP header in JSON:

{
    "source_port": 16383,
    "destination_port": 80,
    "sequence_number": 938872468,
    "acknowledge_number": 938872468,
    "data_offset": 0,
    "urg": false,
    "ack": true,
    "psh": false,
    "rst": false,
    "syn": true,
    "fin": false,
    "window": 16384,
    "checksum": 8876,
    "urgent_pointer": 0
}

This syn-ack TCP header comes to 238 bytes, 60 words.

Nothing hurts like a side-by-side comparison — I bet you silently upvoted my earlier labels of “dignified, elegant, book-scented” and “crude,” didn’t you?

From a software design standpoint, the TCP header reflects careful deliberation — word alignment, cache friendliness, clearly defined value ranges for every field, and while it leaves a few pits behind (window, for instance), it accounts for how it might be extended later (dig your own pit, fill your own pit). Crucially, it is disciplined: every bit earns its keep, and it keeps its own payload as small as it can. This JSON structure, by contrast, follows the TCP header definition to the letter, yet reeks everywhere of improvisation and shallowness.

To make sure it’s correct, we might also need to write a lengthy JSON schema to constrain the data structure. But then, who guarantees that the JSON schema itself is correct?

JSON-centric API design finds itself in exactly this bind. Beyond wasting space, the pervasive attitude of “I haven’t figured out what I need yet — let me put something up there and patch it in later” leaves programmers utterly unconstrained — frontend and backend developers, popsicles in mouth and cigarettes lit, “collide” a “data interface” into existence. Then a few requirements come in, so you add some fields, useful or not; JSON is flexible enough, just stuff them in. Over time the structure grows ever more unwieldy, like Ptolemy’s geocentric model, until it stands on the verge of collapse.

Conversely, if a data structure is less flexible and more constrained, it looks at first as though your hands are tied and your room to maneuver is small — but little do you know, constraints are no bad thing: they force you to consider the system’s design more clearly, and thus to cut a path through the impossible and arrive at better choices.