Measurement

In the Lean Startup framework, measurement is a critical link — it sits between “build” and “learn.” Once all or part of a system has been built, we need to find appropriate means to measure it, so as to gather enough information to feed further learning and reflection (learn). A software feature that cannot be measured is like a doctor’s prescription for the common cold: as for how it finally turns out, heaven knows.

“Heaven knows” is not an attitude an engineer should hold. You say that once feature X is done, system performance will be great, will improve dramatically — then in what circumstances, how great exactly, improved to what level? There has to be a number. There are two important parts here: the number itself, and the context behind the number. Generally speaking, measurement must distinguish context. A number without context is like a theorem stripped of its constraints and premises: nothing wrong with it in itself, but used in the wrong setting it becomes absurd. I often see articles claiming that a single server handled 1M concurrent connections — yet how many of those 1M are active, how many messages flow in and out per second, they avoid entirely. Such a benchmark has some academic value but little practical meaning.

Now let’s look at the number itself. I once attended a training session on software estimation — quite interesting. It covered two properties of numbers: accuracy and precision. Take pi = 3.1415926… If a measurement yields pi = 3, it is relatively accurate but not precise; if a measurement yields 4.1415926586897985384626…, it is precise but not accurate. Seeing this example, most people laugh: without accuracy as a premise, what good is precision, however high? What pi actually is, is basically common knowledge — many can recite thirty-odd decimal places — so you can tell right from wrong at a glance. But with unknown software measurements, we often start playing with precision long before we’ve established accuracy, so jokes like 4.14… naturally follow.

So a good measurement should be an accurate and precise result under a given set of preconditions.

Take the concurrent-connection measurement mentioned above; a better formulation of the result would be (I’m making this up): under 64G memory, an Intel Xeon 5650 (6-core) CPU, a gigabit NIC, and Ubuntu 14.10, running 1M concurrent connections, 10% of them active, with each active connection sending and receiving an average of 10 64-byte packets per second, memory utilization averages 83.76% and CPU utilization averages 92.59% after stabilization.

That software features are easy to measure is a blessing for the ages, but reality is that most systems we build are not very measurable, and even when a system is measurable, its individual components are not. Open your source code and check whether each module (or each key module) has a benchmark — it’s obvious at a glance.

Many software features lack measurability because they fail to achieve a proper separation of concerns. A feature too tightly coupled with the rest of the system naturally loses independent measurability; when many such features pile up, even if the system as a whole is measurable, a serious performance regression between two releases is very hard to pin down to one or a few features and fix, precisely because no individual feature is measurable.

In many large software systems, with features in abundance, performance is sometimes an (implicit) promise. For example, an IPC channel: under the simplest send/receive model (the sender’s simulated messages are already waiting in memory, and the receiver simply discards each message after verifying its integrity), the per-second throughput, error rate, and loss rate amount to a promise to callers — callers design and optimize their subsystems on that premise, and if the premise shifts significantly, it may directly affect the callers’ subsystems. For instance, if in version 1 message throughput is 100k and in version 2 it drops to 50k, the whole system may be in serious trouble.

Hence we may need to introduce the concept of Continuous Benchmarking, or Continuous Profiling. We are fairly familiar with Continuous Integration — it is a guarantee of code quality: the higher the CI frequency, the shorter the time to find the root cause when CI fails (because the intervening code change is smaller), and bugs are nipped in the bud. CB/CP is a similar idea: we want to nip performance problems in the bud, rather than running a performance benchmark months later, discovering a serious problem, and scrambling to fix it.

Just as CI’s premise is good, independent, and (gradually) complete UT suites, CB/CP’s premise is good, independent, and (gradually) complete BT (benchmark test) suites. BT suites must be hardware-agnostic, for only then is independence guaranteed. Given that CB/CP values vertical comparison, as long as the hardware running CB/CP stays the same, we can tell whether each benchmark’s performance went up or down.

If you agree with the above, then you will see Bezos’s 2002 mandate on Amazon’s entire system architecture in a new light (excerpted from an article by a Google engineer; for the original, click “Read the original” at the end of this piece):

1. All teams will henceforth expose their data and functionality through service interfaces.

2. Teams must communicate with each other through these interfaces.

3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols – doesn’t matter. Bezos doesn’t care.

5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6. Anyone who doesn’t do this will be fired.

7. Thank you; have a nice day!

Good God — this is not a CEO’s insight at all; Bezos is plainly gunning for the CTO’s rice bowl!

…

After reading so much of my dry and rather unreliable opinion on software measurement, let’s lighten up and run a little experiment — measure yourself:

1. Take a blank sheet of paper and write down your estimate v1 of how long it will take you to compute 1234 x 9876 (pure arithmetic — no clever tricks, such as turning 9876 into 10000 - 124).

2. Open the timer on your phone and start working it out on paper. When you have the result, note the elapsed time u1.

3. Then write down your estimate v2 of how long it will take you to verify that the result is correct.

4. Start the timer again and begin verifying. When verification is done, note the elapsed time u2.

5. Compute w = (u1 + u2) / (v1 + v2) x 100%

Next time your boss asks how long feature A will take, don’t rush to blurt out the intuitive number that pops into your head — multiply it by w first, then reply. Measure your software often, and measure yourself often too. That’s as much as this programmer can do for you. ^_^