Musings on Software

(I)

In the world of software there’s a fellow named Greenspan. Most people probably aren’t familiar with him, but there are few who’ve written code and don’t know this line:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

This is the so-called “Greenspan’s Tenth Law” (don’t bother looking — there are no first nine). The man never produced anything especially brilliant in his whole career, but he did leave behind this one law, destined to be immortalized in the programmer’s chain of contempt.

As a C programmer, having felt the force of that sentence time and again, I finally charged into the Lisp camp late last year. I started by taking a scalpel to Racket and studied it in a state of rapture; then, unable to resist the temptation, I leapt into the golden club of Clojure and had my head rinsed by the likes of Rich Hickey and David Nolen. Before I stormed in, I harbored an evil ulterior motive: I wanted to stand atop the chain of contempt and look down on all sides. After storming in, I trembled, breaking into a cold sweat. The more I learned, the more I despised myself as utterly worthless.

Learning a language that is “heretical” to you is like opening up a whole new world for yourself, pulling you out of the Dunning-Kruger effect. The feeling is like a C programmer using Python’s REPL for the first time, seeing list and dict for the first time, so beautiful it makes you want to cry. Of course, every language has its own appropriate domain; you can’t judge superiority by whether a language has a REPL. What matters is how much wisdom you can extract from it that you didn’t know before. A Python programmer who learns to read C, who figures out the preprocessor, compiling, linking, loading, and in the process of disassembly “sees” the sinews of the system the way Cook Ding carved the ox — that person will also weep with happiness.

That is the benefit of learning languages that are heretical to you. For a Python programmer, learning C, Erlang, Clojure, or Haskell all count as heresy; learning Ruby does not. These two siblings need remarkably little in the way of a paradigm shift — even Cython and MRI’s GIL (Global Interpreter Lock) are like one big happy family. When learning anything, a paradigm shift is crucial — something like the saying “break it down before you build it up, fail before you succeed.” It is a key to continuous growth.

(II)

C and assembly are like the Taizu Long Fist. In the hands of some nameless servant it’s no better than children roughhousing; in the hands of Xiao Feng, every move is lethal. The syntax itself is extremely simple — you can count the keywords on your fingers and toes — and a hello world takes all of two minutes. But only when you’ve achieved a thorough mastery of the system and cultivated all your inner disciplines can you unleash its tremendous power.

PHP/JavaScript is the Star-Absorbing Art. It’s not hard to practice; with no internal energy you still get started fast, and ready-made modules are everywhere on the web. Take them as your own and your level shoots up immediately. Yet its fatal flaw keeps you wandering at the quasi-first-rate tier, and if you use it badly it can even backfire on you at the critical moment.

Python/Ruby is the Taiji Sword — endlessly varied, easily handling anything from a humble script to highbrow machine learning. But its innate shortcomings in performance and interpreter implementation are its weak point (Guido and Matz are actually hard done by: I gave you a power drill, you insisted on using it to bore through steel plate, and the performance is bad — my fault?). So when you hit a compute-intensive or I/O-intensive problem, dealing with it really wrecks your kidneys.

Erlang/Elixir is like the Eighteen Dragon-Subduing Palms — sweeping, expansive, powerful. But it’s hard to get started, its ideas run deep, few people know it, and you have to grind through it alone. The actor model, supervision trees, message passing, pattern matching — just grasping them thoroughly takes half a year. Master them, and your every move carries the bearing of a grandmaster.

Clojure is like the Nine Swords of Dugu — “wind and thunder are one transformation, mountain and marsh another, water and fire another.” Infinite in variation, yet its core is to meet ten thousand changes with the unchanging. However the requirements shift, you seize the main threads, find the opening, and dissolve it with macros and polymorphism. Code is data and data is code; the light governs the heavy, complexity (objects) is shed for simplicity (functions). On defense, not a drop leaks; on offense, nothing stands in the way.

Haskell is like the Great Shift of the Universe — without deep internal cultivation it’s very hard to comprehend. Lazy computation and monads do exactly this kind of mind-bending force-redirecting work. A program is nothing but a series of transformations between input and output: do you pass data move by move, or do you pass computations — shifting the stars and moving the constellations? Answer that question and you’ve passed through the gate of Haskell.

(III)

Professor Randy Pausch (yes, that name has probably callused the ears of anyone who reads my articles regularly) once told a story. When he was a kid playing football, his coach, during the scrimmage drills, didn’t give them the ball. One kid was unhappy: “Coach, are we playing football here or are we playing football?” The coach stopped the kids and asked:

“How many players take the field in a game?”

“Twenty-two.”

“How many of them have the ball in their hands?”

“One.”

“I’m teaching you how the other twenty-one play.”

Looking back on this story, Randy said: fundamentals, fundamentals, fundamentals. The flashy stuff is like the part of the iceberg above water; what we don’t see is the far more critical part below. So when you learn a language, the syntax is just that football flying back and forth. Catching the ball and throwing it far doesn’t mean you know how to move without the ball, to cleverly position yourself on defense, to appear out of nowhere on offense. Learn a language without grasping its fundamental ideas, and you’ll only ever skim the surface.

(IV)

After writing code for long enough, some things are simply inescapable: a request flowing into the system first has to be authorized and authenticated, then validated, then routed, then put through all manner of transformations, persisted where necessary, and finally output as a response.

So Greenspan isn’t entirely right. For most people, writing software is like writing an ad hoc, informally-specified, bug-ridden, slow implementation of a small subset of a compiler. We just use badly designed, crude, primitive means — a patchwork of classes, functions, and if-else — to cobble together a compiler that only works in one specific scenario.

Or a database. In fact, a database is a compiler and a compiler is a database. It depends on how you look at it.

(V)

It seems we’re no longer in the age of lex/yacc or bison/flex. I once watched a colleague laboriously parse a system’s data files line by line with Perl, never once thinking to write a BNF grammar. BNF simply wasn’t an option for him.

Databases too are gradually losing their stored procedures and triggers. The new generation, born under the web and raised in the startup boom, has classified these contraptions as dirty little vendor-lock-in artifacts and won’t touch them lightly. I myself haven’t written a trigger in years. Recently, dealing with a legacy third-party Java system with no hook interface, I needed to track updates to specific columns in a certain table so that my code could work without modifying the system (and I have no ability to modify a complex EJB system anyway). So I dusted off the old trade and started wielding triggers and temp tables again. A colleague saw it and said: “Huh? How does this thing work with an ORM?”

When Rich Hickey talks about tradeoffs, he says you need at least two solutions before you can even speak of a tradeoff. Yet most of the time finding even one solution isn’t easy, let alone two or three. Unfortunately, almost everyone who writes code has “tradeoff” on their lips whenever they do design.

(VI)

When we choose technologies and get the job done, we forget that software is really in the service of business goals. Any code not optimized for real business goals is just hooliganism. As programmers, it’s easy to slip into a programmer-centric state of mind:

• That’s not a goddamn bug, the user used it wrong.
• The customer who filed this requirement is an idiot.
• The feature is implemented; it’s not my fault nobody uses it.

Business cares about cost/benefit, ROI, time to market, profit; programmers care about tests passing, code committed, and no damn requirement changes.

A friend of mine works at a startup where development is in utter chaos: no code review, no unit tests, no CI. Developers test things themselves for a bit and push to production, no logs written, and a partial system outage is only discovered when users report it. But their business is doing well. Bad software and a mountain of technical debt can always be paid off gradually by hiring better people; but if the business is bad, what does it matter how brilliant the software is or how gracefully the CI pipeline dances?

So programmers shouldn’t complain about being undervalued and having no voice. Business vision is the core. If you don’t cultivate business vision, if you can’t find the point where a product gets used and customers are willing to pay, you’ll always be a hired hand (even as a CTO you’re a hired hand); even as a founder, you’re theoretically a replaceable role.