The Power of Abstraction

One of the markers of human intelligence maturing from childishness into adulthood is the ability to recognize and manipulate numbers. At three or four, we could already rattle off the numbers up to a hundred, forwards and backwards, at will — and yet, for us as children, numbers were still just numbers. Even after counting the twelve grapes on the table in front of us and eating one, we’d still have to painstakingly count them all over again to be sure there were eleven (don’t ask me why I know this so well). At that age, divorced from concrete objects, numbers meant nothing to us.

As we grew older, as our brains developed and the drilling of elementary school took hold, we became able to wield numbers at will — and so effortlessly that we could no longer even feel that this was an abstraction of real life. Cabbage at eight jiao a jin, celery at one yuan a jin, two jin of each, plus a bottle of soy sauce for two yuan forty: we’d pull out six yuan without a second thought and stroll out of the market happy.

By middle and high school, abstraction had already moved up from numbers to a higher plane. Plane geometry and analytic geometry carried us from numbers to figures, while algebra (from concrete numbers to abstract letters) led us up to the level of functions. The brainteaser of our elementary years — “My father’s age is three times mine; in ten years, his age will be twice mine. How old am I now?” — becomes, at this point, an introductory exercise in equations. If my age is x and my father’s age is y, we can set up the equations like this:

y = 3x
y + 10 = 2(x + 10)

But such equations are countless, and for each one we set up we have to compute a solution. So we go a step further and formalize and abstract the problem:

y = a1x + b1
y = a2x + b2

Now we have a solution for x: (b2 - b1) / (a1 - a2). It works like a machine: for any similar problem, we simply plug into it and it produces a solution. This is a formula (or a theorem). Formulas — and the process of proving them — ran through our entire middle-school years.

In college, the degree of abstraction climbs another giant step: we abstract from numbers to relations. Calculus studies and analyzes the change of things and their relation to the things themselves; probability theory studies the likelihood of random events; discrete mathematics studies mathematical logic, sets, graph theory, and so on. All of these are the study of relations. The knotty ailments of middle school become, at the college level, pediatric trivia to be dispatched without argument.

…

Not everyone, however, can keep up with these upgrades in abstraction. Sometimes you get stuck at a certain level (this programmer, for instance, never did get college math straight — skipping class ruins your life) and cannot break through. At such times, the only recourse is relentless practice to harden this mode of thinking into a habit, and then to find its meaning within that hardening. It’s like Zhang Wuji on Ice Fire Island being forced by Xie Xun to memorize martial-arts manuals, or the three-year-old you struggling to familiarize yourself with each number over and over. They mean nothing at that moment; when the day arrives that they must mean something, you will have an epiphany.

Likewise, writing code requires the capacity for abstraction — desperately. If you don’t want to spend your whole life as a junior code monkey, if you want to write code you can feel satisfied with, if you don’t want to be replaced by more advanced coding tools in the future, you need to learn to abstract.

The first level of abstraction is turning a concrete problem into a function (or a class) and solving it with a program. Every programmer who calls themselves a software engineer can manage this level: once you’ve mastered a language’s syntax, you can map the problem onto that syntax and write acceptable code. This is rather like a schoolchild’s grasp of numbers: applicable at will, without confusion.

The second level of abstraction is writing functions that solve many problems. This is like the general solution to the linear system above, (b2 - b1) / (a1 - a2): you create a machine, and that machine can handle everything it can handle. Something like this code:

function getData(col) {
    var results = [];
    for (var i=0; i < col.length; i++) {
        if (col[i] && col[i].data) {
            results.push(col[i].data);
        }
    }
    return results;
}

Seen on its own, it already looks clean, a precise mapping of the concrete problem at hand. But once I abstract out the loop, the filtering inside the loop, and the processing of the filtered results, the following code emerges:

function extract(filterFn, mapFn, col) {
    return col.filter(filterFn).map(mapFn);
}

This is a general solution, a machine. With it, you can solve any dataset filter-and-map problem. Of course, you can abstract further:

function extract(filterFn, mapFn) {
    return function process(col) {
        return col.filter(filterFn).map(mapFn);
    }
}

Note that although the two abstract to similar results, their scope of application is not the same. The latter is more like a machine that produces machines (a function returning a function), and it decouples the problem further. This decoupling not only generalizes the code, it also splits the code’s execution into two stages, decoupling it both in time and at the interface. Thus you can call extract in context A and process in context B to produce the actual result. Contexts A and B need have nothing to do with each other: A need only supply filterFn and mapFn (at system initialization, say), while B need only supply the concrete dataset col (when a web request arrives, say). This temporal decoupling greatly amplifies the code’s power. Interface decoupling works like the universal travel adapter you take on trips: one end of your function plugs into the system in context A, the other into the system in context B.

(Of course, in languages that support currying, the two are in fact equivalent; I’m just looking at the difference from the standpoint of thinking.)

The second level of abstraction isn’t hard to master either: the various patterns of OOP and the higher-order functions of FP are all powerful weapons for helping us reach it.

The third level of abstraction is the construction of foundational models. If at the previous level we were seeking commonality among the problems to be solved, then at this level we seek commonality and connection among the solutions. Take, for example, establishing contracts within your program’s world. In software development, the most fundamental contract is the type system. In strongly typed languages like Java and Haskell, the language itself embodies such contracts. So what about weakly typed languages like JavaScript? You can build the contract yourself:

function array(a) {
    if (a instanceof Array) return a;
    throw new TypeError('Must be an array!');
}

function str(s) {
    if (typeof(s) === 'string' || s instanceof String) return s;
    throw new TypeError('Must be a string!');
}

function arrayOf(type) {
    return function(a) {
        return array(a).map(str);
    }
}

const arrayOfStr = arrayOf(str);
// arrayOfStr([1,2]) TypeError: Must be a string!
// arrayOfStr(['1', '2']) [ '1', '2' ]

Another example. In the world of programs, no one’s perfect. You run a query against the database, and there are two possibilities (ignoring exceptions for now): the result contains a value, or it is empty. This binary outcome is something we deal with almost every day. If every step of your program produces a binary outcome, your code will form a pyramid of if…else structures:

function process(x) {
    const a = step1(x);
    if (a) {
        const b = step2(a);
        if (b) {
            const c = step3(b);
            if (c) {
                const d = step4(c):
                if (d) {
                    ...
                } else {
                    ...
                }
            } else {
                ...
            }
        } else {
            ...
        }
    } else {
        ...
    }
}

Hard to read, and hard to write. Can this binary-structured data be abstracted into a data structure? Yes. You can create a type, Maybe, containing two possibilities, Some and None, and then lay down contracts like this: stepx() accepts a Maybe and returns a Maybe; if the incoming Maybe is a None, return it straight through; otherwise, process it. The new logic looks like this:

function stepx(fn) {
    return function(maybe) {
        if (maybe isinstanceof None) return maybe;
        return fn(maybe);
    }
}
const step1x = stepx(step1);
const step2x = stepx(step2);
const step3x = stepx(step3);
const step4x = stepx(step4);

const process = R.pipe(step1, step2, step3, step4);

As for how to write this Maybe type itself, we’ll leave that aside for now.

(Note: Haskell has exactly this type, Maybe.)

The fourth level of abstraction is making rules — building a world in which an entire problem space can be solved. This is metaprogramming. When people talk about metaprogramming, the first thing they think of is languages like Lisp and Clojure, where “code is data and data is code” and you can manipulate the syntax tree at runtime. Not so fast. A Clojure programmer without strong abstractive power will not necessarily write better metaprograms than a JavaScript programmer who has it. It’s like Lü Bu’s Sky Piercer halberd: in your hands or mine, it’s only good for selfies.

Metaprogramming ability comes in many stages; here let’s discuss the entry level, the kind that doesn’t require language support: the ability to abstract an actual problem into rules. In other words, before founding a company (solving a problem), you first draft the company law (establishing the rules of the problem space), and then found the company according to that law (using the rules to solve the problem).

Say you need to write a program that parses a wide variety of wildly non-uniform feeds, normalizes them into a single data format, and stores them in a database. The source data might be XML, might be JSON, and the way their field names are defined is anything but uniform; even among XML feeds, the same piece of data lives in an attribute in one and in a child node in another. How do we handle that?

Of course we could write a handler function (or class) for each feed and reuse as much of the common part as possible. But that means every new feed you support requires new code.

A better way is to define a language that can map source data onto target data. Once that language is defined, you write a single parser and you’re done, once and for all; adding a feed later just means writing a description in that language. Here is the description of a JSON feed:

{
    "video_name": "name",
    "description": "longDescription",
    "video_source_path": ["renditions.#max_renditions.url", "FLVURL"],
    "thumbnail_source_path": "videoStillURL",
    "duration": "length.#ms2s",
    "video_language": "",
    "video_type_id": "",
    "published_date": "publishedDate",
    "breaks": "",
    "director": "",
    "cast": "",
    "height": "renditions.#max_renditions.frameHeight",
    "width": "renditions.#max_renditions.frameWidth"
}

And here is the description of an XML feed:

{
    "video_name": "title.0",
    "description": "description.0",
    "video_source_path": "media:group.0.media:content.0.$.url",
    "thumbnail_source_path": "media:thumbnail.0.$.url",
    "duration": "media:group.0.media:content.0.$.duration",
    "video_language": "",
    "video_type_id": "",
    "published_date": "",
    "breaks": "",
    "director": "",
    "cast": "",
    "playlist": "mcp:program.0",
    "height": "media:group.0.media:content.0.$.height",
    "width": "media:group.0.media:content.0.$.width"
}

I won’t go into the details of this description language, but by defining a language — or rather, defining rules — we have successfully changed the nature of the problem: from a data-processing program to a parser; the input changes from individual feeds to source files of a description language. Once the parser is written, the problem is solved once and for all. You could even write a UI for this language, so that non-engineers on the team can add support for new feeds with ease.