On Programming Philosophy

Programming philosophy is a grand subject, and I can’t promise to cover it both broadly and deeply in a mere hour or two. Let me instead recommend a good book: The Colon Classroom (冒号课堂), one of the few Chinese classics on programming philosophy. Alas, it is out of print; you can only admire it in a library (I ran into it at the National Library of China a few years ago).

Software ideas keep sprouting up one after another, but their essence is to reduce system complexity, to eliminate repetition, and to minimize code churn. Once you grasp this general direction, understanding any particular programming idea becomes far easier.

Most of the code below is short, clear Python.

By my own (admittedly imprecise) taxonomy, programming ideas fall into a few broad categories:

• Principles
• Paradigms
• Methodologies
• Patterns

Let’s unfold them bit by bit and see where we end up.

Principles

The principles I know of (or can recall on the spot) are mainly these:

• DRY (Don’t Repeat Yourself)
• OCP (Open-Closed Principle)
• SoC (Separation of Concerns)
• IoC (Inversion of Control)
• CoC (Convention over Configuration)
• Indirection (Layering)

“Don’t repeat yourself” is easy to grasp. The second time you write code of the same shape with little variation, a big question mark should flash in your head: am I repeating myself? If so, refactor it, encapsulate it, abstract it, or turn it into a function — the goal is always the same: eliminate the duplication. In my experience, DRY looks elementary, yet many large software companies fail to practice it; copy & paste is everywhere. From the very first line we write, we must hold to this principle, or else — guided by the broken windows theory — code quality slides swiftly into an abyss from which there is no return.

The OCP says that “software should be open for extension and closed for modification.” Suppose you write message-dispatching code with a single main function that handles every message: each time you add a message type, you must modify that function so it can handle the new one. Code that correctly applies OCP gives every message its own callback; the main function merely looks up the callback by message type and invokes it. Adding any new message then requires no change to the main handler at all. That is the most elementary illustration of “open for extension, closed for modification.” Many techniques in software development — inheritance, the Observer pattern — exist precisely to realize OCP.

These two are the most fundamental of principles; everything after them is derived from this foundation.

SoC sounds impressive, but it simply means decoupling: divide a complex system into subsystems and confine change to a subsystem as much as possible. If you were building for the web ten-odd years ago, you’ll remember HTML that mixed semantics with style, where touching a single hair moved the whole body; today’s sites keep HTML and CSS largely separate — what is God’s to God, what is Satan’s to Satan, each tending its own domain. That is SoC. Another classic application of SoC is the MVC design pattern — the system’s logic is split into Model, View, and Controller so that (ideally) a change in one layer does not affect the others.

The IoC principle is “Don’t call me, I’ll call you.” It propelled the software industry from the era of code libraries into the era of application frameworks. Think of libc: all kinds of functions sit there for you to command, and all the control is yours. Now consider an application framework like Django: you don’t control the system; you passively write functions or classes to spec, and the framework calls them when necessary. The benefit of IoC is that high-level details and logic are hidden, leaving the developer to focus on business logic. Write a web app with ChicagoBoss (an Erlang web framework), for instance: your code is essentially sequential and concurrency-agnostic, yet the system executes asynchronously and with heavy concurrency.

The CoC principle comes from Rails (or at least Rails popularized it). It means that, for simplicity’s sake, we write code according to certain conventions (which directory the code goes in, what the file is named, what the class is called), sparing us a great deal of needless trouble (without sacrificing flexibility, since conventions can be overridden by configuration). In Ember, for example, a component is CamelCase in the controller and dash-separated in the template. In Django, drop a file into “app/management/commands” and subclass BaseCommand, and your command runs via “./manage.py xxx”.

The Indirection/Layering principle also serves decoupling: split the system into layers and strictly regulate the calls between them. The most famous example of layering is the ISO/OSI seven-layer model; the most famous example of indirection is the hypervisor. The most famous line in software is: “All problems in computer science can be solved by another level of indirection.”

Paradigms

With principles done, on to paradigms. Two come to mind:

• GP: Generic Programming
• MP: Meta Programming

Many people see GP (generic programming) and think immediately of C++ templates and the STL. This GP is not that GP. Generic programming here is a question of abstraction — decoupling the algorithm from the data it operates on. Suppose we want to evaluate a string expression: “3 * 20 * 7 * 48”. That’s not hard in Python:

s = "3 * 20 * 7 * 48"
def calc(s, sep):
    r = 1
    for t in s.split(sep):
        if t != "":
            r *= int(t)
    return r
calc(s, "*")
20160

What if s is an addition expression? A better approach:

s = "3 * 20 * 7 * 48"
def calc(s, sep):
    op = {'*': operator.mul, '+': operator.add, ...}
    return reduce(op[sep], map(int, filter(bool, s.split(sep))))
calc(s, "*")
20160

In this implementation, the algorithm is abstracted away from the data.

Another example: the function below applies a test to each element of a given list, and if the test passes, applies an action, returning the list of results.

def process(l, test, action):
    def f(x):
        return action(x) if test(x) else None

    return filter(None, map(f, l))

This function applies to all sorts of scenarios — two that look entirely unrelated, say “find every female engineer in the company directory and give them all a 10% raise,” and “send a message to every one of my followers on Weibo who is in Beijing.” Most importantly, once process is written, it serves both (or more) scenarios with essentially no modification. This also shows that one role of GP is to realize OCP.

The principles and paradigms above are independent of any particular language — universal basic ideas. Metaprogramming is not. It depends heavily on what the language can do.

In the narrow sense, metaprogramming means code that can manipulate code as data; in the broad sense, it means code written at a near-language level to make code more dynamic. An example of the latter first:

class dotdict(dict):
    def __getattr__(self, attr):
        return self.get(attr, None)

    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__
dd = dotdict({'a': 1, 'b': 2})
dd.a
1

In Python, accessing a dictionary requires “[]”, but we can use the language’s own magic functions to map the “.” operator onto “[]”, so that we access a dict with “.” (as in JavaScript). A dict’s keys are unbounded; you can’t generate a method for every key, but by appealing to the language layer, it becomes possible. Likewise, if you’re writing an SDK for the Weibo API, you needn’t write one method per API — a single __getattr__ abstracts and unifies them all. That is metaprogramming in the broad sense: making code more dynamic.

Narrow-sense metaprogramming is best illustrated by Django’s ORM:

class TagItem(models.Model):
    class Meta:
        app_label = 'cayman'
        verbose_name = verbose_name_plural = _('related entity')

    tag = models.ForeignKey('Tag', verbose_name=_('category'), null=True)
    content_type = models.ForeignKey(ContentType, default=None, blank=True, null=True)
    object_id = models.PositiveIntegerField(blank=True, null=True)
    item = generic.GenericForeignKey('content_type', 'object_id')

A complex object-relational mapping, solved in this declarative way (you might even regard it as a DSL, a Domain Specific Language). Without metaprogramming, how would you implement such a mapping so elegantly?

Of course, when it comes to metaprogramming power, the Lisp family — which treats code entirely as data (to be expanded and evaluated as lists) — is a cut above. That is why I say metaprogramming ability is language-dependent. I’ve never actually written Lisp, but they say Lisp programmers, when building a system, first write a DSL for that system and then write the code in that DSL. So I’ve heard — I’ve not seen it with my own eyes.

Methodologies

Mainstream methodologies are basically three:

• OOP (Object Oriented Programming)
• AOP (Aspect Oriented Programming)
• FP (Functional Programming)

I won’t discuss OOP here; the term has been talked to death. Note that OOP is a way of thinking, independent of whether a language supports it. C, which does not support OOP, can still produce OOP code (see the device model in the Linux kernel); Python, which supports OOP, is still used by many to write procedural code.

AOP means separating auxiliary concerns from the primary concern — a touch of SoC. In Django we write many views, each with its own logic, but all of them must consider one thing: the user must be logged in (and authorized) before accessing them. That concern has nothing to do with each view’s primary concern, and it shouldn’t distract us. Hence (and for brevity I use the now-deprecated function-based view):

@login_required
def user_view(request, *args, **kwargs):
    user = request.user
    profile = user.get_profile()
    ...

Here the login_required decorator is a fine example of AOP thinking.

AOP is often a decoupling complement to OOP.

After so many years of evolution, OOP has slowly reached its inherent ceiling: to accommodate more business, it abstracts and layers endlessly, until it exceeds what the human brain can comprehend. Some design patterns try to push the vertical pyramid sideways (composite, decorator, and so on), but they cannot change OOP’s tree-shaped, pyramidal structure.

If OOP helps you build hierarchical systems, FP (functional programming) goes the other way: in FP, everything is flat. What you build are functions, as abstract as possible, and then you compose them into a system.

For instance, build a system that merges lists. A good OOP hand might do:

class Base(object):
    def __init__(self, l):
        self.l = l
    def reduce(self):
        raise NotImplemented
class Adder(Base):
    def reduce(self):
        n = 0
        for item in self.l:
            n += item
        return n
class Multipler(Base):
    ...

But with FP, you’d probably do:

def list_op(f):
    def apply(l):
        return reduce(f, l)
    return apply
adder = list_op(operator.add)
multipler = list_op(operator.mul)

Functional programming achieves complex functionality by transforming and composing basic functions, and does so very elegantly. Back to the earlier example:

return reduce(op[sep], map(int, filter(bool, s.split(sep))))

Code at once readable and elegant like this represents the future of writing code. Another example (Haskell):

boomBangs xs = [if x < 10 then "BOOM!" else "BANG!" | x <- xs, odd x]

Even if you’ve never studied Haskell, you grasp the meaning of this code at once.

Functional programming has some or all of the following traits (depending on the language’s capabilities):

• Immutable data
• First-class functions
• Higher-order functions
• Pure functions (no side effects)
• Recursion & tail recursion
• Iterators, sequences
• Lazy evaluation
• curry
• Pattern matching
• Monads….

Many of these ideas align with what high-concurrency development on today’s multi-core, multi-threaded hardware demands. Which is why languages like Erlang and Haskell are increasingly valued and deployed in all kinds of production environments.

Patterns

Patterns are ideas for solving specific problems that recur over and over in system development. Design patterns were first catalogued by the GoF (Gang of Four), then made famous in Java. As languages have evolved, many patterns have been absorbed into the language itself (iterator), some have hardened into how you write code (bridge, decorator), and some are used constantly inside application frameworks without your knowing it — classic MVC, Django’s middleware (chain of responsibility), command (command pattern), and so on. Given the time, I won’t go through them one by one.

Finally: we write code to solve problems, not to flex our muscles. With the big principles in mind, use whichever paradigm, methodology, or pattern fits your hand and solves the problem best. Code first serves functionality, then readability — never some ideology. In other words, don’t do OO for OO’s sake, don’t do MP for MP’s sake; that’s meaningless.