How to Read a Codebase

Last time I talked about the importance of reading source code the way you read a book. Today I want to talk about how to read a codebase. By “a codebase” I mean anywhere from a few thousand lines to tens of thousands, sometimes even hundreds of thousands. Together these lines of code form an organism that accomplishes certain important functions. Take a few famous, full-fledged web frameworks — the founding master Rails, its senior disciple Django, and the junior sister Phoenix:

The contrast is fascinating — Rails and Django both hover around 200k lines, while Phoenix is a full order of magnitude smaller, at roughly 20k. Do they achieve roughly the same functionality? Could the expressiveness of the languages really differ by that much? That doesn’t add up. In fact, what Phoenix implements, benchmarked against Rails, corresponds more properly to actionpack and actionview combined — about 80k lines. And Rails’s built-in activemodel/activerecord should be compared to Elixir’s ecto — which, as it happens, is another 80k-to-20k comparison. This gap reflects a difference in the expressiveness of the languages, and at the same time a difference in the maturity of the frameworks.

A body of code in the tens of thousands of lines is like a book of moderate thickness — reading it is neither overly taxing, nor does it leave you feeling the end came too soon.

Some gigantic codebases — the Linux kernel, for instance — are built on the scale of War and Peace, so vast and magnificent that they drive you to despair, far exceeding what any of us could hope to read and comprehend. The result is that each time we resolve to read them, we pour in enormous effort, only for it to feel like tossing a pebble into a lake: a ripple or two, and the surface quickly settles back to calm.

If you can’t read it all, don’t despair. Divide and conquer: set a small goal first, and read a portion at a time — say the scheduler (20k) or memory management (80k).

Once you’ve chosen a reasonable scale and picked the source you mean to read, you can clear off your desk, set out your Mac, prepare your brushes and inkstone (as it were), set aside anywhere from an hour to half a day, and begin wandering in the ocean of code.

Because my previous article ([link]) compared reading books and reading code from start to finish, many people will inevitably wonder whether this one will draw an analogy with How to Read a Book and offer the same approach: elementary reading, inspectional reading, analytical reading, syntopical reading. And indeed, these reading methods are highly instructive for reading code. Take inspectional reading: reading a large code project follows a similar logic:

• We read the readme in the root directory, or any file that looks like it’s teasing you, begging you to click on it. This is like a book’s preface — it helps you understand the intent of the codebase a little better;
• Next, we focus on the directory structure and the filenames of each source file. These are like a book’s table of contents. If each directory has its own readme, skim that too. Many languages and frameworks follow conventional directory structures (Convention by Contract), so from the directories alone we can quickly tell which parts can be skipped — Django’s management/commands directory, Elixir’s mix/tasks directory, for instance. These directories carry the side plots; read them when you need to, or when you’re bored, and no great harm done;
• Then we begin to thread the main line from the entry point. The main line differs across languages and frameworks — C’s entry is main(), an Erlang/Elixir OTP app’s entry is app:start, Node.js often starts from the root index.js, and so on. Generally speaking, an application’s main line is fairly clear; follow it down and you arrive at a main loop. A framework’s main line is more opaque, because a framework is usually the abstraction extracted from applications.

But I won’t lean too heavily on that comparison here — if you’re really interested, why not read the book yourself (and if you’ve already read it, read it again through the lens of reading code)? After all, it’s far more complete and systematic.

I want to approach this from another angle — the scenarios of reading. Mention “scenarios,” and many people will think of a famous book: Scenario Analysis of the Linux Kernel (Linux 内核情景分析). Its author clearly grasped the essence of reading code: follow a thread and read end-to-end through a self-contained body of content. In different scenarios, the information we have, the information we lack, and the goals we want to achieve through reading all differ — and so, obviously, do the methods. It’s just like reading books: to become wise, read history; to become refined, read poetry; to become rigorous, study mathematics; to become profound, take up philosophy, and so on.

In what follows, I’ll start from several scenarios for reading code and discuss my own humble lessons learned about reading code.

Scenario One: Reading Code to Crack a Case

This is our most common code-reading scenario. At work, you can’t avoid using all sorts of open-source systems (other people’s code). Along the way you’ll hit all sorts of bizarre problems — problems that may stem from an incomplete grasp of the docs, or from copying some sample off the internet that doesn’t quite fit your use case, or from genuinely running into a bug (八阿哥, “the eighth prince” — a homophone for “bug”). As you scramble to solve it, if your colleagues can’t help, if Google/StackOverflow aren’t up to the task, if the forum’s “urgent, waiting online” posts go unanswered, you’ll start to go mad, as if a Dementor has latched onto you and all the beauty and hope in your life begins slipping away.

At this point you have no choice but to be like a detective on CSI — follow a single thread, start dissecting the code, and try to reconstruct the truth from the fog. You shut out all distractions and noise, focus, carry the clue, follow the problem, and read — and only read — the code directly relevant to solving it. This state I call “hunter mode” — we’re like a lion on the African savanna chasing a zebra that has strayed from the herd, gathering the burning cosmos within us into a single point, eyes locked on the direction our prey is running, legs like the wind, momentum like lightning, mind ceaselessly calculating whether the thunderous strike is best delivered by the throat, across the face, or by the hind legs. A stone on the road cuts your foot — it hurts; a bizarre mishap (幺蛾子) hits you mid-sprint — annoying — but neither is a big deal. Even if two rainbows rarely hung over distant Kilimanjaro, gorgeous enough that a selfie posted to WeChat Moments would surely break a hundred likes, you’d have no attention to spare for such unimportant details.

Focus — concentrating force to attack one point and only one point — is the primary way of reading code in this scenario.

Take an nginx cache problem I ran into as an example (举个栗子 — “chestnut,” a meme for 例子, “example”). A year ago, after I took over Tubi TV’s underperforming, hard-to-maintain API system, though I had resolved to rewrite it eventually, the urgent, pressing problem in front of me was improving performance. There wasn’t much room to maneuver at the application layer (the data was already in Redis), so the only place to work was the web layer. Between HAProxy and nginx cache, I chose the latter, because nginx was already widely used in the production environment at the time. Though I’d never used nginx cache, enabling it isn’t hard — follow the docs to set the cache path and size, then in the location where you need caching, set the cache key and enable it; my simple local tests ran fine. Yet in production, requests that should have been cache hits stayed in a state of perpetual miss. I was at a loss; I tried every solution I could find online to no avail. Finally, I decided to compile my own nginx with the DEBUG flag turned on (--with-debug) to log more, and then go hunting through the source.

The cache-related code in nginx cache and upstream isn’t much — a few thousand lines. I skimmed it quickly, then used the log contents to find the relevant processing flow and, at several of the big bailout branches, guessed at the scenarios that might occur. Because nginx’s debug log still wasn’t detailed enough for my needs, I added debug code to each of these overlooked branches, recompiled, and ran it.

Throughout this process, “guessing” played a big role. I remember my undergraduate math teacher — a lovable little old man — whose stock phrase whenever he called on someone and they couldn’t answer was: “Just guess!” He always said that guessing, blind or educated, is a legitimate way to solve a problem, and that great mathematicians are also great conjecturers.

When we read code, we guess at the intent behind a filename, a function name, a variable name, at the intent of a branch, at the intent of a stretch of code — and in the end we combine the results of running it with the printed debug output to confirm our guesses. This is a delightful cat-and-mouse game between reader and author. The more you read, the more you guess, the more you confirm, forming an effective feedback loop (read - guess - verify) — and the higher your odds of guessing right next time.

In the end I pinpointed the problem — it came down to two or three misconfigured settings. The StackOverflow answer was partially correct; it solved the problem for the vast majority of people — failing to ignore the cache-control-related header is something almost every first-time user misses, and it was one of my configuration problems too. But the reason that answer didn’t solve mine was that our production nginx had an inconspicuous setting that disabled the proxy buffer, which caused nginx to skip the cache entirely.

Let’s abstract from the above and see what to watch for when reading code to crack a case:

1. Carry your clue and find, among the pile of code, the code relevant to the problem. In the nginx cache example, the clue was the proxy’s upstream and the cache never hitting, so the code at fault had to do with cache, proxy, and upstream. Flip through the source directories and you’ll quickly pick out the files you need. Since the problem was in cache, among those files, look specifically at cache-related function names, macro names, and code.
2. Focus on reading the selected content and ignore irrelevant noise. As you read, concentrate on finding the paths that could trigger the problem, then deploy the great art of “guess, guess, guess,” adding debug output.
3. Compile and run the modified code, reproduce the problem, analyze the debug output, and then — bingo! Congratulations, you’ve got it. Please proceed to step 5.
4. Didn’t get it right? Go back to step 1. Don’t worry — this isn’t the college entrance exam; until your boss can’t take it anymore and fires you, you always have another “one more bottle” chance (a reference to the “win a free bottle” caps on Chinese drinks).

There’s also a crucial step 5, which I’ll single out. Often, after cycling through several times, we finally bingo at step 3 and are as happy as Chun-Li just having K.O.’d her opponent — we hop up with knees clamped together, both hands flashing a V in the air, so carried away we forget to do step 5.

Joy is fleeting, and so is memory. Throughout the process your goal was so clear, your execution so powerful, that you were “unscrupulous” in reaching it — by any means necessary. Three days later your boss asks: “Xiao Cheng, you’re great. What method did you use to conquer that impossibly stubborn bug?” You rack your brain desperately, but it’s like carrying water in a sieve — frantic effort, and nothing comes. You begin to doubt your existence: is the me of three days ago even the same person as the me of now?

So the crucial step 5 is: a post-mortem (复盘). After solving the problem, don’t rush to accept your colleagues’ thanks and the admiring glances of the goddess (or god) among them. While the memory is still warm, grab Evernote (or whatever) and record the whole process as concisely as possible — the key code, the key paths, the entire chain of guesses that led to the end, and which logs validated a guess as right and which proved a path a dead end (congratulations — your console or terminal should still be open at this point). In short, every means you used along the way should be jotted down like an itemized ledger. Then analyze and summarize:

• What was the root cause of this problem? What’s the code flow that triggers it?
• In reading the code, where did I guess right, and where did I guess wrong?
• If there’s time, which part of the code deserves to be savored and appreciated in detail?
• Next time a similar problem arises, how can I locate the issue in the source faster?

In this process of reading code like cracking a case, if there’s no post-mortem, 70% of your effort is wasted — you spent a lot of time, read a lot of code, and apart from a good result, came away with little. Unfortunately, in the vast majority of work scenarios, we skip this step. I do too (deep in self-examination). While writing this section, I searched my Evernote and dug through my emails; apart from an email from early last year with a few scattered lines about how I enabled nginx cache, there was no other record. Fortunately, at the time I read some nginx cache code along the way and retained a faint impression, so I could still drag it out as an example.

A post-mortem helps you precipitate this kind of information, giving you a chance to review it and then organize and consolidate it into the “knowledge” I talked about in the previous article. Accumulate enough of this and gradually a halo will rise above your head, with “expert” written on it with smug pride (砖家 — “brick expert,” a self-mocking pun on the real word for expert).

Scenario Two: Reading Code to Understand the Theory

The approach in scenario one is passive — reading to deal with a problem — and it’s the only way most people sharpen their coding. It’s like playing Heroes of Might and Magic as a conservative lord: you collect your mines, stockpile troops, never go pick a fight with wandering monsters, and just wait for the enemy to attack. Three months of that and you’ve fought only a few battles — stable, sure, but your experience points grow far too slowly.

How do you level up faster? Go on the offensive! Many algorithms, fundamentals, and theories in computer science leave us with only a vague sense of understanding after we’ve read a book or an article. At that point, reading code is the fastest way to consolidate and deepen that understanding:

• Algorithms: how is a Bloom filter actually implemented? What does gossip protocol code look like in real production? How does the Linux kernel implement the O(1) scheduler?
• Fundamentals: how do you implement a complete REST API framework covering the HTTP 1.1 protocol? How does a packet travel from the OS driver all the way up to the application? What is zero copy? How does the Linux kernel implement zero copy?
• Theory: what is IoC / DI / Pub Sub? How do various frameworks implement these design patterns? What’s the implementation behind the supervisor behavior?

This process is a positive feedback loop, an accumulation driven by the Matthew effect. The more books you read, the more knowledge you hold in your head — and the more questions too. Those questions drive you to read the relevant code to verify and resolve them; read enough code and you feel your theoretical grounding is lacking, and so the cycle repeats, endlessly, and you keep learning. Conversely, read few books and you’ll have hardly any question marks stored in your head, so there’s no point in reading code to verify anything.

Take REST API frameworks as an example — two years ago, while I was still doing web security at Juniper, I read RFC 2616 and its subsequent revisions (7230–7235). At the time, frustrated that there was no good API framework, and since I was studying Clojure, I picked up Liberator. Liberator was inspired by Erlang’s webmachine and implements the decision tree with simple macros, with remarkable elegance. I later skimmed webmachine’s decision tree — pattern matching plus recursion, very beautiful. Unfortunately, the team I was on at the time was rather rigid, with eyes only for Python. Reluctantly I settled for second best and chose Eve, a REST API framework in Python. Eve’s code quality is passable, straightforward and plain — visibly the kind of code you or I would write.

Let me walk through my process of reading webmachine in detail. I came to webmachine entirely through Liberator’s introduction; Liberator’s author said the decision tree came from webmachine and included a diagram. At this point I was like someone who had just practiced Bruce Lee’s Jeet Kune Do and, hearing it derived from Wing Chun, itched to probe Wing Chun’s real substance.

Webmachine’s code is short, only 4,700 lines. Following the filenames, you quickly find webmachine_decision_core.erl, which is the main body to read — about 800 lines. These 800 lines can be split into three parts: the first 150 lines, the scaffolding of the decision tree; the middle 300-odd lines, the concrete implementation of each individual decision; and the remaining 200 lines, helper functions.

The flow of each decision is shown below:

Here the atom naming follows the diagram exactly. Take the atom v3b13, for instance: it means, in the version-3 diagram, the decision node at column b, row 13. This is the first decision: if the service is available, the whole flow continues; otherwise it returns 503 service unavailable.

Once you grasp this, you can follow the diagram to find the code, and the execution flow becomes easy to understand. What follows is simple — walk the flow and look at each decision node’s code one by one, and RFC 2616 comes alive, dancing before your eyes. Let’s look at another example: precondition check, which is v3g11:

decision(v3g11) ->
    ETags = webmachine_util:split_quoted_strings(get_header_val("if-match")),
    decision_test_fn(resource_call(generate_etag),
                     fun(ETag) -> lists:member(ETag, ETags) end,
                     v3h10, 412);

This code reads the list of ETags from the if-match HTTP header, then calls generate_etag through resource_call to generate an ETag, checks whether it matches any item in the ETags list, and if so jumps to v3h10, otherwise returns 412 precondition failed. How does webmachine know how to generate the ETag? This is where the framework’s power lies: it extracts and implements the common parts of the protocol and delegates the business logic to the application that uses it. In other words, generate_etag is a callback that the application must implement. This is IoC.

Explained this far, anyone who knows what ETags do in HTTP, or who is clear on concurrency control schemes, will follow at once; but I believe many will struggle to grasp the application scenario. Let me explain further: suppose Xiao Ming and Xiao Hong are two admins of “Life of a Programmer” (程序人生), and via the API they simultaneously fetch the site’s basic info (name, description, etc.) — v1. Xiao Ming changes the site’s name from 程序人生 to 序员人生 (a nonsense variant made by dropping a character), calls the PUT API, and successfully updates the data to v2. Xiao Hong updates the same data at the same time, but she still edits based on the original v1, so when she submits, she overwrites Xiao Ming’s change. This is the classic concurrency control scenario — the dreaded race condition. How do you handle it (think about how you usually do)? HTTP’s approach is this: when Xiao Ming and Xiao Hong fetch the data, they each also receive a “version number” (just think of it as the data’s SHA1) — let’s call it the ETag. After Xiao Ming’s change, the data’s ETag changes; when Xiao Hong submits with the old ETag, the server checks the current ETag, finds it doesn’t match, and returns 412. This is a simplified optimistic lock.

All this rambling is to make just one point: being able to read code and understanding its application scenario are two different things. But once you truly understand it, your coding skill shoots up. In the future, when updating shared objects in a concurrent environment, a question mark will flash through your head: hmm, could I skip the lock (pessimistic lock) here and consider something like an If-Match mechanism instead?

Back on track. Throughout the whole process above, I was understanding the author’s intent. Once satisfied, I usually ask myself:

• Is there room to optimize this code?
• Are there potential security vulnerabilities?
• Are there unhandled states or exceptions?

In these mere five lines, lists:member is an O(N) operation — and O(N) operations should always catch our attention. We know that for a membership check a set beats a list. From a security standpoint, the split-out ETags is a weak spot: an attacker could craft a sufficiently large and complex if-match header to drag down the processing of a single request, achieving a better DoS effect. As for unhandled states, here we can rest easy — with such a detailed flowchart (state machine) as shown above, this code won’t have major problems.

OK, this chestnut has been roasted long enough — stop. Let’s compare the line counts of the three API frameworks: Liberator 1.2k, webmachine 5k, Eve 12k. Reading Liberator feels like reading the Chu Ci — beautiful but obscure; reading webmachine feels like a math textbook, page after page of recursive derivations; reading Eve feels like reading an undergraduate thesis — the functionality is done and that’s about it, and you come away with little impression. In fact there are some cautionary examples: the whole framework is written too rigidly, making decisions that shouldn’t be the framework’s to make, so that to satisfy some of our requirements we ended up having to fork it and modify the framework’s code — a cardinal sin for a framework (we were using 0.4 at the time; 0.7 when I wrote this).

Let me summarize — the method for reading code to understand the theory isn’t too hard:

1. First use the inspectional reading method described earlier to go through the whole codebase, and find the core code worth reading.
2. Skim this portion of the code and further break down its content. Have pen and paper at hand (or any other handy tool), and record as you go. The best form of record is diagrams. At this stage I don’t recommend software tools (unless you’ve found one so comfortable it fuses man and tool into one).
3. Read this portion closely, combining it with the knowledge you already have and the materials you need to understand it, and guess at and reconstruct the scenario in which some event, message, or flow occurs in the code. Record your guesses. At this point, if you encounter peripheral code (calls to external functions), as long as it doesn’t block your understanding, set it aside for now and first walk the whole process completely and in detail. In this process, be sure to ask lots of questions, and reduce as much as possible the “I thought I understood but actually didn’t” situations.
4. Use inspectional reading to skim the rest of the code; if you find other code worth reading closely, jump to step 2.
5. Use comparative reading (or, syntopical reading) to scan through repos with similar functionality. Try to digest different authors’ implementations in your own words, pay attention to differences between them, and try to judge those differences.
6. Digitize your manuscript with software for easy future review. Text can go straight into a note-taking tool (you could even try GitBook); for diagrams, if you can’t afford tools like Visio or omniGraffle, use whatever works. For how to use them, see my article: How to xxx.

The last step is very time-consuming. Unless you have astonishing willpower, or a teacher’s itch to share your insights, there’s little motivation to do it. Of course, in this scenario we read code leisurely and at ease, with no client or boss cracking a whip behind us, so once understood the memory is very deep — even without a digital document, a flip back through the code restores it.

Self-reflection again: I don’t do step 6 well either; some manuscripts, if not photographed, are lost forever.

The more of this reading you do, the more theories, knowledge, and algorithms you truly understand, the more you become an irreplaceable master. Sometimes, in fact, all we need is to seriously spend a few months really understanding some large project to be, well, the cream of the crop.

Scenario Three: Reading Code for an Energy-Level Jump

High school physics tells us: when an atom is irradiated with light, it absorbs photons and can leap from a low-energy state to a high-energy state — the electron’s orbit, its energy level, makes a transition. This is the same principle as the philosophical notion of quantitative change turning into qualitative change. As a programmer, your career follows the same arc: you slowly climb at work, then plateau and stall, like a stock chart consolidating in a box. Then suddenly, for some unknown stimulus (say, looting “Dragon King’s Divine Power” out in the field while grinding monsters, or reading “Life of a Programmer” *_*), you hit several consecutive limit-ups and jump to another plateau.

To break through the plateau and achieve an energy-level jump, you need to absorb the right “photon.” That photon can be a world-shaking project (Google Maps at Google, Docker at Docker, Taobao at Alibaba — to name a few), but opportunities like that don’t always come to you or me. Most people spend day after day on unremarkable little jobs that only let them level up slowly — like a level-7 hero with a flock of giant eagles who can only grind low-tier mobs like goblins, hellhounds, and griffins every day. At times like these, rather than quietly sink, it’s better to be like the fish of the Northern Sea in Zhuangzi’s tale — dive and rise, store up energy, and wait for the next time to whirl upward on the wind to ninety thousand li.

One way to store up energy for the jump is to read code. What to read? Read the code so fundamental you’ve decided you’ll never write it in your life — the Linux kernel, OTP. With this kind of reading, you never know when you’ll finish, so you need patience and time. Inspectional reading + syntopical reading + mind maps are the methods you’ll use most. Below is a diagram of OTP’s source: after an inspectional reading I circled the parts I’d gradually read, with the bold ones being the parts I’ve finished skimming.

OTP’s code isn’t small, but its coupling is very low, so it ultimately breaks down into several rounds of scenario-two reading. Look at the total volume of code:

1.4M LOC — nearly terrifying. But after removing examples and some irrelevant helper code:

930k — about 45% less. And the first batch of code I circled is a mere 130k; two days of skimming, and at most half a year of close reading. Do this, and half a year from now, your level will surely be a far cry from the Lü Meng of Wu you are today.

This kind of reading can sometimes be very dispiriting, because you’ll run into a great many knowledge gaps and will have to dig through books and references to fill in what you’re missing, slowing the whole reading-comprehension pace — sometimes days pass with no progress at all, and the motivation that was kindled in you begins to taper off. At this point, steady on! These knowledge gaps are gifts from heaven, your best chance to remedy the things you don’t know you don’t know. Take it slow, take it easy, enjoy the joy of acquiring extra knowledge.

After several years of constantly enriching yourself this way, when you write code and build projects, you won’t be far from the state Yu Guangzhong so evocatively captured for Li Bai — “one breath from those embroidered lips, and half the High Tang blooms into being.”

Let me share a small story to close this piece:

In my early years at Juniper, while writing up a summary of the netscreen data plane code, I wanted to understand the IPSec phase 1 SA establishment process better, so I slipped into IKE’s backyard. At the time I didn’t know much about IKEv2; I read a great deal of material before starting on the code. I read the code in a haze — bizarre terms like catcher and thrower left me like the Yellow Emperor lost in Chi You’s fog, unable to find my way. Later, with a colleague’s reminder, I realized all those terms came from baseball; once I’d grasped what those baseball terms meant on Wikipedia, the code started to look endearing.

Years later, I read How to Read a Book for the first time. The author used a long passage comparing the catcher’s skill in baseball to the skill of reading, and the catcher-pitcher relationship to the reader-author relationship. It brought back, with a rush of happiness, that sun-drenched afternoon more than ten years ago, when I sat at a spacious desk, used the diskless SunRay workstation on it to connect to a Solaris server named after the Brothers Grimm tale of Gretel and Hansel, opened IKE’s connection-establishment code in Vim, sipped fragrant coffee, and began to watch a spectacular game of baseball…