The Linux Kernel Memory Model: Understanding the Rules of a Multicore World

The One gives birth to the Two, the Two to the Three, and the Three to all things. All things carry Yin and embrace Yang; through the blending of Qi they attain harmony. — Dao De Jing

In the age of many cores, it is like a multitude of dragons with no leader, each going its own way. Yet a Way persists: events are ordered, relations have rules, and causality is never thrown into chaos. Instructions may be reordered, caches may disagree, but the principle of happens-before cannot be violated. To know that it is so, and to know why it is so — only then can one tame complexity into formlessness and transmute concurrency into the great Way.

Prefatory note: The world of multicore processors is like a distributed system with no unified clock, where every CPU runs to its own rhythm. What the Linux kernel memory model sets out to do is to establish, in this seemingly fragmented environment, a set of rules everyone can agree on. Put plainly, this is not merely a technical problem but a kind of philosophical inquiry — how to establish order within chaos, how to seek the absolute within the relative.

Two or three weeks ago I came across a post on Hacker News discussing the kernel memory model, and I was instantly captivated. Together with GPT-5 and Opus 4.1, I’ve organized these scattered reflections into an essay — GPT-5 handled information extraction, Opus 4.1 served as reviewer, and I wove it together and edited. Unfortunately, I didn’t note the URL at the time, so the original post is no longer findable.

Starting from a Maddening Bug

Consider a real scenario: you’re writing a driver for an input device. The interrupt handler reads data into a buffer, then sets a flag saying “data is ready.” Meanwhile, on another CPU, a user process is calling read(); it first checks the flag, and if the data is ready, reads the buffer.

Sounds simple enough, right? But the moment you run it, you’re stunned: the flag is clearly set, yet what you read is stale data — or even garbage. You might think: this makes no sense — the flag is set, so how can the data not be ready?

This, in fact, is the “willfulness” of multicore processors, and also the core problem the memory model sets out to solve.

Nostalgia for the Simple Life of the Single-Core Era

Remember how simple things were back in the single-core CPU days? A program was like reading a book, page by page from first to last, one instruction after another, all very orderly. You write a value to memory, and the very next instruction that reads it is guaranteed to see it — as straightforward as that.

But in the multicore era, everything changed. To use an analogy: it used to be one person quietly reading a book and taking notes; now it’s several people flipping through the same book at once, each with their own scratch paper (cache). Can Zhang San see what Li Si just wrote? When can he see it? These questions that were never questions before have now all become big questions.

What’s more, to squeeze out performance, modern processors have deployed a pile of “black magic” — instruction reordering, speculative execution, store buffers, multi-level caches… In the single-core era you never noticed these optimizations existed, but the moment you enter a multicore environment, they start making mischief.

Message Passing: The Most Fundamental and Most Headache-Inducing Pattern

Returning to the example from the beginning, let’s work through it carefully. This pattern is called “message passing” in concurrent programming, and it shows up everywhere in the kernel.

// Producer (interrupt handler)
buf = 1;    // store the data
flag = 1;   // set the flag

// Consumer (read system call)
if flag == 1 {
    data = buf;  // read the data
}

Common sense says: once the consumer sees flag become 1, buf should already contain the new data, right? But in a multicore system, this “common sense” frequently fails. Why?

Where’s the problem? Modern processors are too “clever” — they may take it upon themselves to reorder memory accesses. Your code clearly writes buf first and flag second, but the processor might think: “The order doesn’t matter anyway, I’ll go ahead and tell the other CPUs about flag’s new value first.” This reordering is fine from its own perspective (of course it knows the order of what it wrote itself), but the other CPUs are left bewildered.

Memory Consistency: Setting Rules for the Chaos

Faced with this chaos, computer scientists came up with the concept of a “memory consistency model.” Put plainly, it’s about setting rules for multicore processors, like traffic laws — if everyone follows them, nothing goes wrong.

The simplest and most direct is Sequential Consistency. The idea behind this model is quite plain: pretend there’s a dispatcher who collects all the operations from every CPU, lines them up, and executes them one by one. Of course, the order of operations within each CPU cannot change.

Under sequential consistency, our problem is solved. If the consumer sees flag = 1, then by the rules buf = 1 must already have completed, and the new data is guaranteed to be visible.

But the trouble is that while sequential consistency is simple and easy to understand, it has a huge impact on performance. To guarantee such strict ordering, the processor has to give up many optimization opportunities, and it slows down considerably. That’s why real hardware adopts “looser” memory models.

The Real World: Every Architecture Has Its Own Rules

Different processor architectures each have their own memory model. x86 and SPARC use the TSO (Total Store Ordering) model, which is relatively “well-behaved,” only slightly looser than sequential consistency. ARM and PowerPC, by contrast, “let themselves go,” permitting more reordering in exchange for performance.

Now the trouble comes — the Linux kernel has to support all of these architectures! Surely it can’t write a separate set of code for each architecture? That’s why the Linux Kernel Memory Model (LKMM) exists.

LKMM’s strategy is conservative but very practical: assume the worst case. If some behavior is possible on any one architecture, LKMM treats it as possible. Code written this way runs correctly on all platforms — though it may be overly conservative on some.

Events and Relations: The Core Ideas of LKMM

LKMM abstracts program execution into “events” and “relations” — sounds rather mystical, but it’s really just simplifying complex things to make analysis and reasoning easier.

Events are the basic actions in the memory model, and there are mainly three kinds:

• Read events: reading data from memory
• Write events: writing data to memory
• Fence events: memory barriers, used to control the ordering of other events

These events are connected by various relations. The most fundamental is program order — the order of instructions on the same CPU. Note that this is the order written in the program, not the order actually executed; the processor may quietly swap things around.

Another important relation is reads-from. Put plainly: which write operation did a given read operation get its value from? This relation links operations across CPUs and is the key to information transfer in concurrent programs.

Coherence order is the global order of all write operations to the same memory location. To use an analogy, every memory location has a “history ledger,” all writes are recorded on it in order, and all CPUs must see this same order — everyone must reach consensus.

Dependency Relations: The Hardware’s “Free Lunch”

Although processors like to reorder instructions, they can’t do so arbitrarily. When there are dependencies between instructions, the processor must dutifully follow the order. LKMM recognizes three kinds of dependencies:

Data dependency is the easiest to understand: a later operation uses the result of an earlier one. For example, you first read a value, then use that value in a computation, then write the result back — there’s a data dependency between the read and the write, and the processor dares not meddle.

Address dependency means: the address I want to access must be determined by an earlier operation. The classic example is array access — first read the index, then use the index to access the array element.

Control dependency relates to conditional branches: whether to execute this operation depends on the result of an earlier one. This one is a bit special: modern processors will “guess” the branch outcome and execute ahead — sometimes they guess wrong, and then they must roll back and start over.

These dependencies amount to “free guarantees” from the hardware, but don’t celebrate too early — compiler optimizations may optimize some of these dependencies away.

Memory Barriers: The Programmer’s “Trump Card”

When the hardware’s default guarantees aren’t enough, it’s time to invoke the great weapon of the memory barrier. A memory barrier is like a traffic light, telling the processor: “Stop! Wait until the earlier operations are all complete before continuing.”

The Linux kernel provides several kinds of memory barriers:

// Full barrier - the strongest guarantee
smp_mb();  // all prior operations must complete before all later operations

// Read barrier
smp_rmb(); // prior reads must complete before later reads

// Write barrier
smp_wmb(); // prior writes must complete before later writes

Returning to the example from the beginning, we can fix it with memory barriers:

// Producer
buf = 1;
smp_wmb();  // ensure buf is written before flag
flag = 1;

// Consumer
if flag == 1 {
    smp_rmb();  // ensure flag is read before buf
    data = buf;
}

With these two barriers in place, even under the most “unrestrained” memory model, the program is guaranteed to work as expected.

happens-before: Figuring Out Who Comes First

A core concept in LKMM is the happens-before relation. This relation expresses causality: if operation A happens-before operation B, then B must be able to see the result of A.

The happens-before relation arises from:

1. Program order on the same CPU (plus necessary barriers)
2. Cross-CPU synchronization operations (such as release-acquire semantics)
3. The reads-from relation (one CPU reads a value written by another CPU)

This relation has an iron law: it cannot contain cycles. Think about it — if A is before B, B is before C, and C is before A — wouldn’t that be chaos? How can an operation happen before itself?

RCU: The Linux Kernel’s Signature Move

RCU (Read-Copy-Update) is a specialty of the Linux kernel. Its approach is ingenious: readers take no locks; when an update is needed, you copy first, modify the copy, then swap it in. Readers read however they like, while updaters quietly prepare a new version alongside, without interfering with one another.

The essence of RCU lies in the concept of the “grace period.” Simply put: if a read operation began before the grace period started, it must complete before the grace period ends. With this guarantee, RCU knows when it’s safe to throw away the old data.

RCU’s greatest advantage is that read operations have almost zero overhead, making it especially suited to “read-mostly” scenarios — and such scenarios are everywhere in the kernel.

The Memory Semantics of Locks: The Invisible Rules

Spinlocks are the most widely used synchronization tool in the kernel. From the perspective of the memory model, acquiring and releasing a lock both carry special meaning:

• Acquiring a lock (acquire semantics): operations after the acquisition cannot move before it
• Releasing a lock (release semantics): operations before the release cannot move after it

This guarantees the integrity of the critical section — operations inside it stay dutifully within it and don’t “slip” out.

Even more elegantly, the Linux kernel’s locks have transitivity. For example, if CPU A releases a lock and CPU B acquires the same lock, then everything A did before releasing the lock is visible to B after it acquires the lock. This transitivity holds even across different locks, so long as there is a program-order relationship between them.

Data Races: The Easiest Pit to Fall Into

When two operations access the same memory location at the same time, at least one of them is a write, and they may execute concurrently — congratulations, you’ve fallen into the big pit of data races. This is the most dangerous bug in concurrent programs, because its behavior is completely unpredictable.

LKMM invests a great deal of effort into detecting and preventing data races. It analyzes the various relations between operations (happens-before, propagates-before, and so on) to determine whether two operations might execute concurrently. If they might, and the conditions for a data race are met, LKMM will issue a warning.

The principle for avoiding data races is simple: when accessing shared data, either use atomic operations or protect it with a synchronization mechanism such as a lock.

Trade-offs in Practice

With the theory out of the way, what we face in practice is the trade-off between performance and correctness. Using strong memory ordering guarantees (such as sequential consistency) makes programs easy to understand and easy to verify, but performance suffers. Using weak ordering (such as relaxed ordering) gives good performance, but is error-prone and demands extra care.

In the Linux kernel, different scenarios call for different choices:

• Performance-critical hot paths: use relaxed atomic operations, paired with carefully designed algorithms
• Places where correctness is life-or-death: dutifully use strong memory barriers or locks
• Read-mostly scenarios: RCU is the only choice
• Places that absolutely must be strictly ordered: bring out the full memory barrier

Closing Thoughts

The Linux kernel memory model is not merely a set of technical specifications; it is more like a key to understanding the essence of modern computers. In a multicore world, there is no absolute time, no unified perspective; every CPU lives in its own little universe. What the memory model sets out to do is to establish, in this relativistic world, causal relations that everyone can agree upon.

The greatest reward of studying LKMM is the shift in mindset. You can no longer understand programs through a single-threaded lens; you must learn to analyze problems through the lens of “events” and “relations.” This shift is genuinely difficult, but once you grasp it, you truly understand the essence of concurrent programming.

The next time you encounter one of those “what sorcery is this” concurrency bugs, don’t rush to doubt your sanity. Calm down and analyze it with the knowledge of the memory model, and you’ll find that behind every seemingly bizarre phenomenon there is a reasonable explanation. These rules are complex, but it is precisely they that let us write correct code while pursuing the utmost in performance.

In the end, the memory model is a contract between hardware and software — what guarantees hardware promises to provide, what rules software must obey. Understand this contract, and you understand the core of modern computer systems. That understanding will let you navigate with ease through every new technology and new architecture you encounter.