The Myth of Software Supply Chain Management
Why don’t software projects have supply chain management? If we were to define a supply chain for software projects, what would it look like? What security risks would it carry? Do you remember the leftpad incident? Have you lived through the event-stream incident, where a hacker cleverly exploited a package to siphon off your assets without your ever noticing? Software supply chain management — call it dependency management — is becoming more important by the day.
Recently, word that Huawei had dropped Flex from its supply chain flooded everyone’s WeChat feed. For a hardware company, sound supply chain management is what guarantees product quality and production capacity, and therefore the company’s ability to serve its customers. Under the tide of globalization, everything from a single phone to an entire automobile sits atop a supply chain in which hundreds or thousands of companies compete and cooperate to deliver the finished product. If one link in a company’s supply chain breaks, denting capacity is the least of it — what’s truly troublesome is when it affects the product that reaches the customer’s hands. Years ago, at a company where I worked, we hit an intractable problem: our enterprise firewall product would crash inexplicably in customer production environments. The software team poured enormous effort into chasing the various crashes, to no avail, and finally turned to our colleagues in hardware. After much painful investigation, the culprit emerged: a batch of memory chips from a supplier, used in one version of the product. Under heavy read/write traffic and certain ambient temperatures, the chips had a nonzero probability of bit flips. Factory testing could never catch such a bug, but in customer environments it surfaced. When hardware goes bad like this, the only options are generally recall and replacement, since these are SoCs and can’t be swapped piecemeal; but this batch had shipped in enormous volumes, so a recall was out of the question, and we had to solve it in software. In the end we did solve it cleanly — otherwise it would have cost us a fortune.
I never put skill points into hardware, so my grasp of hardware supply chain management is only skin deep. What hardware SCM has to solve is how to coordinate every activity that bears on product delivery so as to guarantee the quality of that delivery. Its core is the BOM (Bill of Material), and on top of that sits Material Planning — the schedule by which the materials the BOM calls for arrive at the factory. A precise and efficient material plan can hold inventory at a beautiful level — “add a hair’s breadth and it is too long; take one away and it is too short.” Finally comes the execution, control, and monitoring of the production plan. Sometimes I tease my hardware friends: we both started out after World War II, but hardware grew real science (operations research, say) and real engineering, while software grew a mysticism steeped in agnosticism (The Mythical Man-Month, say).
I looked at Flex’s self-inflicted mess, laughed, and amid the snark a thought flashed through my mind: Why don’t software projects have supply chain management? If we were to define a supply chain for software projects, what would its BOM be?
So I searched for software supply chain management. Not many people seem to discuss the topic — Google handed me a screen full of supply chain management software. Too simple, too naive. I switched terms: software supply chain, and finally turned up two not-very-deep pieces from outside the industry (Forbes and InfoWorld) [1][2].
So the idea is still a very niche one. I had no choice but to let my thoughts soar freely — or, to spell out the pinyin joke, to just make wild-ass guesses (xiā jī bā cāi).
In the early days of closed-source software, independent software vendors (ISVs) mostly had to buy some components from outside, but there weren’t many suppliers (Borland, dBase, FoxPro), so the supply chain was simple and probably didn’t need managing at all. When some hardware vendors outsourced software development, they’d manage the software as part of the supply chain. I remember classmates who went after graduation to Japanese-outsourcing firms like Neusoft (Dongda Arp) and Dalian Hi-Think, complaining about how “perverse” the Japanese were: never mind algorithm details — even window sizes and positions were spelled out in thousand-page specs, and writing code against the spec was pure translation work. Looking back, weren’t they just treating software as inventory, a screw like any other? Nothing wrong with that.
With the flourishing of the open source movement and the rapid rise of the internet, software development went from “the swallows in the halls of Wang and Xie” to “the homes of ordinary folk.” Community recognition, spiritual rewards, and potential material returns drew millions upon millions of programmers into open source, so that almost every development problem you meet has some module somewhere that solves it. Open source unlocked the copyright problem; the internet solved the distribution problem — modules within arm’s reach (a matter of npm install or mix deps.get) made developers willing to try and depend on the excellent work of those before them, and that dependence in turn greatly spurred the developers providing it to keep building and maintain carefully. And so the wheel began to turn, and the Matthew effect grew ever more pronounced — new open source projects sprang up endlessly, unlocking the ability of ever-smaller teams to build richer internet products, until open source and the internet finally made each other and reached the heights we see today.
Look at the module counts of the mainstream communities:
(Screenshot from: http://www.modulecounts.com/)
Node.js stands alone, the only one to cross the million-module threshold, looking down on all contenders. As of this writing, even the niche Elixir/Erlang has over nine thousand modules, and Rust nearly thirty thousand.
If your project uses Node.js (frontend), Java + Swift (client), Elixir + Rust (backend), Python (data), then congratulations — you have over 1.5 million modules to choose from; or rather, you must pick the right BOM from more than one and a half million suppliers worldwide to build your product. A happy problem.
Flip through the project you have at hand and look at the astonishing number of dependencies:
$ ll forge/deps/ | wc -l
122
$ ll forge-js/node_modules/ | wc -l
1766(These are the dependencies of the forge framework and of the forge js sdk.)
Now look at the BOM of an iPhone 7:
Doesn’t that give you a little secret glee — “hey, compared with software, hardware isn’t all that complicated after all”?
In the internet era, any software project has a staggeringly complex BOM. On one hand, we inherit vast amounts of our predecessors’ work, standing on the shoulders of giants and spending our limited time on the core logic of the product — a good thing. On the other hand, we depend on things whose external interfaces we understand only superficially; the foundation of what we build is more like a black box — whether it’s sound, we don’t fully know, and can’t control. That brings enormous risk. Which is why software supply chain management is becoming an element we can no longer ignore.
Let’s compare it with hardware products.
A hardware BOM is hammered out with many parties (hardware and software teams, suppliers, even finance) from the PRD stage onward, under repeated review; by the time the PRD is finalized the BOM is settled too, and later there may be minor patches, but as long as the boss’s head doesn’t overheat, there won’t be major changes. A software BOM is (implicitly) fixed at project kickoff, casually added to during architecture, and still churning through design, development, even testing. Many big companies introduce a review process when a software product takes on a new dependency, but most of the time the focus lands on whether it’s necessary, where it comes from, whether the license is compliant, whether maintenance is stable — while the quality of the dependency’s code is left to one side; as long as it works, fine.
When the whole industry is used to standing on shoulder after shoulder after shoulder of giants, loss of control is inevitable. Those 122 Elixir dependencies in the example above I could still spend time eyeballing; the 1766 in the JS SDK already exceed the capacity of one engineer, or even a team.
Of course, someone will say these include transitive dependencies — isn’t it enough just to manage the direct ones? Yes and no. By the Pareto principle, we should indeed focus our limited energy on direct dependencies; but unlike hardware, when a supplier screws up in a hardware product, the supplier carries the can, and that supplier can in turn trace it back to its own supplier, layer by layer, until someone at the bottom takes the blame. In software, most of the BOM comes from products under open source licenses — MIT, BSD, GPL — which explicitly state “AS IS” without warranty of any kind. So every BOM in your product is your fault. Whose fault is it that you picked a bad partner?
Since almost no one talks about software supply chain management and almost no one pays attention to it, supply chain attacks on software products often land a clean hit. Flex cutting off Huawei can be seen as a rather clumsy supply chain attack; its blast radius was a single company, its impact limited. A supply chain attack in the software world could be a global catastrophe.
Those of us who work in JavaScript should remember the leftpad incident from a few years back [3]: a programmer, furious that an npm admin had unilaterally transferred one of his modules, “kik,” to the company kik (whose lawyers asked him to rename it, and he refused), went into berserk mode and, in a rage, unpublished all two-hundred-odd modules under his name, washing his hands of npm forever. The guy was prolific but not a big name, and it wouldn’t have caused much of a ripple — except that one of his modules, a dozen or so lines of code called leftpad, was used directly or indirectly by a great many open source projects, react among them, the top dog of the ecosystem. For a while, the CI of virtually every JavaScript project was broken and no new builds could be produced; and frontend engineers got a rare half-day to slack off and play pacman.
The leftpad incident wasn’t a deliberate supply chain attack, but its consequences were profound. It made people realize how fragile the software supply chain is — add a bit of social engineering, and couldn’t an attacker bypass a target’s layered defenses and strike a clean blow on the supply chain side?
Indeed they can. That’s the event-stream incident from last year [4] (again JavaScript, that restless child). It’s a textbook attack: a user called right9ctrl, learning that the author of event-stream could no longer maintain the project and wanted to hand it over to the community, volunteered to take it on, and to earn trust made a few contributions (3.3.5). Then he quietly injected into event-stream a dependency with a very reassuring name: flatmap-stream. The index.js in that dependency’s GitHub repo was perfectly normal, but the index.min.js published to npm carried attack code — and to make the malicious code hard to identify, he loaded it with obfuscation. Because npm doesn’t check, when publishing, whether the code matches what’s on GitHub, right9ctrl had his opening. He shipped the attack as a patch release (3.3.6), so that many people updating dependencies would upgrade automatically; then he published a major version (4.0.0) that removed flatmap-stream, to cover his tracks — since a major bump usually means breaking changes, few people upgrade. That’s how right9ctrl got countless users to run the malicious flatmap-stream without anyone noticing. All that effort — for what? If not to steal victims’ information, then to steal their wealth. And if wealth, what wealth is valuable enough, can be moved without anyone noticing, and is untraceable afterward? Bitcoin! So the code in flatmap-stream targeted bitcore-wallet-client — it monkey-patched its getKeys function, so that when a payment was made, the software would read the private key and quietly transmit it to his own server, and he’d clean out the victim’s wallet. Any software that used both event-stream and bitcore-wallet-client was in trouble. At this point you might ask: who would be foolish enough to use a wallet on the web? Isn’t the audience tiny? Unfortunately, copay, an open source wallet, used electron/cordova for code reuse…
This will certainly not be the last supply chain attack on software systems. I believe that after event-stream’s demonstration effect, more such incidents will follow. For example, this suspected social engineering (though we can’t rule out that derekdreery is a good guy):
There’s not much we can do about it right now. Software supply chains are too complex, and each dependency has too many versions, which makes manual review impractical. Here are the workable approaches I can imagine for the future:
- Self-auditing at the supply chain end, like
npm audit(GitHub appears to already be issuing security alerts based on npm audit results) - Finding problems through machine learning — because no matter how malicious code disguises itself, it has to achieve some effect: connecting to a bizarre IP/domain, accessing data it shouldn’t, and so on
- Adding permission control to the programming language or its dependency system — like the permission requests on a phone: everything is denied by default, and when the software initializes and a dependency asks for extra permissions, the programmer decides whether to grant them. These permission requests, like a dependency file, generate a text that can be reviewed.
I think some companies and some tools will be born in this space in the future. When hardware teams use ERP to manage their BOM, the software team’s BOM management is like a kite drifting in the wind. Perhaps, one day, someone will come to pry loose the cheese here.
[1] why you need to understand your software supply chain: https://www.forbes.com/sites/adrianbridgwater/2018/06/06/why-you-need-to-understand-your-software-supply-chain/
[2] building a model of the software supply chain: https://www.infoworld.com/article/3245800/building-a-model-of-the-software-supply-chain.html
[3] the Hacker News discussion of the leftpad incident: https://news.ycombinator.com/item?id=11340510
[4] Compromised npm Package: event-stream: https://medium.com/intrinsic/compromised-npm-package-event-stream-d47d08605502
