On Engineering Culture: Passion, Inversion of Control, and Open Source

One of Facebook’s most famous engineering cultures is bootcamp. Every new hire, whether a fresh graduate or a grizzled veteran, must be baptized in bootcamp:

Six weeks of training. Each person is assigned a mentor and a great many technical tasks. The mentor meets with you one-on-one every week and introduces you to the people you need to work with to get something done. Everyone’s tasks are the same as those of full-time employees, only somewhat simpler and less urgent. The tasks come from various teams, so you can personally interact with people across teams, review code, and assess whether there is chemistry with a team; team members can also assess you. It is mutual selection.

Tasks can come from any team. The mentor also asks what you like doing: frontend, backend (C++/Java/PHP), web, mobile.

HR also posts high-prio (high-priority) teams on the intranet. They generally recommend high-prio teams, because these are the teams growing rapidly right now and extremely short-handed.

In this process, employees discover teams they are interested in, and teams look for new employees they are interested in. Each team’s manager pitches the team’s characteristics to new employees and why you should come work there (what you can gain from it). It is a bit like an entrepreneur pitching an idea to a VC—except in reverse.

Abstract this, and it is not hard to find the following engineering culture:

1. Connect every engineer’s passion and ability with the work they do

Simon Sinek says:

working hard for something we don’t care about is called stress; working hard for something we love is called passion.

If you want to unleash a person’s maximum ability, you must find a way to make them love what they do, or—to the greatest extent possible (aligned with the company’s goals)—let them do what they most want to do. In most companies, people choose in only one direction: candidates can hardly choose teams inside the company, and teams try to avoid competing for the same candidate—harmony is precious. In this respect, Facebook’s bootcamp gives engineers a chance to choose the team they are interested in, which is invaluable.

The importance of “giving a person as much opportunity as possible to do what they most want to do” is something everyone can appreciate.

2. Inversion of Control in organizational structure

As an organization grows, it is very important to have a middle layer (roughly from manager to director) that gets things done. Middle managers connect top and bottom; so-called execution is basically embodied in them. But the middle layer is precisely where people most easily become what Red in The Shawshank Redemption calls “institutionalized,” becoming rigid and set in their ways under the grind of years. How many companies have a team that exists where it is merely for historical reasons—because once there was a need for a team to do something? As the company grows, that team may no longer be important or even useful, yet it still gets a replacement when people leave, and new headcount keeps being added as the company develops? Then it operates by inertia until one day management finds the financials simply cannot be held together and needs layoffs to cut costs, and cuts the whole team—good people and not-so-good people alike—in one go?

Companies, like people, need metabolism as they grow. Layoffs and reorganizations are top-down tools, surgery—like scraping poison from a bone or cutting off an arm to survive. Metabolism, in biology, is entirely bottom-up. Giving engineers the chance to choose teams they are interested in and letting intellectual resources flow relatively freely inside the company is a kind of metabolism. Yet this bottom-up metabolism needs organizational mechanisms to guarantee it; this is Inversion of Control: the resources a team needs to grow are not fixed and guaranteed, but must be won by the team’s manager attracting others to join and proving the team’s value.

In the traditional sense, the headcount in a manager’s hands is allocated from above by a slap on the forehead; if unused, it may be lost. So, on the eve of headcount expiring, managers—like ministries and commissions rushing to spend their budgets at year-end—will settle for mediocrity if they can find it, rather than end up with neither people nor “budget.” This is one drawback: hiring people you may not need. Because headcount is distributed from above, naturally, cultivating good relations with the boss above (I am not saying good relations with the boss are unimportant) helps tilt resources your way. This is drawback two: the boss above has considerable room for rent-seeking.

A better approach is to centralize resource planning at the company level, setting it slightly below actual need. Whether you win resources depends on the team’s and the manager’s abilities. Moreover, the direction of resource competition shifts from the manager’s boss to newly hired engineers; the manager must instead pitch the team’s value and direction to engineers.

This creates healthy pressure on managers: hot, focal teams can attract more new employees’ favor; obscure teams, if the manager cannot uncover their value, may fail to hire anyone. This forces teams to keep improving technically and in marketing ability (very important!). If a team’s manager lacks ability and cannot learn quickly, it is easy for the whole team to be marginalized along with them and become a casualty of metabolism.

Continuing with Facebook as an example, I suspect that in the past two years there have been quite a few teams inside Facebook building frontend frameworks similar to React. Once React took off and was widely accepted, it in effect declared those teams dead. The elite members of those teams left one after another (to other teams inside the company), and they could not attract fresh blood, so they naturally disappeared or transformed.

I hear Tencent has a similar mechanism internally: small teams grow naturally, compete for resources by ability, and the victor reigns while the loser exits. Very fair. At least far fairer than the so-called forced 5% bottom-ranking elimination, with much less room for rent-seeking.

Beyond fairness, this reverse pressure can produce quick wits and indirectly foster innovation. Because in a company, most products (or projects) are the kind that seem not very interesting and are not at the center of attention. For this, as a manager, if you cannot find a solution (or problem, or direction) that makes engineers feel their blood boil, then your team will inevitably become marginalized. Often, a product feature is not that interesting simply because no one has discovered a broader problem or greater intrinsic value in it—Stephen Chow, through another character’s mouth, says to himself that even a pair of underpants or a sheet of toilet paper has its uses. Managers and teams need to be forced into self-discovery and self-actualization.

Inversion of Control also brings another effect: whether something is good is decided by users and data, not by a PPT reporting to the boss. React’s success is not because Zuck said it was awesome, but because teams inside Facebook adopted React bottom-up in their own production environments, and external usage of React has grown larger and larger; almost all popular UI libraries have been componentized with React. At the same time, internal and external engineers not only use it but also submit many pull requests, forming a healthy ecosystem. That is success in the true sense. By contrast, some of the less successful internal projects I have seen at big companies can only be pushed top-down by upper management relying on authority. Frontline engineers using these tools may feel a fishbone stuck in their throat, hate and abandon them, yet their voices cannot travel upward; even more absurdly, the teams (and their bosses) behind these not-so-excellent internal projects can still receive awards and high praise.

In the Guanzi: “A state has four ways to perish: when orders are sought but not issued, it is called extinction; when issued but held up on the road, it is called obstruction; when the conditions below are sought but do not reach above, it is called blockage; when the conditions below reach above but are stopped on the way, it is called encroachment.” Although the four words “extinction,” “obstruction,” “blockage,” and “encroachment” discuss the seriousness of “law not being established,” what they describe—orders from above going down and conditions below going up—is all a problem of the middle layer. Inversion of Control in organizational structure can effectively prevent this ossification after a company grows large, promote effective allocation and flow of resources, let the company metabolize like a body, and eliminate bad things bottom-up.

3. Embrace open-source culture

Almost every internet company claims to embrace open-source culture, but most companies talk sweetly while in reality merely using a few open-source projects and at most publishing some of their own code; they do not truly understand open-source culture. For a company to embrace open-source culture internally, first see whether these things are done well:

• Every team has one or more projects or products (and corresponding repos) and is responsible for them (proud to be the owner)
• Everyone inside the company can see the code of any project or product
• Any team that depends on another team’s work is encouraged to send pull requests directly to that team’s repo, submitting the code changes they want

The first and second points are the foundation of the third, and the third can be said to be the essence of open-source culture. In our daily work, we often hear excuses like: my feature isn’t done because it depends on Team X’s feature, and they haven’t finished it yet. The bigger the company, the more product lines, the more complex dependencies between features, the more such Mr. Xs there are. Many companies split teams into infrastructure teams and product teams. A new feature in product group A may depend on features x1 and z2 from platform teams X and Z. These two features are not very urgent on X and Z’s roadmaps and have not been scheduled. Then the wrangling begins, and in the end it can only be escalated to managers to coordinate all kinds of tradeoffs before the project can move forward. Yet tradeoffs are terrifying, because someone gains and someone loses; and such matters depend heavily on managers’ coordination and negotiation skills and the chips in their hands. Past a certain point, they easily become bottlenecks in the system, and there are many incentives to use their rent-seeking room. Much of the so-called mountains of documents and seas of meetings in big companies is exactly this kind of exhausting communication and coordination. If the mountains of documents and seas of meetings caused by back-and-forth tradeoffs and negotiations merely lower team efficiency, it is still forgivable; but when tradeoffs gradually turn into company politics—when the rent-seeking room inside becomes a blade for certain managers to build fiefdoms, suppress colleagues, and climb upward—that is a terrible situation and unforgivable.

By contrast, in open-source communities, everything happens bottom-up. Every project has its own priorities; people are basically equal, and a project will not easily change its roadmap because of one person’s will. For example, Xiaoming wants a component A he uses to provide a new feature. He does not need to wait for A’s owner Xiaohong to implement it. After the two communicate about the approach, Xiaoming forks Xiaohong’s code, implements it himself, and sends Xiaohong a pull request asking to add the feature. As A’s owner, Xiaohong has the deepest understanding of A, so she can very reasonably offer revision suggestions (for example, make it more general). After a round or two, the feature is implemented reasonably well and successfully merged. In this process, Xiaohong did not adjust her priorities; she merely acted as a gatekeeper and spent some extra time reviewing (much easier than writing it herself!). Xiaoming did not have to spend time waiting for Xiaohong to finish the feature; instead, he “used his own hands to feed himself” and kept his project moving. Everyone wins.

Of course, within a company, not all inter-team collaboration can be completed this way. For example, a product team asks a platform team to provide a more efficient message distribution platform, involving changes at the architecture level. At this point the product team may not have the ability or energy to make such changes, so coordinating priorities remains the first thing to consider.

In addition, to reach this level of collaboration, dividing product logic and designing an appropriate organizational structure are very important. An organizational structure suitable for this cannot be one team for UI, one for frontend, one for backend, one for iOS, one for Android… Such a structure simply cannot collaborate like open-source software; it can only depend on each other and wait for others’ progress. Conversely, it should be one team for product A, one team for product B, with all kinds of talent inside each team, able to independently complete their respective products—ideally covering or even owning the entire closed loop from user research, product design, feature development, testing, to deployment. Only such teams will have ownership over what they do, and fundamentally it becomes unlikely for them to pass the buck and wait on each other.

If most collaborative relationships within a company’s engineering teams follow this model, a great deal of internal friction can be reduced.

The three things about open-source culture mentioned above actually imply much more:

• Every project must have complete, highly automated processes: UT/CI/codereview/CD/etc. If some basic homework in a pull request is not done well, automated tools will quickly catch it
• Detailed documentation. At minimum:
  • A contribution guide telling others how to submit code and what standards to meet
  • An installation guide/getting started telling others how to run it, plus basic knowledge about the project
• The project owner is reactive, so others get feedback very quickly (whether through tools or human feedback)

Okay. That is enough for now. This article has repeatedly mentioned bottom-up driving force. Let us look back at the engineering culture described above:

• Connect every engineer’s passion and ability with the work they do
• Inversion of Control in organizational structure
• Embrace open-source culture

In fact, it is about making bottom-up driving force possible in organizational structure and thought. I once read a sentence that left a deep impression: agile at scale is trust at scale. A similar principle. Engineers are a formidable species; when their drive is ignited, their efficiency and output before and after can differ by several times or even dozens of times.