Startups and Management: Building an Experiment-Friendly Work Environment

Agile is now a word every tech company wants perpetually on its lips. In an era when CMM (Capability Maturity Model) has lost its luster and Agile reigns supreme, a tech company that does not practice some form of Agile seems like a fallen household from a bygone age. Yet many companies adopt Agile only by introducing a few Agile practices into their process; they never inject Agile’s primal force into the work environment and culture. I have seen companies move from waterfall to scrum, train scrum masters, train employees, invest in all kinds of shiny new tools, run daily standups with great fanfare, and even—almost like playing house—use planning poker to tackle software engineering’s hardest decision: estimation.

But anti-Agile tendencies at the level of engineering culture render all of this futile. Agile implementations often begin with a roar and end with a whimper, fading away without resolution.

The Anti-Agile Side of Company Culture

So what exactly is agility at the level of company culture? I take it to mean continuously exploring, learning, and improving at an iteration pace that is best suited to the team. In fact, many Agile practices strive toward this goal; their users simply lose sight of it along the way—the daily standup being a case in point. This essay begins with the theme of exploration and discusses how we can build a friendly work environment in which engineers can, as much as possible, keep probing and solving all kinds of engineering problems.

Some readers may smile at this point: as engineers, isn’t our very job to keep figuring out solutions to all sorts of problems? Would any company really be foolish enough to suppress engineers’ desire to explore?

Let us look at some counterexamples first:

• Changing two lines of code requires tens of minutes of compilation and linking.
• Testing a new idea (or new feature) without a sandbox or simulator, only on physical hardware (imagine iPhone development without a simulator—every time you would have to install the compiled .ipa onto a phone).
• Fixing a bug with only a few dozen lines of code, yet code review takes weeks and endless nagging before it is approved.
• Completing a feature that will not meet customers (users) for six months or even longer.
• Building a feature with a third-party library under an open-source MIT license or with a new technology (such as AWS Lambda), only to be told that although the solution is greatly simplified, we do not want to introduce more dependencies and burden the team with extra learning.
• Presenting painstakingly built work in a demo and receiving only negative evaluations (not constructive feedback, let alone praise).
• Wanting to try a new AWS feature but being locked out because of insufficient permissions (and requesting permission requires an extremely long approval process).
• Deploying new code to production manually, making a mistake that causes irreparable loss and impact—and then the “intern” involved is fired.
• The company confines you to a particular role, does not want you to join technical discussions of unrelated non-confidential projects (under the pretext of not wasting your time), and may not even let you read the documentation or code.

These are all real examples. Many readers—especially those working at large companies—have surely encountered similar situations. They are like a slow-acting poison, bit by bit eroding employees’ passion and desire to explore. And once that passion and desire are gone, even a luxurious office environment and lunches carefully prepared by Michelin chefs will struggle to bring them back.

Looking at the examples above, some are solvable through engineering—1, 2, and 8; some are solvable through process—3, 4, and 7; and some require cultural change before they can improve—5, 6, and 9. Yet when something is solvable through engineering and remains unsolved, or solvable through process and still unresolved, the root cause is ultimately company culture. When a company culturally does not value building an experiment-friendly environment, the daily reality is one constraint after another: every three steps forward cost two steps back. It is like playing Sanguosha (Legends of the Three Kingdoms): a loyal minister full of fervor holds a great hand of cards, eager to make his mark, serve the ruler, and crush the rebels—only to find the lord sneering in front and a traitor stabbing him in the back, leaving him barely able to protect himself.

Having gone through these counterexamples, I imagine you now have a better understanding of what experiment-friendly means. A company needs to clear away organizational, process, communication, and tooling obstacles to employees’ high-efficiency output, make work more fulfilling, and make exploration and innovation accessible anytime, anywhere.

This is easier said than done. Every company has its own “national conditions,” and it is hard to find a culture that is universally applicable. We can learn from companies widely recognized as Agile, such as Facebook, but it is very hard to copy their culture wholesale. In my view, doing the following things well will move a company’s culture one step closer to experiment-friendly.

The Tooling Side of Agile Culture

Let us start with tools. Problems in tooling yield the most visible returns and are relatively easy to quantify—for example, introducing various automation tools. Anyone who has worked on a software project knows that in any tightly fitted process, the weakest link is inevitably the human. People are most prone to error in repetitive mechanical work. At such times, we should tool the process as much as possible. For example:

• Set up a commit hook on git commit to run linting and static analysis on the code. How many companies waste time and energy in code review on things like code style that tools could handle?
• Use CI tools to automate builds. How many teams still build iOS/Android releases locally?
• Use more appropriate testing tools to test your system. How many Android app developers still laboriously rely entirely on manual acceptance testing across device models, instead of leveraging tools such as AWS Device Farm and Calabash?
• Use repeatable scripts to deploy a software system (such as Ansible or CloudFormation). How many ops teams still deploy systems purely by hand and then nervously hope nothing goes wrong?

Of course, not every tool can replace human handling, but we need to leverage the power of tools as much as possible to reduce repetitive manual labor. Only then can we save more time, reduce all kinds of foolish problems, free ourselves from constantly rushing around putting out fires, and focus our energy on other problems more worth solving.

Tooling is not perfected overnight; it improves continuously within each development cycle. At first, many things have to be done manually, but once the same repetitive work has been done several times, it is time to consider tooling it.

The Agile Path of Process

The earlier-mentioned problem of long code review feedback cycles is a typical process problem. When a team or an individual can barely keep their own heads above water, who has time to read someone else’s diff? So it gets set aside and delayed for as long as possible.

This creates a vicious cycle: the author of the code review (the reviewee) does not get quick responses from reviewers, so they reduce the frequency of sending reviews. That makes each code review larger, which in turn makes reviewers even less willing to make decisions in a short time. In this way, what should have been small pull requests—three to five files, dozens or at most a few hundred lines of code, reviewed daily by one to a few reviewers—turns into work spanning several modules, dozens of files, and thousands of lines of code, requiring joint review by several teams. Under those conditions, how could it not drag on for weeks?

So the first step is to lighten each person’s load slightly, giving them four to five hours per week (adjustable as needed) for code review. Code review can be split into two parts: pre-review and post-review. For a pull request, after it passes CI (CI should have run lint, static analysis, and tests), the reviewee asks the reviewer to sit down together and spend 10 to 30 minutes walking through the code, then decide on the spot whether to allow the merge. This is pre-review. Pre-review does not consume too much of the reviewer’s time, nor does it block the reviewee’s subsequent work too much, and it can clear up most potential problems.

In pre-review, walking through the code together—in person or via video such as Google Hangout—is important. A large portion of code review time is spent helping the reviewer grasp the reviewee’s intent. Walking through it together is the most time-efficient approach for both parties. There is another benefit: in the process of explaining, the reviewee may notice some errors themselves. I wonder whether you have had this experience: you review your own code repeatedly and see no problems, but when you explain it to someone else—even if they are still half-understanding and mulling it over—you discover all sorts of issues yourself. This is because when you read it alone, your mind remains in the mode in which you wrote it; when you explain it to someone else, you switch to the mode of trying to make them understand. A change of angle makes problems easier to spot.

Post-review is used more to uncover hidden risks from a macro perspective. After a week—or after one iteration—a good deal of code has accumulated. The feature may work, but there is no guarantee that there are no global issues. At that point, several engineers sit together for one to three hours (adjustable as needed) and go through all the team’s diffs from that period. Finding and filling gaps is both a good post-hoc remedy and a process of mutual learning.

In short, pre-review has a small workload, is easy to complete quickly, and clears obstacles for subsequent work; post-review, though much heavier, can find and fill gaps, compensate for the shortcomings of pre-review, and promote mutual learning within the team.

A company without suitable processes is doomed to remain small-time and never amount to much; but if it clings to processes too rigidly—forcing their use even when they no longer fit the needs of the work—it will only reduce efficiency. Processes are, after all, patterns discovered in the ongoing process of solving problems. Once these patterns are solidified into processes, similar problems can be handled quickly next time. This is like fast paths and slow paths in software. However, processes cannot remain static; they must change as the company, culture, and people change. The code review process described above would run into many problems in a distributed team spanning time zones. Therefore, processes must adapt on demand and be continuously adjusted until they reach the point that best suits the team.

At the same time, we need to design processes intelligently. Take problem 7 above (wanting to try a new AWS feature but lacking permissions). The process designers sacrificed usability for security. So how can we balance security and usability? We can use sandboxes to separate high-security actions from low-security actions. For AWS, multiple accounts plus consolidated billing is the best solution: the company can create different accounts for different usage scenarios, such as production, development, and backoffice. Production and backoffice are granted advanced permissions only to DevOps/IT, while development can grant advanced permissions to any engineer. This satisfies the process’s security requirements while allowing engineers to experiment freely within a safe sandbox—killing two birds with one stone.

In building an experiment-friendly work environment, as long as we are willing to broaden our horizons and think carefully, tools and processes are relatively easy to implement and produce visible results.

Some Reflections

Finally, there are cultural considerations—that is, managing, empowering, and developing people. For the purposes of this essay, I think three points are important:

• Trust over control
• Commitment over compliance
• Data over authority

Take “data over authority.” It actually addresses this question: what matters more—the personal taste of a superior (authority), or real, persuasive data? I believe no one would say authority is more important than data. Yet in actual work, “a rank higher can crush a person,” and a leader’s likes and dislikes often determine product direction. Many products die the moment a big shot touches them; this is precisely a counterexample to “data over authority.” It can only be eliminated through cultural efforts driven from the top down.

Culture is a vast topic, and it is hard to find concrete experience that is universally applicable. I will not elaborate further here.