Why Tubi?

As a journey draws to a close, a retrospective is the best parting gift. Tubi is a unique startup. From an outsider’s perspective, it is hard to imagine that a company with fewer than thirty engineers maintains five major product lines at once: more than twenty client applications (and counting); over fifty backend services, self-built or third-party (and even third-party ones require deployment, upgrades, metric recording, and other maintenance work); a complete advertising system; a complex data pipeline and personalization engine; and a next-generation transcoding system, just successfully developed, that is comparable to Netflix’s. Hulu has hundreds—perhaps thousands—of engineers doing the same work, while Netflix has thousands. So while I am proud of this, I often wonder: what did we get right to produce such a result? Comrade Mao says in On Practice: Knowledge begins with practice; theoretical knowledge acquired through practice must then return to practice. In this essay, I hope to explore and summarize that theoretical knowledge so that it can guide my next steps in practice.

Let us start with the challenges facing these five product lines and look at Tubi’s approach to R&D. To keep this essay from running too long, I will focus on 1) client software, 2) backend services, and 3) the transcoding system.

Client Software

For a streaming service like Tubi, the more client platforms we cover, the more users we can reach; the more likely a user will open some device at some moment and use our service. So we want to support as many mainstream clients as possible (for the North American market). And for the best experience, naturally, if we can go native, we go native. Thus for strategically important platforms like iOS, Android, and Roku, we build in native code (we will set these aside for now). Over-the-top/smart TV (hereafter OTT) has many categories; apart from FireTV, each individually may not be huge, but in aggregate they are massive. From an R&D perspective, we cannot implement each one separately, so we chose HTML5 apps or hybrid approaches to implement them uniformly.

Even with a unified HTML5 app approach, there are two thorny problems:

• Every manufacturer has its own SDK or API. So to support a large number of OTT product lines, down to implementation details, there are still many, many parts that must be handled separately—especially the video player.
• Because we use HTML5 apps, for each client we also need to deploy its corresponding server side; together with the client-side code, these complete the app’s functionality. With more than a dozen apps, every time we build a feature, beyond the unit tests for the feature itself, end-to-end testing across so many platforms is a headache; deployment even more so.

To meet such challenges, one approach is to throw more people at it, with each person (or each small group) owning one product; the other is to embrace technology and outsource as much repetitive work as possible to well-designed architecture and automated scripts or software. If we are short-handed and have to add more people, then the goal of adding engineers should be to strengthen automation and improve the architecture, not to brute-force execution.

Tubi chose the latter. Our goal is clear: write once, run everywhere.

For the first challenge, we designed a common layer to handle networking, metrics, analytics, and non-video-player features. For the video player, we built an abstract player layer to handle ad playback, ad tracking, and so on at the video-player layer. Finally, on top of this common layer, we build each platform-specific video player.

For the second challenge, we built a complete infrastructure and deployment process. This means that when we launch a new platform, as long as it follows a set of established interfaces, we can quickly produce the corresponding OTT product and then put it online with a single deployment command. Behind that one command lies enormous work—servers must be deployed, metrics collected, and tools such as Consul, Envoy, and Datadog each play their part; code must be packaged properly—and different OTTs require different packaging strategies: some smart TVs have only 50 MB of memory, and React tends to crash on them, so we choose React alternatives as needed and take dead-code elimination to the extreme.

This is how, from a technical standpoint, we reduce the dependence of a rich product line on headcount. Before March of this year, our web/OTT team had only five people (only two senior engineers). We do not do 996; the two senior engineers also have many day-to-day management tasks, yet they keep the product lines for more than a dozen platforms in good order.

Embracing excellent architecture, embracing automation, and turning complexity into simplicity—these are the keys to our success on the client side (and in all of R&D).

Backend Services

For a VOD business, backend services mainly provide the client with an accessible list of videos efficiently and reliably. That sounds easy—isn’t it just exposing a set of APIs that read data from the DB, apply some business logic, and return the data? That is indeed what it is, but doing it—or doing it well—is actually very hard. As I wrote in the policy engine article:

A movie’s availability windows can be complicated, and multiple windows may exist at the same time. Here’s a made-up example:

• U.S. users can access it on Roku and Xbox in Q1.
• U.S. users can access it on web, iPhone, iPad, and Android in Q3.
• Canadian users can access it on Apple TV in March–April.
• U.K. users can access it on FireTV from December 1 to December 15.

When every API request has to evaluate these rules against the viewer’s context (current date, location, and playback platform), efficiency is very low. Even with heavy caching, the improvement to 95th-percentile response time is still marginal. We built a rule parser with Node.js and Jison (for how to use Jison, see my article “How to Happily Write a Small Parser”); its performance was decent but not a fundamental leap. So we took a different path and built a policy engine to solve the problem. During the development of the policy engine, we found that although the data-as-code approach made it extremely fast to compute, in one pass, whether a large number of videos could play in the current context, the resulting recompile time became a bottleneck. If we could not solve that bottleneck, we might have a window in which we served videos to users that should not have been available—a very serious problem.

Two paths lay before us: one was to return to the middle way of the previous rule parser and avoid mistakes; the other was to grind away at recompile time and optimize it to an acceptable range.

We chose the directionally correct but extraordinarily difficult path. Through constant refinement, we eventually reduced recompile time from 10 minutes to 20 seconds. Along the way, we discovered some Elixir issues under extreme conditions (for example, repeated compilation can exhaust atoms) and fed them back to the community; we also built an autocompiler service to handle this potentially general need for on-the-fly recompiling in Elixir services.

The policy engine improved our service performance by one or even several orders of magnitude, allowing various APIs to build whatever features they wanted on top of it. Through the policy engine, we learned a key insight—quantitative change really can bring about qualitative change: when performance improves by one to several orders of magnitude, the whole world looks different. After that, at Tubi we pursued this kind of order-of-magnitude optimization wherever possible, greatly unleashing product and R&D potential, and turning one “doesn’t make sense” problem after another into “why not?”

When building backend services—especially the early API system—a major challenge was: how can average programmers also produce relatively high-quality API interfaces and code? So in addition to good architecture, we provided a simple DSL for defining APIs, which lets many correct things happen automatically. I mentioned this in my “Revisiting API Writing—Overview” series, so I won’t repeat it here. Beyond that, we also provided an internal feed DSL for non-engineers to use, and engines and SDKs for engineers to call, such as the policy engine, materialization engine, and A/B testing SDK. These DSLs, engines, and SDKs hide a great deal of detail beneath simple APIs, so the whole team benefits.

Let me say a bit more about DSLs. About a year and a half ago, I defined a feed DSL to describe how metadata from content partners’ feeds maps to Tubi’s internal systems. Feeds come as XML or JSON; the location and name of each field vary, and even data types differ, but they all contain information such as title, duration, description, starring, image, and video URL. Our previous approach was to write one class per feed. They were broadly similar, but handling details varied greatly, eventually producing a long, messy codebase of dozens of files. After abstracting a feed DSL, we only need to write one parser that performs the corresponding transformations based on the DSL description. Before, adding a new feed took an engineer half a day to a day (copy, paste, and modify); when there was a bug in the core feed logic, an engineer had to spend several days fixing it and ensuring the same problem did not recur across the source code of dozens of feeds. After we rewrote this logic a year and a half ago, the feed parser has barely changed, and adding a new feed can be done quickly by a non-engineer, with very little chance of error. For details, see “The Ability to Abstract, Revisited.”

Transcoding System

Transcoding is a major part of the VOD business: it converts original video files into formats and sizes that each client can play smoothly. At Tubi, we transcode one source video into more than a dozen different bitrates and resolutions, and our partners often give us video assets in batches. As a result, our transcoding queue often has hundreds or thousands of videos waiting to be transcoded.

A two-hour video transcoded into more than a dozen outputs takes more than ten hours on a c4.4xlarge machine. If your entire video library is small and you only have a few—at most dozens—of videos waiting to be transcoded each day, this is not a problem. But at Tubi in 2017, the number of videos available to North American viewers already exceeded Netflix’s, and because of the special nature of our business (videos have multiple playable windows), our entire media library was an order of magnitude larger than the videos we could actually play. So every day we had a huge number of videos waiting to be transcoded. Netflix’s transcoding technology is unmatched globally, especially its parallel transcoding capability, which can accomplish in minutes what others take hours to do. At Tubi, we learned from Netflix’s approach (Netflix has a simple blog post describing its pipeline). Then three and a half engineers, after months of effort, built and released in Elixir a highly scalable transcoding engine, along with a complete solution that starts from content partner upload and runs all the way through transcoding completion. This engine can split a large video file into thousands of chunks; each chunk (or several chunks) is assigned to a single core in a cluster with hundreds of cores for independent transcoding, followed by post-processing. As I said above, when performance improves by one to several orders of magnitude, the whole world presents itself to you in an unprecedented way. Backed by this powerful transcoding capability, our entire content pipeline has moved beyond the vision of traditional VOD vendors and has already set its sights on new territory (which I cannot yet discuss).

Conclusion

Many problems reveal their true difficulty only after they reach a certain scale. Streaming is not an inherently difficult business. On the surface, Tubi and its competitors do not seem very different. Yet while competitors (such as Crackle) pour piles of money into marketing and user acquisition, Tubi puts its money into R&D. In February, at the annual executive offsite in Napa, CEO Farhad told us about his vision for marketing and R&D. He said he hoped that in the next three years, the capabilities we accumulate in technology (especially machine learning) would allow Tubi to spend close to zero on user acquisition, so that we could pour all the savings into R&D and further widen our lead over competitors in R&D capability. So the most important answer to “Why Tubi?”—and our principle—is:

Keep reinvesting the money we earn into R&D.

This sounds commonplace, a truth everyone knows; but knowing is easy and doing is hard—persisting, single-mindedly persisting, is extraordinarily difficult: more than twenty years have passed, and the fate of the “Juda Zhonghua” telecom quartet—Julong, Datang, ZTE, and Huawei—is the best footnote. Xunzi said: “When water accumulates, it forms a deep pool; from it, dragons are born.” Over the past four years, Tubi has endured and invested in exactly this way, bit by bit, until today.

Precisely because of this, in the AVOD (ad-supported VOD) marathon, Tubi started late yet gradually overtook others to become the leader of the first tier. While Tubi aimed its R&D capability at handling the world’s largest premium contents, competitors were still intoxicated by MAU figures built with money; while Tubi was making one key R&D capability after another an order of magnitude better, competitors were still intoxicated by market awareness built with money. The AVOD landscape of contending warlords resembles the Seven Warring States, and Tubi’s competitors look very much like the incompetent, disappointing State of Chu and the State of Qi, which knew only empty talk.

With “keep reinvesting the money we earn into R&D” as the foundation, Tubi has gradually formed its own approach to R&D. The principles I have summarized are:

• Build things that are an order of magnitude better.
• Build architecture, interfaces, engines, SDKs, DSLs, and processes that enable average brains to produce outstanding results.
• Put money into building (or acquiring) tools to solve problems, rather than into people to repeatedly deal with the same problems.
• Build the invisible competitive advantages that take time.