Don't Let Angry Customers Be the Ones to Tell You About Your Bugs

Prologue

After returning from a Sunday-morning hike in the Fragrant Hills, I remembered I had coffee scheduled with a friend and hurriedly opened WeChat to scan a shared bike by the roadside. Mobike’s mini-program told me it was unlocking and not to close the mini-program. Yet the unlocking progress stayed at zero percent for several minutes without changing at all. After waiting a long while, I suspected the network was poor, so I closed the mini-program, reopened it, and scanned again. This time the mini-program simply told me: “The bike you used last time has not been locked yet; you cannot use another bike for now.”

This was absurd. I scanned again and again at a bike that had clearly never unlocked but was considered unlocked. I expected the clever system to notice this anomalous behavior—if a bike deemed to be unlocked and in use is scanned repeatedly by the same user, then the lock must not have opened properly—especially for an old-timer with a credit score of 160 and fairly solid character.

But Mobike stubbornly refused to let me use a bike. The same copy after every scan, especially that final “oh” meant to be cute, mercilessly mocked me as I stood in the cold wind with a runny nose. With no other choice, for a whole Sunday and for several short trips that never required a taxi, I had to use Didi, adding congestion and smog for the people of Beijing. I grumbled in a couple of WeChat Moments posts. A friend at Mobike asked for my phone number, looked into it, and said that once the backend task was processed, my bike would be unlocked for me.

By Monday morning, more than twenty hours had passed. Even the slowest task should have finished. On my way to work, I wanted to test whether my account had recovered, so I casually scanned a bike—and the problem remained. I was beside myself with anger and wrote the following, ready to bid Mobike farewell:

Soon after, two friends pulled me into two separate group chats. One included a BD director; the other included a head of engineering from some department. Under the kind concern of these big shots, my problem was solved instantly—after I had “ridden” for 1,273 minutes, that is, 21 hours, the ghostly bike finally “successfully locked”:

With the problem solved and the money owed to me refunded, I would like to take this opportunity to thank all the friends who helped. In such a black-swan event, my many WeChat friends—and my modest reputation in programming circles—helped enormously. For another user, things would certainly not have gone so smoothly. Her complaints might have vanished into the void; only through customer service, at great time and effort, could she recover the charges that should never have been incurred.

For anyone else, the matter might have ended there, at most becoming fodder for after-dinner conversation. But for a programmer with too much time on his hands and a habit of idle rumination, nothing is more interesting or more worth pondering. On Monday night, back at the hotel, while burning fat and killing calories in the pool, one question kept circling in my mind: how can we make the system better at detecting and handling such problems on its own, instead of relying on angry users to report them?

How Can We Detect and Handle Such Problems Faster?

Author’s note: The analysis below is pure nonsense—it is not based on a series of real data but on many assumptions that are quite possibly unreliable. Nor is the analysis deeply considered; it is just a half-hour burst of fancy. To borrow Uncle Caitou’s line: everything I say is wrong.

According to the published patent, “An IoT-based Public Bicycle Rental System” (https://www.google.com/patents/CN105354935A?cl=zh), the bike-rental flow is as follows (slightly modified):

1. Use the mobile app to scan the QR code on the bike. The app obtains the bike’s unique identifier and sends a rental request to the backend control system. The backend control system also sends an unlock command to the bike’s onboard controller.
2. After the lock opens, the onboard controller sends an unlock-success message to the system (along with GPS location data), and the bike enters the rented state.

In my case, the onboard controller evidently sent an unlock-success message to the system; the system believed the bike had entered the rented state, while the lock had never actually opened. This is very hard to eliminate in practice, because the mechanical locking action cannot form a complete transaction with the chip that sends the unlock-success message.

As I complained above, if a user with good credit “scans open” a bike and then scans the same bike again, there is a very, very high probability that the lock did not actually open. From a product perspective, the system should not stop the user from riding; it should try to unlock again. If the user repeatedly scans, the app should ask whether this bike fails to unlock properly and prompt the user to report it as a problem bike. The user can then scan another bike and ride it.

Of course, this optimization does not apply well to users with average credit, or to users who, after failing to unlock one bike, simply move on to scan another. In many cases, the system will still think you are using a bike and therefore prevent you from renting another. We need to solve this faster and more gracefully.

Let us define a few terms:

1. session: The period from each unlock until the user finishes and locks the bike is a session. Sessions come in the following varieties.
2. active session: A session in progress. When a user scans to unlock and rent a bike, an active session is created. A user can have only one active session.
3. abnormal session: An abnormal active session—for example, one in which no displacement occurs within thirty minutes of unlocking.
4. finished session: A completed session. When the user locks the bike, the current active session ends, and the user becomes eligible to rent another bike.

Some assumptions:

1. 90% of sessions last no more than one hour. The longest session will not exceed one day.
2. If a session has no activity for half an hour (1800s)—meaning the user has neither locked the bike nor is actually riding—the session should be treated as abnormal.
3. After unlocking, the bike’s smart lock sends a heartbeat to the server every N seconds (assume N=10), including the bike ID, user ID, current geographic location, generated displacement (which can be calculated from wheel rotation speed and circumference), and so on. If the user’s app is open, it also sends similar data at the same cadence, including the bike ID, user ID, current geographic location, and displacement calculated from 4G / WiFi / GPS signals.

Given these definitions and assumptions, we can discuss the session-scan approach.

Option One

The most traditional approach is to organize active sessions into a doubly linked list. When an active session ends, remove it from the list and release it. A background scheduled job then scans this list. If it finds an abnormal session, it pulls it out, marks it, and inserts it into a doubly linked list of abnormal sessions. When it reaches its timeout again, exception handling is performed—for example, locking the bike again and ending the session, correcting the billing, or placing it into a queue for offline manual handling.

Option Two

Option One is simple and easy to implement, but its biggest problem is that the scan itself is a linear operation; the O(N) complexity is unavoidable. If N is in the tens of millions or even hundreds of millions, performance will be poor. A complete scan might take several hours, so many abnormalities would be discovered only hours later.

So we need a more efficient data structure to avoid this full-table scan. There is a data structure widely used in the industry with excellent results and O(1) time complexity: the timer wheel. It looks roughly like this:

Depending on business needs, a timer wheel can have different total slots. The interval between slots is called a tick. Suppose one tick is 1s and there are 128k total slots; then this timer wheel can hold sessions timing out within at most 1.5 days (128 × 1024 / 86400), which satisfies our assumptions above. The timer wheel maintains a current tick, indicating where the clock has advanced. If the linked list corresponding to the current position contains sessions, those are all timed-out sessions that need to be pulled out and handled by one or more workers. If a new session is created at some tick, then, under the assumptions above, we set an 1800s timeout for the session, so it is inserted at current tick + 1800s and organized with a doubly linked list. The next time the current tick reaches that position, the session will be handled.

Logically, the timer wheel is as described above; in implementation, it is an array plus linked lists, as follows:

The advantage of the timer wheel is that handling timed-out sessions is an O(1) operation: the position pointed to by the current tick is exactly the list of timed-out sessions to be handled, which is very efficient and avoids a full-table scan to find sessions that need handling. However, it has some extra update costs. For example, while a user is riding, the bike sends a heartbeat every 10s. We can use changes in GPS location and generated displacement to decide whether to refresh the session. If a refresh is needed, the session must be removed from its current position in the timer wheel and placed in a new position, thereby delaying the timeout task’s handling of that session. With tens of millions of sessions refreshing every 10s, there would be millions of linked-list operations per second, inserting and removing constantly—very uneconomical. One approach is to record a time diff in the session when refreshing. When the current tick arrives and a worker processes timed-out sessions on the list, if it finds that a session’s time diff is not 0, it reinserts the session at the appropriate position in the timer wheel, greatly saving work during refresh (see patent: www.google.com/patents/US8805988).

Another optimization for the timer wheel is, if the linked list at each slot is very long, to use an array + linked list within the slot: hash sessions into different positions in the array and then link them. For example, with an array length of 32, an original linked list of length 1024 can be distributed across 32 linked lists, averaging 32 sessions each. In this way, we can use 32 workers to process these timed-out sessions concurrently.

This can greatly shorten the handling time for abnormal sessions and thereby win back one heartbroken user after another.

Some Thoughts Beyond Technology

At Tubi TV, we often say that engineers should have ownership of the products they are responsible for. Ownership is easy to talk about, but putting it into practice requires a great deal of extra effort. Building a feature for a business line is not that troublesome; what is troublesome is:

1. Which data points along which paths should be sent to the monitoring system (e.g., StatsD)? After launch, which graphs, monitors, and alerts should be added to which dashboards on the monitoring platform (e.g., Datadog)? What is the runbook when a potential problem occurs?
2. Which error logs should be sent to the error-tracking system (e.g., Sentry)?
3. How is this feature provisioned? How is it deployed? If new resources are used, how are those resources described (infrastructure as code)?
4. How do we ensure performance does not degrade?
5. Is A/B testing necessary, and at what scale?
6. Are the existing analytics events (instrumentation) complete? If not, how should they be extended? Do the ETL jobs need to change? Who will analyze and process the newly collected data?

Among these, 1 and 2 both help us detect and handle problems promptly when they surface.

Whenever a system problem is discovered by a customer and the customer is “forced to let out a final roar,” it is nothing short of a disgrace for the programmers who built the system. Why do we start responding only after a user reports a problem to us? Did we know the problem existed and pretend not to see it? Or did we have no idea it existed at all? The former looks like a character problem, but eight times out of ten it is a natural reaction after programmers have been burned out; the latter is caused by the technical team’s usual inattention to monitoring and alerting. In our daily work, we care a great deal about problems that unit tests fail to catch: we immediately add the relevant unit tests so that CI can exit with an error the next time the same problem occurs. Yet for production problems, we rarely trace them back to the source—starting from whether the corresponding data points are missing and following the chain all the way to whether monitors and alerts are complete.

A couple of days ago, while reviewing our production incidents in November, a colleague summarized what he had learned from handling production issues on the Microsoft China Bing team and introduced the concepts of TTD / TTA / TTE / TTR. I found it very helpful and list them here:

• TTD (Time to Detect): The time from when a problem occurs to when it is detected (whether by tools, the engineering team, or even users). The shorter the TTD, the better; detection by tools is far preferable to discovery and reporting by users. For most of our current problems, TTD is within 5 minutes.
• TTA (Time to Aware): The time from when a problem is detected to when the first person responsible for it becomes aware of it. TTA should also be as short as possible, but the human factor introduces many variables. We use Opsgenie: after it receives an alert, it notifies the on-call developer by text message and phone call, and it often takes 5–10 minutes before the developer actually becomes aware of the problem. If no one responds within 20 minutes, Opsgenie calls every member of the dev team. So in most cases our TTA fluctuates widely, roughly between 5 and 40 minutes.
• TTE (Time to Engage): The time from when someone becomes aware of the problem to when work on the problem begins. Sometimes the on-call developer may be driving, in an environment without network access, or without a computer at hand, so she needs to contact the next person she can reach who has the resources to handle the problem. This also involves many human factors, and our TTE fluctuates widely, roughly between 5 and 30 minutes.
• TTR (Time to Resolve): The time from when work on the problem begins to when it is resolved. Production problems are often of a few fixed types, and we have corresponding runbooks. Simple problems (e.g., restarting a service) can be resolved in a few minutes; medium problems (e.g., adding more resources) within half an hour; complex problems may not have a suitable solution right away, but we can work around them, and they can also be resolved within half an hour to an hour.

For the R&D team, we need to shrink TTD / TTA / TTE / TTR as much as possible, so that we can discover problems before users do and resolve them before users go crazy. To shrink TTD / TTA / TTE / TTR, we need to know how good/bad we are—what the max / min / mean of these metrics currently are—so that we can improve in a targeted way.

Improvement is inseparable from measurement! Improvement is inseparable from measurement! Improvement is inseparable from measurement!

Important things must be said three times.

For example, if TTD is too long, is it because monitoring data is not collected sufficiently? Or because the data is collected but no corresponding alerts are set? Or because alerts are set but the thresholds are too loose? By tracing back layer by layer, we can find and fix the monitoring problems and then observe whether the TTD curve declines in the next few incidents.