Alexa: The Goddess of My Dreams

Prologue

At noon my wife sent me a message: Last night, you called out a woman’s name in your sleep—twice.

I nearly choked on my lunch. My sleep-talk is normally lightly base64-encoded; how had it suddenly switched to plaintext output? I hurriedly asked, “Xiaobao or Xiaobei?”

She replied with one line: Alexa, stop. Alexa, play &@*#$(@.

I silently wiped the sweat from my brow. It seemed that in the voice-goddess showdown among Siri (Apple), Cortana (Microsoft), and Alexa (Amazon), I had tilted completely toward Alexa—so much so that I was rendezvousing with her in my dreams. Don’t ask me why Google Assistant isn’t on the list—it didn’t even bother to take a goddess’s name. Demerit. (Actually, I don’t own any G devices—I’m planning to try Google Home when I get the chance.)

Alexa was the last to arrive at our home.

Siri has been by my side since 2011, but she was only a dispensable playmate and never really helped me with anything. Over time, we had nothing left to say to each other. Recently, because of Alexa, I thought of her again and played with her a few times. Unfortunately, the conversation was still stiff and lifeless; she’d keep opening a web page, as if to say, “Whatever—go look it up yourself. Don’t waste my time.” It killed any desire I had to keep talking to her.

Cortana arrived a few months ago alongside HoloLens. To be honest, she is much more engaging than Siri, and her voice is much softer. But I can’t wake her from her dreamless sleep every time I want to talk to her—it’s too much hassle. Over time, I gradually forgot about her.

Alexa, by contrast, moved into our home just a few days ago with the Echo Dot. Her voice is more beautiful and gentler than Siri’s or Cortana’s, and she feels more flesh-and-blood. More important, she is always there when called: say “Alexa,” and she will do something for me—or generously announce, “I’m sorry, I can’t understand what you mean.”

These past few days, the first thing I do after coming home and hugging my wife and kid is to get cozy with her.

Alexa, play family music in Pandora.

Alexa, volume 6.

Alexa, do I need to bring umbrella tomorrow?

At dinner, I still can’t resist:

Alexa, set a 30 min timer. (making sure Xiaobao finishes dinner within the allotted time)

Alexa, stop playing music. Give me latest news.

Alexa, remind me to deal with insurance at 9pm. (add a todo list item)

Almost every time, she satisfies my basic needs just right. So how do the Echo (or Echo Dot) and Alexa together pull all this off?

Exploration

Building a voice-interaction system like Echo/Alexa requires at least several technologies: speech recognition, natural-language understanding, machine learning (deep learning), and artificial intelligence. But technology is just technology; turning technical capability into a product involves many other considerations. Don’t underestimate a short exchange—“Alexa, do I need to bring umbrella tomorrow?” The complexity the machine handles far exceeds what we imagine:

1. “Alexa” acts as the wake word, waking the sleeping device so it can receive the subsequent voice input.
2. The speech must be converted into text flawlessly.
3. Through machine learning and other techniques, the system combines context (the three elements of narrative: time, place, and person) to extract the intent in the text.
4. It invokes the relevant services to obtain information and organizes that information into natural language.
5. It synthesizes the text into speech and sends it back to the user.

To achieve all this, you need purpose-built hardware:

The first step is waking the device. Hardware can’t run all the time; it should start only when it “hears” a specific wake word, much like wake-on-LAN. The wake word is “Alexa” on Echo, “OK, Google” on Google Home, “Cortana” on HoloLens, and a long press of the Home button on iPhone. The main purpose of a wake word is to ensure the hardware responds only to specific conversations. First, it saves processing meaningless speech; second, it ensures the device won’t “eavesdrop” on every conversation in the home. This is crucial: privacy must remain sacrosanct (otherwise we wouldn’t dare speak Mandarin at home—we’d have to switch to Wenzhou dialect). At first I was quite unhappy with this setup, because having to prefix every sentence with “Alexa” felt very unnatural. But once I understood the privacy rationale, I let it go.

As far as public information shows, the wake word on Echo is not programmable; you can’t change it to your goddess’s name—say, “Cynthia” or something. So it is likely baked in at the hardware level, or at least in firmware—partly for security (users can’t change it), but mainly to keep wake-up latency as low as possible.

Once the wake word is recognized, Alexa sends the following conversation to the cloud for analysis.

At this point, the audio stream “do I need to bring umbrella tomorrow?” is converted into text. This seemingly simple action is actually quite complex. I ran a few ex(Q)periments (A):

• At a restaurant, while everyone was chatting, I talked to Alexa at a slightly louder volume. She recognized it correctly.
• After waking Alexa, my daughter talked to her. She recognized it correctly.
• After waking Alexa, my daughter talked to her while I generated some background noise in English. She recognized it correctly.

Once the text is correctly converted, Alexa needs to use natural-language understanding to analyze the intent it expresses and invoke the relevant service—this is the hardest part. The key phrase “bring umbrella” represents my intent: Will it rain tomorrow? If it rains, I need an umbrella; if not, I don’t. This is a decision tree: the root is the weather, and the result is whether to bring an umbrella. I’m not familiar with NLP, so I won’t go into detail here.

Based on the NLP result, Alexa knows my actual intent is to find out whether it will rain tomorrow. Given that intent, she looks up a weather skill registered on AWS that can serve it, sends a service request to that skill, and uses the result to tell me what I need to know, helping me decide whether to carry rain gear.

For a weather-service query, we need two elements: time and location.

• Time: My utterance mentioned “tomorrow,” so the time is tomorrow. If I don’t mention it, the time defaults to the current local time.
• Location: Since I didn’t mention a location, Alexa has to infer it from other information. GPS can provide location, but for an Echo used at home, there’s no need to build in a GPS chip. Besides GPS, we can get a not-very-precise location via IP. So here I think Alexa knows via IP that I’m in Cupertino, CA (an experiment connecting Echo over VPN is still missing here). Of course, another possibility is that Alexa is tied to my Amazon account, so she can also get my location from my home address.

That way, whether it will rain tomorrow in Cupertino can be retrieved from the weather service.

The next step is dialogue management—turning the answer into meaningful text. The weather service tells Alexa tomorrow’s weather and the chance of rain, and Alexa needs to convert that into natural language: “No rain is expected in Cupertino, CA tomorrow.” Based on the user’s question, she should further add, “So you don’t need to bring an umbrella with you.” But it seems Alexa hasn’t reached that step yet; she just tells me it won’t rain.

In this context, the person isn’t very important; time and place are enough. But in many conversations, the person matters a great deal. For example, with “Alexa, play Mozart Effect in Pandora,” Alexa reads the Pandora skill, which reads the stations under my account. If there is something related to Mozart Effect, she plays it (Alexa will say: “Getting Mozart Effect from Tyr’s Pandora account”); if not, she searches Pandora, and if it exists, prompts me to add it and plays it. So how does Alexa know information about “me”? She gets it from the account currently logged in to Alexa (and the household account). In multi-user scenarios, this setting can be switched with “Alexa switch.”

Finally, the text is synthesized into speech. The difficulty here is producing speech that is as life-like as possible—natural in tone, coherent, and full of emotion. In this respect, Alexa is streets ahead of Siri.

Outlook

AI-powered voice input is the next great battlefield, no less significant than the Battle of Fei River in the Jin dynasty, the Central Plains War in Republican China, the Battle of Waterloo in Europe, or the American Civil War. So the internet giants have all joined the fight. For now Amazon is ahead of Apple and Google, but the final outcome is far from certain.

Amazon didn’t start particularly early, but it has seized the initiative in this fight—a testament to the soundness of its strategy.

Let’s start with hardware. Apple was originally in a league of its own on this map: the iPhone’s enormous penetration and Siri’s three- or four-year head start made Apple look invincible by any measure. Unfortunately, the initial version of Siri was less than ideal: the voice was stiff, too few services were available, and it constantly directed people to web-search results, which was even more frustrating than simply refusing the user. After a few rounds of that, no one wanted to use Siri except to tease it now and then. But a voice assistant is well suited to the home, the car, and any situation where using your hands is inconvenient—and the iPhone is precisely not the ideal context. If I have to pull out my phone and long-press the Home button to start voice input, wouldn’t it be faster to just search directly? Apple TV 4, from the same family, should have been a great vehicle for Siri, but it was expensive and tied to the TV, so ordinary families kept their distance.

Amazon’s Echo/Echo Dot made a full-throated push to get voice assistants into the home. When my wife is cooking in the kitchen, she doesn’t have to stop what she’s doing; she just says a few words to Alexa to play music, read the news, set a timer, or listen to a joke. It’s very convenient. The Echo Dot’s price is friendly enough—a few dozen dollars—and with Amazon’s unmatched sales network, it gained an early advantage in hardware. Other competitors—Google and Microsoft—are at a disadvantage in both hardware and hardware sales capability. Google Home has the same positioning as Echo. As for Microsoft, it’s still unclear what its strategy actually is: HoloLens is too niche, and Windows laptops have the same problem as the iPhone—my hands are already on the keyboard, and you want me to add my mouth too?

In hardware, Amazon has a huge advantage, and Google is giving chase. But for now, everyone has occupied only the home smart-terminal market; other markets remain vast, especially automotive hardware. It’s still too early to say who will win.

Now platform openness. Among Siri, Alexa, Google Assistant, and Cortana, Amazon has the clearest and most usable platform strategy, one that is consistent with its AWS strategy. Building a skill on Alexa Skills isn’t too difficult; two AWS services released late last year—Polly and Lex—extended this capability to a much broader range of use cases, even third-party hardware platforms. Third-party manufacturers can build their own smart hardware and pair it with software using the Lex API, easily solving the voice-interface problem for smart hardware—work that previously only big companies had a chance to do. Startups that don’t make smart hardware and don’t want to tie their products to Echo can also use the Lex API to build their own voice-interaction systems, as demonstrated at re:Invent:

Google once moved slowly on voice platforms. Google Now (the predecessor of Google Assistant) had been around for so long, yet Amazon Alexa ruthlessly left it several lengths behind. After some painful reflection, Google picked up the pace and made a huge leap in API openness and ease of use. Especially after acquiring api.ai last September, its momentum has brought it right on Amazon’s heels. Building an action on api.ai is simple (from an interaction standpoint, much better than Alexa), and the concepts are basically the same as Alexa’s (intent corresponds to intent, entity to slot, etc.), using the same approach. So the huge advantage Amazon Alexa has built up around skills may gradually be eroded.

As for Microsoft and Apple—well, they’ll just have to catch up slowly.

In terms of AI capability, the major players are probably not that far apart, and they all have enough capital to compete for talent. The focus of competition is who can spread their API to the most hardware and users, collect the largest-scale voice data, and thereby gain the upper hand in data.

That’s enough for now. Next time I have a chance, I’ll write about how to build an Alexa skill or a Google action.