Touch and Feedback Systems

Today I went to a classroom at Stanford to hear a lecture called “EngX: The Digital Sensory System,” which fulfilled a long-held wish of mine: to sit in a Stanford classroom and take a proper class like everyone else. My advice — if you ever get the chance to visit Silicon Valley on business, check event.stanford.edu first and see whether there’s a lecture that appeals to you.

The lecture was divided into three segments: Seeing (computer vision), Touching (haptics), and Learning (Texts are knowledge). Each was superb — especially the first two (perhaps because I couldn’t quite follow the third) — and the one that set me thinking most was the second theme: touching. Let me first give a rough account of what each segment covered, and then share my reflections on touch and feedback.

The first was Associate Professor Fei-Fei Li on computer vision. What inspired me most:

• The Cambrian explosion and a plausible explanation for it (book: In the Blink of an Eye)
• How the human brain handles 3D/2D (optical illusions)
• The enormous challenges of computer vision

The road of evolution:

• To see is to recognize objects (how to correctly identify objects, which involves a complex taxonomy)
• To see is to recognize people (face recognition, action recognition)
• To see is to tell the story of the complex visual world (how a computer can describe, as a human would, the rich meaning a single image conveys)

The second was Associate Professor Allison on computational touch — more on that later.

The third was Professor Chris Manning on natural language processing. Professor Manning has an NLP course on Coursera (already finished); if you’re interested, you can watch the recordings. What interested me most in this topic was his discussion of how to extract, from a small amount of contextual information, the sentiment a piece of language is trying to express. This involves quite complex models: you need to build a sentiment-intensity score for each word, judge the sentiment of the whole sentence from the parse tree, and also understand the humor of whoever wrote it — for example: “For the love of God, the cameras are on, give me something?” NLP is truly hard work. As a novice, I freely admit that I need to brush up on this area (I signed up for that Coursera course but have barely watched any of it).

Let’s return to the second theme: computational touch. Which brings us to the question posed at the top of this piece: if you had to lose your senses one by one (suppose Shaka unleashed his Tenbu Hōrin, stripping away the five senses), which one would you want to keep to the very end?

Most people, myself included, answer “sight” first. But in fact, the loss of touch is the most terrifying, because you get no feedback whatsoever — and much of human behavior is built on feedback. She gave an example: most patients with tactile loss end up bedridden for life, with one exception — a man who insisted on using his vision to give himself feedback. To walk normally, he had to train himself to stare at the ground, learn through “seeing” that his foot had landed, and then take the next step. Sometimes he’d catch sight of a pretty woman on the road and couldn’t resist a glance (a normal male reflex) — and then down he’d go, flat on his face. The story is funny and heartbreaking at once; the importance of touch is plain to see.

At this point I had basically drifted off, thinking: if we look at this the other way around — if the various devices we build could effectively deliver appropriate feedback to the brain — then the eyes could be freed from what they’re doing and go do something else, like looking at pretty women. ^_^

A car’s entire driving system is an excellent feedback system. Once you’re familiar with that feedback, you can release the clutch and shift gears without staring at them; the same goes for turning the wheel, hitting the brake, and pressing the accelerator. If every action in driving required your eyes, the car would be undrivable.

The two little bumps on the F and J keys of a computer keyboard are also an excellent feedback system: put your hands there and you know where to type without looking.

Building appropriate feedback systems creates good user experience. Think of the bedside light in a hotel, the zipper on a backpack, the tear-open notch on a package — once you’re familiar with it, you can use it freely without your eyes.

If only our phones and all the so-called “smart devices” around us could free our eyes to a greater degree. For instance, the iPhone does more than just multi-touch (convenient as that is, it doesn’t fundamentally free the eyes): when a finger touches different areas, it could produce appropriate feedback so the user knows what they just did. Here, “appropriate feedback” matters — don’t just default to vibration (game controllers only know that one trick for so-called force feedback, and it isn’t much fun). For example, some spots might give the finger a slight suction feel when pressed (think of a manual-transmission car slipping into neutral), others a push-back feel, and so on.

After mulling it over, I believe this direction will produce some disruptive technologies on the order of multi-touch. We’ll see.