Wagner James Au / New World Notes

Above: Me in a more innocent time (by which I mean 2024)

Tech giants are still insisting XR headsets will go mass market, and now, in the latest version of this article of faith, that they'll somehow be integrated into personal superintelligence. While HMDs have many great niche applications, especially for short term use, I'm now convinced they'll never go mass market as our default computing device.

My deeper doubts started after my good friend and staggeringly smart colleague, designer Amber Case (a former research fellow at MIT Media Lab and Harvard’s Berkman Klein Center for Internet & Society), raised limitations with XR headsets in a recent essay, bringing up points that never occurred to me before.

Briefly put, she argues that there are neurological and even evolutionary barriers to the effectiveness of an XR headset — because touch and even smell are essential to how we acquire knowledge:

I asked my colleague Dr. David Sisson, a Ph.D. in Neurophysiology, why XR wasn’t taking off as quickly as the media promised. He pointed out that audio and visual information — the kind conveyed in HMDs — is only part of how we make sense of what we’re experiencing.

Then there’s the quality of touch conveyed through force feedback in most XR rigs.

“Without touch, there’s no ‘intimacy’. You’re not really interacting with what’s going on,” David told me. “You can hit a ball — and hear the ‘crack’ in VR — but you’re not feeling anything other than a little jerk in the controller that makes you feel like there’s some inertia happening.”

Headsets and their up-close screens, in other words, deprive our minds of the tactile and situational context which feeds into how we build memories…

The famous moment in French literature when the aroma of a cookie sends the author flying backward into his childhood memories has been validated by recent peer-reviewed studies — and strongly suggests digital memories formed apart from smells will be less vivid or indelible. Indeed, a recent study suggests that our sense of smell “may facilitate transfer of learning to other sensory domains”.

Read the whole thing here. These are barriers that seem difficult if not impossible to overcome for mainstream adoption, but they're rarely discussed alongside the usual culprits that supposedly hold XR headsets back: cost, weight, battery life, etc. Amber's essay strongly suggests, even if the XR industry does get to the fabled point where the headset is the size and weight of sunglasses, most of us will still find the experience unsatisfactory, at least for most use cases.

But surely there's a solution to the neurological challenges she raised? I checked with two XR experts: Avi Bar-Zeev, who's spent most of his career developing various VR/XR projects, including the Apple Vision Pro, and Jeremy Bailenson of Stanford, who's led groundbreaking work in virtual reality for over two decades.

What they told me was surprising:

 

Above: Amber Case trying Google Glass, an early, disastrous attempted at an XR HMD – read her essay here

"Seems reasonable," Avi told me, after giving Amber's essay a read. "I don't love HMDs either. But never say never. They're the best option for now, but one day we'll make a holodeck."

As for XR lacking tactile or olfactory interactivity: "Touch screens have some haptics at least. But the only way we get real touch and smell is for either matter printing like Star Trek [to be created] or BCI. Both are pretty dangerous."

By BCI, he means a Brain–Computer Interface — like the kind that reportedly (and horribly) killed monkeys during Neuralink trials. As for the holodeck and replicator of Star Trek, those are likely a lifetime or two away from being feasible, if ever.

Bailenson also acknowledges the sensory barriers to XR raised in Amber's essay.

"Touch and Smell are critical for sensemaking in the world," as he puts it to me. "The technological progress on haptics has slowed due to the dominance of cheap, consumer VR, but it is a question of when, not if, technology can reproduce the forces that one encounters in the real world.  We are talking decades not years, but there is no true barrier other than time and money. " (Emphasis mine, because when a speculative technology is decades away from feasability, it's difficult to make any assumptions about its consumer appeal.)

In the short term, Bailenson points to some low budget experiments his Stanford team has performed, to simulate olfaction:

"We have done some fun studies in the lab with 'dime store' scent implementations to surprisingly strong behavioral responses, such as dangling a cue tip dipped in vanilla off the headset to prime a donut scent." (PDF report of that here)

The far greater challenge is not the technology itself, he goes on, but the tech giants like Apple and Meta, who insist on promoting XR headsets for common, everyday consumer use.

Bailenson believes that's not where HMDs shine:

"The biggest barrier to market adoption is not perfect realism, but use case matching," as he puts it. "The technology companies are trying to foist a model of all day use upon VR, whereas my research indicates that it is not a medium for daily use, but best one reserved for special experiences (i.e., my DICE model).  VR gets used plenty when it solves a hard problem."  

By "DICE", as he explains in this paper [PDF]:  

This approach recommends that we primarily use VR for dangerous, impossible, counterproductive or expensive experiences that are not easy to implement in the real world:

Training firefighters, rehabilitating stroke victims, learning art history via sculpture museums and having a visceral, perceptual experience of the Earth’s future to understand climate change for example, all fit squarely in DICE. 

Alternatively, one does not necessarily need to don a headset to check email, watch television and conduct general office work. Such applications work better on 2D screens. By not putting such use cases needlessly into VR, society can avoid some of its challenges… [We] should focus on areas where VR is transformational and clearly the best medium for the job. [Emphasis mine]

Again, Bailenson is the world's foremost expert on virtual reality, yet the way he talks about it is utterly different from how Meta and Apple market VR. Or for that matter Avi, who helped develop Apple VisionPro, but seems to see current headsets only as an intermediary step, or even a necessary evil. ("I don't love HMDs either.")

So if the experts say HMDs aren't ideal for everyday use, but for very specific use cases (Dangerous, Impossible, Counterproductive, Expensive), why do Meta and Apple keep promoting their headsets as products destined to go mass market, for, say, watching movies or working remotely?

That's a subject for a whole other article, though it's certainly in part due to echo chamber excitement among XR enthusiasts in those companies, and also partly from cynical, short term boosting of the company's stock price. (Again, when writing my book, I could find no evidence that Meta even researched VR's well-documented propensity to give females motion sickness, still another open topic.)

For XR developers, however, this is all worth keeping in mind, and tempering their own expectations for the market they're creating for, and the kind of applications most likely to succeed.

If they asked me, I would not roll the dice beyond DICE.

An earlier version of this article appeared on my Patreon.