Pretty Soon, Your VR Headset Will Know Exactly What Your Bedroom Looks Like

Imagine a universe where Meta, and every third-party application it does business with, knows the placement and size of your furniture, whether you have a wheelchair or crib in your living room, or the precise layout of your bedroom or bathroom. Analyzing this environment could reveal all sorts of things. Furnishings could indicate whether you are rich or poor, artwork could give away your religion. A captured marijuana plant might suggest an interest in recreational drugs.

When critics suggest that the metaverse is a giant data grab, they often focus on the risks of sophisticated sensors that track and analyze body-based data. Far less attention has focused on how our new “mixed reality” future—prominently hyped at last week’s Meta Connect conference—may bring us closer to a “total surveillance state.”

At the Meta Connect conference last week, Mark Zuckerberg took the stage to talk about legions of interactive holograms invading our physical space through new mixed reality augments in the company’s Quest 3 headset. This comes just a few months after Apple inaugurated the age of spatial computing by announcing that its Vision Pro headset would blur digital content with real life. All of these devices rely on external-facing sensors to understand their position relative to their physical surroundings, virtual content like augments, and other devices. This sensor data and the resulting environmental awareness that these devices and their responsive owners obtain is generally known as spatial mapping and spatial data.

The risks of this spatial information have not received as much attention as they deserve. Part of this is because few people understand this technology, and even if they do, it does not seem as scary as tech that is developed to monitor our eyes or surreptitiously record someone at a distance. Concepts like “point clouds,” “scene models,” “geometric meshes,” and “depth data” can be explained away as technical jargon. But allowing wearables to understand their surroundings and report back that information is a big deal.

We should anticipate that companies, governments, and bad actors will find ways to use this information to harm people. We have already seen how location data can be used by bounty hunters to harass people, target women seeking reproductive health care, and do an end run around the Fourth Amendment. Now imagine a spatial data positioning system that is far more precise, down to the centimeter. Whether wearing a headset or interacting with AR holograms on a phone, the real-time location and real-world behaviors and interests of people can be monitored to a degree not currently imaginable.

Built irresponsibly, this technical infrastructure will also undermine our security and safety. Imagine applying this technology to map a military installation like the Pentagon or enabling mixed reality in grade schools and health clinics. It would be akin to having a 3D “Marauder’s Map” out of Harry Potter where every nook and cranny of our world is revealed as well as the real-time locations of every real person and digital augment. If lawmakers were worried about women receiving targeted ads on their way to a health clinic or Juul buying ads on Cartoon Network, that’s nothing compared to a reality where virtual dancing babies peddle health information in a doctor’s office or promote vaping in a school bathroom. Not to mention, companies would also understand who engaged with these virtual objects, where, when, and for how long.

Meta states it wants to build mixed reality in a manner that is “trustworthy, inclusive, and privacy-preserving,” but it is unclear how it or Apple or Niantic or any of the other companies building spatial maps can achieve this. One major problem is that few companies have even acknowledged the risks of this technical infrastructure, so it is difficult for them to begin communicating publicly about what they are doing to mitigate these challenges. AR headset developer Magic Leap has been one of the few companies to explicitly discuss spatial data in its privacy policy, while Meta quietly released a primer on spatial data last week. Both companies emphasize that it is the user’s choice to share mapping data, but this puts the onus on individuals to either protect the privacy of their environments or lose access to the primary selling points of these headsets. Of course, once this data is shared with a tech company, they get to keep it forever. Maps can’t be deleted.

Even if companies are more transparent about their mapping ambitions, they could also do more to share the wealth. Privacy laws generally require companies to provide users with access to data, and future legislation like the EU Data Act aim to facilitate more user-friendly access to this sort of device-level information. Yet, companies aren’t making their maps available to their users. Quest 3 will automatically build a rudimentary map of the walls, floors, and furniture of the user’s immediate environment. Vision Pro will have the same capabilities. But even though both Meta and Apple are leading members of the Data Transfer Initiative, there’s no way to pull any of this information off the headset or share it across devices.