Virtual reality headsets open a window into the mind

Data from the motion sensors inside virtual reality (VR) headsets can be used to reconstruct proxy brainwave signals, revealing what users perceive while wearing the high-tech goggles.

The AI-based system that does this is called BraVeSpy, and its insights could open avenues in neuroscience research, gaming, and a host of other areas^[1].

“This effectively turns an ordinary commercial, off-the-shelf VR headset into a low-cost brain-computer interface, without adding any specialized neural hardware,” says KAUST researcher Tao Ni, who led the international team with collaborators from the National University of Singapore, City University of Hong Kong, and Hon Hai Security Research Institute in Taiwan.

“But it also demonstrates that VR headsets need better security safeguards because the data could expose personal information without a user’s consent,” adds Ni.

VR headsets combine a head-mounted display with motion sensors that track the user’s movements. Crucially, the sensors can also detect subtle movements around the eyes, such as those caused by pupil dilation, that reveal where the user is focusing their attention. Previous research suggests that these eye responses are controlled by the same neural pathways that produce changes in the brain’s electrical activity, which can be measured using electroencephalography (EEG).

“The motion data and the brainwaves are two echoes of the same underlying process,” explains Ni.

To get inside a user’s head, BraVeSpy first separates these subtle eye signals from the movements of the head and body. It then uses a generative AI model to translate the eye signals into proxy EEG traces — estimates of the brain’s electrical activity. The model learned this relationship by comparing eye movements with brain activity recorded from 25 volunteers as they viewed a selection of websites, apps, and videos while wearing VR headsets and brainwave sensors.

A second AI model then analyses the proxy EEG traces to work out what the user is seeing. The researchers trained this model to recognize the brain-activity patterns associated with different types of images, using EEG recordings from the volunteers as they viewed pictures from a range of categories.

Once fully trained, BraVeSpy could use motion-sensor data alone to identify where a user was looking and what they were typing, as well as which app, website or video they had opened. Across these tasks, it achieved an overall accuracy of more than 85%.

However, these initial experiments tested BraVeSpy on relatively limited sets of different media. Ni says its models could be fine-tuned by collecting new data every time someone uses their headset. Eventually, such systems could serve as an “intelligent copilot that lives inside the headset, understands what the user is seeing and doing, and provides real-time guidance.”

The findings also reveal a serious privacy risk: many VR headset apps collect motion-sensor data without requesting users’ permission and do not protect it with security restrictions. This could allow malicious apps to access the information. To guard against that, the researchers showed that adding small amounts of interference to obfuscate the sensor data could prevent misuse, while still allowing the headset to track motion normally.

“Our broader message is a call to action for VR vendors to address the privacy risks posed by unrestricted access to motion sensors,” says Ni.