The team of six discovered that exposed eye-tracking data can be exploited to decipher what users enter on the device's virtual keyboard. This attack, named GAZEploit and shared with WIRED, enabled the researchers to reconstruct passwords, PINs, and messages typed with users' eyes.
"Based on the direction of the eye movement, the hacker can determine which key the victim is now typing," says Hanqiu Wang, one of the leading researchers involved in the work. The team identified the correct letters in passwords 77 per cent of the time within five guesses and 92 per cent of the time in messages.
The boffins did not gain direct access to Apple's headset to observe what users were viewing. Instead, they deduced what people were typing by remotely analysing the eye movements of a virtual avatar created by the Vision Pro. This avatar can be utilised in Zoom, Teams, Slack, Reddit, Tinder, Twitter, Skype, and FaceTime applications.
The Tame Apple Press touted this attack as another example of Apple being first. However, much of it was due to the ease of access to the biometric data—something that Apple has fixed.
The GAZEploit attack comprises two main components. Initially, the team developed a method to identify when someone wearing the Vision Pro is typing by analysing the 3D avatar they share.
They trained a recurrent neural network, a type of deep learning model, using recordings of 30 people's avatars as they completed various typing tasks. When users type with the Vision Pro, their gaze fixates on the key they intend to press before swiftly moving to the following key.
Wang said that when we are typing our gaze will show some regular patterns that these patterns are more prevalent during typing than when browsing a website or watching a video while wearing the headset.
"During tasks like gaze typing, the frequency of your eye blinking decreases because you are more focused," Wang said.
In essence, looking at a QWERTY keyboard and moving between the letters is a distinct behaviour.
The second part of the research involves using geometric calculations to determine the keyboard's position and size.
The only requirement is that as long as hackers get enough gaze information to accurately recover the keyboard, all following keystrokes can be detected. Apple helpfully provides this information, with its super cool security features.
In a series of lab tests, without any prior knowledge of the victim's typing habits, speed, or keyboard placement, the researchers could predict the correct letters typed with 92.1 per cent accuracy in messages, 77 per cent for passwords, 73 per cent for PINs, and 86.1 per cent for emails, URLs, and webpages.