Virtual Keyboards for VR Make Any Surface a Workspace

The ultralight alternative

Paper Boat Creative / Getty Images

The best keyboard for typing in virtual reality (VR) may turn out to be one that's not really there.

Meta CEO Mark Zuckerberg posted a video recently of himself trying a virtual keyboard in a Quest 2 headset. Experts say such keyboards could be the future of productivity in VR.

"The next evolution of virtual keyboards will use robust hand tracking technology to accurately track the movement of all of the user's fingers," D.J. Smith, the chief creative officer of the mixed reality firm The Glimpse Group, told Lifewire in an email interview. "Until recently, the inaccuracies of the hand tracking prevented a user from being able to do this; however, the current advances in AI and machine vision are rapidly making this type of virtual typing viable."

Zuckerberg states that he attained a speed of approximately 100 words per minute using the virtual keyboard, while Meta CTO Andrew Bosworth reports hitting 119 words per minute.

Their typing process appears to be a research project by Meta that advances the state of the art in the keyboard for VR. What stands out is the lack of any hardware accessories, such as wristbands, to capture their finger movements.

Currently, developers can create virtual keyboards anchored to a surface on Quest, using hand-tracking and requiring users to tap the surface for calibration. However, in practical terms, even a minor discrepancy in the height between the virtual and actual surfaces can lead to unintended keystrokes.

"To date, virtual keyboards have been used as the main input device for text in VR; however, typing was slow because there was no way for a user to type naturally with all of their fingers," Smith said. "Creating a more effective keyboard experience is a vital component in making the productivity use case viable."

Currently, users can "spawn" a virtual keyboard within their VR experience and either select each letter with a single VR laser pointer or use the "hunt-and-peck method" using a single finger on each virtual hand, Smith said.

The current advances in AI and machine vision are rapidly making this type of virtual typing viable.

"Both of these systems are difficult to use and have a very slow rate of input," he added. "There is also an option for a VR user to purchase a specific type of physical keyboard that pairs to the VR environment. In this case, the user can see a digital twin of their physical keyboard within the VR environment and type normally on the paired keyboard. Unfortunately, this system of typing is not widely used because most VR users do not own the specific type of keyboard that is capable of pairing to the VR environment."

Virtual keyboards might even be vulnerable to hackers. A team of computer science experts at the University of California Riverside has shown that spyware can monitor and capture human movements. AI algorithms can then interpret this data into words with an accuracy rate of 90 percent or higher. For example, spyware could potentially record this password if you pause your virtual gaming session to access your Facebook messages by "air typing" your password on a virtual keyboard displayed by the headset.

"Basically, we show that if you run multiple applications, and one of them is malicious, it can spy on the other applications," Abu-Ghazaleh said in the news release. "It can spy on the environment around you, for example, showing people around you and how far away they are. And it can also expose to the attacker your interactions with the headset."

shutter_m / Getty Images

One future option for virtual keyboards could be using your eyes to type. The system would look at each key and then click on a controller or wait a predetermined moment for the system to select that key, Jason Yim, the CEO of the mixed reality firm Trigger XR, said in an email. But so-called "gaze-tracking" isn't easy to use.

"I have a fond but painful memory of watching my friend, who forgot his physical controller, trying to enter a long-hidden password for Wi-Fi," he added. "It was 20 minutes of him typing with his head and swearing profusely in the middle of a coffee shop."

Get the Latest Tech News Delivered Every Day