A Reality Check for AR/VR

In March 2018, as the annual Game Developer Conference (GDC) was about to kick off in San Francisco’s Moscone Center, the conference organizers, UBM Game Network, released a report titled, “State of the Game Industry 2018.” Responding to the question “Which platform(s) are you currently developing games for?” only 19% said they were developing for the VR headsets, and 7% for AR headsets. Similarly, when asked “Which platforms do you anticipate your next game will be released on?” only 17% selected VR headsets, and 7% selected the AR headsets.

In their replies to both questions, PCs, Xbox, PlayStation, and smartphones/tablets outranked AR and VR as the preferred game development platform. The report’s authors notes: “A number of high-profile virtual reality headsets launched in 2016, and our survey results suggest [development] interest in the tech has risen steadily since then, but may currently be on a bit of a downswing ... This is the third year in a row we’ve asked respondents whether they believe VR/AR is a long-term, sustainable business, and for the first time we’re witnessing a rise in the number of ‘No’ answers.”

The attitude in the media and entertainment markets may be seeping into the enterprise market. Greenlight Insights is an analyst firm that focuses on AR/VR adoption in professional markets. Rounding out the year with some predictions culled from the firm’s reports and surveys, Greenlight Insights analyst J.C. Kuang writes:

“Even though 2018 saw VR hardware become more specialized and affordable, gradually gravitating away from enthusiasts into mainstream appeal, major firms will not be primed for mass adoption ... The increasingly conspicuous lack of available content funding from major xR [extended reality] providers, as well as an ongoing struggle within the industry to develop and propagate viable content monetization strategies, are concerns ...”

However, despite the reality checks and the pauses in the much hyped consumer and enterprise markets, AR/VR for the design, engineering and manufacturing sectors continues to steadily evolve. GPUs and workstations, sporting the “VR-ready” moniker are widely available from vendors such as NVIDIA and Dell, allowing design engineering teams to create VR/AR content. How they interact with that content is changing as well, integrating new innovations such as voice command, haptic feedback (simulated tactile sensation), and more.

Voice-Controlled Becoming a Reality

Realwear showcased its latest AR application running on the Android-powered HMT-1 headset at the 2018 Virtual Reality Strategy (VRS) Conference. The head-mounted AR gear has a rotatable boom mike, positioned by the user’s mouth. It’s designed to listen to and respond to the user’s voice commands. This feature is important as users in the company’s targeted industries (power & utility, construction, and manufacturing, to name but three) tend to prefer to keep their hands free for other tasks, such as annotation, equipment operation, maintenance, repair and so on.

Visual Vocal’s software let you add audio comments and hand-drawn markups to captured VR scenes, to be shared with multiple collaborators. Image courtesy of Visual Vocal.

“We learned a lot while working to perfect our voice-recognition algorithm,” says Tom Dollente, director of Product Management, Realwear. “We fine-tuned our system so it can detect the direction of the voice. If someone nearby happens to utter a command phrase (such as Home), the headset won’t execute it. We also learned to use commands like ‘Terminate,’ which has distinct consonants, highly unlikely to be confused with other words, and is not a phrase someone might say frequently.”

At the Autodesk University 2018 conference in Las Vegas, John SanGiovanni, CEO and cofounder of Visual Vocal, demonstrated his company’s product, a mobile VR app with voice memo features. Visual Vocal’s software works with reality capture data from other sources, such as Google’s Cardboard app. The company uses Chirp, a partner technology, to let multiple mobile device users join the same VR collaboration session in real time. In doing so, it eliminates the need to email a link or transfer a file for shared viewing. The voice memo feature and the finger-drawn annotations lets users add audio comments to VR design review sessions.

Grab Hold of Your Models

Conceived in the dorm room of its co-founder and CEO Meron Gribetz, Meta is now a recognized name in the professional AR market. The company’s partnership with the leading mechanical CAD software maker SolidWorks ensures that SolidWorks CAD users can easily export their CAD assemblies into Meta’s AR environment, viewable with Meta headsets.

Gribetz describes Meta’s AR interaction as “gestural computing.” The headset is equipped with depth sensing cameras IMUs (inertial measurement units), which allows the device to monitor the user’s hand position and correlate it to the 3D holographic space. Therefore, a user can grab, hold, move, and rotate 3D holographic models with natural motion, without relying on clumsy game controllers.

Meta’s AR setup lets you view digital content as holograph-like objects inside the real world view. Image courtesy of Meta.

uSens, another exhibitor at the VRS Conference, uses hand tracking to allow users to employ natural interactions (such as grabbing, clutching or pinching). The company offers a developer kit—hardware with stereo cameras—called Fingo.

“Our algorithm acts like X-ray; it can see through the skin and select the bone joints,” says Dr. Yui Fei, uSens co-founder and CTO. The soft-tissued human hand presents challenges for the computer to recognize and track. Therefore, uSens uses a technique similar to motion-capture technology, where the system tracks the easily identifiable finger joints. By applying machine learning, uSens developed an algorithm that can identify the theoretical positions of the finger joints even when some are obscured by the hand’s angle.

Dueling Data Formats

Before AR/VR content can be interacted with, however, it needs to be created. That process too, is getting easier for design engineering teams. Unity and Unreal, the two major game engines with stakes in the emerging AR/VR markets, recognize that establishing an easy data pipeline from common 3D formats and CAD to AR/VR is one way to speed adoption.

At last year’s SIGGRAPH, Unreal (owned by Epic Games) released a beta version of Unreal Studio, designed to help CAD users bring their CAD models into the Unreal game engine environment, a preparatory step for many AR/VR applications. Unreal Studio includes Datasmith, a tool to move CAD data into the Unreal engine. Similarly, Unity offers PiXYZ, a partner product to bring CAD models into the Unity game environment.

The Wild, a startup developing cloud-based VR collaboration solutions, now offers the ability to directly import models from SketchUp (a common modeler for architecture) to its VR platform. It currently supports import of other CAD models as OBJ or FBX models.

As more leading firms in the game and entertainment take note of the engineering and design space’s growing appetite for AR/VR, particularly for in training and design review, lingering data transfer inefficiencies from CAD to AR/VR are being resolved.

Statistics from the GDC report and analysts suggest adoption of AR/VR is now entering a much more realistic phase, with less tolerance for speculations and hype. It also indicates, both for the consumer and the professional spaces, mobile implementation of AR/VR may pave the way, largely due to its affordability.