Daydream Labs and Google Research created an improved Mixed Reality to combat some of the downfalls of Virtual Reality.

By combining machine learning, 3D computer vision, and advanced rendering techniques, Google claims that they can now “remove” headsets and show a person’s identity, focus and full face in mixed reality.

Basically, mixed reality is imposing what’s happening within the VR on to a real world scenario. It’s a way of letting you watch someone interacting in VR. Currently, it’s difficult to tell what they are seeing in VR and if you were in VR with another participant, it tends to rely on avatars. In mixed reality, it “relies on a calibrated VR setup including a headset, a green screen, and a video camera, combined with accurate tracking and segmentation, you can see the “real world” and the interactive virtual elements together.”

The only thing is, faces are often obstructed by VR headsets in mixed reality. In a blog post by Avneesh Sud, a software engineer at Google Research, they have solved this issue by combining several technologies together.

They construct a dynamic 3D model of the person’s face, capturing facial variations as they blink or look in different directions. They also used a modified VR headset with eye-tracking capabilities, to capture the person’s eye-gaze from inside the headset. Combining that, they create the illusion of the person’s face by aligning and blending the 3D face model with a camera’s video stream. Finally, imposing that visual into the virtual world, which requires calibrating between the tracking system and the external camera.

These are automated so movements look natural, resulting in a complete view in mixed reality including the entire face and where their gaze lands.

While they were focused on mixed reality, they believe it could help enhance communication and social interaction in VR, including video conferences.

More details on the technology here.