Oculus Quest 2, Facebook’s virtual reality system, will receive an update that will allow developers to integrate real-life videos from the VR headset’s sensors into their games to produce mixed reality experiences. Its new application programming interface, Passthrough API Experimental, will make it possible for the developers to personalize how the setting of a player looks via their VR headset by exhibiting the real world onto particular in-game surfaces, and by using filters and effects.
Oculus stated that it could allow more synergic teleworking by integrating real-life keyboards and desks of workers to enhance productivity as well. Given that this is a mixed reality experience, users would also be capable of interacting with virtual content without having to give up on the capability to engage with their pets or roommates. In a forthcoming software update, Oculus said that the API will initially be available for Unity developers, "with support for other development platforms coming in the future."
Regarding privacy, the company stated:
Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device.
Oculus announced that it will release a production version "later this year", implying that developers would be able to ship their games that employ Passthrough to players soon.