Microsoft has patented some interesting ideas in the past, some of which might surface in the future. These include an artificial speech-generating system, personalized travel diary automation, a multi-functional Surface pen - perhaps one that turns into an earpiece -, and more.
Now, a patent recently granted to Microsoft has emerged, featuring live picture overlays for mixed reality systems. In essence, the system in question would recognize the people present in pictures being viewed by a user and automatically display information associated with these people in real time.
To summarize the procedure through which this would work, initially, a monitoring process would continuously attempt to detect pictures of faces in a mixed reality environment. Once a face is detected, its edges or borders will be identified, with a canvas object then being generated, coinciding directly with the face. Facial recognition could then be used to discover its identity. Finally, information associated with the recognized person will then be applied to the canvas, appearing as an overlay or replacement for the actual picture being viewed in mixed reality.
Notably, although augmented and virtual reality systems do allow for annotation of information with objects, the scope of this technology does not yet extend to pictures of real people. Furthermore, the already present information augmentation techniques do not take into account the user"s personal connection with any recognized subject in the photo. Through this patent, implementation of both these ideas has been taken into consideration.
Sources from which the information could be attained may include social media applications, instant messaging services, and more. Similarly, facial recognition searches may either be limited to comparisons with contacts present in the user"s local database or be extended to utilize search engines and social networks.
Microsoft unveiled the second generation of its mixed reality headset, the HoloLens 2, in February. Perhaps a future iteration of HoloLens might feature the above-described system. However, as always is the case with patents, there is no guarantee as to that. You can listen to our views regarding the HoloLens 2 Development Edition here.