Apple working on an onscreen awareness functionality for Siri

The latest iOS 18.2 beta 2 began rolling out on Monday to developers. The update brings a host of new features, including ChatGPT in Settings, "Create Image" in Notes, Find My upgrade, Mail app badges, and more. The stable update is expected to land sometime next month.

Now, it has also been discovered that Apple is working on an advanced feature for its chat assistant, Siri. As a part of its Apple Intelligence features, Apple has provided developers with APIs that will make Siri more contextually aware of what's happening on the screen. Reportedly, there is a new API that lets developers make the onscreen content on their apps available to Siri and Apple Intelligence. From Apple's documentation:

When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"

With the latest iOS 18.2 beta update, the ChatGPT integration with Siri lets you ask questions about images, videos, PDFs, and presentations and get more information about them. For example, you can ask Siri, "What's this document about?," to which Siri will take a screenshot, hand it over to ChatGPT and give you the requested information.

However, as noted by MacRumors, the ChatGPT integration is not the same as the onscreen awareness feature described in the Apple Documentation. Apple has outlined this onscreen awareness feature as Siri's capability to understand, analyze, and take action on the content on the screen. For instance, if someone texts you a contact number, you can ask Siri to add it to your contacts list.

Interestingly, the said feature isn't available even in the latest iOS 18.2 beta 2 update, and ChatGPT can, for now, only access screenshots. It appears that it could be a part of the several Siri features that may not arrive until next year.