Twitter is currently in the process of rolling out an update which will overhaul the way it handles photos. The social network started offering users the ability to include photos with tweets in 2011, but as it is difficult to display the entirety of an image within the Twitter timeline, the service auto-crops them to generate previews using a mechanism based on face detection.
Naturally, not all tweeted photos include a face, so auto-cropping can sometimes get it awkwardly wrong, often focussing unintentionally on body parts or the empty space within photos, making some tweets look quite odd until they"re clicked, and context is restored.
Auto-cropping of images on Twitter is (hopefully) about to get a lot better though, as the social network has been training a neural network, or in other words, AI, to more appropriately auto-crop images.
The company showcased its efforts with a few before and after comparison (left and right respectively) images, shown below, demonstrating its improved auto-cropping function:
The system is based on saliency prediction, meaning that the neural network is "trained" to recognise areas of an image that the human eye naturally gravitates toward. This could be areas of high contrast, animals, faces, text or any object that grabs the attention of a real-life human.
Twitter has been training the neural network to recognise the above areas of interest quickly, and notably to discard areas of non-interest, so it happens fast enough to be completely transparent to the image uploader, as explained in the original blog post:
"these two methods allowed us to crop media 10x faster than just a vanilla implementation of the model and before any implementation optimizations. This lets us perform saliency detection on all images as soon as they are uploaded and crop them in real-time"
The changes are currently in the process of being rolled out to everyone on Twitter"s website, as well as the iOS and Android clients. It"s unclear whether third-party clients will be affected by the change at this stage.