Due to the rise of spammy results in search engines, Reddit quickly became a reliable source for consumers seeking answers to various queries, including product recommendations, travel advice, and more. Recognizing this trend, Google began prioritizing Reddit in its search results over other websites.
Recently, Google expanded its partnership with Reddit to gain access to Reddit"s Data API, which delivers real-time, structured, and unique content from the platform. Through this API, Google will receive improved signals that will help it better understand Reddit content, use it for training, and display it in search results.
Following the paid deal with Google, Reddit has now blocked other search engines from accessing its content. As a result, Microsoft Bing and other search engines that rely on Bing"s search index will not have access to Reddit content.
Reddit spokesperson Tim Rathschmidt released the following statement to The Verge regarding this issue:
"This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI."
From this statement, we can infer that Reddit wants companies to pay to train their AI models on the publicly available content generated by hundreds of millions of users on the Reddit platform.
A Microsoft spokesperson said the following regarding Reddit blocking Bing and other search engines:
"We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site."
Early this year, OpenAI signed a deal with Reddit to get access to Reddit Data API. This trend of major tech companies like OpenAI and Google securing exclusive access to valuable data sources like Reddit through paid partnerships raises significant concerns about the future of open internet and the consolidation of power in the digital landscape. Smaller organizations and independent researchers, lacking the financial resources to compete, will be left behind, potentially hindering innovation.
It is imperative that regulatory bodies like the FTC scrutinize these deals to ensure fair competition and protect the interests of the broader community.
Source: SearchEngineLand