Just over a year ago, Microsoft launched its "Bing it On" challenge, where it asked users to view search results from Bing and Google and pick which offered the best answer to their search queries. At the time, Microsoft said it commissioned an independent study where it claimed Bing search results were preferred by a group of 1,000 people over Google"s by "nearly 2-to-1 in blind comparison tests."
This week, Ian Ayres of the Freakonomics website has put Microsoft"s claims to the test in his own study. The site questions the number of participants in Microsoft"s original study, which suggests that 1,000 people is simply not big enough of a pool to come up with an accurate final result. Ayres also questions why Microsoft does not release the results of its online "Bing It On" challenge, which millions have taken since the site launched.
Freakonomics conducted its own survey, using the online "Bing It On" website. The Ayers said:
We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”). This is not even close to the advertised claim that people prefer Bing “nearly two-to-one.” It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge, if, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.
This caused Microsoft to post its own response on the official Bing blog. Matt Wallaert, Bing"s behavioral psychologist, defended the number of people used for its original study, saying, "A 1,000 person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is just 1,500 people." As far as releasing the data on the Bing It On website, Wallaert says it"s not being posted because of privacy concerns. He stated:
It isn"t conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it. So we simply don"t track their results, because the tracking itself would be incredibly unethical. And we aren"t basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).
Wallaert ended the blog with an invitation for anyone to send him an email for questions about the Bing-Google study and the Bing It On challenge.
Source: Freakonomics, Microsoft | Image via Microsoft