When Microsoft first announced the Insider Program, it was met with an exuberant reception. The new initiative from the software giant would open up Windows development to the masses, and allow fans to follow along as the company's engineers worked on improving and expanding the capabilities of what was deemed 'the last version of Windows'.
Despite that earlier enthusiasm, however, many among the Microsoft faithful have since become somewhat disillusioned with it. As the myriad tweets lampooning the ninja cat will tell you, something's gone awry. It's also not just a problem that has to do with perception; with the release of both major feature updates to Windows 10 this year, Microsoft faced significant challenges with regard to the quality of their release.
No time for that. Ninja cat and taco hat jokes are a priority.
— WPD (@wpdapp) October 6, 2018
In the case of the October 2018 Update, in particular, Microsoft has even had to take the embarrassing step of pulling the update while it works to fix an issue that is deleting some users' documents. Far worse is the fact that these issues had been reported by Insiders as much as three months ago, and yet were somehow still unresolved. All of this raises the question: has the Insider Program lost its focus?
Back when the first Technical Preview of Windows 10 was flighted to Insiders, the focus of the endeavour seemed to have been inviting Windows enthusiasts to chime in and help the company navigate the treacherous waters of OS development without risking another disaster à la Windows 8. Within two weeks, the program had 1 million users, and had accrued 200,000 pieces of feedback. A bafflingly large number by any measure.
But this was just the beginning. By the time Windows 10 officially launched, there were 5 million registered Insiders; that number grew to 10 million back in March of 2017, and is likely even larger today. With all those testers and all that feedback, Microsoft gained an enormously large pool of beta testers for its OS, and possibly a previously unthinkable amount of telemetry and other data to look at.
Yet, the program's success seems to have also been its Achilles heel. As Microsoft began to rely less on professional independent testers and more on Insiders, it got a lot more data, yes, but that data was not always the most useful, especially when it comes to testing the OS.
Among the millions of Windows Insiders, most are simply fans of the company and use it as a means of telling Microsoft what they would like to see in Windows in the future. That's not a bad thing, but if Microsoft is relying so heavily on Insiders for testing, there should be a more concerted effort to weed out problems before they become real issues. Open Feedback Hub now, for example, and among the top 10 most upvoted items, only two pertain to actual problems users encountered in their use of Windows 10.
Both of these are about Edge crashing at times, and thus refer to the same issue. Then, there's the fact that while both provide a problem to fix, they give little information about what exactly could be causing the issue, something a team of professional testers would be far more adept at.
Sure, Microsoft can, and has, used artificial intelligence and the telemetry it necessarily gets from Insiders to supplement its bug-bashing efforts but as is becoming clearer now, those efforts aren't working out nearly as well as the company would like.
In the last year, both of the major feature updates released by Microsoft have suffered from major, often deal-breaking issues. Following the release of the April 2018 Update, many users saw their systems freeze after using Chrome. Google's browser is among the most popular programs on Windows, and the entire OS freezing with its use is no small thing.
Last week, the company released the October 2018 Update for Windows 10, and this release was similarly beset by various issues from day one. Some users were unable to install the update on their PCs due to issues with Intel drivers, while others reported their documents being deleted. All of this culminated in Microsoft taking the unprecedented step of pulling the update until it had fixed the problems, especially those pertaining to data loss.
To make matters worse, we soon learned that Windows Insiders had reported data loss issues in the Feedback Hub almost three months ago. However, since the number of users affected was not particularly large, and the feedback reports only received a few upvotes, they were buried under the barrage of requests for a prettier UI or other feature requests.
Sampling of Feedback Hub reports of data deletion from as early as 3 months ago pic.twitter.com/xlfsuEbh1d
— Rafael Rivera (@WithinRafael) October 6, 2018
The Insider Program allows Microsoft to gain an enormous amount of feedback, but that also means that the substance of that feedback may not always be up to par, especially when it comes to quality control. Needless to say, whatever AI tricks the company is relying on for sorting feedback were likely unsuccessful in flagging these issues. Moreover, the fact that it also chose to simply skip the Release Preview ring this time around didn't help matters much, either.
Those comments about too many ninja cats I referred to earlier may be annoying, but they do underline a new facet to the Insider Program: it's no longer just about beta testing, it's now one of Microsoft's most important marketing and branding tools. Take the following interview from Insider Program head Dona Sarkar, where she tries to describe it to people who don't know what being an Insider means:
“Where is the roadmap for new #Windows and #Office features?" if you wanna be in the know, @donasarkar will share how you can be a part of @windowsinsider + @OfficeInsider @ #MSIgnite https://t.co/pNJyoA1kSJ @MSTCommunity #WindowsInsiders #officeinsiders #microsoft365 pic.twitter.com/GVt91DGwwl
— Dux Raymond Sy (@meetdux) September 13, 2018
Note that while she emphasises the role of the Insider Program in helping Microsoft figure out where to take Windows next, she does not mention, at all, the other important function of the Insider Program: testing Windows.
What can Microsoft do now? That the Insider Program is such a success or that it's now one of Microsoft's most prominent brands for interfacing with its community are not at all bad things, but all of this does mean that Microsoft needs to realise that it can't just treat the Insider Program like a normal beta.
Firstly, it's clear that the company needs to stop relying on the Insider Program so much for testing, and needs to supplement the data it receives from Insiders with some old school dedicated in-house testers. Back in 2014, the decision to lay-off a large number of the Windows team's QA team was considered unthinkable, but that is exactly what happened. Microsoft needs to rehire some of the many dedicated testers that it has laid off since 2014. The Insider Program is good for learning what users want, and informing the Windows team of what features to add, but it is not doing enough to help stave off major issues from making their way to public releases.
The company also needs to significantly reassess how Feedback Hub works. There is clearly room for improvement when it comes to how its AI algorithms sort and rate feedback, especially those issues that may be critical in nature but are not experienced by too many people. User-generated content will always be lacking in terms of expertise and quality, but Microsoft's recent decision to allow users to also indicate the severity of a bug may go a long way in helping it avoid similar mishaps in the future.
What further measures the company finally decides to take in order to fix things, only time will tell, but hopefully, the embarrassment of having to pull a major feature update just days after its release should make it clear to Microsoft that things are not working out as they stand. The recent changes are a step in the right direction, and hopefully Microsoft will continue to build on them.
33 Comments - Add comment