How Microsoft Crowdsourced the Making of Office 2010

For several years, Denise Carlevato has studied millions of mouse clicks and keystrokes made by anonymous computer users from all over the world. Her objective: to make Microsoft Office better fit the way millions of people work.

“We were making many decisions based on … what customers wanted us to do.” –Microsoft VP P.J. Hough

Months before Microsoft rolled out the latest version of its productivity suite, Office 2010, 9 million people downloaded its beta version to test the software and provide feedback. As part of the program, Microsoft collected 2 million comments from beta testers. An additional 600 people participated in Microsoft’s Virtual Research Lab, where Carlevato and her colleagues could observe how people were using new features.

In a sense, it was a massive, controlled crowdsourcing project. That’s just what you have to do to cater to as broad an audience as possible, says Carlevato, who has worked as a Microsoft usability engineer for 10 years.

“We do our darnedest to make sure the features we put into our product are the things people ask for,” Carlevato told Wired.com. “We know from watching them work that they really need it.”

Though some tech observers have predicted Microsoft’s downfall after falling behind in the smartphone game and being one-upped by Apple in market capitalization, many agree that reports of the software titan’s inevitable demise are greatly exaggerated. Microsoft as a whole remains hugely profitable, and Office has consistently been the most lucrative part of Microsoft’s business, raking in billions of dollars each quarter, even exceeding sales of Windows.

And although Google offers a competitive productivity suite, Google Docs, for free, Microsoft still has a major advantage: 67 percent of U.S. online consumers regularly use Office, while only 4 percent use Google Docs, according to Forrester Research.

“In some ways, the ‘Office versus Google Docs’ debate doesn?t merit a lot of consideration — it?s still no competition,” said J.P. Gownder, a Forrester analyst. “In terms of usage and penetration, Google Docs remains a failure ?- so far, anyway.”

But staying in the lead with productivity software isn’t easy, and to retain the loyalty of millions, Microsoft goes to great lengths to determine what customers want. For Office 2010 beta, Microsoft included a feature called “Send a Smile,” a comment box for testers to submit feedback and suggestions for improvement. Of the 2 million Send a Smile comments, 81,000 included the senders’ e-mail addresses so the engineers working to improve Office could follow up with them.

To analyze the Send a Smile feedback, Microsoft built a database and programmed algorithms to classify and tag comments under certain categories, while filtering out biased feedback and useless drivel. From that point, researchers manually read every single comment to determine necessary tweaks and additions to Office.

A major new feature birthed from customer feedback was an online broadcasting tool for users to share PowerPoint 2010 presentations by simply sending around a URL, according to P.J. Hough, corporate vice president of Office Program Management.

“We were making many decisions not based on what others were doing, but on what customers wanted us to do,” Hough said. “We did research on customers that led us to the path.”

Microsoft also invites select users to participate in its Virtual Lab, where they are instructed to perform specific tasks such as formatting a section of a Word document, or changing the background color of a PowerPoint presentation. After lab participants completed each task, Carlevato and her colleagues analyzed their history to observe the actions they took.

The Virtual Research Lab is especially useful when users struggle to finish a task, because researchers can examine why they are becoming confused or taking too long and work to resolve the problem. This is what usability researchers call “unarticulated needs,” said Carlevato.

Going beyond virtual testing, Microsoft engineers also worked one-on-one with human subjects at the University of Washington to create the brand-new Office web apps suite released in mid-June. A group of 26 students testing an early version of Office web apps met with engineers every two weeks to discuss features they wanted or didn’t like in the web-based suite.

Microsoft’s product planner Char Popp helped lead the field trial, and she said it was crucial to work in a human environment with live feedback, because this was the very first version of Office web apps.

“While the team was building it and giving insights and information, it was still all like a big jigsaw puzzle,” she said. “Toward the end when this all came together I had the opportunity to go out and see how the web apps were actually going to live with real people.”

Early reviews of the Office web apps (including Wired’s) weren’t stellar, citing some missing key features and elements, but Microsoft stressed that this was a version-one product, and it would continue to collect feedback to improve the service.

“With any release as big as Office you can’t do everything all at once,” said Hough, who added that any unaddressed shortcomings are “part of our engineering road map.”

Fortunately for Microsoft, there are 360 million Windows Live users with access to the Office web apps who can help crowdsource the solutions.

See Also: