We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response.
Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us
   -- Ray Ozzie, Chief Software Architect         

Our Mission is to accelerate software innovation through trustworthy experimentation

Introduction

The Experimentation Platform enables product groups at Microsoft and later on will enable developers using Windows Live to innovate using controlled experiments with live users.   The platform enables testing new ideas quickly using the best-known scientific method for establishing causality between a feature and its effects: randomized experimental design.   The basic methodology in controlled experiments is to expose a percentage of users to a new treatment, measure the effect on metrics of interest, and run statistical tests to determine whether the differences are statistically significant, thus establishing causality.  The diagram below explains the concept:

Additional analysis against the collected data can provide more insights and enable testing other hypotheses.   Runtime monitoring, while the experiment is running, will enable early auto-aborts of experiments that are clearly underperforming, providing graceful failure (by auto-reverting everyone to Control).  The safety provided by this architecture encourages quicker deployments without extensive testing, and also allows for automated ramp-up from a tiny percent of users receiving treatment to a larger percentage since there are no adverse affects to the customer experience.

The platform encourages faster launches, which will in turn result in faster failures of bad ideas and better understanding of good ideas. 

1. Controlled Experiments on the Web: Survey and Practical Guide (July 30th, 2008) appeared in the journal of Data Mining and Knowledge Discovery.
2. The amazing story of Semmelweis and the Semmelweis Reflex (Aug 24, 2008)
3. Bloodletting: Why Controlled Experiments are Important (May 19, 2008)
4. Q&A with Mary Jo Foley (April 16, 2008).   (CNET news, Liveside, Germany, Japan, Poland, Turkey)
5. White paper (March 2008):  Seven Pitfalls to Avoid when Running Controlled Experiments on the Web by Thomas Crook, Brian Frasca, Ronny Kohavi, Roger Longbotham
6.  
   Emetrics 2007 talk in Washington DC (October 2007): Practical Guide to Controlled Experiments (includes multiple new examples).
7. We published a paper that appeared in KDD 2007 (August 2007): Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO by Ronny Kohavi, Randy Henne, and Dan Sommerfield.
8. A second paper in IEEE Computer (September 2007) is available: Online Experiments, Lessons Learned by Ronny Kohavi and Roger Longbotham (copyright IEEE, published with here with permission).

     

9. is Live at Microsoft!  

   Several teams have successfully integrated the Experimentation Platform into their product development environments and are running live experiments on various Microsoft web sites