Experimentation Platform

Experimentation Platform (ExP) is a team of 60+ Data Scientists, Software Engineers and Program Managers. Our mission is to accelerate innovation through trustworthy experimentation. Most major products such as Bing, Cortana, Edge, Exchange, Identity, MSN, Office client, Office online, Photos, Skype, Speech, Store, Teams, Visual Studio Code, Windows, Xbox use our platform ExP to run trustworthy Online Controlled Experiments – aka A/B tests.

ExP team members regularly contribute to conferences through publications and tutorials, some of which we share on this group page.

You can read about our work by visiting our Publications tab or by reading our blog posts:

• A/B Testing Infrastructure Changes at Microsoft ExP
• A/B Interactions: A Call to Relax
• Deep Dive Into Variance Reduction
• For Event-based A/B tests: why they are special
• A/B Testing and Covid-19: Data-Driven Decisions in Times of Uncertainty
• Patterns of Trustworthy Experimentation: Pre-Experiment Stage
• Diagnosing Sample Ratio Mismatch in A/B Testing
• p-Values for Your p-Values: Validating Metric Trustworthiness by Simulated A/A Tests
• Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Work 
• A/B Testing Across Products
• Metric computation for multiple backends
• It takes a Flywheel to Fly: Kickstarting and Keeping the A/B testing Momentum
• Patterns of Trustworthy Experimentation: During-Experiment Stage
• Alerting in Microsoft’s Experimentation Platform (ExP)
• Beyond Power Analysis: Metric Sensitivity in A/B Tests
• More Trustworthy A/B Analysis: Less Data Sampling and More Data Reducing
• A/A’/B Testing: Evaluating Microsoft Teams across Build Releases
• Data Quality: Fundamental Building Blocks for Trustworthy A/B testing Analysis
• Patterns of Trustworthy Experimentation: Post-Experiment Stage
• STEDII Properties of a Good Metric