Improving the Sensitivity of Online Controlled Experiments
by Utilizing Pre-Experiment Data
by Alex Deng, Ya Xu, Ron Kohavi, Toby Walker
To appear in WSDM 2013, Feb 4-8, 2013, Rome, Italy. PDF.
Online controlled experiments are at the heart of making data-driven decisions at a diverse set of companies, including Amazon, eBay, Facebook, Google, Microsoft, Yahoo, and Zynga. Small diﬀerences in key metrics, on the order of fractions of a percent, may have very signiﬁcant business implications. At Bing it is not uncommon to see experiments that impact annual revenue by millions of dollars, even tens of millions of dollars, either positively or negatively. With thousands of experiments being run annually, improving the sensitivity of experiments allows for more precise assessment of value, or equivalently running the experiments on smaller populations (supporting more experiments) or for shorter durations (improving the feedback cycle and agility). We propose an approach (CUPED) that utilizes data from the pre-experiment period to reduce metric variability and hence achieve better sensitivity. This technique is applicable to a wide variety of key business metrics, and it is practical and easy to implement. The results on Bing’s experimentation system are very successful: we can reduce variance by about 50%, eﬀectively achieving the same statistical power with only half of the users, or half the duration.