Skip to main content

Experimentation Platform

Accelerating software innovation through trustworthy experimentation

Home: ExP Platform
Talks
Journal Survey
Encyclopedia article
Objective Bayesian AB
Dilution
Rules of Thumb
Two Stage
Large Scale
Puzzling Outcomes
CUPED
Experiments at Microsoft
Practical Guide (short)
Tracking Users' Clicks an
Seven Pitfalls
Semmelweis Reflex
Bloodletting
Power Calculator
What is a HiPPO
Pitfalls of Long Term
Appears in the journal of Data Mining and Knowledge DiscoveryPDF
 

Abstract
The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.
What others are saying
 

-----------------------------------------

 

@article{controlledExperimentsSurvey
    author = {Kohavi, Ron and Longbotham, Roger and Sommerfield, Dan and Henne, Randal M.},
    title = {Controlled Experiments on the Web: Survey and Practical Guide},
    journal = {Data Mining and Knowledge Discovery},
    issue_date = {February  2009},
    volume = {18},
    number = {1},
    month = feb,
    year = {2009},
    issn = {1384-5810},
    pages = {140--181},
    url = {http://dx.doi.org/10.1007/s10618-008-0114-1},
    doi = {10.1007/s10618-008-0114-1},
    acmid = {1485091},
    publisher = {Kluwer Academic Publishers},
    address = {Hingham, MA, USA},
    keywords = {A/B testing, Controlled experiments, MVT, MultiVariable Testing, Website optimization, e-commerce},
}

 

ACMRef: Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled Experiments on the Web: Survey and Practical Guide. Data Mining and Knowledge Discovery 18, 1 (February 2009), 140-181. DOI=10.1007/s10618-008-0114-1 http://dx.doi.org/10.1007/s10618-008-0114-1


 Quick link to this page: http://bit.ly/expSurvey