Skip to main content

Experimentation Platform

Accelerating software innovation through trustworthy experimentation

Home: ExP Platform
Talks
Journal Survey
Encyclopedia article
Objective Bayesian AB
Dilution
Rules of Thumb
Two Stage
Large Scale
Puzzling Outcomes
CUPED
Experiments at Microsoft
Practical Guide (short)
Tracking Users' Clicks an
Seven Pitfalls
Semmelweis Reflex
Bloodletting
Power Calculator
What is a HiPPO
Pitfalls of Long Term
KDD 2015 Keynote
 
Online Controlled Experiments:

Lessons from Running A/B/n Tests for 12 years

Ron Kohavi
 

 

Talk slides in PPT and in PDF

 

DOI: 10.1145/2783258.2785464


Talk Video

 

 

Abstract
 

The Internet provides developers of connected software, including web sites, applications, and devices, an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using trustworthy controlled experiments (e.g., A/B tests and their generalizations). From front-end user-interface changes to backend recommendation systems and relevance algorithms, from search engines (e.g., Google, Microsoft’s Bing, Yahoo) to retailers (e.g., Amazon, eBay, Netflix, Etsy) to social networking services (e.g., Facebook, LinkedIn, Twitter) to Travel services (e.g., Expedia, Airbnb, Booking.com) to many startups, online controlled experiments are now utilized to make data-driven decisions at a wide range of companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale (e.g., hundreds of experiments run every day at Bing) and deployment of online controlled experiments across dozens of web sites and applications has taught us many lessons.  We provide an introduction, share real examples, key lessons, and cultural challenges.

 

 

To reference keynote talk:

 

Ron Kohavi. 2015. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '15). ACM, New York, NY, USA, 1-1. http://bit.ly/KDD2015Kohavi.  DOI=http://dx.doi.org/10.1145/2783258.2785464

@inproceedings{Kohavi:2015:OCE:2783258.2785464,
 author = {Kohavi, Ron},
 title = {Online Controlled Experiments: Lessons from Running A/B/N Tests for 12 Years},
 booktitle = {Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
 series = {KDD '15},
 year = {2015},
 isbn = {978-1-4503-3664-2},
 location = {Sydney, NSW, Australia},
 pages = {1--1},
 url = {http://bit.ly/KDD2015Kohavi},
 doi = {10.1145/2783258.2785464},
 acmid = {2785464},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {a/b testing, controlled experiments},
} 

 

 

 

Photos, thanks to Carmen Lee from encapture:

 

 

 

 

 

Geoff Webb and Thorsten Joachims, the KDD 2015 program co-chairs

 

 

People voting on A/B quiz

 

 
Few people remained standing: worse than random  voting
 
 
 
 
 
Diane Tang
 
 
Diane Tang
 
 
 
 
Shortcut to this page: http://bit.ly/KDD2015Kohavi