PCD 1/25/08 Kohavi

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments or A/B tests. In this talk, I’ll provide multiple real-world examples of control experiments that were run at Microsoft and Amazon, many with very surprising results. Significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person’s Opinion (HiPPO). I’ll review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational).

Ronny Kohavi is the General Manager for Microsoft's Experimentation Platform, a team whose mission is to build a platform that will accelerate software innovation through trustworthy experimentation. Controlled experiments, also called A/B tests, allow evaluating ideas through randomized assignment of users to a Control group or different Treatment groups. The methodology is practically the only scientific method we know to establish causal relationships between ideas and metrics of interest.

Prior to joining Microsoft in 2005, Ronny was the director of data mining and personalization at Amazon.com, where he was responsible for personalization, automation, search engine marketing (SEM), consumer behavior / data mining, site experimentation, and automated e-mail. His teams introduced several features estimated to be worth several hundred million dollars in incremental revenue. Prior to Amazon, Ronny was the Vice President of Business Intelligence at Blue Martini Software, where he led the engineering group responsible for the data collection, analysis, visualization, reporting, and campaign management modules in Blue Martini's applications. Prior to joining Blue Martini, Kohavi managed the MineSet product, Silicon Graphics' award-winning product for data mining and visualization. MineSet was based in part on MLC++, a machine learning library developed at Stanford University.

Ronny received a Ph.D. in Machine Learning from Stanford University and a BA from the Technion, Israel. He was the General Chair for KDD 2004, he co-chaired KDD 99's industrial track with Jim Gray, and he co-chaired the KDD Cup 2000. He was an invited speaker at Emetrics 2007, the SF ACM Data Mining SIG in 2006, Emetrics 2004, KDD 2001's industrial track, the National Academy of Engineering in 2000.

Titles and abstracts for previous years are available by year and by speaker.

Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO

Practical Guide to Controlled Experiments on the Web:
Listen to Your Customers not to the HiPPO