CS547 Human-Computer Interaction Seminar   (Seminar on People, Computers, and Design)

Fridays 11:30am-12:30pm PT · Gates B3 · Open to the public
Previous | Next
Ramesh Johari
Stanford University
Peeking at A/B Tests: Why it matters, and what to do about it
March 3, 2017

I'll describe a novel statistical methodology that has been deployed by the commercial A/B testing platform Optimizely to communicate experimental results to their customers. Our methodology addresses the issue that traditional p-values and confidence intervals give unreliable inference. This is because users of A/B testing software are known to *continuously monitor* these measures as the experiment is running. We provide *always valid* p-values and confidence intervals that are provably robust to this effect. Not only does this make it safe for a user to continuously monitor, but it empowers her to detect true effects more efficiently. I'll cover why continuous monitoring is a problem, and how our solution addresses it.

The talk will be presented at a level that does not presume much statistical background.

Joint work with Pete Koomen (Optimizely), Leo Pekelis (Optimizely/OpenDoor), David Walsh (Stanford). (This work was done under my relationship with Optimizely as a technical advisor.)


Ramesh Johari is an Associate Professor at Stanford University, with a full-time appointment in the Department of Management Science and Engineering (MS&E), and courtesy appointments in the Departments of Computer Science (CS) and Electrical Engineering (EE). He is a member of the Operations Research group and the Social Algorithms Lab (SOAL) in MS&E, the Information Systems Laboratory (ISL) in EE, and the Institute for Computational and Mathematical Engineering (ICME).