CS547 Human-Computer Interaction Seminar (Seminar on People, Computers, and Design)
Fridays 11:30am-12:30pm PT · Gates B3 · Open to the public|
Ramesh Johari
Stanford University
Peeking at A/B Tests: Why it matters, and what to do about it
March 3, 2017
I'll describe a novel statistical methodology that has been deployed by the commercial A/B testing platform Optimizely to communicate experimental results to their customers. Our methodology addresses the issue that traditional p-values and confidence intervals give unreliable inference. This is because users of A/B testing software are known to *continuously monitor* these measures as the experiment is running. We provide *always valid* p-values and confidence intervals that are provably robust to this effect. Not only does this make it safe for a user to continuously monitor, but it empowers her to detect true effects more efficiently. I'll cover why continuous monitoring is a problem, and how our solution addresses it. The talk will be presented at a level that does not presume much statistical background. Joint work with Pete Koomen (Optimizely), Leo Pekelis (Optimizely/OpenDoor), David Walsh (Stanford). (This work was done under my relationship with Optimizely as a technical advisor.) |
|