Tips on A:B testing

If you’re doing A:B testing, you should read this paper and the accompanying presentation by Kohavi, Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. It is a comprehensive primer on how to do A:B testing well.

Some nuggets of goodness culled from the paper in no particular order (some sound obvious but read the paper to get more context). It will take you 20 minutes and you’ll use the lessons you learn for years:

1. Agree on evaluation criteria UP FRONT (vs after the fact analysis)
2. Ensure sample size is sufficiently large to have high confidence in results (with small samples, testing a version against itself can show wide ranges in performance)
3. Be truly random, and consistent when allocating users to groups
4. Ramp experiments from 1% of users to [50%] of users to get results fast [subject to day of week variations]. Auto abort if the new version is significantly worse.
5. Account for: robots skewing results, “newness” effect, time of day, day of week
6. Understand why as well as what (e.g. is lower performance caused by slower load time? incompatible browser types or screen sizes? etc. If fixable, fix and retest.)
7. Integrate constant testing into culture and systems
8. Test all the time.

Discovered via Eric Reis