jump to navigation

Tips on A:B testing November 4, 2008

Posted by jeremyliew in A:B testing, product management.

If you’re doing A:B testing, you should read this paper and the accompanying presentation by Kohavi, Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. It is a comprehensive primer on how to do A:B testing well.

Some nuggets of goodness culled from the paper in no particular order (some sound obvious but read the paper to get more context). It will take you 20 minutes and you’ll use the lessons you learn for years:

1. Agree on evaluation criteria UP FRONT (vs after the fact analysis)
2. Ensure sample size is sufficiently large to have high confidence in results (with small samples, testing a version against itself can show wide ranges in performance)
3. Be truly random, and consistent when allocating users to groups
4. Ramp experiments from 1% of users to [50%] of users to get results fast [subject to day of week variations]. Auto abort if the new version is significantly worse.
5. Account for: robots skewing results, “newness” effect, time of day, day of week
6. Understand why as well as what (e.g. is lower performance caused by slower load time? incompatible browser types or screen sizes? etc. If fixable, fix and retest.)
7. Integrate constant testing into culture and systems
8. Test all the time.

Discovered via Eric Reis


1. SBL - Video audio Tagging - November 7, 2008

These points are very helpul for doing A:B testing well. Thanks for this great post.
SBL – Video audio tagging

2. Which user experience research tools a startup should use, and when « Lightspeed Venture Partners Blog - December 4, 2008

[…] experience research, with others including the ethnographic field studies made popular by Ideo, the A:B testing becoming standard for web 2.0, customer feedback, focus groups etc. With so many tools at your […]

3. Richard - January 29, 2009

You should take a look at http://blog.mixpanel.com/2009/01/ab-testing-your-website-with-mixpanel/

Apparently it’s a service to help startups A/B test a bit easier.

4. Does CVV affect e-commerce conversion rates? | Intellivative Payment Solutions without Borders - June 3, 2009

[…] Tips on A:B testing from Lightspeed Venture Partners […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: