jump to navigation

Usability testing slows down launch but speeds up success October 10, 2008

Posted by jeremyliew in A:B testing, product management, UI, usability.

I hate hearing the term “user error”. Good usability testing should eliminate most user error. I am a big proponent of A:B testing with live users. However, often a small usability test can quickly highlight any big problems before you go live so that you are working from a better starting point.

Many developers like to quickly prototype and push code out quickly, and I am a fan of this. However, if taken to an extreme, it can lead to products based on incorrect assumptions. Noah Kagan notes that at Mint:

We did surveys, user testing and psychological profiles. This was extremely useful in identifying the types of users we may have on the site and especially for seeing how people use the site. I never really did this before and was AMAZED how people use the site vs. what I expected

As Noah points out, usability testing can be easy and cheap. What it requires is simple:

1. Determine what you are trying to test. This is usually a list of the form, “How easy is it to complete task X?”

2. Recruit representative users. If you’re testing a new user experience, Craigslist is fine for this [Tip – if your core user is a middle america, . But make sure that your testers are truly representative. Your existing user base is another good place to find testers, but make sure that you’re not just listening to just your loudest users. The key is to pre-qualify the users to ensure that they are “average”.

3. Do the test. First ask them what their first impressions of the site are. What captures their attention? What would they do first? Ask the users to speak out loud during the session, explaining what they are thinking at all times. Then ask them to complete the tasks that you have listed. Watch and listen. Note what they find easy, what they find confusing, and what they don’t find at all.

This can be incredibly frustrating for you. You’ll think that some things are “obvious” that are not, or you’ll be shocked to see how unfamiliar users are with your site, or even with how browsers work. Remember that your role is to learn, not teach. Don’t touch the screen, the keyboard or the mouse; don’t point out how to do anything (even when they are “doing it wrong”, even if it is a “basic mistake”). You can provide encouragement and reassurance, or ask questions about why they did something, but that is it. You’ll be surprised at what you see.

The key is to internalize that there is no such thing as “user error” and there are no “stupid users”. If users are having problems achieving the tasks that you laid out for them, then the fault lies with your site. You’ll need to review your UI.

I prefer to do these usability tests over webex with users at their own computers. This makes the interaction as natural as possible for the testers. As an added bonus, you can then record both their screencast and the phonecall for later review.

Usability testing does not have to be a lot of work/ You only need to test five users to uncover most usability problems.

The most striking truth of the curve is that zero users give zero insights.

When you’ve completed your usability test, go back and makes some changes, but then come back and test again:

You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with 5 users has found 85% of the usability problems, you will want to fix these problems in a redesign.

After creating the new design, you need to test again. Even though I said that the redesign should “fix” the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn’t. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.

Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left – they will have to wait until the third test to be identified.)

Finally, the second test will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs. These important issues are often obscured in initial studies where the users are stumped by stupid surface-level usability problems that prevent them from really digging into the site.

This can all feel like overhead when all you want to do is launch. Trust me, it isn’t overhead. Getting this stuff closer to right the first time will only help you reach your goals faster.


1. dp - October 10, 2008

i wish mint would do usability tests to see how painful it is to categorize checks – takes up >80% of my time on their site.

2. Ravi Kannan - October 11, 2008

Can’t agree with you more. At JobsByRef.com we have been doing exactly this at this time. We are allowing a small group of people with different skills both technical and non-technical to use the service and monitoring their behavior across the site. This has and is helping us in fixing lot of the usability issues up-front before we open it up to everyone.

3. Clay Whitehead - October 13, 2008

Great post! I particularly liked the focus on “no such thing as user error.” I wrote on this topic recently as well: http://inveterateoptimism.blogspot.com/2008/10/five-tips-for-effective-usability.html

4. Lee Drake - October 14, 2008

Excellent post. I summarized, and added some notes based on entrepreneurial software we’re developing for a client here:


5. These Links are Hyper | Juicer - Eric Nelson's Blog - October 18, 2008

[…] Usability testing slows down launch but speeds up success […]

6. The OPLIN 4cast » Blog Archive » OPLIN 4Cast # : Screen Readers, Usability Testing, Elections, AOL - October 22, 2008

[…] Usability testing slows down launch but speeds up success […]

7. Which user experience research tools a startup should use, and when « Lightspeed Venture Partners Blog - November 7, 2008

[…] in A:B testing, UI, product management, usability. trackback I recently posted about how usability testing can slow down launch but speed up success. But usability testing is just one of many elements of user experience research, with others […]

8. Three meta-rules for usability design « Lightspeed Venture Partners Blog - November 20, 2008

[…] are by definition experts, and this can lead to a blindness to possible modes of error. Just five user tests will turn up the biggest problems. C: (3,6,7&8) Give users visual clues as to what does […]

9. Good Usability - November 23, 2008

I agree with most of what you said here. I wouldn’t advocate spending a lot of time asking participants first impressions etc. You put them into critique mode and they get an unnatural introduction to the homepage. This depends on the website though.

I also wouldn’t say usability testing slows down launch. Leaving usability testing out will quicken your launch in the same way as missing out any other essential part of a successful project.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: