Which user experience research tools a startup should use, and when November 7, 2008

I recently posted about how usability testing can slow down launch but speed up success. But usability testing is just one of many elements of user experience research, with others including the ethnographic field studies made popular by Ideo, the A:B testing becoming standard for web 2.0, customer feedback, focus groups etc. With so many tools at your disposal, which user experience research tools should you use and when?

Jakob Nielsen recently posted about this topic, and concluded that it depends on what phase of product management you are in. For startups, my summary of his work is below:

Ideation: At the very beginning of the process you are looking at new ideas and opportunities. In this phase, aside form the founders vision, ethnographic field studies, focus groups, diary studies, surveys and data mining of webwide behavior can all be useful. Most startups will not have access to proprietary user data of existing products to identify additional opportunities.

Pre-launch: Once you’ve settled on a product idea and are working towards (beta) launch, you want to improve design and functionality as much as possible to minimize risk and maximize the likelihood for a successful launch. In this phase rely on tools such as cardsorting, paper prototype and usability studies, participatory design, desirability studies and field studies (including closed alpha launches to “friendlies”) to improve the user experience.

Post-launch: Once you’ve pushed the product out you will have live data that you can use to compare the product both to itself and to its competition. In this phase, usability benchmarking, online assessments, customer emails, surveys and A/B testing will be your primary tools

Nielsen provides some additional frameworks to differentiate when to use different forms of user experience research in his post. The site is a good resource about user experience in general.

Usability testing slows down launch but speeds up success October 10, 2008

I hate hearing the term “user error”. Good usability testing should eliminate most user error. I am a big proponent of A:B testing with live users. However, often a small usability test can quickly highlight any big problems before you go live so that you are working from a better starting point.

Many developers like to quickly prototype and push code out quickly, and I am a fan of this. However, if taken to an extreme, it can lead to products based on incorrect assumptions. Noah Kagan notes that at Mint:

We did surveys, user testing and psychological profiles. This was extremely useful in identifying the types of users we may have on the site and especially for seeing how people use the site. I never really did this before and was AMAZED how people use the site vs. what I expected

As Noah points out, usability testing can be easy and cheap. What it requires is simple:

1. Determine what you are trying to test. This is usually a list of the form, “How easy is it to complete task X?”

2. Recruit representative users. If you’re testing a new user experience, Craigslist is fine for this [Tip – if your core user is a middle america, . But make sure that your testers are truly representative. Your existing user base is another good place to find testers, but make sure that you’re not just listening to just your loudest users. The key is to pre-qualify the users to ensure that they are “average”.

3. Do the test. First ask them what their first impressions of the site are. What captures their attention? What would they do first? Ask the users to speak out loud during the session, explaining what they are thinking at all times. Then ask them to complete the tasks that you have listed. Watch and listen. Note what they find easy, what they find confusing, and what they don’t find at all.

This can be incredibly frustrating for you. You’ll think that some things are “obvious” that are not, or you’ll be shocked to see how unfamiliar users are with your site, or even with how browsers work. Remember that your role is to learn, not teach. Don’t touch the screen, the keyboard or the mouse; don’t point out how to do anything (even when they are “doing it wrong”, even if it is a “basic mistake”). You can provide encouragement and reassurance, or ask questions about why they did something, but that is it. You’ll be surprised at what you see.

The key is to internalize that there is no such thing as “user error” and there are no “stupid users”. If users are having problems achieving the tasks that you laid out for them, then the fault lies with your site. You’ll need to review your UI.

I prefer to do these usability tests over webex with users at their own computers. This makes the interaction as natural as possible for the testers. As an added bonus, you can then record both their screencast and the phonecall for later review.

Usability testing does not have to be a lot of work/ You only need to test five users to uncover most usability problems.

The most striking truth of the curve is that zero users give zero insights.

When you’ve completed your usability test, go back and makes some changes, but then come back and test again:

You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with 5 users has found 85% of the usability problems, you will want to fix these problems in a redesign.

After creating the new design, you need to test again. Even though I said that the redesign should “fix” the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn’t. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.

Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left – they will have to wait until the third test to be identified.)

Finally, the second test will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs. These important issues are often obscured in initial studies where the users are stumped by stupid surface-level usability problems that prevent them from really digging into the site.

This can all feel like overhead when all you want to do is launch. Trust me, it isn’t overhead. Getting this stuff closer to right the first time will only help you reach your goals faster.