jump to navigation

Which user experience research tools a startup should use, and when November 7, 2008

Posted by jeremyliew in A:B testing, product management, UI, usability.

I recently posted about how usability testing can slow down launch but speed up success. But usability testing is just one of many elements of user experience research, with others including the ethnographic field studies made popular by Ideo, the A:B testing becoming standard for web 2.0, customer feedback, focus groups etc. With so many tools at your disposal, which user experience research tools should you use and when?

Jakob Nielsen recently posted about this topic, and concluded that it depends on what phase of product management you are in. For startups, my summary of his work is below:

Ideation: At the very beginning of the process you are looking at new ideas and opportunities. In this phase, aside form the founders vision, ethnographic field studies, focus groups, diary studies, surveys and data mining of webwide behavior can all be useful. Most startups will not have access to proprietary user data of existing products to identify additional opportunities.

Pre-launch: Once you’ve settled on a product idea and are working towards (beta) launch, you want to improve design and functionality as much as possible to minimize risk and maximize the likelihood for a successful launch. In this phase rely on tools such as cardsorting, paper prototype and usability studies, participatory design, desirability studies and field studies (including closed alpha launches to “friendlies”) to improve the user experience.

Post-launch: Once you’ve pushed the product out you will have live data that you can use to compare the product both to itself and to its competition. In this phase, usability benchmarking, online assessments, customer emails, surveys and A/B testing will be your primary tools

Nielsen provides some additional frameworks to differentiate when to use different forms of user experience research in his post. The site is a good resource about user experience in general.


1. Amy Jo Kim - November 9, 2008

great stuff — thanks for posting!

2. Justin Barrie - November 9, 2008

Jeremy – Ideo are fantastic at the ‘building’ end of design. Another great user experience company to check out is run by a mate of mine in Redwood Shores – Cheskin. Their thinking is fantastic and they have been a bit of an inspiration for our approach to user experience.


3. Tim Jaeger - November 10, 2008


One tool I find indispensible is OmniGraffle Pro. It is mac-based (hope that isn’t too bad for all the PC users out there..), but there are tons of custom stencils you can make make for it that scale as vectors (or vectorially, whichever..).

I wrote a post on it, you can check it out:


4. Alison Murdock - March 18, 2009

How many start-ups do user testing beyond the immediate families of the 3-10 employees? How many people do they test and how much do they spend – time- and money wise?

jeremyliew - March 19, 2009


This varies widely, but it is a good idea to do user testing on non employees who are in the target market and not familiar with the product.

In general, doing usability testing with 5 people is enough to uncover most usability problems (http://www.useit.com/alertbox/20000319.html). This can be done in a day or two and you can recruit testers via craigslist or a similar resource for $20/hr and do the testing via webex and a telephone so it should be cheap.

Of course, you have to be willing to listen to the feedback (vs attributing it to “user error” – there is no such thing as user error in usability testing!) and make changes to address issues that come up. That may take considerably more time and money

5. Quora - September 26, 2010

Does user research get the respect it deserves in the Valley? Why or why not?…

User research gets the respect it deserves in software companies, but not in internet companies. However, this is starting to change. In the pre internet days, when everything was software and you couldn’t easily change something after you launched it…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: