How a change in wording increased response rates by 15 percentage points July 9, 2012
Posted by jeremyliew in copy, product management.1 comment so far
I’m a big fan on focusing on getting the “copy” (the words on the page) right to drive behavior. I’ve posted in the past about Cialdini’s great book, Influence, The Psychology of Persuasion, and how the principles outlined in it can be used for structured brainstorming to improve copy to drive the results that you want. Of course, all this needs to be A:B tested, but it provides great ideas to test.
Sunday’s NY Times has a great story on how behavioral science can help drive policy, and how a change in copy helped increase tax collection in the UK by 15 percentage points:
One early success story involves an attempt to collect taxes from people who fail to pay on time. Most British citizens pay their taxes promptly because it is a simple tax system with few deductions, so that most taxes are collected via payroll withholding. (That’s “make it easy” in action.) But small-business owners and individuals with significant nonpayroll income are expected to save up the money to write a check to the government, and some of them fail to pay on time.
In such cases, the government’s first step is to send a letter asking for payment within six weeks, after which sterner, more expensive measures are taken. The tax collection authority wondered whether this letter might be improved. Indeed, it could.
The winning recipe comes from Robert B. Cialdini, an emeritus professor of psychology and marketing at Arizona State University, and author of the book “Influence: The Psychology of Persuasion.”
People are more likely to comply with a social norm if they know that most other people comply, Mr. Cialdini has found. (Seeing other dog owners carrying plastic bags encourages others to do so as well.) This insight suggests that adding a statement to the letter that a vast majority of taxpayers pay their taxes on time could encourage others to comply. Studies showed that it would be even better to cite local data, too.
Letters using various messages were sent to 140,000 taxpayers in a randomized trial. As the theory predicted, referring to the social norm of a particular area (perhaps, “9 out of 10 people in Exeter pay their taxes on time”) gave the best results: a 15-percentage-point increase in the number of people who paid before the six-week deadline, compared with results from the old-style letter, which was used as a control condition.
The tax authorities estimate that this initiative, if rolled out across the country, could generate £30 million of extra revenue annually. And note that sending an effective letter doesn’t cost any more than sending a bad one.
If you’re in product management and you haven’t read Cialdini’s book, go out and buy it now. Nine out of ten product managers already have! 😉
Startup CEO New Years Resolutions January 6, 2011
Posted by jeremyliew in 2011, culture, growth, HR, new years resolutions, product management.2 comments
I asked the CEO’s of the companies that I work with, “What are your company related new years resolutions?”. Each had a different spin, but they mostly fell into a few themes of:
- Great people
- Improve the product
- Stay Lean
- Grow fast
- Internalize culture and values
Lisa Marino of RockYou, a leading developer of social games and advertising solutions for social media, is focused on building a great team with more gaming DNA to improve the quality of the games it publishes:
Make RockYou the place talent wants to be.
Many of the companies had resolutions focused on their product management and development. From the social gaming companies, Will Harbin of Casual Collective, which publishers the popular game Backyard Monsters, said simply:
Mo’ money, mo’ fun
As games like CityVille have shown, mo’ fun usually leads to mo’ money! [And congrats to Alex Le, cofounder of a prior investment, Serious Business, whose first game since the Zynga acquisition is CityVille.]
Scott Albro of Focus, a knowledge sharing community for business people, has fully embraced the idea of constant iteration for 2011, taking the best practices of social media to the business media world.
Plan big, increment small. Meaning: set big, audacious goals for the year, but understand how the little things you do every day link together so you can achieve those big goals.
Encourage more product suggestions from our engineering team. Our engineers often come up with great ideas, and they are the most productive when they work on these ideas. in 2011, I’d like to encourage a culture where this happens as much as possible.
Shawn Gupta of OhLife, a personal journaling tool over email, has similar sentiments in using metrics to drive product innovation:
Make metrics a core part of our product development. It will be a lot easier for us to make improvements to our product when we have data-driven discussions and decisions.
Although the industry is in much better shape than the dark days of 2008 and 2009, many CEOs have fully embraced the continue to internalize the lean startup principles that came out of those years. Joe Greenstein of Flixster, the leading movies app on all social and mobile platforms, wants to take big risks with small dollars.
Stay hungry, stay foolish.
Pursue our passion, build lasting strategic relationships and most importantly use our cash wisely.
Put the pedal through the floor.
Traffic,traffic,traffic! For media sites like ours you are either growing traffic or or you are dying. Our whole focus this year will be on finding new users, improving the user experience and increasing their engagement with our sites.
Two ecommerce companies that have seen tremendous growth are paying attention to their core values and culture to keep their organizations coherent as they dramatically increase in size. Andy Dunn of Bonobos, a vertical web retailer, will be driving attention to one of their core values in 2011:
Focus on self-awareness, the core trait of leaders, both people and firms.
We will measure our traction in three ways:
1. Knowing we are becoming self-aware as a team. Measure of success: 360 degree reviews done smartly (meaning efficiently and not dreaded by all involved).
2. Marketing who we are, not who we may want to be one day. Measure of success: more than doubling our customer base without doubling our cost per acquisition.
3. Developing products based on knowledge of our strengths and curating products based on knowledge of our weaknesses. Measure of success: sales growth and gross margin return on investment.
Brian Lee of Shoedazzle, a personalized fashion etailer for women, is also focused on one of their core values, great customer service:
Treat every client like they are part of your family.
Simplify – Challenge ourselves to simplify, we can strive for perfection in the next version.Celebrate – Take the time to celebrate small wins in all areas of the company, and do it every day as they happen.Feedback – Continue to foster an environment where everybody on the team communicates feedback, good and bad, in an open and honest way.
Focusing on copy can dramatically improve user response June 18, 2009
Posted by jeremyliew in copy, product management, UI, usability.2 comments
A couple of years ago I posted about how improving copy is an easy way to increase user interaction. Bokardo recently posted on a similar topic, about how microcopy can improve your user interface:
Microcopy is extremely contextual…that’s why it’s so valuable. It answers a very specific question people have and speaks to their concerns right on the spot. And because its so contextual, microcopy isn’t always obvious. Sometimes you have to hunt to find the right words. (or create an error notification service like I did) How to discover these hurdles? Talk to people! Why aren’t they adopting your software? What concerns do they have? What are they worried about? Successful salesmen know the power of these small turns of phrase. They have an arsenal of them for every situation.
Here are some other examples:
- When signing up for a newsletter, say “this low-volume newsletter”
- When people add their emails, say “we hate spam as much as you do”
- When subscribing for something free, say “you can always unsubscribe at any time”
- When selling an paid-for web application, be sure to let people know if you have a free trial.
- When storing customer’s information, say “You can export your information at any time”
- If offering optional account creation, say “If you create an account, you’ll be able to track your package”
All of these microcopy examples have one thing in common: they help to alleviate concerns of would-be customers. They help to reduce commitment by speaking directly to the thoughts in people’s heads. That’s why this copy can be so short yet so powerful.
Don’t be deceived by the size of microcopy. It can make or break an interface.
Writing good copy may not feel as heroic as implementing a huge new code base, but it can often by just as effective in increasing user interaction.
How to use popularity lists to influence your users behavior May 20, 2009
Posted by jeremyliew in Consumer internet, product management, social media, user generated content.10 comments
Two years ago I asked if crowds generated wisdom, or simply crowdiness. Today’s WSJ has a really interesting article on the same topic, concluding that popularity is self fulfilling and that arbitrary top 10 lists can meaningfully increase the popularity of the items on that list:
A more-recent study demonstrates that popularity in the music world, even unearned, breeds more popularity. Researchers enlisted more than 12,000 volunteers to rate and download songs from among 48 chosen for their relative obscurity. Some of these volunteers were lied to: At a certain stage in the experiment, popularity rankings for this group were reversed, so the least-downloaded songs were made to appear most-downloaded.
Suddenly, everything changed. The prior No. 1 began making a comeback on the new top dog, but the former No. 47 maintained its comfortable lead on the old No. 2, buoyed by its apparent popularity. Overall, the study showed that popularity is both unstable and malleable.
I think music and other entertainment sources are an interesting case study because in these industries the problem of discovery is quite difficult to solve for many bands/movies/writers etc, as well as for consumers, and these top 10 lists can help solve that problem. However, it isn’t just improving discoverability that is important, but also the perception of popularity of the discovered items as another study found:
Another group of researchers demonstrated this with restaurant diners in Beijing. Table cards at Mei Zhou Dong Po, a Szechuan restaurant chain, touting the five most popular items boosted ordering of these items by 13% to 20%, according to a forthcoming paper by a team from Peking University and Duke University. “Part of it is reassurance that something is good and worth buying,” says Bill Paul, a restaurant-menu designer.
Calling these items popular is crucial, the researchers found, because other table cards that highlighted five sample items but made no claim on their popularity had little effect on sales. And the diners liked following the pack: “Diners who were exposed to the popularity information treatment are more satisfied,” says co-author Hanming Fang, a Duke economist.
These findings are consistent with one of Cialdini’s principles of persuasion, social proof.
The WSJ article mentions another study where a hotel tried to get customers to reuse towels. Claiming that 75% of people who stayed in the same room as the customer reused their towels increase towel reuse rates by 300% over the control message as you can see in the left hand column of the chart below.
Like the hotel, social media sites, e-tailers and other companies that are trying to influence their users’ click paths can use claimed or actual popularity to get their users to do more of what they want.
Three meta-rules for usability design November 20, 2008
Posted by jeremyliew in iphone, product management, UI, usability.2 comments
I recently saw a useful deck from Create with Context about how people really use the iPhone. The company did a bunch of ethnographic research on how ordinary users (not power users) used the iPhone, and in particular, what were common mistakes made. Based on their research, here are their eight rules of thumb for designing for the iPhone:
1. Take advantage of learned behaviors
2. Avoid interaction inconsistencies
3. Provide clear conceptual link across widgets
4. Put space between action widgets
5. Plan for accidental overswiping
6. Don’t rely exclusively on multitouch
7. Provide visual feedback for taps
8. Provide interaction affordances
The presentation provides examples of all of these rules and what sort of errors are possible when they are not followed, and is well worth the 10 minutes it takes to review.
These useful rules of thumb fall into three metarules for design that make sense under almost all circumstances:
A: (1&2) Use UI standards whenever you can
When I was GM of Netscape I drove some bad design decisions for the Netscape 8 browser based on my idea of “efficiency”. The worst one was moving the browser dropdown menus to the right hand side of the application window, up a level so that it was on the same level as the logo and the name of the current web page. See a screenshot here (look at the top right hand corner of the page, in the black background with blue text).
We did this with the best intentions – to save some pixel height and thereby create more space on the screen for the actual webpage. Of course, since this “improvement” was different from the way that every other app in the world works, many users were confused and couldn’t find the dropdown menus at all. It was a bad idea.
I think that tagclouds, once all the UI rage, have since faded from the scene for the same reason. A few years ago they were the hallmark of web 2.0, all over the home pages of Delicious, Technorati and Flickr. But they are gone from all of these home pages now. The majority of users (not early adopters) found them to be too different from the standard forms of navigation that they were used to (left hand nav bars, top nav bars, search, headline-teaser) and didn’t use them.
B: (4&5) Make sure that user error is not catastrophic
This is obvious, but worth mentioning. The team building a product are by definition experts, and this can lead to a blindness to possible modes of error. Just five user tests will turn up the biggest problems.
C: (3,6,7&8) Give users visual clues as to what does what
This is an area where designers sometimes come with the wrong instincts. Making things look good is definitely not the same as making them easy to use. In fact, sometimes these factors are contradictory.
One common example is in formatting a page to fit a screen. While the most aesthetic design has all the elements of a page fitting neatly into a grid so that nothing is “cut off” by the fold of the screen, in fact having partial images disappearing off screen is an important visual cue to users that there is more below the fold. Similarly, overly stylized icons can confuse a user as to what they represent. Also, UI that is too subtle can leave a user confused as to what is clickable and what is not, and what they should be clicking on next. Why are some web ads and social network profiles so garish and “ugly”? Because they serve not to please the eye, but to draw the eye, and they do that job well.
Keep these three simple meta rules in mind when you’re designing your user interactions.
Which user experience research tools a startup should use, and when November 7, 2008
Posted by jeremyliew in A:B testing, product management, UI, usability.6 comments
I recently posted about how usability testing can slow down launch but speed up success. But usability testing is just one of many elements of user experience research, with others including the ethnographic field studies made popular by Ideo, the A:B testing becoming standard for web 2.0, customer feedback, focus groups etc. With so many tools at your disposal, which user experience research tools should you use and when?
Jakob Nielsen recently posted about this topic, and concluded that it depends on what phase of product management you are in. For startups, my summary of his work is below:
Ideation: At the very beginning of the process you are looking at new ideas and opportunities. In this phase, aside form the founders vision, ethnographic field studies, focus groups, diary studies, surveys and data mining of webwide behavior can all be useful. Most startups will not have access to proprietary user data of existing products to identify additional opportunities.
Pre-launch: Once you’ve settled on a product idea and are working towards (beta) launch, you want to improve design and functionality as much as possible to minimize risk and maximize the likelihood for a successful launch. In this phase rely on tools such as cardsorting, paper prototype and usability studies, participatory design, desirability studies and field studies (including closed alpha launches to “friendlies”) to improve the user experience.
Post-launch: Once you’ve pushed the product out you will have live data that you can use to compare the product both to itself and to its competition. In this phase, usability benchmarking, online assessments, customer emails, surveys and A/B testing will be your primary tools
Nielsen provides some additional frameworks to differentiate when to use different forms of user experience research in his post. The site is a good resource about user experience in general.
Tips on A:B testing November 4, 2008
Posted by jeremyliew in A:B testing, product management.4 comments
If you’re doing A:B testing, you should read this paper and the accompanying presentation by Kohavi, Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. It is a comprehensive primer on how to do A:B testing well.
Some nuggets of goodness culled from the paper in no particular order (some sound obvious but read the paper to get more context). It will take you 20 minutes and you’ll use the lessons you learn for years:
1. Agree on evaluation criteria UP FRONT (vs after the fact analysis)
2. Ensure sample size is sufficiently large to have high confidence in results (with small samples, testing a version against itself can show wide ranges in performance)
3. Be truly random, and consistent when allocating users to groups
4. Ramp experiments from 1% of users to [50%] of users to get results fast [subject to day of week variations]. Auto abort if the new version is significantly worse.
5. Account for: robots skewing results, “newness” effect, time of day, day of week
6. Understand why as well as what (e.g. is lower performance caused by slower load time? incompatible browser types or screen sizes? etc. If fixable, fix and retest.)
7. Integrate constant testing into culture and systems
8. Test all the time.
Discovered via Eric Reis
How robust are communities? November 3, 2008
Posted by jeremyliew in communities, product management, usability, user generated content.4 comments
Wired has an article in its November issue about Urban Baby and You Be Mom. Urban Baby is an anonymous forum for Moms. Like 4chan, its anonymity makes for a mix of candid discussions, raw honesty and trolling, but with a mommy bent (think cheating, divorce and public schools). Says Wired:
Then in May, UrbanBaby, which was purchased by CNET in 2006, launched a redesign. All hell broke loose.
The changes weren’t huge, but each of them subtly altered the flow of conversation. CNET added a wide sidebar on the site to create space for ads. This reduced the reading area, a big problem on a board with hundreds of comments per hour. Discussions had been organized chronologically, but immediately after the relaunch, the default setting had “most popular” threads at the top, even if they had been started days earlier. Worse, you had to refresh your browser to see new posts. UrbanBaby users went nuts, demanding a return to the old design.
They soon got it. But not from UrbanBaby. A week after CNET rolled out the hated redesign, a couple of work-at-home computer programmers—longtime UrbanBaby users themselves—launched a rival site called YouBeMom.
They perfectly re-created the look and feel of the old boards. Better yet, they made improvements, including a souped-up search engine and privacy controls that make sure your spouse can’t use your computer to find out what you’ve been posting. They also set up a blog to capture users’ requests for site improvements and to outline what YouBeMom plans to do about them.
Within days, there was a mass exodus of users from UrbanBaby to the new site. CNET won’t give out traffic figures, and neither will the owners of YouBeMom. But I logged on to both sites recently and compared how often people posted. I’d estimate that YouBeMom has three times the traffic of UrbanBaby. That’s just how fragile a social application can be.
I found much higher comment volume and more vibrant conversations at YouBeMom as well when I looked at conversations on similar themes on both sites. The moral of the story according to Wired:
People have a very sophisticated sense for their online hangout—if you mess up the feel of it, or impede the ways they want to schmooze online, they’re gone.
What a terrific parable about the importance of community. What is strange though is that the traffic stats don’t appear to bear it out:
According to Compete, not only is Urban Baby far bigger that You Be Mom, but the redesign actually seems to have dramatically grown usage.
Sometimes communities are more robust than you think. Redesigns almost always create a lot of negative feedback when they first occur because all users hate change. You have to leave a little time to pass for users to get used to the changes before you can truly judge if the redesign has been a success or a failure.
There are three classes of user within social media, creators, curators and consumers. It may well be that many of the Urban Baby creators moved to YouBeMom, but the 90% of social media consumers, who read but don’t write, stayed at Urban Baby.
Do any readers have experiences of the impact of redesigns on a community?
Applying game design to building apps (and games!) October 28, 2008
Posted by jeremyliew in game design, game mechanics, product management, usability.2 comments
Dan Cook has posted the slides of his recent talk, “The Rescuing Princess Application“:
My talk was on building an application that rescued princesses. The goal was to give interaction designers some insight into how game design might be applied to the domain of more utilitarian applications.
The notes fields are heavily annotated with more details about each visual. For those of you who attended, this deck also includes a third section on game design patterns that I didn’t have time to cover in the time allotted.
This is a terrific presentation for product managers and designers, as well as for people building web games. Read it – it will take you about 10 minutes
Also, check out Dan’s list of recommended reading:
Dan’s blog
Chemistry of Game Design
What activities can be turned into games?
A Theory of Fun, Raph Koster
Shufflebrain: Putting the Fun in Functional
Dan particularly recommends the Shufflebrain presentation. The Shufflebrain team recently launched Photograb*, their first game of their own, that applies many of their principles of game design. Check it out on Facebook.
* Lightspeed provided the seed round of financing for Shufflebrain.
Usability testing slows down launch but speeds up success October 10, 2008
Posted by jeremyliew in A:B testing, product management, UI, usability.9 comments
I hate hearing the term “user error”. Good usability testing should eliminate most user error. I am a big proponent of A:B testing with live users. However, often a small usability test can quickly highlight any big problems before you go live so that you are working from a better starting point.
Many developers like to quickly prototype and push code out quickly, and I am a fan of this. However, if taken to an extreme, it can lead to products based on incorrect assumptions. Noah Kagan notes that at Mint:
We did surveys, user testing and psychological profiles. This was extremely useful in identifying the types of users we may have on the site and especially for seeing how people use the site. I never really did this before and was AMAZED how people use the site vs. what I expected
As Noah points out, usability testing can be easy and cheap. What it requires is simple:
1. Determine what you are trying to test. This is usually a list of the form, “How easy is it to complete task X?”
2. Recruit representative users. If you’re testing a new user experience, Craigslist is fine for this [Tip – if your core user is a middle america, . But make sure that your testers are truly representative. Your existing user base is another good place to find testers, but make sure that you’re not just listening to just your loudest users. The key is to pre-qualify the users to ensure that they are “average”.
3. Do the test. First ask them what their first impressions of the site are. What captures their attention? What would they do first? Ask the users to speak out loud during the session, explaining what they are thinking at all times. Then ask them to complete the tasks that you have listed. Watch and listen. Note what they find easy, what they find confusing, and what they don’t find at all.
This can be incredibly frustrating for you. You’ll think that some things are “obvious” that are not, or you’ll be shocked to see how unfamiliar users are with your site, or even with how browsers work. Remember that your role is to learn, not teach. Don’t touch the screen, the keyboard or the mouse; don’t point out how to do anything (even when they are “doing it wrong”, even if it is a “basic mistake”). You can provide encouragement and reassurance, or ask questions about why they did something, but that is it. You’ll be surprised at what you see.
The key is to internalize that there is no such thing as “user error” and there are no “stupid users”. If users are having problems achieving the tasks that you laid out for them, then the fault lies with your site. You’ll need to review your UI.
I prefer to do these usability tests over webex with users at their own computers. This makes the interaction as natural as possible for the testers. As an added bonus, you can then record both their screencast and the phonecall for later review.
Usability testing does not have to be a lot of work/ You only need to test five users to uncover most usability problems.
The most striking truth of the curve is that zero users give zero insights.
When you’ve completed your usability test, go back and makes some changes, but then come back and test again:
You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with 5 users has found 85% of the usability problems, you will want to fix these problems in a redesign.
After creating the new design, you need to test again. Even though I said that the redesign should “fix” the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn’t. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.
Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left – they will have to wait until the third test to be identified.)
Finally, the second test will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs. These important issues are often obscured in initial studies where the users are stumped by stupid surface-level usability problems that prevent them from really digging into the site.
This can all feel like overhead when all you want to do is launch. Trust me, it isn’t overhead. Getting this stuff closer to right the first time will only help you reach your goals faster.
Lightspeed Venture Partners Blog 