How to implement reporting and analytics for your startup July 30, 2008
Posted by jeremyliew in A:B testing, analytics, product management, start-up, startup, startups.2 comments
Andrew Chen has a good post on how a startup should think about implementing analytics that I think applies to companies of all sizes and is worth reading. He notes:
In general, a philosophy on the role of analytics within a startup is:
If you’re not going to do something about it, it may not be worth measuring.
(Similarly, if you want to act to improve something, you’ll want to measure it)
Don’t build metrics that aren’t going to be part of your day-to-day operations or don’t have potential to be incorporated as such. Building reports that no one looks at is just activity without accomplishment, and is a waste of time.
He goes on:
Metrics as a “product tax”
In fact, one way to view analytics is that they are a double-digit “tax” on your product development process because of a couple things:* It takes engineers lots of time and development effort
* It produces numbers that people argue about
* It requires machines, serious infrastructure, its own software, etc
* Fundamentally, it slows down your feature developmentAs a rough estimate, I’ve found that it takes between 25-40% of your resources to do analytics REALLY well. So for every 3 engineers working on product features, you’d want to put 1 just on analytics. This may seem like a ton (and it is), but it throws off indispensible knowledge that you can’t get elsewhere, like:
* Validating your assumptions
* Pinpointing bottlenecks and key problems
* Creating the ability to predict/model your business to make future decisions
* It tells you which features actually are good and what features don’t matter
I recommend reading the whole thing.
One additional piece of advice that I’ve found helpful:
1. Ask the product owners to use excel to mock up EXACTLY the reports that they would like to use, whether charts, tables, graphs, including time periods and mock data. This is way better than PRDs when it comes to reporting.
2. Go line by line through these reports with the product owners and ask them “what decision will you make with this data”. If the answer is “none” or if it is for investigation or theortical purposes rather than frequent operating decisions, cut the report out.
Happy analytics!
Is Social Media a business? July 29, 2008
Posted by jeremyliew in advertising, business models, social media, social networks.10 comments
I am a fan and subscriber to the paper version of Technology Review, but was disappointed in their cover story in the current edition, where Bryant Urstadt looks at the current state of the social network sector and concludes that social networking is not a business (free registration required). The article essentially looks at CPMs in the current business (which are low), concludes that revenues are low relative to traffic, and it might all just be a fad.
I have to admit to being biased about social media, but I think that the author’s lack of knowledge in this area (he typically writes for Rolling Stone and Harper’s) really shows. As examples of how poorly social networking sites are doing he proffers four pieces of evidence:
1. MySpace will fall $100m short of its revenue predictions this year. This means that it will only do $650m in revenue and only grow revenue by 100% according to Goldman Sachs.
2. Facebook will only do $50m in EBITDA this year.
3. Ning won’t tell him their revenue
4. CPMs for social media sites are lower than that of Technology Review
Maybe I’m a glass half full kind of guy, but I’d call the first two pieces of evidence pretty promising! The third is hardly surprising as very few private companies want their revenues to be publicly disclosed. And the fourth is a completely specious argument; I’m sure that the Technology Review’s website’s traffic is tiny and that its ads are bundled with that of the print publication, so any sort of comparison is meaningless.
That being said, MySpace and Facebook are far and away the two most successful social media sites at monetizing so far. It is fair to say that click through rates and CPMs are low relative to other forms of online media. The author thinks that targeting is the answer to raise CPMs. I think that is part of the answer, but I don’t think it is the whole answer. It is certainly the answer for social media apps like Flixster (a Lightspeed portfolio company) and Dogster, both of which offer a very targeted audience to endemic advertisers. In these cases, CPMs are not in the sub $1 range, but are comparable to other internet media sites with similarly targeted traffic, often in the single digit or low double digit range.
For the social games category of apps, likely the answer is free to play games with virtual goods models. This is the direction that the rest of the gaming industry appears to be moving towards, and social games are a subset of that trend.
For the vast majority of broad reach social media sites though, I think that the answer lies in a new ad standard for social media. The thing that differentiates social media sites from other forms of online media is not just user generated content, it is also that users are willing to affiliate themselves with brands. This takes many forms, from friending Scion on Myspace to putting a Natasha Bedingfield style on your Rockyou photo slideshow, to buying one of your Top Friends a Vitamin Water. These willing user affiliations/endorsements of brands are clearly valuable to marketers of those brands. Right now though, these deals are being negotiated on a one off basis; they look more like business development deals than selling ads off of a rate card. It will take a while for the social media industry to establish standards for selling this incredibly valuable inventory to brands, but I suspect that this will happen over the next 12-36 months.
There is an interesting parallel to search advertising here. In 2000, search inventory was monetized like every other form of online inventory, through banner ads. It wasn’t until Overture, and later Google, adopted the text ad-CPC standard that the distinctive thing about search inventory, user intent, was appropriately monetized. This created a new category of advertising that is now larger than banner advertising. Although some might disagree, I believe that a similar opportunity will eventually be unlocked by social media once the right ad unit standards emerge
In the interim though, targeting and scale go a long way. As Myspace has shown, $650m here, $650m there, and pretty soon, you’re talking about real money!
Why do kids and tweens buy virtual goods vs why do teens buy virtual goods? July 28, 2008
Posted by jeremyliew in game design, game mechanics, games, games 2.0, gaming, kids, teens, virtual goods, virtual worlds.3 comments
A recent post on KZero, asks, “Will luxury brands drive the growth of virtual goods?”:
… tweens and teens (KT&T) will play a big part in the growth. This group has ‘less of an issue’ paying real money for virtual goods – their decision-making process does not take into account its a virtual good – they just want the product and see technology as invisible.
So worlds targeting KT&T (kids, tweens, teens) clearly have a major opportunity to create strong revenues here, as long as the products are right. But which types of product are right?
Obviously the ones that are demanded are the right sorts of products. But what is this demand? Certainly in the younger worlds and to a certain degree in older worlds, some virtual goods are viewed as status symbols – a ‘badge’ that sends out a message that the owner (wearer) has something unique / purchased / earned, that others do not have. And in this context, there’s a lot of perceived value associated with the item.
Izzy Neis though thinks that the kids and tweens are motivated quite differently than teens:
Kids/Tweens have less “ownership” and responsibility and ABILITY than teens do. For tweens the status is more broad than teens – who care less about the meaning & depth of an item and more about what “luxury” means to them.
Tweens/Kids seem much happier to own the item itself, as well as show it off and play with it. They remind me more of Dragons & Treasure. As in many tales of lore (oh, how I love the folksy stuff), Dragons want want want, then horde it all. It’s as much of a self-congratulations in ownership as it is a play thing or show-n-tell.
Teens are more “look at me, look at me” (to quote Kat in “Ten Things I Hate About You”). The name of the item and the social-style-competition is much bigger a pay off than the actual day-to-day use or “play time”.
In the virtual worlds, from what I’ve seen, kids are just as psyched to EARN/PURCHASE as they are own – and just like kids playing with their clutter in real life, kids yearn to earn all sorts of silliness that moms & dads won’t buy them in the real world. Empowerment.
And in the teen worlds – it’s the display of the purchase and how it makes others around react that shows the “BIG” payoff.
This dichotomy seems like an important distinction to people building social media sites and virtual worlds.
Kids who are collecting virtual goods for the love of ownership (perhaps enjoying the control that they do not have to buy things in the real world) need less of a social experience to justify their purchases. THis is good news because (i) creating a social environment implies communication between members, which is more fraught with risk when it comes to kids and (ii) if kids enjoy “soloing” the demand for virtual goods can start earlier in the lifecycle of the company. But ownership for it’s own sake can get boring when you own enough stuff, so there may be a more well defined limit on how long any virtual world can hold a kids attention.
Teens (and adults), who more focused on how others react to their virtual goods than in the ownership of the goods themselves, require a social environment to justify and validate their purchases. Ownership is a performance, one of the three ways that social networks are different from other forms of online communication. This social environment takes longer to develop, and hence demand for virtual goods can be delayed versus a more “soloing” environment. But the flipside is that as the social environment changes and evolves, you can maintain an ongoing demand for virtual goods to stay current with the environment, just as the fashion industry does in the offline world.
I’d be interested to hear from readers if they agree with this dichotomy.
7x more interest in free to play games than subscription games July 25, 2008
Posted by jeremyliew in games, games 2.0, gaming, virtual goods.5 comments
Worlds In Motion summarizes key findings from the new Parks report on gaming.
Of the 2,000+ online gamers polled, only hardcore gamers were found to be interested in subscribing to an MMORPG service, whereas “social, dormant, and leisure gamers all show significant interest in a free-to-play, microtransaction-based model.”
The study’s numbers show that 14% of gamers currently not playing MMORPGs would be interested in playing them if they could for free, while only 2% of gamers from the same group were interested in playing an MMORPG requiring subscriptions.
According to Parks Associate’s director of broadband and gaming, Yuanzhe (Michael) Cai, the barriers to entry with subscription-based MMORPGs, such as time and money, are too high for potential customers. Free-to-play models, however, offer flexibility and enable players to choose how much they want to invest based on interest level and play patterns. Thus, the firm believes that microtransaction models have the best potential to grow the U.S. MMORPG audience.
I continue to be very bullish on the free-to-play/digital goods model for gaming companies over the next few years.
Online video CPMs are north of $15 July 24, 2008
Posted by jeremyliew in advertising, user generated content, video.4 comments
TV Week reports on a new report from the Diffusion group on video advertising:
Professional Web programming yields very high CPMs, the report found. The CPMs for long-form online content are $40 today and will reach nearly $46 in 2013. Meanwhile, CPMs for short clips are clocking in at about $30 and will rise to a little over $34 in five years.
The CPMs for user-generated video will have the smallest rise, from only $15 today to about $17 in 2013.
While many have viewed this negatively because User gen video ad rates are lower than professional programing, I think that $15 CPMs with no content creation costs sound pretty good to me!
How Kosmix employs enterprise 2.0; a guide for other startups July 22, 2008
Posted by jeremyliew in blogs, communication, email, enterprise 2.0, enterprise infrastructure, IM, management, start-up, startup, startups, wiki.1 comment so far
Anand Rajaraman, co-founder of Lightspeed portfolio company Kosmix, posts about how to stop email overload and break silos using wikis, blogs, and IM.
We hit the email wall at my company Kosmix recently. When we were less than 30 people, managing by email worked reasonably well. The team was small enough that everyone knew what everyone else was doing. Frequent hallway conversations reinforced relationships. However, once we crossed the 30-person mark, we noticed problems creeping in. We started hearing complaints of email overload and too many meetings. And despite the email overload and too many meetings, people still felt that there was a communication problem and a lack of visibility across teams and projects. We were straining the limits of email as the sole communications mechanism.
We knew something had to be done. But what? Sri Subramaniam, our head of engineering, proposed a bold restructuring of our internal communications. He led an effort that resulted in us relying less on email and more on wikis, blogs, and instant messaging. Here’s how we use these technologies everyday in running our business.
* Blogs for Status Reports
* The Wiki for Persistent Information
* Instant Messaging for Spontaneous DiscussionsThe effects of the communication restructuring have been immediate and very visible. They include a lot less email and almost none on weekends; better communication among people; and 360 degree visibility for every member of the Kosmix team. After we instituted these changes, everyone on the team feels more productive, more knowledgeable about the company, has more spare time to spend on things outside of work.
Anand goes into detail as to how blogs, wikis and IM are used by all employees, and how this has streamlined the communications in the company. I highly recommend reading the whole thing.
How many A:B tests do I have to run before it is meaningful? July 21, 2008
Posted by jeremyliew in A:B testing, product management.2 comments
Inside Facebook has a good post about how to not screw up your A:B testing that is a useful reminder about how many tests you need to run before you know that the results are statistically significant.
The author notes:
How many [tests] do we need to declare a statistically significant difference between [a design leading to action success rate] of p1 and one of p2? This is readily calculable:
* number of samples required per cell = 2.7 * (p1*(1-p1) + p2*(1-p2))/(p1-p2)^2
…
(By the way, the pre-factor of 2.7 has a one-sided confidence level of 95% and power of 50% baked into it. These have to do with the risk of choosing to switch when you shouldn’t and not switching when you should. We’re not running drug trials here so these two choices are fine for our purposes. The above calculation will determine the minimum and also the maximum you need to run.)
Thus, if you did this number of tests and found that the difference in action success was greater than (p1-p2), then you would have a 95% confidence level that the design being tested is responsible for the increase in success rate, and you would move to a new best practice.
The author reminds developers to adhere to A:B testing best practices, including:
# Running the two cells concurrently
# Randomly assigning an individual user to a cell and make sure they stay in that cell during the test
# Scheduling the test to neutralize time-of-day and day-of-week effects.
# Serving users from countries that are of interest.
One thing that immediately emerges from this formula is that you don’t need that many tests to determine if a new design is working. For example, testing a design that anticipates increasing success from .5% to .575% only needs about 52k tests. For apps and websites that are at scale, this does not take very long.
The danger is that, because of the overhead of putting up and taking down tests, “bad” test designs stay up for too long, exposing too many users to a worse experience than usual. While some people consider A:B testing to be splitting users into equal groups, there is no such requirement. I’d advise developers to size their test cells to be x% of their total traffic, where x% is a little more required to hit the minimums calculated above over a week. This neutralizes time of day and day of week effects, minimizes the overhead of test set up, and ensures that not too many users are exposed to bad designs. It also allows multiple, independent tests to be run simultaneously.
Post mortems on two failed startups from their founders July 19, 2008
Posted by jeremyliew in Entrepreneur, failure, management, start-up, startup, startups.12 comments
Monitor110 recently shut down after raising $20m over three rounds. One of the co-founders wrote a portmortem of Monitor110, highlighting 7 mistakes that the company made:
1. The lack of a single, “the buck stops here” leader until too late in the game
2. No separation between the technology organization and the product organization
3. Too much PR, too early
4. Too much money
5. Not close enough to the customer
6. Slow to adapt to market reality
7. Disagreement on strategy both within the Company and with the Board
(found via Brad Feld)
At the other end of the spectrum is a post mortem of a bootstrapped two person startup that shut down last month after building for 1.5 years but not raising any venture capital. This founder’s lessons learned are more tactical, but no less important:
1. If your idea starts with “We’re building a platform to…” and you don’t have a billion dollars in capital, find a new idea. Now.
2. It’s a marathon, but it’s a marathon made of sprints
3. Initial conditions matter. A lot.
4. Developing in a vacuum never works.
5. Beware the chicken and the egg.
6. Prototype any 3rd-party libraries that you’ll be depending upon, before you base your product on them.
7. If you’re doing anything other than building your project and getting users, it’s premature.
8. The product will take longer than you expect. Design for the long-term.
9. People have an incentive not to crush your dreams. Take everything they say with a grain of salt.
10. Know your limitations.
(found via Brian Green)
I would recommend entrepreneurs to read both posts.
“Those who cannot remember the past, are condemned to repeat it.” — George Santayana
People like to have fun. Who would have thunk it? July 16, 2008
Posted by jeremyliew in apps, facebook, iphone.1 comment so far
There has been much handwringing about how silly facebook apps are, and how it would be so much better if they were more useful. But Facebook users have voted with their mouse buttons, as the O’Reilly report in May showed:
According to a Medialets survey, it seems that iPhone users have voted in exactly the same way, with almost half iPhone apps being games or entertainment:
Girls (and boys) just want to have fun.
Runescape stats: 250k peak concurrent users! July 15, 2008
Posted by jeremyliew in games, mmorpg, Runescape.1 comment so far
Today GigaOm has an interview with the CEO of Jagex, publisher of Runescape, a free to play MMOG with more players in the west than World of Warcraft. Some stats to give you an idea of their scale:
– New content (questions, items, etc.) added to RuneScape every two weeks
– Peak concurrency: 250,000
– Average player time: 12.5 hours/week
– 250 RuneScape shards for up to 2,000 players each. Unlike many MMORPGs, player characters are not bound to a single shard.
– Main player demographics: 60 percent are from the U.S., 25 percent from the EU, smaller percentages from Australia/New Zealand and Canada. Player age typically 8-20, approximately 80 percent between 10-16.
Very impressive.
Lightspeed Venture Partners Blog 