Search improvements are more about understanding queries better, not understanding results better October 2, 2007
Posted by jeremyliew in Search, yahoo.trackback
Yahoo’s new search product has just launched to positive reviews. I think it’s pretty great as well.
The Yahoo search blog outlines most of the new changes, but they can be summarized in one phrase; the new Yahoo search understands queries better. This gets instantiated in a two different ways:
1. Yahoo recognizes certain classes of queries and brings relevant vertical search results (e.g. video, photos, news, local, music etc) back into the body of the search results. Both Ask and to a lesser extent Google (with its Onebox) also do this. e.g. try searching for evolution of dance and get the video at the top, or Justin Timberlake and get a capsule of information on the singer at the top.
2. Yahoo aggressively helps users to refine their queries. Although average query length has been creeping upwards, search is still an iterative process for many users. People will type a search string, review the results, realize that it isn’t what they wanted, and improve the query string. Yahoo doesn’t just correct typos and suggest longer search terms that start with the same words, but it also recognizes concepts that are similar. (Ask does something similar in the left rail). E.g. try searching for King Henry VIII and get a suggestion for Catherine of Aragon. It also watches for hesitation as users type and auto-magically makes suggestions when they are needed.
As John McKinley (the ex CTO of AOL) pointed out a few weeks ago, AOL Search had this vision nailed over a year ago, but when new management came in they instituted a much sparser, more “Google-like” model instead. Henry Blodget thinks that this change tanked AOL’s search revenues in Q2, causing it to miss the quarter.
Yahoo has been rolling some of these search improvements out incrementally, and as a result, Compete found that they have both higher search fulfillment rates than Google. More of Yahoo’s search queries turn into clicks than Google’s, suggesting that users are finding what they want more often.
These improvements involve some deep technical problem solving. Finding a way to identify when to show results from a vertical search engine, and understanding related search concepts at scale are both difficult problems. You can cheat and do this with a big list, but that rapidly runs out of steam given the long tail of search queries. This isn’t just a lick of UI paint over the same old search engine.
The question will be whether this is enough. Yahoo has put a lot of time and effort into product innovation. I believe that there are three phases of competition in a consumer technology market; first distribution matters most, then product, and finally brand. Search may have already passed into the “brand” phase of competition. If people believe that Google has the best search, then they may not even try Yahoo’s search to be proved wrong. Yahoo will need to do more than ship great product (as it has done with this product), it needs to also find a way to drive trial from users who have Google as their default.
Lightspeed Venture Partners Blog 
“Fulfillment” may be somewhat of a misleading term to use here. A Google search results page which gives me the info I’m looking for on the page itself, is not going to get a click-thru to the linked site. Short cuts, snippets, Universal search etc. Another case where more click-thrus is not good is when you need to click through to more sites per search results page on Yahoo to finally get through to the page which contains the info you’re looking for. Unless Compete is measuring the rate at which the Back button is used to go back to Yahoo from a linked-to page, I’d say this is a possibility. Search quality *is* a rather hard quantity to measure and quantify in one nicely packaged number. Compete is as likely to be measuring Frustration as Fulfillment, using the methodology described..
Excellent point about understanding queries!
I remember reading in John Battelle’s book _The Search_ that the real challenge in search is determining *intent*. If word got out about a search engine delivering highly improved accuracy in presenting what you’re really after, its timesaving and targeting potential would be extremely attractive and undeniable. (That goes for both regular and sponsored search.)
One thing about the Internet is user loyalty is diffcult to maintain. Online loyalty is very fickle, and if people discover something easier and more reliable that’s just a URL or click away, they will try it and adopt it.
We recently posted an analysis on this. Search improvements are about maximizing the Average Relevance / Recall Effort ratio. Yahoo did a couple of things right in this release:
1. It improved Average Relevance by understanding queries better.
2. It decreased Recall Effort by automatically suggesting related queries.
Lot more to be done, but certainly a great start.
Fulfillment is a great metric to judge a search engine on!
It seems to me that when search engines really seek to understand your query and refer you to content, they become more of a recommender system than a search engine.