Monday, November 28, 2005
Best Search Engine Study
We know how spammy Google can be for some searches, but according to a blind study they do provide the best results. Not very surprising; the others are full of spam. What is surprising is that their lead isn't very big.
The study is being limiting to only the top three results from the main three engines (Google, Yahoo, MSN), but you can't really list them all and expect people to carefully evaluate the results of each engine. I presume that for most general (non-sex/drug) keywords you aren't going to run into much spam in the top three results so I guess they are mostly measuring relevancy rather than least spamminess, but they do go together.
There certainly are some flaws in the study. There is no way to vote a tie. And as some people suggest in the comments, by limiting to only the big three engines they are missing out on possibly better results (yeah right, there is a reason they are the big three). But when you run your own version you can set it up better. ;-)
You can participate in the ongoing study by choosing the best results for a search term you provide. You only get one official vote (monthly?) but can try it as many times as you want.
In my first run at the blind test I chose the Google results, but all three were pretty good and two only differed by one link. This was for a normal keyword that is not highly competitive.
My second try was for tramdol. Guess what I got. That is right, they all sucked. It was hard to decide, but after looking at each page I decided on the slightly less spammy of the three. In case you are wondering how non spammy the results were, here are the winners:
New to tramdol
Buy Tramdol 20% OFF - Drugs.MD - FREE Consulta ...
Tramadol
And finally I gave lolita a try. I really got far better results than I expected. No porn or spamdexing pages in the top three results from any engine. I chose this one on informational quality of results assuming I wanted information on the book or movie.
Two were pretty good, giving both movie and book information, but one of those listed three IMDB and Amazon pages. How they got three results for two different pages doesn't look good to me. The third included a young girl picture site; they were clothed and from what I saw not in revealing clothes, but still. It also returned a site for a bar named Lolita, which was not what I was looking for.
This time the winner was Yahoo, only because of the duplicate IMDB page and commercial bias of Google's top results. By process of elimination I am sure you have already guessed who returned the more questionable results, MSN.
I kind of figured when I saw it, but a manual Google search for lolita confirmed it. Due to Google's stacking of related pages from the same site (which in this case were identical with slightly different URLs) we got the odd looking results. I really don't know how you are supposed to handle that case in a study any better though. In a study I worked on a couple years ago we did the same thing, we determined that you just have to include everything otherwise it is not comparable to the other engines. They included it as one of the ten results on the page so we looked at it as a separate result.
The study is being limiting to only the top three results from the main three engines (Google, Yahoo, MSN), but you can't really list them all and expect people to carefully evaluate the results of each engine. I presume that for most general (non-sex/drug) keywords you aren't going to run into much spam in the top three results so I guess they are mostly measuring relevancy rather than least spamminess, but they do go together.
There certainly are some flaws in the study. There is no way to vote a tie. And as some people suggest in the comments, by limiting to only the big three engines they are missing out on possibly better results (yeah right, there is a reason they are the big three). But when you run your own version you can set it up better. ;-)
You can participate in the ongoing study by choosing the best results for a search term you provide. You only get one official vote (monthly?) but can try it as many times as you want.
In my first run at the blind test I chose the Google results, but all three were pretty good and two only differed by one link. This was for a normal keyword that is not highly competitive.
My second try was for tramdol. Guess what I got. That is right, they all sucked. It was hard to decide, but after looking at each page I decided on the slightly less spammy of the three. In case you are wondering how non spammy the results were, here are the winners:
New to tramdol
Buy Tramdol 20% OFF - Drugs.MD - FREE Consulta ...
Tramadol
And finally I gave lolita a try. I really got far better results than I expected. No porn or spamdexing pages in the top three results from any engine. I chose this one on informational quality of results assuming I wanted information on the book or movie.
Two were pretty good, giving both movie and book information, but one of those listed three IMDB and Amazon pages. How they got three results for two different pages doesn't look good to me. The third included a young girl picture site; they were clothed and from what I saw not in revealing clothes, but still. It also returned a site for a bar named Lolita, which was not what I was looking for.
This time the winner was Yahoo, only because of the duplicate IMDB page and commercial bias of Google's top results. By process of elimination I am sure you have already guessed who returned the more questionable results, MSN.
I kind of figured when I saw it, but a manual Google search for lolita confirmed it. Due to Google's stacking of related pages from the same site (which in this case were identical with slightly different URLs) we got the odd looking results. I really don't know how you are supposed to handle that case in a study any better though. In a study I worked on a couple years ago we did the same thing, we determined that you just have to include everything otherwise it is not comparable to the other engines. They included it as one of the ten results on the page so we looked at it as a separate result.
