Google Starts Ranking Twitter Search Results Pages
Over the last couple of weeks it appears that more and more Twitter search results pages have started ranking on Google for “news” type queries.
This is interesting because Twitter has made no efforts to ensure the search pages are SEO friendly and even goes as far as blocking Google from spidering the pages using robots.txt.
User-Agent: *
Disallow: /search
Disallow: /*?
Google has obviously decided, algorithmically or manually, that even thought they don’t know what content is on these Twitter pages they are probably high quality enough to warrant high rankings.
The screenshot below is from the “gaza” search result. You can see that Google isn’t spidering the page and therefore can’t generate much of a description or use the pages correct title.
The actual pages are likely being discovered in two different ways. The first is via the usual link discovery method where Google spots lots of links to a page and ranks it based on link data.
The second method Google might be using is to generate the Twitter results pages themselves. If there is a particular keyword that Google wants more results for they can just plug that keyword into the Twitter search page and generate a brand new page to suit.
We know Google is filling in forms on thousands of websites every day to try and index the deep web so it comes as no surprise that they are doing the same with Twitter.
The only surprising thing is that Twitter is actually working against Google rather than embracing SEO and using it to build marketshare and perhaps even revenue.
Patrick Altoft is Director of Search at Branded3, a Leeds SEO & Digital Agency specialising in SEO, Web Design, Development & Social Media.
Comments
Read the 19 comments below, or add your own!
Only a matter of time I guess, however the more worrying thing for me from your post, is Google’s disregard for parameters defined within the robots.txt.
If someone has gone to the effort of defining content to be disallowed, surely that is the decision of the webmaster and not of Google itself!!!
I think that Google has changed stance on whether they want search results indexed. The fact they are filling in forms and producing the results shows that they actually want to index the pages and make their own decision on whether they let them rank or not.
Personally I think a Twitter search results page is a good addition to the results for breaking queries, perhaps it might end up getting a onebox result.
Google (via Matt Cutts) do say that they don’t want to index other engines’ search results, so I think Twitter are doing the right thing with their robots.txt.
That said, hashtags are a different paradigm – they do represent added value. Twitterers adding hashtags to report the news in realtime is a useful resource that should be in the index.
I hope this is a manual decision, not some glitch of the algorithm because it would mean that Google is taking Twitter seriously and might get to addressing the issue of URL shorteners not passing juice.
More than 60,000 pages listed at the moment …
http://www.google.com/search?q=site:search.twitter.com&filter=0
Amazing, didn~t fully realize that twitter search ranks on 1st page for gaza.
http://www.google.com/search?hl=en&q=site%3Asearch.twitter.com%2Fsearch&btnG=Search returns 1 result with repeated results omitted.
I think the use of hashtags denotes content that the user wants to be found/shared/et cetera, so that’s fine.
What user are you talking about?
Makes superb sense on Google’s part. The mind reels at what SEO folks could do to utilize this information….
Actually Google should take a note of a company called Boilingpage — http://www.boilingpage.com which tracks the hottest pages on the web based on recent tweets in twitter. And I find the results to be amazing! They do have search box in which I can search for a topic and it brings all interesting pages related to the keyword. Google should try something like this!
Twitter and revenue will never mix
Why would Twitter block only part of the sub-domain? I suspect there is something hidden in the sub-domain that answers some of the question.
As Danny in the ReadWriteWeb comments points out this is not Twitter-specific.
It’s merely because the page itself carries authority, and the term [#gaza] exists in the URI, on page, and in links to the page. Try a Google search for [gaza] and you’ll get much the same results!
That said, the way Twitter is architected means that more and more hashtags will rise up the SERPs. Not because they’re subverting Google’s algorithms or being treated differently, but because of the other, long established SEO factors in play on a popular page.
Dan
Do a search for “#gaza” “%23gaza ”
out of the 293 exact phrase matches
Alot of them contain the link where it is follow or no follow the “search.twitter.com” text is there.
http://search.twitter.com/search?q=%23gaza
http://tinyurl.com/7cn6zu <– some have this instead
It makes prefect sense mathematically from a pure link reference standpoint.
do a search for gaza +”www.twitter.com”. you get 247,000
gaza +”www.cia.gov” 2,420,000 for gaza +”www.cia.gov”
gaza -”www.twitter.com” 110,000,000 for gaza -”www.twitter.com”
gaza “www.freegaza.org” 24,100 for gaza “www.freegaza.org”.
hehe, the twitter storm continues
The problem with indexing a search index is that it may be unreliable and controlled by end users. In theory, anyone can control the results pages on Twitter by merely tweeting about it and tossing in their links.
Personally, I don’t think Twitter Search should be in the Google Search. It’s so dynamic and can be easily manipulated.
Instead, I would love if Google made their own integrated search for Twitter. Similar to how they have News, AP News, Books, etc. The idea is that it gets it’s own attention that these are “Live” or similar type of results.
I suspect these pages are being linked from relevant sources, thus the presence of them on SERPs. For instance, for our user conference, we linked to our hashtag #infusioncon on our blog and our Website and thus, the Twitter Search page became relevant.
~Joseph
@JoeManna
Finally a way to observe the growth of twitter? http://www.google.com/search?hl=en&q=site%3Atwitter.com
Counting 52M pages at the moment.
Google Starts Ranking Twitter Search Results Pages
1
Thanks Alot!
Wow sounds fantastic to me!
Although I don’t have much to “advertise” I am raising awareness for the Madeleine McCann campaign! I now have over 10,700 followers so things can only get better!
Feel free to follow me! I’ll follow you back in just a few hours,
http://www.twitter.com/PrayMaddyMcCann
I amazed so many people would want to search on an old washed up Tyneside footballer… (!)
10 trackbacks
March 23, 2009 at 3:31pm
March 23, 2009 at 5:30pm
March 23, 2009 at 8:06pm
March 23, 2009 at 10:33pm
March 25, 2009 at 10:26pm
March 27, 2009 at 7:53pm
April 2, 2009 at 5:33pm
May 12, 2009 at 6:05pm
June 9, 2009 at 9:06pm
November 9, 2009 at 11:21pm