Google Starts Ranking Twitter Search Results Pages

by Patrick Altoft on March 23, 2009

Over the last couple of weeks it appears that more and more Twitter search results pages have started ranking on Google for “news” type queries.

This is interesting because Twitter has made no efforts to ensure the search pages are SEO friendly and even goes as far as blocking Google from spidering the pages using robots.txt.

User-Agent: *
Disallow: /search
Disallow: /*?

Google has obviously decided, algorithmically or manually, that even thought they don’t know what content is on these Twitter pages they are probably high quality enough to warrant high rankings.

The screenshot below is from the “gaza” search result. You can see that Google isn’t spidering the page and therefore can’t generate much of a description or use the pages correct title.

The actual pages are likely being discovered in two different ways. The first is via the usual link discovery method where Google spots lots of links to a page and ranks it based on link data.

The second method Google might be using is to generate the Twitter results pages themselves. If there is a particular keyword that Google wants more results for they can just plug that keyword into the Twitter search page and generate a brand new page to suit.

We know Google is filling in forms on thousands of websites every day to try and index the deep web so it comes as no surprise that they are doing the same with Twitter.

The only surprising thing is that Twitter is actually working against Google rather than embracing SEO and using it to build marketshare and perhaps even revenue.

Patrick Altoft is Director of Search at Leeds based digital & SEO agency Branded3. Patrick also runs Blogstorm.

You can get our blog posts delivered for free by email every day - simply add your email address to the box below or alternatively grab the RSS feed.

Read some similar posts

Post category: Google Share it :

{ 18 comments… read them below or add one }

Peter Young 23 Mar 2009 at 9:57 am: Only a matter of time I guess, however the more worrying thing for me from your post, is Google’s disregard for parameters defined within the robots.txt.

If someone has gone to the effort of defining content to be disallowed, surely that is the decision of the webmaster and not of Google itself!!!
andymurd 23 Mar 2009 at 10:35 am: Google (via Matt Cutts) do say that they don’t want to index other engines’ search results, so I think Twitter are doing the right thing with their robots.txt.

That said, hashtags are a different paradigm – they do represent added value. Twitterers adding hashtags to report the news in realtime is a useful resource that should be in the index.

I hope this is a manual decision, not some glitch of the algorithm because it would mean that Google is taking Twitter seriously and might get to addressing the issue of URL shorteners not passing juice.
Patrick Altoft 23 Mar 2009 at 10:40 am
Find me on Twitter: I think that Google has changed stance on whether they want search results indexed. The fact they are filling in forms and producing the results shows that they actually want to index the pages and make their own decision on whether they let them rank or not.

Personally I think a Twitter search results page is a good addition to the results for breaking queries, perhaps it might end up getting a onebox result.
More comments from Patrick Altoft
RogerW 23 Mar 2009 at 12:55 pm: More than 60,000 pages listed at the moment …

http://www.google.com/search?q=site:search.twitter.com&filter=0
AntÃ³nio 23 Mar 2009 at 4:28 pm: Amazing, didn~t fully realize that twitter search ranks on 1st page for gaza.

http://www.google.com/search?hl=en&q=site%3Asearch.twitter.com%2Fsearch&btnG=Search returns 1 result with repeated results omitted.
Tweetplate 23 Mar 2009 at 5:58 pm: I think the use of hashtags denotes content that the user wants to be found/shared/et cetera, so that’s fine.
Patrick Altoft 23 Mar 2009 at 6:19 pm
Find me on Twitter: What user are you talking about?
More comments from Patrick Altoft
Barbara Ling, Virtual Coach 23 Mar 2009 at 6:33 pm: Makes superb sense on Google’s part. The mind reels at what SEO folks could do to utilize this information….
Jim Collins 24 Mar 2009 at 12:04 am: Actually Google should take a note of a company called Boilingpage — http://www.boilingpage.com which tracks the hottest pages on the web based on recent tweets in twitter. And I find the results to be amazing! They do have search box in which I can search for a topic and it brings all interesting pages related to the keyword. Google should try something like this!
Twat 24 Mar 2009 at 2:09 am: Twitter and revenue will never mix
carlos del rio 24 Mar 2009 at 11:24 pm: Why would Twitter block only part of the sub-domain? I suspect there is something hidden in the sub-domain that answers some of the question.
Dan 25 Mar 2009 at 10:26 am: As Danny in the ReadWriteWeb comments points out this is not Twitter-specific.

It’s merely because the page itself carries authority, and the term [#gaza] exists in the URI, on page, and in links to the page. Try a Google search for [gaza] and you’ll get much the same results!

That said, the way Twitter is architected means that more and more hashtags will rise up the SERPs. Not because they’re subverting Google’s algorithms or being treated differently, but because of the other, long established SEO factors in play on a popular page.

Dan
Yellow SEO 25 Mar 2009 at 7:06 pm: Do a search for “#gaza” “%23gaza ”
out of the 293 exact phrase matches
Alot of them contain the link where it is follow or no follow the “search.twitter.com” text is there.
http://search.twitter.com/search?q=%23gaza
http://tinyurl.com/7cn6zu <– some have this instead

It makes prefect sense mathematically from a pure link reference standpoint.

do a search for gaza +”www.twitter.com”. you get 247,000
gaza +”www.cia.gov” 2,420,000 for gaza +”www.cia.gov”
gaza -”www.twitter.com” 110,000,000 for gaza -”www.twitter.com”
gaza “www.freegaza.org” 24,100 for gaza “www.freegaza.org”.
rates 26 Mar 2009 at 3:02 am: hehe, the twitter storm continues
Joseph Manna, Infusionsoft 26 Mar 2009 at 4:27 pm: The problem with indexing a search index is that it may be unreliable and controlled by end users. In theory, anyone can control the results pages on Twitter by merely tweeting about it and tossing in their links.

Personally, I don’t think Twitter Search should be in the Google Search. It’s so dynamic and can be easily manipulated.

Instead, I would love if Google made their own integrated search for Twitter. Similar to how they have News, AP News, Books, etc. The idea is that it gets it’s own attention that these are “Live” or similar type of results.

I suspect these pages are being linked from relevant sources, thus the presence of them on SERPs. For instance, for our user conference, we linked to our hashtag #infusioncon on our blog and our Website and thus, the Twitter Search page became relevant.

~Joseph
@JoeManna
oyy 26 Mar 2009 at 10:16 pm: Finally a way to observe the growth of twitter? http://www.google.com/search?hl=en&q=site%3Atwitter.com
Counting 52M pages at the moment.
www.twitter.com/PrayMaddyMcCann 02 Feb 2010 at 12:57 am: Thanks Alot!
Wow sounds fantastic to me!

Although I don’t have much to “advertise” I am raising awareness for the Madeleine McCann campaign! I now have over 10,700 followers so things can only get better!

Feel free to follow me! I’ll follow you back in just a few hours,
http://www.twitter.com/PrayMaddyMcCann

{ 21 tweetbacks }

19 palpitt (Palpitt) 02/09/2010 at 3:58 pm: Pas de concurrence entre Twitter search et Google : le second va tout simplement avaler le premier http://tinyurl.com/d9edps
20 LVRealEstate (John Novak) 02/09/2010 at 3:58 pm: From Patrick Altoft at Blogstorm.com: “Google Starts Ranking Twitter Search Results Pages” http://tinyurl.com/d9edps
21 adnene (Adnene) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages : http://tinyurl.com/d9edps
22 mparent77772 (Marc Parent) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages

http://tinyurl.com/d9edps
23 Samiam500 (samiam500) 02/09/2010 at 3:58 pm: Google showing Twitter pages in search results http://tinyurl.com/d9edps
24 catchmedia (Derrick Stembridge) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
25 oliolaya (olaya de marcos) 02/09/2010 at 3:58 pm: Uppss http://tinyurl.com/d9edps
26 BelfastCity (BelfastCity) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
27 heatherjackson (heatherjackson) 02/09/2010 at 3:58 pm: i have a chance yet for heatherjackson to rank #1 – Google started ranking twitter pages for ?news? type queries: http://tinyurl.com/d9edps
28 cchcreative (Celso Hora) 02/09/2010 at 3:58 pm: Man, this is huge!! Google starts to ranking twitter search results page!!! D
29 askpavel (Pavel Israelsky) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
30 javier_lorente (Javier Lorente) 02/09/2010 at 3:58 pm: Google empieza a devolver p
31 pimpmywordpress (PimpMyWordpress) 02/09/2010 at 3:58 pm: Reading: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
32 dave_woods (Dave Woods) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
33 TraffiCO (TraffiCo) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps Please RT
34 rj3sp (rj3sp) 02/09/2010 at 3:58 pm: Reading: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps
35 NoExpAffiliates (NoExpAffiliates) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages http://tinyurl.com/d9edps Please RT
36 patrickaltoft (Patrick Altoft) 02/09/2010 at 3:58 pm: Google Starts Ranking Twitter Search Results Pages: Over the last couple of weeks it appears that more and more .. http://tinyurl.com/d9edps
37 kieronmayers (Kieron Mayers) 02/09/2010 at 3:58 pm: twitter posts being ranked by Google – http://tinyurl.com/d9edps
38 eni_kao ([Enikao]) 02/09/2010 at 3:58 pm: [Nutek] Google inclut les twitts dans ses recherches. Vu par hasard avec une google alert. http://is.gd/oBIj (mais via @palpitt)
39 marcosnobre (Marcos Nobre) 02/09/2010 at 3:58 pm: Google indexing Twitter pages despite robots.txt denial: http://is.gd/oBIj