Stealth Googlebot or Google Operating System?
Website monitoring company Net Applications has discovered that around 30% of the traffic originating from Google HQ has identifying information stripped from the user agent string.
This traffic, which people are assuming is Google employees, has the operating system identifier stripped from the user agent string.
Speculation is that Google is testing a Google OS amongst employees:
The new data showed that a percentage of Internet users in Google’s offices (principally based in the company’s Mountain View headquarters) are using an operating system that essentially shields itself from detection by stripping traffic of identifying information. Vizzacarro describes this data, known as a user agent, as a string of information that a computer uses to identify itself. Removing it (possibly via a proxy server) means that outsiders like Net Applications can’t tell which operating system a particular Web user is using. (Net Applications uses other methods, like a Web site’s JavaScript to detect other information about a user and determine that the traffic is coming from Google.) About 11% of Google’s Web traffic currently shows up like this. The level fluctuates daily, Vizzacarro says. A few days ago it was around 30%.
Whilst this might be true there are other explanations for the data. My first thought was that Google was using a stealth spider pretending to be a normal user to build up a more accurate picture of the web. Googlebot can’t see images, scripts, Flash and accept cookies. A stealth Googlebot could do all of these things and help build a better index in the future.
Patrick Altoft is Director of Search at Branded3, a Leeds SEO & Digital Agency specialising in SEO, Web Design, Development & Social Media.
Comments
Read the 12 comments below, or add your own!
They could be trying to use it to detect cloaked sites?
Do these “users” pull down the robots.txt?
There are a few huge UK sites that use strange cloaking methods to help them with their SEO rankings.
I don’t think that they would risk investing $ into G’OS.
1. If it’s free, there is less $ in free OS, not G thing.
2. If it’s paid OS, no one going to buy it.
3. Creating OS is not the same as creating browser, it takes hundreds/thousands of people and many years.
Not really sure. Interesting thoughts though.
I don’t think the traffic we’re seeing is a bot. The referer, at least for the last few, has actually been a Google search results page where they search for the last token in the URI (e.g. my-wordpress-post-title) and use the “site” operator to restrict it to the site they were looking for the post on (e.g. site:mywordpressblog.com). They’ve done that exact same thing at least 3 different times on different days, viewed that one post, and then were done.
I suspect it’s something to do with new Googlebots. I’ve seen some unwelcome and disruptive Googlebot-Image behaviour recently.
A Google OS would be great. Since Windows 7 isn’t looking as it’s going to perform much better than Vista another player would be more than welcome.
Hi,
In my point of view if Google bot see image,script,flash and accept cookies is to be better for Google now Google algorithm is can’t see these thing . I feel better to change the algorithm .
1. GOOGLE OS? I did hear a while back that Google was looking at developing a custom version of Ubuntu, possibly for in-house use.
2. GOOGLE OS NOT WORTH $$$: I have to disagree with Vygantas, if Google makes an OS and offers it for free they can indeed find a way to make money from it – some ideas: a) charge to upgrade for more features, b) increase their customer base especially if it comes pre-installed with Google Apps, c) could be **carefully** ad-supported in one way or another
3. DETECT CLOAKED SITES: Sounds like a reasonable assumption, and one that Google and their users would benefit from.
Just reading what’s here I’d place my bet on detecting cloaked sites since its such a necessary, and belated, move. Google, and other search engines, really should have a way to detect cloaked pages/sites.
I think they should make it an opensource OS. They are well-known and they could bring all experts like me into the realm of making an OS. Today, if you wanted a good OS. You need years of study and thousands of experts.
Charles, I think if Google is developing a Ubuntu in-house for internal use, that a natural extension of that in the future could include some type of public release.
Its something to do with new Googlebots. I’ve seen some unwelcome and disruptive Googlebot-Image behaviour recently.