Wordpress Crawl Rate Tracker Plugin

by Patrick Altoft on February 25, 2008

With the advent of Google Analytics and other JavaScript based solutions a lot of webmasters have lost the ability to easily analyse log files to see how search engines are interacting with their site.

Google Webmaster Central offers a nice graphing feature so you can see the crawl trends but there is no way to access meaningful data. Studying crawl rate is hugely important because it helps you find out which pages are trusted most by the search engines (especially Google). The more Google trusts a page the greater the frequency it will re-visit that page. If you build a few deep links and see your crawl rate for the target page suddenly increase you can be confident that the links are highly trusted and that you should try to get some more.

Today we are releasing a plugin for Wordpress that allows you to track crawl activity for MSN, Yahoo & Google from your dashboard.

Access crawl data for 3 main search engines
See the exact times a spider accessed any page on your site
Monitor your crawl rate over time
Discover how crawl rate is related to PageRank

Download the plugin here.

Screenshots

Crawl Rate

A big thanks to Open Flash Charts for the graph function.

You can get our blog posts delivered for free by email every day - simply add your email address to the box below or alternatively grab the RSS feed.

Read some similar posts

Post category: Blogging, Search Engine Optimisation Share it :

{ 111 comments… read them below or add one }

Fahmishah 25 Feb 2008 at 11:37 pm: thanks for sharing
Matthew 26 Feb 2008 at 12:43 am: I have been looking forwards to seeing a plugin like this created since you mentioned something similar (or the same) a few months back.

Quick question. About an hour ago I added a new post to my blog and then a few minutes later I installed the plugin. For that new post I see 3 crawls with in 40 minutes from Google to that new page. I am just curious as to why google would visit at 6:50:06 and then just 4 or 5 minutes later and then just over 30 minutes after that. Is that common?

google 25th Feb 2008 07:32:11
google 25th Feb 2008 06:54:51
google 25th Feb 2008 06:50:06
Patrick Altoft 26 Feb 2008 at 12:47 am
Find me on Twitter: I expect that loads of people will be asking questions like this, sometimes Google spiders my homepage 5 times an hour.

We may never know why.
More comments from Patrick Altoft
Matthew 26 Feb 2008 at 12:55 am: I am seeing the same also on my home page. The 4th visit just came along in the last hour. I will guess that is a good thing and carry on what I am doing knowing I may never know why. LOL.
James Mann 26 Feb 2008 at 1:36 am: Excellent plugin Patrick. It wasn’t even a minute after installing and activating that I was seeing results.

This plugin looks like it is going to be one of my fav tools. Keep up the great work Patrick and thanks
Dave 26 Feb 2008 at 3:02 am: Holy S**t this is awesome! You just made life a lot easier. Thank you for creating this.
Patrick Altoft 26 Feb 2008 at 3:43 am
Find me on Twitter: I’ve just fixed a bug that was stopping the MSN favicon displaying. The new version works but if you have the old version you need to rename the msnbot.ico file to msn.ico to get the icon to display.
More comments from Patrick Altoft
Teach Me $$ 26 Feb 2008 at 5:13 am: Thanks a lot,
Now lets watch google and other two. This is cool now we can watch them…

Vijay
Rasheed 26 Feb 2008 at 10:27 pm: Cool Plugin !
It will give us more details about google activity in our blogs.
Thanks.
Becky 26 Feb 2008 at 10:40 pm: Thanks for the great plugin, another set of analytics for me to stare at
Paul 27 Feb 2008 at 7:23 am: Hi,

Nice plugin. But I have had it installed for 12 hours now and I can’t see any activity at all. This leads me to believe that there ia a probem with the installation. Are there any special installation instructions? Are there any file permissions that need to be set?

Cheers
Elliot 27 Feb 2008 at 8:23 am: Excellent plugin – great idea!

When using this plugin under WP mu, I had to make a small change, since the open flash chart doesn’t seem to follow 301s (301ing domain.com to http://www.domain.com, with http://domain.com as the ‘home’):

if (!$_GET['page_url']) {
$dir = get_option(’home’) . ‘/wp-content/plugins/’ . basename(dirname(__FILE__));
+ // Hack, since we’re redirecting to www. and this pooches the graph data
+ $dir = preg_replace( ‘/http:\/\//’, ‘http://www.’, $dir );
$url = $dir .’/sbtracking-chart-data.php?chart_data=1′;
$url = urlencode( $url );
Maik 27 Feb 2008 at 10:02 am: Hello you have a little bug in your script.
I have chance Script from
require ('sbtracking-setup.php'); require ('sbtracking-log.php'); require ('sbtracking-report.php');
to
require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-setup.php'); require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-log.php'); require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-report.php');
MFG Maik
Betty 28 Feb 2008 at 5:03 am: So all I have to do is upload the plugin and activate the plugin?
Because I can’t find it anywhere after activation…what am I missing?
Thanks
Patrick Altoft 28 Feb 2008 at 5:05 am
Find me on Twitter: In your Dashboard there should be a link to “Crawl rate tracker”.
More comments from Patrick Altoft
Betty 28 Feb 2008 at 6:03 am: Found it..Thanks a lot
I was searching it under Options
Dan 28 Feb 2008 at 8:55 am: Hey,

Do I need to include ? Because my theme is highly customized and I do not have it currently.

I think you should mentioned something about it.
Patrick Altoft 28 Feb 2008 at 10:06 pm
Find me on Twitter: It should work as soon as you install and activate it. Is it still showing nothing?
More comments from Patrick Altoft
Patrick Altoft 28 Feb 2008 at 10:07 pm
Find me on Twitter: Not sure what you mean by ?
More comments from Patrick Altoft
Metty 29 Feb 2008 at 2:48 am: Hi there.
THX for your nice Plugin, but i can’t activate it.
I get an error message, in german, but it meany “could not be activated because it causes a fatal error”

Maybe you can help?
Patrick Altoft 29 Feb 2008 at 3:28 am
Find me on Twitter: Not really. I don’t know much about the program. Maybe it conflicts with another plugin?
More comments from Patrick Altoft
m.o.m. 29 Feb 2008 at 11:34 am: Nice Plugin. Thank you!
Pink 01 Mar 2008 at 12:22 am: Thanks for a great plugin!
Bier Blogger 02 Mar 2008 at 4:31 pm: Thanks from the beer Blogger from Germany!
jessie 03 Mar 2008 at 2:49 am: Thanks for sharing this plugin. I’m pretty sure it will benefit many Wordpress users like myself.
Chetan 06 Mar 2008 at 2:53 am: A great plugin..
Using it since a couple of days and seeing really good results and crawl stats.
Mike 11 Mar 2008 at 10:47 am: Very nice plugin, thanks a ton!
Mike 13 Mar 2008 at 7:00 am: I’m getting all the data but the Flash isn’t working and I’ve tried it on a couple different computers and browsers to be sure. Any ideas what it might be?
Patrick Altoft 13 Mar 2008 at 7:15 am
Find me on Twitter: Sorry I don’t know.
More comments from Patrick Altoft
Mike 13 Mar 2008 at 8:40 am: Oh well, I can still see the data myself. Thanks a lot for the plugin, working great to see how things are going.
Carla 20 Mar 2008 at 3:12 am: Thanks for this!

I don’t see nothing on the chart. I’m wondering if that’s because because my blog has only been on the self-hosted WordPress platform for about two weeks. It used to be on WordPress.com and I wasn’t able to redirect my traffic, so it might be that the blog looks brand new to Google.
Jaan Kanellis 07 Apr 2008 at 5:56 pm: Anyone know of a tools like this for non-Word Press websites?
aksamuzeri 15 Apr 2008 at 11:17 pm: thanks for your share, i will use this pack my blog,
Oleh 24 Apr 2008 at 5:15 pm: Big thanks for your share, i’ll use this plugin on my blog. I think that it will be good idea to translate this article into my native Ukrainian.
Google Webmaster 05 May 2008 at 7:02 pm: Thank you my friend i setup now. And i listed my blog

http://www.googleyardim.com/ i am document write. Please search crawl rate. Thank you
Google Webmaster 05 May 2008 at 7:03 pm: thank you i setup now my google help blog.
emrinho 15 May 2008 at 5:12 pm: Any chance that you will update this to work with WP 2.5 ???
Patrick Altoft 15 May 2008 at 5:28 pm
Find me on Twitter: Yes, next week.
More comments from Patrick Altoft
Chris Donner 20 May 2008 at 12:04 am: Nice i’m using it already for a long time.
Heinz Hübner 20 May 2008 at 12:06 am: Is this tool also available for non Wordpress Blogs?!
Patrick Altoft 20 May 2008 at 8:49 am
Find me on Twitter: No but I’m sure a php programmer could alter it to fit a php site.
More comments from Patrick Altoft
Fritz Huebchen 21 May 2008 at 5:21 pm: I guess i’ve seen a similar tool for non wordpress blogs…
Heinz Hübner 21 May 2008 at 11:33 pm: Found it, called Piwik. Working similiar to Analytics.
Angela Kukral 21 May 2008 at 11:35 pm: Also tried this tool, it’s perfect for my purposes.
jocuri 27 May 2008 at 7:49 pm: Thanks for this useful plugin.
jocuri 30 May 2008 at 9:32 pm: Keep up the good work! 10q
online izle 02 Jun 2008 at 7:41 am: Found it, called Piwik. Working similiar to Analytics
Mark Schurze 02 Jun 2008 at 11:34 pm: Pwiki is as powerful as analytics itself and will catch data in realtime.
Typo3 06 Jun 2008 at 10:02 am: Thanks – this is a nice tool – i work with this System now – I think it is also Google Analytics very good!

LG
Allan 08 Jun 2008 at 9:20 pm: When will it work with wordpress 2.5?
Patrick Altoft 08 Jun 2008 at 9:22 pm
Find me on Twitter: Not sure, nobody seems to know why it won’t work.
More comments from Patrick Altoft
ovidiu 11 Jun 2008 at 12:06 pm: hi there,
a couple of questions:

I got this plugin to work by manually creating the table. but still, after a few hits, I cannot see any graphs, I only see hits. Should the graph immediately appear?

also if I look at the detail report and compare with your screenshots, I seem to be missing the little images like yahoo and google in front of them..

anything else I need to do? If you want to I can send you screenshots of what I see right noww
Patrick Altoft 11 Jun 2008 at 12:17 pm
Find me on Twitter: I would wait a couple of days and see if the graph starts to work. Check the image locations, they might be getting pulled from the wrong folder?
More comments from Patrick Altoft
ovidiu 11 Jun 2008 at 2:10 pm: thx.

found the problem, had to change from:
$dir = get_option(’home’) . ‘/wp-content/plugins/’ . basename(dirname(__FILE__));
to:
$dir = get_option(’home’) . ‘/wordpress/wp-content/plugins/’ . basename(dirname(__FILE__));

had to insert wordpress as I installed my blog into a folder, not into the root, I guess it would be easier to use (not sure about this) blog_url instead of get_option(’home’)
This fixed the problem with the missing graph and the missing icons.
ovidiu 12 Jun 2008 at 5:35 pm: strange, even after the above change, the icons for the search engines didn’t appear, so I had to hardcode the path like this:

$text .= ”

for it to work.
Manish 15 Jun 2008 at 9:22 am: From the hacking point of view, it is advisable that we upgrade the WP as soon as any upgrade is available. However, the article states this great plugin doesn’t work in 2.5 and later…

Does anybody have any idea when this great plugin be available for WP 2.5 and later?

Thanks,
Mnis
ovidiu 17 Jun 2008 at 4:37 pm: one urgent question:

I just noticed search engines are hitting all my pages twice. Once without trailing slash and once with.

so pacura.ru/contact and pacura.ru/contact/ both get crawled…

I woudl like to avoid duplicate content – is this the case here? – I guess so…

Any ideas where this could result from? I just remember after wp 2.2 this was a big issue… I could use this plugin: http://www.agnivo.com/webmasters/wordpress-22-permalink-trailing-slash-duplicate-pages-fix-94.html to add a trailing slash, but is this the way to go?

Basicallly I am more interested to find out why search engines hit my site with both versions…
ovidiu 17 Jun 2008 at 4:38 pm: sorry for double posting, just found out it concerns also posts, not only pages
ro 20 Jun 2008 at 1:32 pm: thanks a lot for your plugin, pretty cool!

ronald
oronjo.com
sell content on your site
Movie 24 Jun 2008 at 6:06 pm: Nice Tools,Thank you very much.
jocuri fotbal 24 Jun 2008 at 8:47 pm: awesome plugin thank you
Jocuri 11 Jul 2008 at 10:58 am: Thanks for this useful plugin!!!
Jocuri 11 Jul 2008 at 11:00 am: awesome plugin thank you!!!
Xarj 21 Jul 2008 at 10:49 am: Just installed the plugin on Wordpress 2.6 and it seems to work fine!

Can anyone else confirm the Crawl Rate Tracker is compatible with Wordpress 2.6 ?
Equa 30 Jul 2008 at 2:10 pm: Thanks For Plugin.
Hequ 18 Aug 2008 at 4:33 am: great plugin thanks.
Jaan Kanellis 03 Sep 2008 at 12:47 am: What happens when the graph is not working right (showing any spider data) but the table data is correct?
John Pash 05 Sep 2008 at 11:37 am: Isn’t there some sort of plugin standards council? Or a document that states, “when you create a plugin, don’t hide the options page in the dashboard”. That sort of thing. It seems like every plugin author wants to be unique and have their little bit of php stand out (or hidden in this case!)

Ok, enough complaining. I love the plugin! But put it in the Setting panel please. I was ready to delete the thing because I thought it was broken. Nope, end-user error
Sofhal Jamil 07 Sep 2008 at 4:44 am: tks for sharing this information. Salam.
Posicionamiento Web 18 Sep 2008 at 7:04 pm: Escelente plugin para seguir el trafico de los robots.
Referáty 21 Sep 2008 at 12:35 pm: Great plugin, really! Thanks I use this plugin on my 3 sites.
Jocuri 22 Sep 2008 at 5:40 am: Thank you my friend i setup now
Ilka 22 Sep 2008 at 4:17 pm: WOW!!! Nice Plugin! Thanks a lot!

Illy
ed 03 Oct 2008 at 1:12 am: Cool Plugin !
It will give us more details about google activity in our blogs.
Thanks.
trends 07 Oct 2008 at 9:19 pm: A great plugin, i use it since may and it works correctly. Thank You!!
Milan 13 Oct 2008 at 8:25 pm: This plugin is in conflict with FireStats. I have been using FireStats for some time and maybe two weeks ago when I created new site powered by WordPress 2.6.2, I searched for something that I could use to track bot’s visits. I found this plugin and since it was like something I wanted, I activated it on that new site and also on one my other WordPress 2.6.2 site which is already ranked in search engines, just to see how it is working.

As days were passing, there where zero bots visits on both sites. That was strange to me since on that old site as soon as I posted new content Google would index it (via ping).

Then when I searched on Google if that new site was indexed, I saw that there is some problem. On Google’s SERP for that my site, title of the page and that description below would be text like “Table ‘firestatsdb:wp_sbtracking’ doesn’t exist.” (firestatsdb is example of name of FireStats’ database). I opened Google’s cache for that page and there was only that text. Then I checked Yahoo and same thing there. Next I checked Google’s index for that my older site and for some pages it appeared same thing.

So, this was happening: when some bot would come to page, this plugin would try to connect to FireStats’ database and sbtracking table in it. Since obviously there isn’t such table there it would return page to bot with title and text in body written above. Of course there was no any data written about that bots visit.

I first turned off plugin on that old site and Google started to index it normally so it is obvious that it is problem with this plugin. Luckily that old site is not some commercial one for me, so I didn’t get any loss, but if I used it on some commercial one, there would be problems. That’s why I’m writing this publicly: to people be aware of this potential problems.

(I must also note that FireStats is installed as standalone (version 1.5.12-stable) and I’m using its plugin for WordPress (I’m writing this because there are to types of installation for FireStats used on WordPress))

What I expect from you now is to fix this bug (I’ll send this bug to FireStats’ author too; maybe you could together see what problem is) and to write about this in this post (or whatever “official” page of plugin is) to people first test this plugin, like for example, to uncomment that line in sbtracking-log.php where Firefox is seen as bot and then to see what is happening.
Omry Yadan 16 Oct 2008 at 9:37 pm: Hi,
I am the author of FireStats.
I looked at the code, and it appears that the reason is that this plugin uses direct mysql calls and not going through the $wpdb object.
FireStats opens it’s own connection to the database, and since PHP MySQL works the way it does, further queries mysql queries that does not specify an explicit mysql connection handle uses the last open mysql connection.

a solution for that would be change the Rate tracket plugin to work with the $wpdb object (which keeps a handle of the mysql connection.

so instead of something like:
$result = mysql_query($sql);

you would do
global $wpdb;
$result = $wpdb->get_results($sql);

make sure you check the api of wpdb (which is based on the ezsql class)
Danh ba web 2.0 20 Oct 2008 at 12:06 pm: It’s very useful for me! Thanks you very much
Keep up the good work !
fil 20 Oct 2008 at 2:00 pm: Interesting clause certainly…..
ovidiu 28 Nov 2008 at 9:31 am: jsut testing the plugin with wpmu – will report back later
Ankur Gandhi 01 Dec 2008 at 12:16 pm: I update my blog to 2.6.3 and this plugin is not working…
Chart is not created. What to do…

Pls reply
izle 02 Dec 2008 at 4:44 am: It is impressive. I started to use it. Thanks.
Franky 05 Dec 2008 at 11:14 am: Thanks for a great plugin!
Free Ads portal 25 Jan 2009 at 12:41 pm: this is great plugin thanx for you help
amanhem 31 Jan 2009 at 12:36 am: thx for this pack
Web design Brisbane 05 Feb 2009 at 9:38 pm: So very cool ! Another essential plugin. A Stumble for your efforts.
Gustavo 06 Feb 2009 at 10:04 pm: Hi Patrick,

Seems I’m having the same problem as some of the guys, no graphs, at all. I have the plugin working for some weeks so loads of data. I’m on Wordpress 2.7. Any ideas?

Cheers,
Gustavo
James Colon 15 Feb 2009 at 7:14 am: Thanks for a great plugin.
NextGenLinks 21 Feb 2009 at 12:45 pm: I loaded this plugin on one of my blogs last week and just remembered to check the stats this morning – what a huge difference it makes to checking the spider stats. I could see instantly which spiders were visiting which pages and how frequently. Ever thought of putting a “donate” option in it – I’d gladly make a donation for it.
Blog Shot 22 Feb 2009 at 9:04 am: Anything that gives you more data on what the search engines are up to has to be a good thing. Thanks for the info.
Financedom 03 Mar 2009 at 11:25 am: That’s cool,Also you can check Crawl Rate in Google webmaster tools…
bilale 20 Mar 2009 at 9:40 pm: Thanks
It will give me more details about google activity in my blogs.

Thank you very much.
Cairns Bookkeeping 21 Mar 2009 at 10:00 pm: I don’t know if I agree with the statement: “The more Google trusts a page the greater the frequency it will re-visit that page.”

Isn’t crawl frequency determined by how often the page changes rather than the page authority? For example a news site would get crawled more frequently than an average web site due to the frequency of updates?
ucuz tatil 03 Apr 2009 at 11:45 am: thank you
Olympic Reporting 08 Apr 2009 at 1:48 pm: So, this way it is important to the sites to crawl.
San Francisco Bay Area places to visit 19 Apr 2009 at 4:56 pm: I find good correlation between crawl rate and SERP position/traffic, but recently very poor correlation between Page Rank and anything on
my wordpress travel blog.
lesen texte 29 Apr 2009 at 12:18 pm: Well nice plugin but the crawl rate is not that interesting for me… better would be some plugs for checking link popularity and things like that. Would be happy if you could write about such plugins
JobsEmployment 02 May 2009 at 7:57 pm: Yes, I must admit that this is good plugin to get and install. Plus, it may even help me to focus more on quality rather than quantity when it comes to blogging. In th epast I was just happy to get google to crawl my page. Now, I know I have to figure why and how often they do it. Thank you.
Makka 13 May 2009 at 7:15 am: Another useful plugin – thanks! Trouble is, there are so many WP plugins I’d like to use I’m gonna need an expanded Plugins folder. Or get rid of some of the other plugins, ‘cos this one’s a keeper!
Navaro 19 May 2009 at 8:27 am: Thanks for sharing this
Cam G 23 May 2009 at 6:04 am: Great tool, thanks for making it available, uploaded and running.

Like others on this site, I was surprised at how often google hits, seems at odds with what I assumed would be the frequency rate.
Ella 05 Jun 2009 at 1:37 pm: Hi…! This rocks. You rock. Thank you for sharing and helping people out. I send you a virtual beer
Ervin Blaylock 07 Jun 2009 at 3:35 pm: Great Plug-in just what i been looking to be able to track my campaign
and not have to wait to see the result and free did not hurt either will be
coming back for more look forward to a long relationship

Ervin
upx 07 Jun 2009 at 5:38 pm: How many CPU load on this Plug-in (My host has daily cpu load limited)
Goldstar 08 Jun 2009 at 6:20 am: Is it really about trust ? I thought if the page changed a lot then Google crawls more often? Maybe it’s a combination of both?
Ovidiu 08 Jun 2009 at 9:09 am: @Patrick: your wordpress site is generating a duplicate Mime header when sending the new comment notifications:

X-Amavis-Alert: BAD HEADER SECTION Duplicate header field: “MIME-Version”

it generates emails like:
MIME-Version: 1.0
MIME-Version: 1.0

I had the same problem with wpmu, but don’t know exactly how to solve this with simple wordpress
Chris DOnner 18 Jun 2009 at 12:24 am: I love this tool!
best free software 23 Jun 2009 at 9:01 pm: This plugin is fantastic. Thank you very much !
richardbaxterseo 29 Jun 2009 at 9:52 am
Find me on Twitter: Hi Patrick

Is this plugin tested / updated for WP v2.8? Thanks,
More comments from richardbaxterseo
Patrick Altoft 29 Jun 2009 at 10:27 am
Find me on Twitter: No. I have not updated it for a while.
More comments from Patrick Altoft
kissamosblogger 03 Jul 2009 at 10:16 am
Find me on Twitter: Thank you for providing us such useful info, this widget is exactly what I was searching for!
More comments from kissamosblogger