WordPress Crawl Rate Tracker Plugin
With the advent of Google Analytics and other JavaScript based solutions a lot of webmasters have lost the ability to easily analyse log files to see how search engines are interacting with their site.
Update: A new version of this plugin compatible with WordPress 3+ is now available here.
Google Webmaster Central offers a nice graphing feature so you can see the crawl trends but there is no way to access meaningful data. Studying crawl rate is hugely important because it helps you find out which pages are trusted most by the search engines (especially Google). The more Google trusts a page the greater the frequency it will re-visit that page. If you build a few deep links and see your crawl rate for the target page suddenly increase you can be confident that the links are highly trusted and that you should try to get some more.
Today we are releasing a plugin for WordPress that allows you to track crawl activity for MSN, Yahoo & Google from your dashboard.
- Access crawl data for 3 main search engines
- See the exact times a spider accessed any page on your site
- Monitor your crawl rate over time
- Discover how crawl rate is related to PageRank
Screenshots
A big thanks to Open Flash Charts for the graph function.
Post category: Blogging, Search Engine Optimisation Tweet
Patrick Altoft is Director of Search at Branded3, a Leeds SEO & Digital Agency specialising in SEO, Web Design, Development & Social Media.
Comments
Read the 180 comments below, or add your own!
thanks for sharing
I have been looking forwards to seeing a plugin like this created since you mentioned something similar (or the same) a few months back.
Quick question. About an hour ago I added a new post to my blog and then a few minutes later I installed the plugin. For that new post I see 3 crawls with in 40 minutes from Google to that new page. I am just curious as to why google would visit at 6:50:06 and then just 4 or 5 minutes later and then just over 30 minutes after that. Is that common?
google 25th Feb 2008 07:32:11
google 25th Feb 2008 06:54:51
google 25th Feb 2008 06:50:06
I expect that loads of people will be asking questions like this, sometimes Google spiders my homepage 5 times an hour.
We may never know why.
I am seeing the same also on my home page. The 4th visit just came along in the last hour. I will guess that is a good thing and carry on what I am doing knowing I may never know why. LOL.
Excellent plugin Patrick. It wasn’t even a minute after installing and activating that I was seeing results.
This plugin looks like it is going to be one of my fav tools. Keep up the great work Patrick and thanks
Holy S**t this is awesome! You just made life a lot easier. Thank you for creating this.
I’ve just fixed a bug that was stopping the MSN favicon displaying. The new version works but if you have the old version you need to rename the msnbot.ico file to msn.ico to get the icon to display.
Thanks a lot,
Now lets watch google and other two. This is cool now we can watch them…
Vijay
Cool Plugin !
It will give us more details about google activity in our blogs.
Thanks.
Thanks for the great plugin, another set of analytics for me to stare at
Hi,
Nice plugin. But I have had it installed for 12 hours now and I can’t see any activity at all. This leads me to believe that there ia a probem with the installation. Are there any special installation instructions? Are there any file permissions that need to be set?
Cheers
It should work as soon as you install and activate it. Is it still showing nothing?
Excellent plugin – great idea!
When using this plugin under WP mu, I had to make a small change, since the open flash chart doesn’t seem to follow 301s (301ing domain.com to http://www.domain.com, with http://domain.com as the ‘home’):
if (!$_GET['page_url']) {
$dir = get_option(‘home’) . ‘/wp-content/plugins/’ . basename(dirname(__FILE__));
+ // Hack, since we’re redirecting to www. and this pooches the graph data
+ $dir = preg_replace( ‘/http:\/\//’, ‘http://www.', $dir );
$url = $dir .’/sbtracking-chart-data.php?chart_data=1′;
$url = urlencode( $url );
Hello you have a little bug in your script.
I have chance Script from
require ('sbtracking-setup.php');
require ('sbtracking-log.php');
require ('sbtracking-report.php');
to
require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-setup.php');
require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-log.php');
require (ABSPATH . 'wp-content/plugins/search-bot-tracking/sbtracking-report.php');
MFG
Maik
So all I have to do is upload the plugin and activate the plugin?
Because I can’t find it anywhere after activation…what am I missing?
Thanks
In your Dashboard there should be a link to “Crawl rate tracker”.
Found it..Thanks a lot
I was searching it under Options
Hey,
Do I need to include ? Because my theme is highly customized and I do not have it currently.
I think you should mentioned something about it.
Not sure what you mean by ?
Hi there.
THX for your nice Plugin, but i can’t activate it.
I get an error message, in german, but it meany “could not be activated because it causes a fatal error”
Maybe you can help?
Not really. I don’t know much about the program. Maybe it conflicts with another plugin?
Nice Plugin. Thank you!
Thanks for a great plugin!
Thanks from the beer Blogger from Germany!
Thanks for the plugin, awesome.
Thanks for sharing this plugin. I’m pretty sure it will benefit many WordPress users like myself.
A great plugin..
Using it since a couple of days and seeing really good results and crawl stats.
Very nice plugin, thanks a ton!
I’m getting all the data but the Flash isn’t working and I’ve tried it on a couple different computers and browsers to be sure. Any ideas what it might be?
Sorry I don’t know.
Oh well, I can still see the data myself. Thanks a lot for the plugin, working great to see how things are going.
Thanks for this!
I don’t see nothing on the chart. I’m wondering if that’s because because my blog has only been on the self-hosted WordPress platform for about two weeks. It used to be on WordPress.com and I wasn’t able to redirect my traffic, so it might be that the blog looks brand new to Google.
Anyone know of a tools like this for non-Word Press websites?
thanks for your share, i will use this pack my blog,
Big thanks for your share, i’ll use this plugin on my blog. I think that it will be good idea to translate this article into my native Ukrainian.
Thank you my friend i setup now. And i listed my blog
http://www.googleyardim.com/ i am document write. Please search crawl rate. Thank you
thank you i setup now my google help blog.
Any chance that you will update this to work with WP 2.5 ???
Yes, next week.
Nice i’m using it already for a long time.
Is this tool also available for non WordPress Blogs?!
No but I’m sure a php programmer could alter it to fit a php site.
I guess i’ve seen a similar tool for non wordpress blogs…
Found it, called Piwik. Working similiar to Analytics.
Also tried this tool, it’s perfect for my purposes.
Thanks for this useful plugin.
Keep up the good work! 10q
Found it, called Piwik. Working similiar to Analytics
Pwiki is as powerful as analytics itself and will catch data in realtime.
Thanks – this is a nice tool – i work with this System now – I think it is also Google Analytics very good!
LG
When will it work with wordpress 2.5?
Not sure, nobody seems to know why it won’t work.
hi there,
a couple of questions:
I got this plugin to work by manually creating the table. but still, after a few hits, I cannot see any graphs, I only see hits. Should the graph immediately appear?
also if I look at the detail report and compare with your screenshots, I seem to be missing the little images like yahoo and google in front of them..
anything else I need to do? If you want to I can send you screenshots of what I see right noww
I would wait a couple of days and see if the graph starts to work. Check the image locations, they might be getting pulled from the wrong folder?
thx.
found the problem, had to change from:
$dir = get_option(‘home’) . ‘/wp-content/plugins/’ . basename(dirname(__FILE__));
to:
$dir = get_option(‘home’) . ‘/wordpress/wp-content/plugins/’ . basename(dirname(__FILE__));
had to insert wordpress as I installed my blog into a folder, not into the root, I guess it would be easier to use (not sure about this) blog_url instead of get_option(‘home’)
This fixed the problem with the missing graph and the missing icons.
strange, even after the above change, the icons for the search engines didn’t appear, so I had to hardcode the path like this:
$text .= ”
for it to work.
From the hacking point of view, it is advisable that we upgrade the WP as soon as any upgrade is available. However, the article states this great plugin doesn’t work in 2.5 and later…
Does anybody have any idea when this great plugin be available for WP 2.5 and later?
Thanks,
Mnis
one urgent question:
I just noticed search engines are hitting all my pages twice. Once without trailing slash and once with.
so pacura.ru/contact and pacura.ru/contact/ both get crawled…
I woudl like to avoid duplicate content – is this the case here? – I guess so…
Any ideas where this could result from? I just remember after wp 2.2 this was a big issue… I could use this plugin: http://www.agnivo.com/webmasters/wordpress-22-permalink-trailing-slash-duplicate-pages-fix-94.html to add a trailing slash, but is this the way to go?
Basicallly I am more interested to find out why search engines hit my site with both versions…
sorry for double posting, just found out it concerns also posts, not only pages
thanks a lot for your plugin, pretty cool!
ronald
oronjo.com
sell content on your site
Nice Tools,Thank you very much.
awesome plugin thank you
Thanks for this useful plugin!!!
awesome plugin thank you!!!
Just installed the plugin on WordPress 2.6 and it seems to work fine!
Can anyone else confirm the Crawl Rate Tracker is compatible with WordPress 2.6 ?
Thanks For Plugin.
great plugin thanks.
What happens when the graph is not working right (showing any spider data) but the table data is correct?
Isn’t there some sort of plugin standards council? Or a document that states, “when you create a plugin, don’t hide the options page in the dashboard”. That sort of thing. It seems like every plugin author wants to be unique and have their little bit of php stand out (or hidden in this case!)
Ok, enough complaining. I love the plugin! But put it in the Setting panel please. I was ready to delete the thing because I thought it was broken. Nope, end-user error
tks for sharing this information. Salam.
Escelente plugin para seguir el trafico de los robots.
Great plugin, really! Thanks I use this plugin on my 3 sites.
Thank you my friend i setup now
WOW!!! Nice Plugin! Thanks a lot!
Illy
Cool Plugin !
It will give us more details about google activity in our blogs.
Thanks.
A great plugin, i use it since may and it works correctly. Thank You!!
This plugin is in conflict with FireStats. I have been using FireStats for some time and maybe two weeks ago when I created new site powered by WordPress 2.6.2, I searched for something that I could use to track bot’s visits. I found this plugin and since it was like something I wanted, I activated it on that new site and also on one my other WordPress 2.6.2 site which is already ranked in search engines, just to see how it is working.
As days were passing, there where zero bots visits on both sites. That was strange to me since on that old site as soon as I posted new content Google would index it (via ping).
Then when I searched on Google if that new site was indexed, I saw that there is some problem. On Google’s SERP for that my site, title of the page and that description below would be text like “Table ‘firestatsdb:wp_sbtracking’ doesn’t exist.” (firestatsdb is example of name of FireStats’ database). I opened Google’s cache for that page and there was only that text. Then I checked Yahoo and same thing there. Next I checked Google’s index for that my older site and for some pages it appeared same thing.
So, this was happening: when some bot would come to page, this plugin would try to connect to FireStats’ database and sbtracking table in it. Since obviously there isn’t such table there it would return page to bot with title and text in body written above. Of course there was no any data written about that bots visit.
I first turned off plugin on that old site and Google started to index it normally so it is obvious that it is problem with this plugin. Luckily that old site is not some commercial one for me, so I didn’t get any loss, but if I used it on some commercial one, there would be problems. That’s why I’m writing this publicly: to people be aware of this potential problems.
(I must also note that FireStats is installed as standalone (version 1.5.12-stable) and I’m using its plugin for WordPress (I’m writing this because there are to types of installation for FireStats used on WordPress))
What I expect from you now is to fix this bug (I’ll send this bug to FireStats’ author too; maybe you could together see what problem is) and to write about this in this post (or whatever “official” page of plugin is) to people first test this plugin, like for example, to uncomment that line in sbtracking-log.php where Firefox is seen as bot and then to see what is happening.
Hi,
I am the author of FireStats.
I looked at the code, and it appears that the reason is that this plugin uses direct mysql calls and not going through the $wpdb object.
FireStats opens it’s own connection to the database, and since PHP MySQL works the way it does, further queries mysql queries that does not specify an explicit mysql connection handle uses the last open mysql connection.
a solution for that would be change the Rate tracket plugin to work with the $wpdb object (which keeps a handle of the mysql connection.
so instead of something like:
$result = mysql_query($sql);
you would do
global $wpdb;
$result = $wpdb->get_results($sql);
make sure you check the api of wpdb (which is based on the ezsql class)
It’s very useful for me! Thanks you very much
Keep up the good work !
Interesting clause certainly…..
jsut testing the plugin with wpmu – will report back later
I update my blog to 2.6.3 and this plugin is not working…
Chart is not created. What to do…
Pls reply
It is impressive. I started to use it. Thanks.
Thanks for a great plugin!
this is great plugin thanx for you help
thx for this pack
So very cool ! Another essential plugin. A Stumble for your efforts.
Hi Patrick,
Seems I’m having the same problem as some of the guys, no graphs, at all. I have the plugin working for some weeks so loads of data. I’m on WordPress 2.7. Any ideas?
Cheers,
Gustavo
Thanks for a great plugin.
I loaded this plugin on one of my blogs last week and just remembered to check the stats this morning – what a huge difference it makes to checking the spider stats. I could see instantly which spiders were visiting which pages and how frequently. Ever thought of putting a “donate” option in it – I’d gladly make a donation for it.
Anything that gives you more data on what the search engines are up to has to be a good thing. Thanks for the info.
That’s cool,Also you can check Crawl Rate in Google webmaster tools…
Thanks
It will give me more details about google activity in my blogs.
Thank you very much.
I don’t know if I agree with the statement: “The more Google trusts a page the greater the frequency it will re-visit that page.”
Isn’t crawl frequency determined by how often the page changes rather than the page authority? For example a news site would get crawled more frequently than an average web site due to the frequency of updates?
thank you
So, this way it is important to the sites to crawl.
I find good correlation between crawl rate and SERP position/traffic, but recently very poor correlation between Page Rank and anything on
my wordpress travel blog.
Well nice plugin but the crawl rate is not that interesting for me… better would be some plugs for checking link popularity and things like that. Would be happy if you could write about such plugins
Yes, I must admit that this is good plugin to get and install. Plus, it may even help me to focus more on quality rather than quantity when it comes to blogging. In th epast I was just happy to get google to crawl my page. Now, I know I have to figure why and how often they do it. Thank you.
Another useful plugin – thanks! Trouble is, there are so many WP plugins I’d like to use I’m gonna need an expanded Plugins folder. Or get rid of some of the other plugins, ‘cos this one’s a keeper!
Thanks for sharing this
Great tool, thanks for making it available, uploaded and running.
Like others on this site, I was surprised at how often google hits, seems at odds with what I assumed would be the frequency rate.
Hi…! This rocks. You rock. Thank you for sharing and helping people out. I send you a virtual beer
Great Plug-in just what i been looking to be able to track my campaign
and not have to wait to see the result and free did not hurt either will be
coming back for more look forward to a long relationship
Ervin
How many CPU load on this Plug-in (My host has daily cpu load limited)
Is it really about trust ? I thought if the page changed a lot then Google crawls more often? Maybe it’s a combination of both?
@Patrick: your wordpress site is generating a duplicate Mime header when sending the new comment notifications:
X-Amavis-Alert: BAD HEADER SECTION Duplicate header field: “MIME-Version”
it generates emails like:
MIME-Version: 1.0
MIME-Version: 1.0
I had the same problem with wpmu, but don’t know exactly how to solve this with simple wordpress
I love this tool!
This plugin is fantastic. Thank you very much !
Hi Patrick
Is this plugin tested / updated for WP v2.8? Thanks,
No. I have not updated it for a while.
Thank you for providing us such useful info, this widget is exactly what I was searching for!
Great plugin…Any chance that this will be updated soon?
why would it need updates? its working perfectly with wp 2.8.4 and wpmu 2.8.4
We have been using the plugin for some time – works like a charm! Thanks for sharing, got anything in dev? We don’t mind being guinea pigs … lol
Hmm nice plugin…searched for something like it for days. But i miss the opportunity to block single pages from being crawled..or i just dont see it?
But anyway…you saved me lots of time dude, thanx
Cool Great!
Thanks for the great plugin, another set of analytics for me to stare at
Nice plugin
I like it !
Thanks for a great plugin.
This is a good one. I am actually addicted to watching realtime update of access log files. Hopefully this plugin will save me some time.
Thanks.
Guys, how do you remove Technorati and add Yandex instead. Believe it or not, one of my sites is swamped by Yandex indexing (even thought it has nothing to do with Russia) and gets a few hits from Yandex. It would be good to see the stats on the graph.
Thanks for the plugin. It is great to see how often I am being crawled.
Nice Plugin. Thank you!
tks for this plugin!
What a great tool. Some more info to digest to help strategise marketing campaigns. Thanks guys.
Ein wunderbares Plugin, das mir Einschätzungen über die Wirkung von Deeplinks und allgemein von Links nebst deren Trust ermöglicht.
Herzlichen Dank dafür!
Excellent plugin
Thank you, this will save me from having to keep checking all the different webmaster sites.
is it possible to find out non index pages in this wordpress plugin?
i need list of pages which is not index in google from my blog
is it possible?
Cool plugin and I have been using it on my blog for about two years now. I am wondering if it has been updated since it’s release?
I am using it on Wp 2.9.2 with no issues, but am about to install it on a friend’s new blog and wanted to see if the download on this page is the most current.
Best regards,
Jerry
This was the best find of the night! Thanks for the plugin.
This is a great plugin. We were wondering for a while why our male and female models pages were not ranking well but it was all about the robots’ frequency of visits. Many thanks!
Thank you very much Great post…thanks for share this. 5225
Thanks for a great plugin!
It is a nice information and now have this plugin installed in my wpress blog.
Thats a fine plugin, really funny and useful, thanks!
This is nice…. thanks for this
Great post. This plugin is awesome. Thank for the share.
Thanks for this useful plugin!!!
Any partiulcar reason you don’t submit this plugin to wordpress official site?
I have been waiting for this for a while!! Thanks. By the way, I had problems too, but Omry Yadan tip helped me a lot
Thanks Omryyyyyyyy!!!!
greata rticle..just what i needed..thank you very much for sharing
Very cool. Definitely something I need!!
Very cool – thanks for sharing this. I am going to add it to my blog asap.
Plugin problems in my theme
Fatal error:
Fatal error: Cannot redeclare curpageurl() (previously declared in /home/content/21/5301021/html/blog/wp-content/themes/magazine-premium/functions.php:1901) in /home/content/21/5301021/html/blog/wp-content/plugins/crawlrate/sbtracking-log.php on line 69
Great post. This plugin is awesome. Thank for the share.
Really nice plugin, however I’m having some issues, graph is not showing, all I get is a blank square with an url inside it “…wp-content/plugins/CrawlRate/sbtracking-chart-data.php?chart_data=1″ (removed the domain name) Any idea on how to fix this? Btw I’m on wp 2.9.2
Thanks for a great plugin!
yes, this definately looks like something worth trying. Thank you.
Great script. I’m looking at using this for my site (non WP). Also, bing has changed their bot name recently (they use msn and bing in the agent).
Rob
Gonna try it also.
Great tool, using it since a few days now and really glad with it.
thank you very much this plugin is very useful for me.
It would be better to support Chinese search engines such as baidu,sogou,youdao and soso.
Awesome plugins and many thanks for sharing…This is definitely something worth installing for all blogs.
What do you think of pop-under exchanges? I’m having trouble getting visitors to my site,
so I’m thinking about trying one. Are they effective????????
This is a good selection of sites you made. My favorite of your selection is Ben Hulse Design.
Hi there.
THX for your nice Plugin, This is definitely something worth installing for all blogs.
Hi,
I’ve just copied the plugin but it does not come up. I activate it but can’t find any settings
Any suggestions how I can make it work?
Awesome plugin, but after I updated to WP 3.1 the graph function is not working anymore.
My issue is the same as Tom, i am using WordPress 3.1 and not getting the graph, any solution to this? would like to see how it works thanks today is March 16, 2011
I am also using this wordpress plugin – Thanks
love this plugin! Howver need to work on my interpretation of the information, somthing i struggle with on google analytics. Be kind i have just started on my web design word press jopurney!
Hi, I found this plugin to be awsome. Great work mate!!
Very handy! I was using AWstats on my wordpress page and it looked like i was getting alot of traffic…. turns out it was spiders!
ohwell least the search engines are finding me
Thanks again
Great plugin. Thanks. Now I don’t have to go into Awstats all the time!
Great plugin. I’ve downloaded and can see the stats already.
Well done.
I am using it on 3.1.2 and it seems to be broken. I see a couple other comments to the same effect. Any solution??
This is an incredibly cool plugin. Thanks so much for doing this and releasing it free! I especially loved that it went back through my logs and filled in data from the last month. Neat to see data instantly. I’m using it on my freelance web design portfolio site and am fascinated to watch how Google crawls my site.
I’m working on a roundup of useful SEO/Internet Marketing WordPress plugins for a blog post, and this one is certainly in the top 3. Excellent work.
Hi,
I downloaded the plugin, but the chart does not show up. It just shows a blank square. I am using Firefox. Is there any other thing I need to do?
Regards,
Garry
In the event you keep on submitting helpful articles or reviews on your web site, there is always a larger opportunity that a number of individuals will go to your web page.
I really like this plugin as I’ve used it before but it doesn’t seem to be working with the latest version of WordPress, any chance an update is going to be coming soon?
This tool is really great. No more guessing but raw backed up statistics!
Thanks for sharing this with us!
Tool Crawl Rate Tracker Plugin is really, really great.
Thanks very much
Great plugin!
i love this plugin because using it since a couple of days and seeing really good results and crawl stats. Thanks…
Great post. This plugin is awesome. Thank for the share.
Great post. This plugin is awesome. Thank for the share.
I guess now in 2012 WP has different plug ins. It would be great if you could update this post.
73 trackbacks
February 25, 2008 at 10:45pm
February 26, 2008 at 11:29am
February 26, 2008 at 12:39pm
February 26, 2008 at 1:19pm
February 27, 2008 at 5:59am
February 27, 2008 at 5:01pm
February 28, 2008 at 1:26pm
February 29, 2008 at 4:00pm
March 1, 2008 at 1:43am
March 1, 2008 at 4:16am
March 2, 2008 at 3:43am
March 7, 2008 at 6:23pm
March 10, 2008 at 2:01am
March 10, 2008 at 3:48pm
March 17, 2008 at 6:37am
March 18, 2008 at 10:58pm
March 23, 2008 at 8:41am
March 25, 2008 at 5:15am
March 26, 2008 at 4:21am
April 4, 2008 at 2:49am
April 7, 2008 at 5:55pm
April 8, 2008 at 4:33pm
April 19, 2008 at 4:53pm
April 28, 2008 at 3:09pm
May 6, 2008 at 12:55pm
May 21, 2008 at 6:27pm
June 24, 2008 at 2:31pm
June 25, 2008 at 11:27am
June 27, 2008 at 1:34pm
August 21, 2008 at 9:46pm
September 1, 2008 at 6:42am
September 5, 2008 at 8:02am
September 6, 2008 at 3:08pm
September 6, 2008 at 3:46pm
September 17, 2008 at 2:34pm
September 17, 2008 at 3:46pm
September 17, 2008 at 5:04pm
September 18, 2008 at 10:19am
September 18, 2008 at 7:07pm
September 26, 2008 at 3:26pm
September 26, 2008 at 5:01pm
September 30, 2008 at 8:14pm
October 1, 2008 at 4:56pm
October 1, 2008 at 4:58pm
October 8, 2008 at 4:01am
November 5, 2008 at 12:48am
November 14, 2008 at 7:58am
March 21, 2009 at 2:44pm
April 8, 2009 at 9:41pm
April 28, 2009 at 5:02pm
May 2, 2009 at 3:56pm
May 31, 2009 at 12:26am
June 6, 2009 at 11:50am
June 21, 2009 at 6:15pm
July 1, 2009 at 8:46pm
July 3, 2009 at 4:07pm
July 18, 2009 at 2:26am
August 31, 2009 at 9:11am
September 29, 2009 at 4:23pm
October 16, 2009 at 7:09am
November 2, 2009 at 1:25pm
November 22, 2009 at 2:41pm
December 7, 2009 at 6:33pm
January 5, 2010 at 6:05pm
January 6, 2010 at 4:45am
January 20, 2010 at 10:56am
April 13, 2010 at 5:59am
May 11, 2010 at 3:03pm
May 17, 2010 at 9:01am
September 1, 2010 at 3:27am
December 7, 2010 at 8:15pm
September 25, 2011 at 6:14pm
September 26, 2011 at 11:09am