Since releasing the Wordpress Crawl Rate plugin a few people have been asking whether their crawl rate is “normal” or not. The answer is that crawl rates are determined by a number of factors so there really isn’t a normal crawl rate.
Here are some of the main factors that affect crawl rate:
- Trust
- PageRank
- Number of pages / site structure
- Number & quality of incoming links
- Frequency content changes
- Type of site
- Whether you run Adsense
- How often you ping Google using RSS
If you have a very large site you will get a much higher number of pages being crawled per day but the actual rate each individual page is crawled will be lower. If a search engine finds your content is changing a lot they will increase your crawl rate to take this into account and vice versa.
Google tends to crawl the web and load all the links they find into a “To be crawled” database without duplicating entries. Yahoo appears to crawl links as and when they find them without worrying about crawling pages too often. Sometimes Yahoo crawls the same page a few times per hour whereas Google usually just visits a page once every day or so depending on your crawl cycle.
The end result of this behaviour is that the more inbound links your site has the more Yahoo will crawl your pages. Small sites with lots of links will see higher Yahoo activity while large sites will almost always see more Google activity.
Below are some screenshots of the Crawl Rate Tracker in action. Although the data is only for a couple of days I find that the actual crawl rates are pretty consistent so the data is reliable.
Coolest Gadgets
Coolest Gadgets has 73,000 pages and a massive 350,000 incoming links. Google spiders over 11,000 pages per day and sometimes hits the homepage every 3 minutes!
Blogstorm
Blogstorm has 500 pages and 100,000 incoming links. Because it has lots of links Yahoo spiders the most.
GadgetVenue.com
Gadget Venue has 15,000 pages and 40,000 links. Because it has lots of pages we see that Google spiders the most pages per day.
Joost de Valk
Joosts SEO blog has 540 pages and 65,000 links. Again we see Yahoo spidering the most.
Self Made Minds
Self Made Minds has 330 pages and 70,000 incoming links – Yahoo spiders the most.
You can get our blog posts delivered for free by email every day - simply add your email address to the box below or alternatively grab the RSS feed.