Has Google Stopped Indexing Blogs?

by Patrick Altoft on September 15, 2008

This might seem an overly dramatic headline but it seems that Google is having trouble indexing certain Wordpress blogs.

DaveN spotted the issue first when he realised his new blog posts were missing from the Google search index even though Google had spidered them and included them in the blog search results.

In what might be a related issue Google has been alerting people (myself included on 2 Wordpress blogs) to “General HTTP Errors” in the Webmaster Tools console.

As far as I can tell from looking at crawl logs Google accessed these pages just fine so why are the errors appearing and what are they?

You can get our blog posts delivered for free by email every day - simply add your email address to the box below or alternatively grab the RSS feed.

Read some similar posts

Published in: Blogging, Google

{ 3 trackbacks }

Have You Got Your Free Backlinks From Yahoo Yet? | Brad Blogging: 09.15.08 at 9:24 pm
Google is Creating Problem with SEO Blogs ? | Tech At Hand Dot Net | Philippines, Technology, SEO and Blogging: 09.16.08 at 9:22 am
The Weekly Insider 9-15-08 to 9-19-08: 09.19.08 at 11:17 pm

{ 23 comments… read them below or add one }

1 Tom 15/09/2008 at 3:04 pm: I’m getting general HTTP errors in my WMC for distilled too - though I’m actually seeing it for some regular HTML pages as well as individual blog posts
2 DazzlinDonna 15/09/2008 at 3:16 pm: Yep, I have those same errors for SEO Scoop as well. I’m guessing it’s a reporting glitch, but maybe it’s an actual problem. Hope G either fixes - or tells us more.
3 Internet Business Blogger 15/09/2008 at 3:21 pm: I just went into webmaster tools and get the same thing. Some of the posts though are older posts so I am not sure what the problem is. I was worried about the links themselves so I clicked through and they are there.

Anyway, I created a post on the weekend that should have ranked very high for “Huricane Ike Cancun” looked the next day and I was not in the first 30 results so I am apt to agree with you about the indexing of new blog posts.

Hopefully this is just an error on Googles side
4 Ken Jones 15/09/2008 at 4:17 pm: I noticed the same thing on my WM Tools report a couple of days ago, but just assumed it was a temporary glitch because the date of the error coincided with the day last week when I installed the latest WP update. I put up a new post later that day and G’s cache of my homepage shows that post (and a cache time that’s only a couple of minutes after I hit the publish button) so it all seemed hunky-dory after that.
Haven’t had any other crawl errors reported, but if it’s affecting a lot of other WP users as well it might be something more serious. I’ll have to have a deeper look into my stats and see if it shows up anything untoward.
5 Joshua Sciarrino (schreno) 15/09/2008 at 4:38 pm: Actually, I’d say it’s your reports that are in error.

Blogstorm looks fine.
Dave Naylor looks fine.
DazzlinDonna looks fine.

Everything is okay. Just another glitch in google’s products.
6 imnotadoctor 15/09/2008 at 4:55 pm: I am having this issue as well on my wordpress blog. How odd.
7 Dan 15/09/2008 at 5:09 pm: Conspiracy alert!! Only issues with WP and not Blogger sites? Hmmmm….
8 David Bradley 15/09/2008 at 6:57 pm: I can see half a dozen of those general http errors for my site all dated September 9, I suspect it’s a transient error, but more to the point all the pages in question are #1 in the Google SERP for their titles
9 don 15/09/2008 at 10:32 pm: Interesting. I was having similar issues and thought, it was my developer’s fault.
And like Dave above, those particular titles were in number one slots.

In my case I have been using wordpress as a CMS. Maybe its time to abandon ship and go over to DRUPAL.
10 Jenin 16/09/2008 at 12:56 am: I seem to be having a similar problem and I’m really frustrated. Since the 9th of September it appears that google has stopped indexing my new blog posts. I don’t know what to do about this. At first I thought it was a problem with my blog, but I realise that others are facing a similar problem. Any news on whether Google has acknowledged this problem or made any announcements about how it will be solved??
11 Brad Blogging.com - Personal Blog Tips And Blog Help 16/09/2008 at 1:58 am: I hardly think that Google is going to stop indexing blogs..

They are accounting for more internet content than any other type of media on the internet. I’m sure it’s just something like a feedburner glitch and will be over soon.
12 Nick Stamoulis 16/09/2008 at 3:02 am: Good to hear it’s probably just a tool error as opposed Google not indexing them anymore..phew!
13 Dexter | Tech At Hand Dot Net 16/09/2008 at 8:55 am: I also have the same errors in my Log
14 David Bradley 16/09/2008 at 10:07 am: I see the errors for another of my sites that isn’t strictly speaking a blog and doesn’t run Wordpress
15 Sébastien Billard 16/09/2008 at 10:41 am: Got the same generic HTTP errors and I use Dotclear. Google is also very slow to refresh the index pages which is unusual as my blog has some decent popularity.
16 Tom Goering 16/09/2008 at 6:37 pm: My blog stopped having pages indexed in Google Blog Search the day I was excepted for AdSense for Feeds. Blog Search updates from the site’s feed.

My posts however do make it to the regular index on Googlebots regular rounds.

I deactivated Feedsmith in wordpress and everything is back to normal (except the benefits of feedsmith that is)
17 DazzlinDonna 16/09/2008 at 9:11 pm: I’ve made this comment elsewhere, but I think it deserves to be shared here as well: One seorefugee member found his blog indexing issue to be related to the switch to adsense for feedburner. (see http://www.seorefugee.com/forums/showthread.php?t=10624) Perhaps this will help at least some folks track down their problems.
18 JohnMu 17/09/2008 at 10:50 am: Hi Patrick, the “General HTTP Errors” should be disappearing from your account soon — it seems that this was an intermittent reporting issue in Webmaster Tools. It’s not an error you need to be worried about, sorry for the scare! If it doesn’t disappear from your account in the next day or so, it’d be great if you could post in the Webmaster Help groups at http://groups.google.com/group/Google_Webmaster_Help-Tools/browse_thread/thread/d40a45ed11f1534e/
19 Peter Young 17/09/2008 at 11:06 am: Had the same with a number of wordpress blogs - both .com and hosted.

Is the problem merely with Wordpress blogs, or have any other blog vendors been affected?
20 Jon 17/09/2008 at 5:12 pm: I had one http error on 8th September, but have been OK since. I’m still on WP 2.5 - wonder if the version makes any difference?

Cheers, Jon
21 David Bradley 17/09/2008 at 6:20 pm: Apparently, Google is now aware of this issue. To quote from JohnMu, who is presumably staff: “This appears to be an issue in the reporting side of things in
Webmaster Tools. As far as I know, the “General HTTP errors” should be
gone soon :-). Thanks for bringing it to our attention, you guys are
the greatest!!”

http://groups.google.com/group/Google_Webmaster_Help-Tools/browse_thread/thread/d40a45ed11f1534e#
22 Scott McAndrew 17/09/2008 at 9:55 pm: I had several HTTP errors for both of the blogs I actively run last week. Google crawls one of the blogs immediately after posts are made and new posts generally go straight onto the blog search results with the web search results soon after. In the past few days I’ve noticed significantly more lag than I’d become accustomed to.
23 Chris at Watch TV Shows Online 08/11/2008 at 7:13 pm: I don’t know about that, but I do know that recently a site of mine took some severe drops in Google search engine results pages. So something is going on!