Don’t waste time blocking SEOmoz Linkscape
There is a lot of discussion today about SEOmoz Linkscape and whether you can/should be blocking it. Here is my advice – don’t bother.
Unless your site is only linked to from your own websites and you can block them all (in which case adding a big footprint by adding a meta tag is a bad idea) then there is no point in blocking the spider/crawler/indexer/scraper.
The simple fact is that you can only block your own websites. There is no way of stopping SEOmoz getting your link data because they can quite easily spider all the websites that are linking to you.
I’m not saying this is a bad thing, just something to be aware of before you panic about blocking them.
My question to Rand:
Ok so assuming I did block my own websites surely all my competitors could still see my link information because I am unable to block crawling of all the sites that link to me? Correct?
If so then there seems little point in blocking it.
Rand answered:
@patrickaltoft – yes, that’s right. You’d need to get all of your link partners to block in order for us not show links to your sites/pages/
Post category: Search Engine Optimisation Tweet
Patrick Altoft is Director of Search at Branded3, a Leeds SEO & Digital Agency specialising in SEO, Web Design, Development & Social Media.
Comments
Read the 5 comments below, or add your own!
Another point: Consider the potential downside of Linkscape crawling your sites, versus the work it’ll take to completely block them. I understand they’ll be grabbing competitive data. There are already dozens of other tools doing the same thing, behind firewalls and in private. Does it really pay to go to the effort?
Blocking crawlers and bots depends on the them actually taking notice of your meta or robots.txt anyway. I’m sure linkscape is legit and would follow the commands but the only reason you you should be blocking bots is if they are having an adverse affect on bandwidth /server loads.
How adversely does a bot normally impact a websites bandwidth? Has anyone released any data concerning this?
One good point that was brought up in this is, even if you opt out, couldn’t they just get the data in an inverse fashion? For example:
You have 5 websites, all linked to each other. Site 1 is like “I want out”, but all the rest are fine with being crawled. Couldn’t the bot just see that 2-5 are linked to 1? I guess the question from this is if site 1 in fact cross-linked to the others, which a bot wouldn’t know unless it crawled 1, unless i’m mistaken elsewhere.
I had a website that confused a bot once, and it just went in circles and generated 500,000 hits a day, without impacting my bandwidth to such a degree that it was a problem. I’d like to hear other examples where these web bots have caused a problem.
Hi Max. All my hosting is unlimited bandwidth so I don’t care about spiders and bots. Bandwidth is cheap these days so unless you have a huge forum site or something it really isn’t an issue.
Another point: Consider the potential downside of Linkscape crawling your sites, versus the work it’ll take to find the linking information on competitors
TBH anything that will help to remove some of the repetitiveness of link building and link building research is worth it.
Personally I like the Linkscape tools and think that SEOMOZ have done a great job releasing this info – they didn’t need to, they could have kept it in house. There’s nothing really in the data that can’t be found without some hard work and the only real contentious issues for me are as Michael Martinez points out http://www.searchenginejournal.com/seomoz-linkscape-new-backlink-checking-tool-reviewed/7826/ – the data is inconclusive (as are all backlink tools ) and the page strength feature is subjective.