How to Block Crawlers, Spiders and Bots from Websites
Aug 02, 2014
Sometimes disallowing crawlers on your website using robots.txt is not sufficient. In spite of this measure crawlers are still able to come on your website. In such a situation you need to block crawlers. Read on to find how you block crawlers.