While everybody is busy getting their website more and more exposed to the search engines (SEO) – you might want to restrict search engines to crawl your website.
You may have your own development website or may have your company’s web based HR system – which you don’t want to expose to the world. Search Engines will find your website if you do not restrict them. If you want to restrict the complete website just upload a file named “robot.txt” in the root folder and add the following line in it:
User-agent: *
Disallow: /
If you want to restrict a specific part (files in a specific directory) – add the following line:
User-agent: *
Disallow: /yourdirectory
If you want a certain Search Engine to be restrcited – (for example Google): add the following line:
User-agent: googlebot
Disallow: /yourdirectory
Some of the other Search Engine Bots are as follows:
MSN/Bing: bingbot
Yahoo: Yahoo Slurp (Currently using bingbot).
Baidu: Baidu Spider