{"id":549,"date":"2016-03-22T19:16:40","date_gmt":"2016-03-22T13:16:40","guid":{"rendered":"http:\/\/hasibul.info\/blog\/?p=549"},"modified":"2016-03-22T19:17:03","modified_gmt":"2016-03-22T13:17:03","slug":"restrict-search-engines-from-finding-your-website","status":"publish","type":"post","link":"https:\/\/hasibul.info\/blog\/2016\/03\/22\/restrict-search-engines-from-finding-your-website\/","title":{"rendered":"Restrict Search Engines from finding your website"},"content":{"rendered":"<p style=\"text-align: justify;\">While everybody is busy getting their website more and more exposed to the search engines (SEO) &#8211; you might want to restrict search engines to crawl your website.<\/p>\n<p style=\"text-align: justify;\">You may have your own development website or may have your company&#8217;s web based HR system &#8211; which you don&#8217;t want to expose to the world. Search Engines will find your website if you do not restrict them. If you want to restrict the complete website just upload a file named &#8220;robot.txt&#8221; in the root folder and add the following line in it:<\/p>\n<p style=\"padding-left: 30px;\"><em>User-agent: *<\/em><br \/>\n<em> Disallow: \/<\/em><\/p>\n<p>If you want to restrict a specific part (files in a specific directory) &#8211; add the following line:<\/p>\n<p style=\"padding-left: 30px;\"><em>User-agent: *<\/em><br \/>\n<em>Disallow: \/yourdirectory<\/em><\/p>\n<p>If you want a certain Search Engine to be restrcited &#8211; (for example Google): add the following line:<\/p>\n<p style=\"padding-left: 30px;\"><em>User-agent:\u00a0googlebot<\/em><br \/>\n<em>Disallow: \/yourdirectory<\/em><\/p>\n<p>Some of the other Search Engine Bots are as follows:<\/p>\n<p>MSN\/Bing: bingbot<br \/>\nYahoo: Yahoo Slurp (Currently using bingbot).<br \/>\nBaidu: Baidu Spider<\/p>\n","protected":false},"excerpt":{"rendered":"<p>While everybody is busy getting their website more and more exposed to the search engines (SEO) - you might want to restrict search engines to crawl your website. You may have your own development website or may have your company's web based HR system - which you don't want to expose to the world. Search&hellip;<\/p>","protected":false},"author":1,"featured_media":550,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13,18,40,17,6,12,203],"tags":[266,267,260],"how-to":[],"_links":{"self":[{"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/posts\/549"}],"collection":[{"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/comments?post=549"}],"version-history":[{"count":0,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/posts\/549\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/media\/550"}],"wp:attachment":[{"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/media?parent=549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/categories?post=549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/tags?post=549"},{"taxonomy":"how-to","embeddable":true,"href":"https:\/\/hasibul.info\/blog\/wp-json\/wp\/v2\/how-to?post=549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}