Help
)-
Kitchisaki
- Newbie
- Posts: 8
- Joined: 30 Jul 2010, 23:42
- 13
- Location: Just above the earth's atmosphere
Help
Hello there every one .Is it possible to restrict bots from visiting and viewing a website example Googlebot,Yahooslurp,Msn and others.If so please inform me,any help would be greatly appreciated.
Th3 only wh1t3 cat that has black str1p3s
-
bad_brain
- Site Owner
- Posts: 11636
- Joined: 06 Apr 2005, 16:00
- 19
- Location: In your eye floaters.
- Contact:
Re: Help
sure, usually you do this via the robots.txt file stored in the website root:
http://www.robotstxt.org/robotstxt.html" onclick="window.open(this.href);return false;
the well-known crawlers like google, yahoo (slurp), msn follow what you have set in robots.txt, but there are rogue crawlers that ignore it, you can block these via .htaccess file (also in the website root) by the IP, like:
http://www.robotstxt.org/robotstxt.html" onclick="window.open(this.href);return false;
the well-known crawlers like google, yahoo (slurp), msn follow what you have set in robots.txt, but there are rogue crawlers that ignore it, you can block these via .htaccess file (also in the website root) by the IP, like:
Code: Select all
Order deny,allow
Deny from 192.168.0.1
Deny from 192.168.0.3
-
Kitchisaki
- Newbie
- Posts: 8
- Joined: 30 Jul 2010, 23:42
- 13
- Location: Just above the earth's atmosphere
Re: Help
wow this useful
i need it... thank you and great link B_B
i need it... thank you and great link B_B
There is an UNEQUAL amount of good and bad in most things, the trick is to work out the ratio and act accordingly. "The Jester"