httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "RuneImp" <r...@imptech.net>
Subject Re: robots.txt
Date Thu, 17 Jan 2002 23:52:38 GMT
Put this in your file

User-agent: *
Disallow: /


-=- RuneImp
ImpTech - Web Design, Hosting & Computer Tech
http://imptech.net
rune@imptech.net


----- Original Message ----- 
From: "Raquel Rice" <raquel@thericehouse.net>
To: <users@httpd.apache.org>
Sent: Thursday, January 17, 2002 2:31 PM
Subject: robots.txt


I'm not certain whether this is "on topic" or not.  I'm going to
take the chance and ask.

Is it possible to use "robots.txt" to totally block certain robots
from a domain?

I have a domain from which certain sites have been gone for 2 years.
 However, Scooter insists on probing for them.  It wouldn't be bad
except that I have a perl script that sends an email whenever a
robot reports broken links.  Scooter just found all those sites
which have been gone for 2 years and my perl script sent me ... all
at once ... 181 emails.

At least the other robots don't hit so hard in one shot!

Phew!  I feel better now.  If this isn't a topic for here, does
anyone know where to look?

-- 
Raquel
============================================================
Men never do evil so completely and cheerfully as when they do it
with religious conviction.
  --Blaise Pascal


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message