httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "RuneImp" <>
Subject Re: robots.txt
Date Thu, 17 Jan 2002 23:52:38 GMT
Put this in your file

User-agent: *
Disallow: /

-=- RuneImp
ImpTech - Web Design, Hosting & Computer Tech

----- Original Message ----- 
From: "Raquel Rice" <>
To: <>
Sent: Thursday, January 17, 2002 2:31 PM
Subject: robots.txt

I'm not certain whether this is "on topic" or not.  I'm going to
take the chance and ask.

Is it possible to use "robots.txt" to totally block certain robots
from a domain?

I have a domain from which certain sites have been gone for 2 years.
 However, Scooter insists on probing for them.  It wouldn't be bad
except that I have a perl script that sends an email whenever a
robot reports broken links.  Scooter just found all those sites
which have been gone for 2 years and my perl script sent me ... all
at once ... 181 emails.

At least the other robots don't hit so hard in one shot!

Phew!  I feel better now.  If this isn't a topic for here, does
anyone know where to look?

Men never do evil so completely and cheerfully as when they do it
with religious conviction.
  --Blaise Pascal

The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:> for more info.
To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message