httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Taylor" <>
Subject Re: [users@httpd] robots.txt
Date Thu, 02 Jan 2003 08:52:07 GMT
robots.txt-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

robots.txt is a file placed in the webroot used to control how
spiders "crawl" through your site for search-engine indexing. It's
nothing to worry about, all search engines look for it as they index
your site.

HTH :)

Chris Taylor - - The guy with the PS2 WebServer -

- ----- Original Message ----- 
From: Jeff Cohen 
To: Apache Group 
Sent: Thursday, January 02, 2003 8:37 AM
Subject: [users@httpd] robots.txt

Hi All,

Does anyone of you get the robots.txt file requests from the

It seems that I'm starting to get it almost every 3-4 hours a day,
every time from a different IP.

Is there a Virus going outta there that I'm not aware of?

How can I block these requests for this file on the root directory?

Help would be very appreciated,

Jeff Cohen

Version: PGPfreeware 7.0.3 for non-commercial use <>


View raw message