httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Taylor" <ch...@x-bb.org>
Subject Re: [users@httpd] robots.txt
Date Thu, 02 Jan 2003 08:52:07 GMT
robots.txt-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

robots.txt is a file placed in the webroot used to control how
spiders "crawl" through your site for search-engine indexing. It's
nothing to worry about, all search engines look for it as they index
your site.

HTH :)

Chris Taylor - chris@x-bb.org - The guy with the PS2 WebServer -
http://www.x-bb.org/chris.asc

- ----- Original Message ----- 
From: Jeff Cohen 
To: Apache Group 
Sent: Thursday, January 02, 2003 8:37 AM
Subject: [users@httpd] robots.txt


Hi All,

Does anyone of you get the robots.txt file requests from the
internet??

It seems that I'm starting to get it almost every 3-4 hours a day,
every time from a different IP.

Is there a Virus going outta there that I'm not aware of?

How can I block these requests for this file on the root directory?

Help would be very appreciated,

Jeff Cohen

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA+AwUBPhP9tCqf8lmE2RZkEQKEHwCeMpNY8p9rPMLbujJRHXbWzT43hh0Al2WB
lSk9t4IpcOePAc9VV8HxOrE=
=SiLQ
-----END PGP SIGNATURE-----


Mime
View raw message