httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark J Cox <m...@ukweb.com>
Subject Re: Suggestion to help robots and sites coexist a little better
Date Mon, 15 Jul 1996 07:57:45 GMT
>  1) robots send a new HTTP header to say "I'm a robot", e.g.
>       Robot: Indexers R us

Being able to distinguish a robot access on individual requests may become
a problem for search engines as users write modules that tailor their
output.  An Apache module that tags the dictionary or just the usual porn
words onto the end of every request that comes from a robot is trivial to
write.  Many search engines now ignore words in comments for this very
reason (and now you can find many pages that have pages of keywords tagged
to the end with text colors designed to make them not noticable)

Mark
Mark Cox, Technical Director, UK Web ------ http://www.ukweb.com/~mark/


Mime
View raw message