httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roy T. Fielding" <field...@liege.ICS.UCI.EDU>
Subject Re: No HOST header solutions?
Date Mon, 03 Jun 1996 06:10:16 GMT
> I guess robots.txt is from pre-CGI days... the "Useragent" thing
> seems more than a little useless.

That's probably because you don't use MOMspider to help maintain
your site.  If you did (or used other benign robots), then it would
be clear as to why different robots would be assigned different
restrictions.

We (Martijn, myself, and a couple other robot authors) came up with
the /robots.txt stuff near the end of '93, so I guess that would be
a couple months after CGI.  It was back when the www-talk list was
much like new-httpd is now (though less focused).  It was definitely
before virtual hosts and CGI scripts being in uncontrolled directories,
which are the only real problems with the current format.

......Roy

Mime
View raw message