httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roy T. Fielding" <field...@liege.ICS.UCI.EDU>
Subject Re: robot denial
Date Tue, 23 Jul 1996 03:16:25 GMT
> So if "/robot" is bad (Koster seems to think it's ok), then what would
> be valid?

What I mentioned -- "indexer", "link checker", "pre-fetcher", etc.
Each type of robot has different characteristics -- lumping them into
one category is a bad thing.


View raw message