commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henri Yandell <flame...@gmail.com>
Subject Robots.txt parser
Date Thu, 11 Nov 2004 18:01:04 GMT
Following on from a mail on the httpclient-dev list; I'm interested in
submitting a codebase of mine into Commons that parses robots.txt
files. It would definitely be of use with HttpClient's future plans
but I'd like to keep it stand-alone and not hidden away inside
HttpClient.

Is there any interest? 

Probably the biggest -ve point for it is that as the robots.txt RFC is
very well written and unlikely to change in the future, it's not a
component that is likely to change beyond an option to use HttpClient
as its GET mechanism.

http://www.osjava.org/norbert/

If there is interest; is Norbert a bad name? :) I should probably go
with the simpler NoRobots name as that's the RFC title.

Hen

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org


Mime
View raw message