commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henri Yandell <>
Subject Robots.txt parser
Date Thu, 11 Nov 2004 18:01:04 GMT
Following on from a mail on the httpclient-dev list; I'm interested in
submitting a codebase of mine into Commons that parses robots.txt
files. It would definitely be of use with HttpClient's future plans
but I'd like to keep it stand-alone and not hidden away inside

Is there any interest? 

Probably the biggest -ve point for it is that as the robots.txt RFC is
very well written and unlikely to change in the future, it's not a
component that is likely to change beyond an option to use HttpClient
as its GET mechanism.

If there is interest; is Norbert a bad name? :) I should probably go
with the simpler NoRobots name as that's the RFC title.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message