hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeetendra Mirchandani <je...@amazon.com>
Subject Re: AW: Robots.txt support?
Date Fri, 23 Dec 2005 10:17:39 GMT

I found this: http://www.osjava.org/norbert/changes-report.html
Any comments on using this? Or any other suggestions?

Thanks,
Jeetu



On Fri, 2005-12-23 at 10:55 +0100, Ingo Meyer wrote:
> > -----Urspr√ľngliche Nachricht-----
> > Von: Jeetendra Mirchandani [mailto:jeetu@amazon.com] 
> > Gesendet: Freitag, 23. Dezember 2005 10:38
> > An: httpclient-user@jakarta.apache.org
> > Cc: Jeetendra Mirchandani
> > Betreff: Robots.txt support?
> > 
> > Hi,
> > 
> > Can some one point me to some extension to HTTP Client to 
> > honor robots.txt?
> 
> How should HttpClient do that???
> This is not in scope of HttpClient.
> 
> You should load the robots.txt and parse it.
> 
> > 
> > Please CC me on the reply as I am not on the list.
> > 
> > Thanks,
> > Jeetu
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: 
> > httpclient-user-help@jakarta.apache.org
> > 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message