lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <e...@ehatchersolutions.com>
Subject Re: robots.txt
Date Mon, 20 Oct 2008 13:59:01 GMT

On Oct 20, 2008, at 8:58 AM, Alexander Aristov wrote:
> Just wonder if Nutch takes into consideration rules from the  
> robots.txt file
> while crawling a site.

Wrong e-mail list, but yeah, Nutch supports robots.txt considerations.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message