lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Aristov" <alexander.aris...@gmail.com>
Subject robots.txt
Date Mon, 20 Oct 2008 12:58:37 GMT
Hi all,

Just wonder if Nutch takes into consideration rules from the robots.txt file
while crawling a site.

-- 
Best Regards
Alexander Aristov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message