nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Reardon <irnu...@gmail.com>
Subject Server Delay when crawling
Date Fri, 13 May 2005 19:16:37 GMT
What is a safe number to delay between page requests from the same
host?  I want to crawl as much information as possible in the shortest
amount of time, but I also don't want to hurt the server i'm
crawling....  What do you guys use?  I am using 5 seconds right now.

Mime
View raw message