nutch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gal Nitzan" <gal.nitza...@gmail.com>
Subject RE: Nutch and Crawl Frequency
Date Thu, 19 Apr 2007 20:26:50 GMT
As it is right now... You answered the question yourself :-) ...

Separate db's and the whole ceremony...


> -----Original Message-----
> From: Briggs [mailto:acidbriggs@gmail.com]
> Sent: Thursday, April 19, 2007 10:02 PM
> To: nutch-user@lucene.apache.org
> Subject: Nutch and Crawl Frequency
>
> Nutch 0.9
>
> Anyone know if it is possible to be more granular regarding crawl
> frequency?  Meaning, that I would like some sites to be crawled more
> often then others. Like, a news site should be crawled every day, but
> your average business website should be crawled every 30 days.  So, is
> it possible to specify a crawl frequency for specific urls, or is it
> only global for within the crawl db?  I suppose I could have several
> crawldbs or something like that, and deal with it.. but, just curious.
>
> Thanks
> --
> "Conscious decisions by conscious minds are what make reality real"



Mime
View raw message