nutch-agent mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniele Menozzi <>
Subject Re: Pages/s rate decreasing
Date Mon, 26 Sep 2005 17:24:15 GMT
On  13:01:15 26/Sep , Jay Pound wrote:
> I crawl with 100 threads, but you need to have 10% available for outbound
> traffic so if your connection is a cablemodem with only 512kb upload you
> will not be able to download at 10mbit but 5mbit, your numbers are low, most

I have also 10Mb in upload..
What machine do you usually use?

> of the machines I run can crawl 30pages a sec with 3mbit 2xT1's and 42pages
> a sec with 9mbit cablemodem connection, just run more threads, that is if
> your hardware can handle it.

but I thought my machine was powerful enought to run 50/100 threads.. I
cannot understand, because the cpu isn't really used, an even memory is
quite unused...

> you can also try and run the -noparse flag and
> parse the data later or on a different faster machine!!!

ok, I'll try with this,but it seems to have no relevant effect..

Thank you!!!

View raw message