hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: AbstractNIOConnPool memory leak?
Date Thu, 27 Dec 2012 21:48:32 GMT
On Thu, 2012-12-27 at 12:30 -0800, Sebastiano Vigna wrote:
> On 26 December 2012 13:10, Oleg Kalnichevski <olegk@apache.org> wrote:
> 
> >
> > Just out of curiosity, why are using an asynchronous HTTP client for a
> > web crawler? I personally would consider a blocking HTTP client a much
> > better choice for a heavy duty web crawler.
> >
> 
> Well... how would you manage 10-20K simultaneously opened connections with
> a synchronous client?

But why would you want a web crawler to have 10-20K simultaneously
opened connections in the first place?

Oleg

>  We tried with that number of threads each with a
> DefaultHttpClient, but the problem is that as soon as there is any
> contention (even on logging) it slows down terribly everything.



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message