Return-Path: X-Original-To: apmail-hc-httpclient-users-archive@www.apache.org Delivered-To: apmail-hc-httpclient-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 98C7AE78D for ; Sat, 5 Jan 2013 21:34:01 +0000 (UTC) Received: (qmail 38730 invoked by uid 500); 5 Jan 2013 21:34:01 -0000 Delivered-To: apmail-hc-httpclient-users-archive@hc.apache.org Received: (qmail 38691 invoked by uid 500); 5 Jan 2013 21:34:01 -0000 Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "HttpClient User Discussion" Delivered-To: mailing list httpclient-users@hc.apache.org Received: (qmail 38679 invoked by uid 99); 5 Jan 2013 21:34:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jan 2013 21:34:01 +0000 X-ASF-Spam-Status: No, hits=1.0 required=5.0 tests=SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of vigna@di.unimi.it does not designate 216.139.250.139 as permitted sender) Received: from [216.139.250.139] (HELO joe.nabble.com) (216.139.250.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jan 2013 21:33:55 +0000 Received: from tom.nabble.com ([192.168.236.105]) by joe.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1TrbN0-0000hq-CO for httpclient-users@hc.apache.org; Sat, 05 Jan 2013 13:33:34 -0800 Date: Sat, 5 Jan 2013 13:33:34 -0800 (PST) From: vigna To: httpclient-users@hc.apache.org Message-ID: <1357421614373-18641.post@n7.nabble.com> In-Reply-To: <1356644912.1167.1.camel@ubuntu> References: <1356284256900-18554.post@n7.nabble.com> <1356316434027-18555.post@n7.nabble.com> <1356349223.9005.16.camel@ubuntu> <1356411707335-18567.post@n7.nabble.com> <1356437653.19934.4.camel@ubuntu> <1356454434981-18569.post@n7.nabble.com> <1356556228.20672.8.camel@ubuntu> <1356644912.1167.1.camel@ubuntu> Subject: Re: AbstractNIOConnPool memory leak? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org > But why would you want a web crawler to have 10-20K simultaneously > opened connections in the first place? (I thought I answered this, but it's not on the archive. Boh.) Having a few thousands connection open is the only way to retrieve data respecting politeness (e.g., not banging the same site too often). I have another question: is there any suggestion for parameters of the asynchronous client in case of several thousands parallel requests (e.g., for the IOReactor)? We are experimenting both with DefaulHttpClient and DefaultHttpAsyncClient, and with the same configuration (e.g., 4000 threads using DefaultHttpClient or 64 threads pushing 4000 async requests into a default DefaultHttpAsyncClient) we see completely different behaviours. The sync client fetches more than 10000 pages/s, the async client speed fetches 50 p/s. Should we increase the number of threads or the I/O interval of the IOReactor? -- View this message in context: http://httpcomponents.10934.n7.nabble.com/AbstractNIOConnPool-memory-leak-tp18554p18641.html Sent from the HttpClient-User mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org For additional commands, e-mail: httpclient-users-help@hc.apache.org