Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <1f3854d50806130734p384c6361v22aa5df0323c97d@mail.gmail.com>
Date: Fri, 13 Jun 2008 16:34:51 +0200
From: "Quintin Beukes" <quintin@last.za.net>
Sender: quintin@skywalk.co.za
To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Subject: Re: Higher Throughputs
In-Reply-To: <1213361635.6206.21.camel@ubuntu>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <1f3854d50806121307w4bdb5cddp51b670838015e444@mail.gmail.com>
	 <48518ED9.1060502@apache.org>
	 <1f3854d50806122335o2779f5d4ifccd285158ba57b9@mail.gmail.com>
	 <1213344477.6206.9.camel@ubuntu>
	 <1f3854d50806130118n39f5afc4lf1f8aec16baa5717@mail.gmail.com>
	 <1213348987.6206.14.camel@ubuntu>
	 <1f3854d50806130256y28b157fcnccea39aac2422473@mail.gmail.com>
	 <1213361635.6206.21.camel@ubuntu>

How would I increase this?

Q

On Fri, Jun 13, 2008 at 2:53 PM, Oleg Kalnichevski <olegk@apache.org> wrote:
> On Fri, 2008-06-13 at 11:56 +0200, Quintin Beukes wrote:
>> Not getting expected rate when running inside tomcat.
>>
>> Here are my benchmark results for running as an application (direct
>> connections):
>> +++++++
>> Benchmark 'Benchmark http clients' results.
>> -------------------------------------------
>> Total time taken in milliseconds:
>> HttpClient 4.x: 234337
>> HttpCore 4.x: 199655
>> HttpURLConnection: 209033
>>
>> Total bytes transfered: 10488590000
>>
>> Benchmark: HttpURLConnection
>> Average transfer rate (bytes/second): 50176720
>>
>> Benchmark: HttpCore 4.x
>> Average transfer rate (bytes/second): 52533572
>>
>> Benchmark: HttpClient 4.x
>> Average transfer rate (bytes/second): 44758576
>> +++++++
>>
>> Up to 52.5 MB/s using HttpCore. HttpClient being the slowest in this
>> case. This was doing 10000 connections each downloading a 1 MB static
>> file from Apache over a Gbit ethernet connection.
>>
>> Through Tomcat I was topping 8mb/s using the current HttpCore layout.
>> But I'll let you know where the exact problem lies. I'm assuming it
>> has something to do with the way I am writing output to tomcat's
>> output stream. Tomcat can definitely handle super speeds as well (with
>> static files), so I'm sure there is a way to get this all be faster.
>>
>
> One important thing to watch out for is the proper sizing of the
> HttpClient connection pool. Per default HttpClient allows only two
> concurrent connections to the same host. So if you are hitting Tomcat
> with 100 concurrent connections, you are very likely to end with 100
> Tomcat worker threads contending over 2 HttpClient connections. This
> setup will not scale well for obvious reasons. Make sure HttpClient
> allocates enough outbound client side connections for all inbound server
> side connections.
>
> Hope this helps
>
> Oleg
>
>> If you want the above benchmark code, you can get it .class and .java
>> in the following to jars:
>> http://quintin.dev.junkmail.co.za/httpclient/new/benchlib.jar
>> http://quintin.dev.junkmail.co.za/httpclient/new/benchhttp.jar
>>
>> I use the following command to run it (also shows the versions of
>> httpclient/commons I'm using):
>> java -cp benchhttp.jar:benchlib.jar:libs/commons-codec-1.3.jar:libs/commons-logging-1.0.4.jar:libs/httpclient-4.0-alpha4.jar:libs/httpcore-4.0-beta1.jar:libs/httpcore-nio-4.0-beta1.jar:libs/httpmime-4.0-alpha4.jar
>> test.AppBench
>>
>> Q
>>
>> On Fri, Jun 13, 2008 at 11:23 AM, Oleg Kalnichevski <olegk@apache.org> wrote:
>> > On Fri, 2008-06-13 at 10:18 +0200, Quintin Beukes wrote:
>> >> I'm doing ethernet. Localhost is also quite unpredictable, as it can
>> >> go too fast. Doing ethernet I know what the limits are, ie. 100mbit or
>> >> 1gbit.
>> >>
>> >> Eitherway, I setup some benchmarks and got very high troughputs. I
>> >> also spent some time figuring out HttpCore, so it's much faster than
>> >> all the others now.
>> >>
>> >> My question is then more of the sort, what could the reason for this
>> >> be? It's probably not HttpClient's fault though. Any ideas how I can
>> >> find the bottleneck?
>> >>
>> >
>> > I am not sure I understand the problem. Is it that HttpClient tends to
>> > be slower than HttpCore or that you are not getting the expected
>> > throughput rate when running HttpClient inside Tomcat?
>> >
>> > Oleg
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> >
>> >
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>


-- 
Quintin Beukes

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org