hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Higher Throughputs
Date Thu, 12 Jun 2008 21:02:17 GMT
Quintin Beukes wrote:
> Hey,
> I made some basic servlets to test throughputs for HttpClient,
> HttpCore and sun's sun.net.www.protocol.http.HttpURLConnection (or
> something along those lines, I don't have an API at hand now).
> I basically just put up an apache session listening on port 8580, and
> then Tomcat on port 80. In tomcat I dumped 3 servlets, each using one
> of the 3 methods to retrieve a 51kb page from the apache session.
> The benchmarks was done with "ab" with 10 second timeouts. The results
> are as follows (in byte readings, not bit readings).
> Doing a direct request to apache I get a throughput of about 33mb/s.
> Using HttpURLConnection, 4.7MB/s.
> Using HttpCore 3.6MB/s
> Using HttpClient 6.03mb/s
> The request times were in general best with apache/HttpClient, maxing
> at 38ms for the longest request.
> This was done at a concurrency of 1 connection. Pushing this up to 5,
> the HttpCore's connection times stayed the same, but throughput
> dropped to 980kb/s. HttpClient's connection times now came to the same
> as HttpCore (230ms for longest request), but it's throughput increased
> to 7.1mb/s.
> This is all good and well, but my question is this. Are there any ways
> I can increase these throughputs? Before just jumping into
> optimizations suggestions and referals to documentation, could someone
> perhaps just enlighten me into why these speeds are so different. And
> especially so low?

I do not want to jump into conclusions here, but I seriously suspect 
something is wrong with your test cases. In particular I do not 
understand why you are using Tomcat to host the code your are testing. 
What is the point of that?

I made an extensive comparison of HttpCore vs HttpClient (both versions 
3.1 and 4.0). HttpCore can easily achieve a throughput of 10,000 (small) 
   requests per second on a fairly average hardware.


> I am obviously using HttpCore wrong for it to be so slow (I mean,
> HttpClient is built on core, so it's impossible to be faster). It's
> just that the documentation for these are sparse, and I'm having a
> very hard time finding my way around.
> And beyond this, Tomcat itself gives me fantastic speeds reading
> static data. Where are the bottlenecks?
> The HttpCore setup I used was the "Basic GET Requests" example
> available on the site. I modified it a bit to allow URL input and
> target server specified through parameters passed to the servlet.
> Beyond that I didn't change anything. My assumption is that
> connections aren't being reused. In fact, I know this is the case, so
> it logically and obviously removes the validity of the above
> benchmarks for HttpCore. I'm not going to remove them though.
> I would really like to get throughputs similar to direct Apache
> connections (at least 33+ mbyte/s, which is round about the general
> limit of hard drive transfer rates). Full ethernet throughput is
> preferable and I'm sure possible. Downloading large files I get speeds
> of up to 54mbyte/s with Apache. I'm sure this is limited only to the
> hard drive I tested with, as it's a gigabit connection, and this is
> just less than half it's full potential. I haven't done tests using
> memory transfer. Will try it when I can diagnose the bottlenecks I
> currently have, and then use them as measurements to gain insights
> into the speed of the program itself by taking as much of the other
> bottlenecks in the system away as possible (in this case the
> limitation of data transfer rates from the hard drives).
> The three benchmark servlets I used are here:
> HttpClient: http://quintin.dev.junkmail.co.za/httpclient/Bench.java
> HttpCore: http://quintin.dev.junkmail.co.za/httpclient/BenchCore.java
> HttpURLConn: http://quintin.dev.junkmail.co.za/httpclient/BenchURL.java
> And a utility class I use in the above:
> http://quintin.dev.junkmail.co.za/httpclient/StreamBinder.java
> Any insights would be greatly appreciated.

I'll try to look at the benchmark code over the weekend. But I would 
_strongly_ suggest measuring the performance of each respective 
framework by using direct connections to the target server. Once you 
have established the baseline you can work on optimizing the code when 
running inside a servlet container.


> regards,

To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

View raw message