Mailing-List: contact tomcat-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <tomcat-dev@jakarta.apache.org>
Message-ID: <42934E73.4030507@apache.org>
Date: Tue, 24 May 2005 17:55:31 +0200
From: Remy Maucherat <remm@apache.org>
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
MIME-Version: 1.0
To: Tomcat Developers List <tomcat-dev@jakarta.apache.org>
Subject: Re: Hybrid (NIO+Multithread, SSL enabled) architecture for Coyote
References: <1116582672.20510.23.camel@pcpujol> <428DB67F.9030805@apache.org>
 <20050523142052.GA30748@ac.upc.edu> <429224A4.6070704@apache.org>
 <20050524152820.GA24522@ac.upc.edu>
In-Reply-To: <20050524152820.GA24522@ac.upc.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Vicenc Beltran Querol wrote:
> Hi,
> 
> I've repeated the tests on the hybrid architecture using the AB.
> You can find them attached to this mail. I've run the AB with several 
> concurrency levels, ranging from 20 to 10000. You can see all the
> results in a plot.

-c 20 -k is basically the only thing I am interested in. This is not a 
realistic test, it just measures the raw performance rather than the 
scalability.

About your previous bench results: obviously the performance of the 
regular HTTP connector is going to suck once the amount of connections 
exceeds maxThreads. As most threaded servers, it scales by increasing 
the amount of threads, and I believe it will perform relatively well (at 
the expense of resources).

>>Running a test with ab (ab -k -c 20 -n 20000 
>>http://host:8080/tomcat.gif) would take 30s, and would make comparisons 
>>easy (basically, I actually know what the tests do ...), and will be an 
>>actual measurement of throughput.
> 
> I've been studying the behavior of the AB and I've several doubts 
> about the significance of the results when trying to measure the throughtput
> of a server. In my opinion, the AB is a great way to test the impact of
> architectural modifications on the internal latency of the tomcat execution
> pipeline, but not a deterministic way to compare the throughput of two servers.
> In the following paragraphs I try to justify this idea (hope that in a
> comprensible way) :)

Yes, you need to use ab to test either in localhost, or using a gigabit 
network.

> The first thing that makes me suspect about the reliability of the obtained results
> is that this benchmark produces a different workload intensity for each tested server.
> I mean that, given a number of concurrent clients simulated, the AB produces a higher
> number of requests as lower is the response time for the server (a new request is issued
> when the previous is completed). This behavior will always favour an architecture
> with lower internal latencies, even when it manages concurrency worse. This is the case of
> the tomcat multithreaded architecture. Other architectures, for instance the Hybrid or any
> other using non-blocking operations with readiness selectors, will always obtain 
> worse results for low loads (remember that the select operation introduces an internal
> latency of 15-30ms since data is ready in a channel, with the purpose of getting more
> channels ready during that period). 
> When the simulated number of concurrent clients is increased (and especially
> when the number of threads in the pool is lower than the number of emulated
> clients), the multithreaded architecture starts suffering. You can check
> the plots for the throughput, number of keep-alive requests, errors or connect
> time to create your own opinion.

Well, that's precisely the reason why we never used non blocking IO in 
the past :)

> In conclusion, it must be taken into account that using this benchmark to compare
> the throughput of several architectural proposals can lead to wrong conclusions,
> especially when WANs (instead of fast LANs) are used for the evaluation.

Yes.

> This reasoning indicates that this test is more precise to compare the response
> time between two server architectures than to evaluate its performance, because
> the network latency (between the server and the client) can bias the obtained results.
> 
> Finally, I miss a "think-time" as one of the configuration parameters of the AB. It
> reduces the "realism" of the test and makes not possible to test the performance 
> of the server in terms of "user sessions" instead of individual requests.

Again, I am not interested in a real world test here like you would do 
on your server when putting it in production, and where you want to see 
if your app reaches its performance targets, but a measurement of raw 
performance.

> PS: I'm very interested in your opinion (I mean the community) about my reasoning
> about the adequate use of the AB for throughput comparisons...

I don't see any results in your email, BTW.

R�my

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org