tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vicenc Beltran Querol <vbelt...@ac.upc.edu>
Subject Re: Hybrid (NIO+Multithread, SSL enabled) architecture for Coyote
Date Tue, 24 May 2005 15:28:20 GMT
Hi,

I've repeated the tests on the hybrid architecture using the AB.
You can find them attached to this mail. I've run the AB with several 
concurrency levels, ranging from 20 to 10000. You can see all the
results in a plot.


> Running a test with ab (ab -k -c 20 -n 20000 
> http://host:8080/tomcat.gif) would take 30s, and would make comparisons 
> easy (basically, I actually know what the tests do ...), and will be an 
> actual measurement of throughput.


I've been studying the behavior of the AB and I've several doubts 
about the significance of the results when trying to measure the throughtput
of a server. In my opinion, the AB is a great way to test the impact of
architectural modifications on the internal latency of the tomcat execution
pipeline, but not a deterministic way to compare the throughput of two servers.
In the following paragraphs I try to justify this idea (hope that in a
comprensible way) :)


The first thing that makes me suspect about the reliability of the obtained results
is that this benchmark produces a different workload intensity for each tested server.
I mean that, given a number of concurrent clients simulated, the AB produces a higher
number of requests as lower is the response time for the server (a new request is issued
when the previous is completed). This behavior will always favour an architecture
with lower internal latencies, even when it manages concurrency worse. This is the case of
the tomcat multithreaded architecture. Other architectures, for instance the Hybrid or any
other using non-blocking operations with readiness selectors, will always obtain 
worse results for low loads (remember that the select operation introduces an internal
latency of 15-30ms since data is ready in a channel, with the purpose of getting more
channels ready during that period). 
When the simulated number of concurrent clients is increased (and especially
when the number of threads in the pool is lower than the number of emulated
clients), the multithreaded architecture starts suffering. You can check
the plots for the throughput, number of keep-alive requests, errors or connect
time to create your own opinion.


In conclusion, it must be taken into account that using this benchmark to compare
the throughput of several architectural proposals can lead to wrong conclusions,
especially when WANs (instead of fast LANs) are used for the evaluation.

Let me explain with a simple example, assuming that the AB simulated 10 concurrent
clients...

Lets suppose two architectures, A and B:

A -> Internal service time for the tomcat.gif file: 1ms
B -> Internal service time for the tomcat.gif file: 5ms


And lets suppose two different scenarios:

First, assuming a zero network latency... (in practice, a Gbit LAN)
 - The observed throughput for A will be 10000 replies/sec (10*(1/10^-3))
 - The observed throughput for B will be  2000 replies/sec (10*(5/10^-3))
 - The speedup observed between A and B is 5.


Later, assuming a  200 ms network latency... (in practice, a WAN)
 - The observed throughput for A will be  49,75 replies/sec (10*(1/(10^-3 + 200^-3)))
 - The observed throughput for B will be  48,78 replies/sec (10*(1/(5^-3 + 200^-3)))
 - The speedup observed between A and B is 1,019.


This reasoning indicates that this test is more precise to compare the response
time between two server architectures than to evaluate its performance, because
the network latency (between the server and the client) can bias the obtained results.


Finally, I miss a "think-time" as one of the configuration parameters of the AB. It
reduces the "realism" of the test and makes not possible to test the performance 
of the server in terms of "user sessions" instead of individual requests.

> 
> Note: your patch still seems bad, as the file add is represented as a 
> diff. This is likely not going to be patcheable.

I'm very sorry but I missunderstood you again... :(

Again... do I send the modified files "as is", instead of a diff?


Thanks for the comments,

Vicenç

PS: I'm very interested in your opinion (I mean the community) about my reasoning
about the adequate use of the AB for throughput comparisons...




> 
> Rémy
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-dev-help@jakarta.apache.org


Mime
View raw message