tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pier P. Fumagalli" <p...@apache.org>
Subject Re: Catalina/Tomcat performance/functionality
Date Tue, 15 Aug 2000 20:30:33 GMT
Roy Wilson wrote:
> 
> Since I am not (yet) a Tomcat/Catalina developer, there may not be
> much reason to take these performance related questions too seriously,
> if at all. I hope, however, they might prove useful.

I believe you didn't really understand what we mean by performance...

> In the Catalina proposal, the following comments were made concerning
> performance.
> 
> "No formal benchmarks have been published by the Tomcat development
> team describing the performance of Tomcat 3.x or Catalina."

Yes, and that's true. I know that some of us ran some performance tests
stressing the different web servers and different http stacks, but we
don't yet have a formal performance report.

> QUESTION 1: What is meant by "formal" benchmark? See below on data
> collected (but not on Tomcat) using ApacheBench: is this a formal
> benchmark, an informal benchmark, or neither?

What you did is not even a benchmark :) What you measured is the latency
time between codeflux.com and jakarta.apache.org, over the INTERNET...
Just a simple example: if I try to run ApacheBench on my machine using
as URL my host name (http://www.betaversion.org/) I get these results:

Server Software:       
Apache/1.3.12                                      
Server Hostname:        www.betaversion.org
Server Port:            80
Document Path:          /
Document Length:        1358 bytes
Concurrency Level:      1
Time taken for tests:   0.384 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      17810 bytes
HTML transferred:       13580 bytes
Requests per second:    26.04
Transfer rate:          46.38 kb/s received

Running the _same_ test from the codeflux gateway I get:

Server Software:        Apache/1.3.12
Server Hostname:        www.betaversion.org
Server Port:            80
Document Path:          /
Document Length:        1358 bytes
Concurrency Level:      1
Time taken for tests:   6.470 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      17810 bytes
HTML transferred:       13580 bytes
Requests per second:    1.55
Transfer rate:          2.75 kb/s received

WHHHATTTTT??? Am I nuts? from 26 requests per seconds to 1.55? What's
wrong with my site? Nothing it's my test methodology. In the second case
I am measuring network latency.

> "(In addition, I would take any benchmark that purports to predict how
> *your* application will perform on a particular server with a very
> large grain of salt :-)."

Well, as far as I can see, our benchmarks are more accurate than yours.

> I agree: this is a big can of worms (to be opened later, if ever).

No, it's not... Performance testin (as Mike Pogue from IBM docet) is a
very intricate matter, but certainly, in some situations, can be
resolved. Of course we cannot say which server or servlet engine is
faster, but we can make real-world examples and give a test methodology
that in your environment can be followed and used by you to determine
what's best for your needs.

> "However, simple tests with tools like ApacheBench and Microsoft's
> stress test indicate that Catalina is faster than Tomcat on serving
> static resources (substantially faster if caching is enabled,
> basically on par with Apache itself), and within a few percentage
> points on serving "Hello, World" type servlets and JSP pages."
> 
> It would be nice to see the results of at least the ApacheBench test
> since ApacheBench is potentially available to all developers via the
> ApacheBench Gateway at codeflux.com.

CodeFlux is (as the name says) a gateway that merely measures the
network latency times between their host and whatever URL you enter...
It's not a web-server benchmark, it's a network benchmark.

> [... skipped digression over codeflux tests...]

> QUESTION 2: Using the criteria given above, does this mean that the
> Apache web server is not a "good" web server? (I understand that, per
> the goals of the ASF, functionality should dominate performance).

Given that the test metodology is wrong this question is not pertinent.
Try to re run YOUR test with this URL:
<http://codeflux.com/images/tell.gif>
Apache 1.3.4 processes approximately 36 requests per seconds... Whoha!

> QUESTION 3: Does Catalina have any performance REQUIREMENTS? On page
> 12 of "Java Platform Performance", Steve Wilson and Jeff Kesselman of
> Sun suggest the need to "reach a consensus about the performance
> requirements for your system. If you don't, how will you know when
> your product is fast enough". If such advice doesn't apply here, why
> not?

I don't agree with Wilson and Kesselman on that. Our requirements (after
of course our requirements of being compliant to the spec and bug-free)
are to be "faster". And the codebase is clean enough to see where our
performance weak points are. We know how current VMs are working, what
is an overhead and what is not. I believe we can achieve very good
results only looking at our code...

> [... skipped descriptions of what's in the files...]
> 
> QUESTION 4: will Catalina traffic look more like URL1 or URL2 or some
> mix of the two or some other mix.

None of the above. You're measuring the network.

> QUESTION 5: what will be the consequences of answering question 4 for
> user-level performance?

With your test-case, just buy more bandwidth, and sit your host on the
AXXS network (http://www.axxs.net/)

> "Only modest efforts to tune Catalina have taken place to date
> (compared to the substantial -- and successful -- tuning efforts
> expended between Tomcat 3.1 and 3.2), so it can be assumed that there
> is still significant room for improvement through the usual techniques
> of isolating hot spots and optimizing them."
> 
> While I agree that this is the usual technique, there are those who
> suggest it may not be enough. Some suggest that performance is
> functionality: if it takes 10 minutes to login to a system, it isn't
> functional in an meaningful sense (I am not predicting that such an
> event would occur, only noting that it did occur elsewhere). In the
> horror story I mentioned in an earlier post, there were no
> REQUIREMENTS up front. It was assumed that intelligent design, coding,
> tuning. and a bigger pipe/hammer/etc. would be enough. They weren't.
> Sorry for the preachiness.: please educate me.

As I told before, maybe we don't have _requirements_ bu we know what's
going on and how to improve our performances. We just need programmers
time to go in and optimize.

> If there is a version of Tomcat or Catalina that is up that can be
> accessed using the codeflux.com ApacheBench Gateway, I am happy to
> volunteer for some additional monkey-test work :-) like that done in
> producing the data shown in the attachment.

Don't worry. Using codeflux is completely useless in terms of
benchmarking the internals of any web server.

	Pier

Mime
View raw message