commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glenn Nielsen <gl...@mail.more.net>
Subject Re: [TBM] Thread Benchmark example test results
Date Wed, 10 Dec 2003 16:13:43 GMT
Brian S O'Neill wrote:
> The BenchTest has a few flaws. I wrote a few new tests and was shocked 
> by the numbers. Running with two threads, each with half the iterations 
> was over one hundred times slower than the single thread test. This 
> makes no sense.
> 

Yes, this is a work in progress.  Thanks for the feedback.

> The main problem is the warmup thread. After you call join on the 
> thread, it is useless. I would expect calling start again to throw an 
> exception, but instead nothing seems to happen. Either way, thread 0 is 
> always dead. This throws off the results for the single thread test, and 
> the normalized results are way off too.

Not doing a warmup could also impact the test results.  It would increase
the time it takes for the single thread test to complete due to HotSpot
optimizing the code.

I modified the code to instantiate a seperate object for doing the warmup.
When I ran the tests it verified your conclusion that thread[0] was not being
executed.  Wierd, that is not what I would have expected.  Do you know why
this is the case?

> 
> The second problem is including garbage collection times as part of the 
> test. I don't recommend forcing a call to Runtime.gc(). If you're trying 
> to determine how much time is spent collecting garbage, run with 
> verbose:gc. If you want to see how many objects are allocated, run with 
> a profiler.

If you run these tests with a profiler that could also impact the results.
The instrumentation the profiler uses will affect the benchmark results.

Yes, trying to get data on the impact of GC is problematic, I mention
that in the docs.  There are many other things which can impact the
performance such as using different JVM args like -server, heap size,
gc options.  Getting accurate data on the impact of GC may require
capturing the output from -verbose:gc and post processing it.

I still think it is a good idea to try and include in the benchmark
tests the impact of GC on performance.

> 
> When I commented out the warmup section and all the gc() related calls, 
> the results were more like what I would expect. Doubling the threads and 
> halving the iterations resulted in performance which was the same or 
> only slightly worse. After these changes, here are my results:
> 

GC does have a significant impact on performance.  Not performing the
Full GC  prior to starting the test could impact the test results.
For example, you execute a series of benchmark tests, during the
execution of the single thread for a benchmark a Full GC is triggered.
That would significantly impact the time taken to execute the
benchmark with a single thread.

Developing accurate benchmarks is challenging, especially when you
are trying to benchmark java code running within a JVM.

Regards,

Glenn


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org


Mime
View raw message