harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Mazzocchi <stef...@apache.org>
Subject Re: [performance] performance measurement of HDK
Date Tue, 01 Aug 2006 20:29:29 GMT
Vladimir Strigun wrote:
> Our project is now in a good shape and we started encouraging people
> to try real applications on top of Harmony. One of the important parts
> of the feedback will be general impression of stability, reliability
> and performance of the HDK. We are doing pretty good in fixing bugs
> and developing new functionality in HDK, and it might be a good time
> for us to start thinking of performance as well. It might be a little
> bit preliminary, but do we consider having any performance targets for
> us? If yes, we may need to focus at least some of our efforts on
> benchmarking and tuning overall performance of HDK.
> 
> One of the main questions here is what should be the targets for us
> and how should we measure our performance. There are several ways for
> measuring performance, such us commercial benchmarks, free benchmarks,
> application startup, small micro suites, etc. Some of free benchmarks
> have been mentioned in JIRA issues and dev list, nevertheless at the
> moment we don't have any goals for performance. In spite of
> application enabling initiative it might be good to consider publicly
> available benchmarks as the additional list of the software
> applications which we would enable on Harmony.
> 
> So I suggest to start discussing performance techniques and methods
> that can be used for comparing performance between RI and HDK. I think
> in case we do not consider performance issues, we can get negative
> feedback from users even if application starts without any errors,
> exceptions, etc.
> 
> One of the benchmarks that was mentioned is DaCapo[1]. It's a free
> open-source benchmark suite and I believe it can be used for regular
> performance measurement of HDK. I've tried to find other free suites
> and got the following list:
> 
> Telco - this one mostly stresses BigInteger/BigDecimal functionality
> GcOld - the purpose of this one is clear from the name :)
> SciMark - java benchmark for scientific and numerical computing
> Linpack java - well-known benchmark solving linear equations The
> Plasma Benchmark - creates an animated display by continuously summing
> four sine waves in an applet
> JavaWorld Benchmark - benchmark for low-level operations: loops,
> accessing variables, method invocation, arithmetic operators, casting,
> instantiation, exception handling, thread creation and switching.
> CaffeineMark 3.0 - low-level benchmark suite, including sieve of
> Eratosthenes, sorting, logic ops, method invocation, floating point,
> simple graphics and GUI ops
> JavaGrande benchmark suite - a set of benchmarks stressing different
> areas of java.
> 
> Having in mind that the list of publicly available benchmarks is not
> too big, sometimes it will be necessary to create micro benches for
> some of patches (for instance, Harmony-935). IMO micro should be
> started in case we change some code that the bench covers.
> 
> Other interesting and possibly more productive way for comparing
> performance between different implementations are to use non-free
> benchmarks. For instance, we can use benchmarks from Spec[n], like
> SpecJVM, SpecJBB, SpecJAppserver. Unfortunately first we should get
> license for it, but I believe this issue can be solved within the help
> of companies participating in Harmony :)
> 
> Thoughts? Comments?

one thing that happened in mozilla-land that catalized the community in
fixing leaks and performance issues was adding profiling information to
the tests and start plotting them overtime.

Not only that gives an idea of the evolution of the program performance
overtime, but it also keeps people honest because profiling is not
something that should be done once and being forgotten but something
that should be considered part of the feature of the program.

-- 
Stefano.


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message