harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Pasko <egor.pa...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Thu, 23 Nov 2006 10:28:42 GMT
On the 0x228 day of Apache Harmony Stefano Mazzocchi wrote:
> > Our client (default) compilation path was tuned a long time ago and it
> > probably makes sense to have another round. What we initially did was
> > running some script executing the given set of workloads trying to find the
> > best configuration for our VM. Having said that I suggest we choose the
> > right set of applications/benchmarks, so we can start our tuning once
> > again.
> 
> Maybe it's the analog microelectonic guy in me talking, but every time I
> hear something like "let's get reasonable defaults", I think of
> introducing a variation and a feedback to reach a local minimum and
> stabilize the system.
> 
> I know very little about how DRLVM works, but would it be feasible to
> start with such "reasonable defaults" and introduce a random variability
> to the way the JIT works alongside a very simple method profiler and see
> if the performance increase? think of you trying out different things
> and see if they work better... but done by the JVM as it runs.
> 
> Keep in mind I'm a total newbie in virtual machine design (or CPU
> architectures for that matter, despite my degree in microelectronics..
> well, to be fair, I was doing analog not digital circuits) so bear with
> me if I'm saying stupid things :-)

not stupid)) Searching optimal parameters automatically is a good
idea. First of all, we need to find out, what the function is that we
want to minimize. That is, what benchmarks we should rely on when
composing the set of parameters for best-of-all default mode. Then a
genetic algorithm may suit for searching optimal parameters.

AFAIR, DRLVM default mode was tuned to
* HelloWorld program
* Eclipse startup
* some long running benchmarks

In general, there is a simple controversy: fast startup and fast
execution. We want both, everywhere. That is why we have
heuristic-based recompilation. But, sadly, there is no best-for-all
set of parameters yet, and won't be in the nearest future.

We can try to tune DRLVM default mode for HelloWorld + Eclipse +
SciMark + DaCapo, would it be enough for users to be happy with
Harmony on their first glance? I dunno, actually. IMHO, almost all
java users taking care of performance know about client and server
modes and what applications are best for each mode.

> > Currently we have in mind the following list:
> > - HWA (Hello World Application)
> > - SciMark
> > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > - Anything else?
> > 
> > What do you think about this? Any additions to the list? Comments?
> > Questions?
> 
> The problem I have with this is that I feel that each one of such
> scenario might require different tuning parameters... and if that is the
> case, you end up with the 'short blanket' problem: you improve here and
> you decrease there.
> 
> An 'adaptive' scenario, on the other hand, would allow us to:
> 
>  1) avoid trying to find the optimal defaults (since we can't possible
> test every scenario that will be useful in a way that is consistent with
> real world usage)
> 
>  2) avoid the blanket problem, each VM can adapt to the scenario of use
> 
>  3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> changes and 'retune' itself if the environment changes.
> 
> Of course, there is a price to pay in such 'fedback variability' systems
> since they have to find the minima over and over again.
> 
> So, another solution is to have a JVM "tuning parameters discovery mode"
> that you can run and you turn such "parameter finding" autoprofiling
> on... and the JVM dumps the tuning results for you on disk which you can
> later use to initialize the JVM on your own.
> 
> Not sure how feasible or complicated to write this is, but wow does this
> sound on paper?
> 
> 
> > Thanks,
> > ---
> > Sergey Kuksenko
> > Intel Enterprise Solutions Software Division
> > 
> > 
> > On 11/17/06, Stefano Mazzocchi <stefano@apache.org> wrote:
> >>
> >> Alexey Varlamov wrote:
> >> > Stefano,
> >> >
> >> > It is a bit unfair to compare *debug* build of Harmony with other
> >> > release versions :)
> >>
> >> I'm simulating what a journalist with a developer could do.
> >>
> >> If there is a way to make it compile in 'release mode' (if such a thing
> >> exists), I'll be very glad to redo the benchmarks.
> >>
> >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> >> > switches)?
> >>
> >> Right. No switches. I'm simulating what users do when they get the JVM:
> >> they run "java"... and if it's now fast enough they buy a new box.
> >>
> >> Having command line tuning parameters is mostly useless since most
> >> people don't know the internals of a JVM well enough to guess what
> >> parameters to tune anyway.
> >>
> >> So, what people will do once they get an harmony snapshot is "java
> >> my.class.Name <http://my.class.name/>" and see the results.
> >>
> >> I want to simulate that and compare it to the same exact experience they
> >> will get with other virtual machines for a variety of common scenarios
> >> (number crunching, xml processing, http serving, database load, etc...)
> >>
> >> I will focus on the server because that's there the apache action (and
> >> my personal interest) is.
> >>
> >> So, like I said, if there are 'compile time' switches that I can use to
> >> turn 'release mode' on, please tell me and I'll re-do the tests.
> >>
> >> > 2006/11/17, Stefano Mazzocchi <stefano@apache.org>:
> >> >> There are lies, damn lies and benchmarks.... which don't really tell
> >> you
> >> >> if an implementation of a program is *faster* but at least it tells
> >> you
> >> >> where you're at.
> >> >>
> >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM,
I
> >> >> was able to start running some benchmarks.
> >> >>
> >> >> The machine is the following:
> >> >>
> >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep
16
> >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >> >>
> >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> >> bogomips 6410.31 (per CPU)
> >> >>
> >> >> There is nothing else running on the machine (load is 0.04 at the time
> >> >> of testing).
> >> >>
> >> >> The various virtual machines tested are:
> >> >>
> >> >> harmony
> >> >> -------
> >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> >> Foundation or its licensors, as applicable.
> >> >> java version " 1.5.0"
> >> >> pre-alpha : not complete or compatible
> >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >> >>
> >> >> sun5
> >> >> ---
> >> >> java version "1.5.0_09 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >> >>
> >> >> sun6
> >> >> ----
> >> >> java version " 1.6.0-rc"
> >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >> >>
> >> >> ibm
> >> >> ---
> >> >> java version " 1.5.0"
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> >> pxa64dev-20061002a (SR3) )
> >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> >> j9vmxa6423-20061001 (JIT enabled)
> >> >> J9VM - 20060915_08260_LHdSMr
> >> >> JIT  - 20060908_1811_r8
> >> >> GC   - 20060906_AA)
> >> >> JCL  - 20061002
> >> >>
> >> >> bea
> >> >> ---
> >> >> java version "1.5.0_06 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> >> >> BEA JRockit(R) (build
> >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >> >>
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >> >>
> >> >> command: java jnt.scimark2.commandline
> >> >>
> >> >> NOTE: bigger number is better
> >> >>
> >> >> Sun6
> >> >> Composite Score: 364.5832265230057
> >> >> FFT (1024): 220.8458713892794
> >> >> SOR (100x100):   696.1542342357722
> >> >> Monte Carlo : 149.37978088875656
> >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> >> LU (100x100): 430.1617273683819
> >> >>
> >> >> BEA
> >> >> Composite Score: 359.13480378697835
> >> >> FFT (1024): 303.8746880751562
> >> >> SOR (100x100):   454.25628897202307
> >> >> Monte Carlo : 93.23913192138497
> >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> >> LU (100x100): 413.8627835924175
> >> >>
> >> >> Sun5
> >> >> Composite Score: 332.84987587548574
> >> >> FFT (1024): 216.5144595799027
> >> >> SOR (100x100):   689.429322146947
> >> >> Monte Carlo : 25.791262124978065
> >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> >> LU (100x100): 414.99493895566377
> >> >>
> >> >> IBM
> >> >> Composite Score: 259.8249218693683
> >> >> FFT (1024): 296.8415012789055
> >> >> SOR (100x100):   428.974881649179
> >> >> Monte Carlo : 89.15159857584082
> >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> >> LU (100x100): 339.8042037225181
> >> >>
> >> >> Harmony
> >> >> Composite Score: 113.65082278962575
> >> >> FFT (1024): 203.76641991778123
> >> >> SOR (100x100):   224.37761309236748
> >> >> Monte Carlo : 9.063866256533116
> >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> >> LU (100x100): 65.6410280487242
> >> >>
> >> >> In this test harmony is clearly lagging behind... at about 30%
> >> >> performance of the best JVM, it's a little crappy. Please note how
> >> FFT's
> >> >> performance is not so bad awhile monte carlo is pretty bad compared
to
> >> >> BEA or IBM.
> >> >>
> >> >> Overall, it seems like there is some serious work to do here to catch
> >> up.
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 2: Dhrystones
> >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> >> )
> >> >>
> >> >> command: java dhry 100000000
> >> >>
> >> >> NOTE: bigger is better
> >> >>
> >> >> NB: I modified the code to accept the count at input from the command
> >> >> line!
> >> >>
> >> >> sun6:     8552856 dhrystones/sec
> >> >> sun5:     6605892
> >> >> bea:      5678914
> >> >> harmony:   669734
> >> >> ibm:       501562
> >> >>
> >> >> The performance here is horrific but what's surprising is that J9 is
> >> >> even worse. No idea what's going on but it seems like something is
not
> >> >> working as it should (in both harmony and J9)
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >> >>
> >> >> command: java Sieve 30
> >> >>
> >> >> NB: I modified the test to run for a configurable amount of seconds.
> >> >>
> >> >> sun6     8545 sieves/sec
> >> >> sun5     8364
> >> >> bea      6174
> >> >> harmony  1836
> >> >> ibm       225
> >> >>
> >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> >> lagging behind.
> >> >>
> >> >> Stay tuned for more tests.
> >> >>
> >> >> --
> >> >> Stefano.
> >> >>
> >> >>
> >>
> >>
> >> -- 
> >> Stefano.
> >>
> >>
> > 
> 
> 
> -- 
> Stefano.
> 
> 

-- 
Egor Pasko


Mime
View raw message