harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Varlamov" <alexey.v.varla...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Thu, 23 Nov 2006 11:32:45 GMT
[snip]
> We can try to tune DRLVM default mode for HelloWorld + Eclipse +
> SciMark + DaCapo, would it be enough for users to be happy with
> Harmony on their first glance? I dunno, actually. IMHO, almost all
> java users taking care of performance know about client and server
> modes and what applications are best for each mode.

Err, SciMark looks quite specific in this list, and I suspect it will
provoke that "short blanket" scenario. Probably we should not include
it for the default mode tuning.
As a side note, any user cares about performance but not all dare to
tune it. This is the main motivation for measuring OOB at the first
place...

>
> > > Currently we have in mind the following list:
> > > - HWA (Hello World Application)
> > > - SciMark
> > > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > > - Anything else?
> > >
> > > What do you think about this? Any additions to the list? Comments?
> > > Questions?
> >
> > The problem I have with this is that I feel that each one of such
> > scenario might require different tuning parameters... and if that is the
> > case, you end up with the 'short blanket' problem: you improve here and
> > you decrease there.
> >
> > An 'adaptive' scenario, on the other hand, would allow us to:
> >
> >  1) avoid trying to find the optimal defaults (since we can't possible
> > test every scenario that will be useful in a way that is consistent with
> > real world usage)
> >
> >  2) avoid the blanket problem, each VM can adapt to the scenario of use
> >
> >  3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> > changes and 'retune' itself if the environment changes.
> >
> > Of course, there is a price to pay in such 'fedback variability' systems
> > since they have to find the minima over and over again.
> >
> > So, another solution is to have a JVM "tuning parameters discovery mode"
> > that you can run and you turn such "parameter finding" autoprofiling
> > on... and the JVM dumps the tuning results for you on disk which you can
> > later use to initialize the JVM on your own.
> >
> > Not sure how feasible or complicated to write this is, but wow does this
> > sound on paper?
> >
> >
> > > Thanks,
> > > ---
> > > Sergey Kuksenko
> > > Intel Enterprise Solutions Software Division
> > >
> > >
> > > On 11/17/06, Stefano Mazzocchi <stefano@apache.org> wrote:
> > >>
> > >> Alexey Varlamov wrote:
> > >> > Stefano,
> > >> >
> > >> > It is a bit unfair to compare *debug* build of Harmony with other
> > >> > release versions :)
> > >>
> > >> I'm simulating what a journalist with a developer could do.
> > >>
> > >> If there is a way to make it compile in 'release mode' (if such a thing
> > >> exists), I'll be very glad to redo the benchmarks.
> > >>
> > >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> > >> > switches)?
> > >>
> > >> Right. No switches. I'm simulating what users do when they get the JVM:
> > >> they run "java"... and if it's now fast enough they buy a new box.
> > >>
> > >> Having command line tuning parameters is mostly useless since most
> > >> people don't know the internals of a JVM well enough to guess what
> > >> parameters to tune anyway.
> > >>
> > >> So, what people will do once they get an harmony snapshot is "java
> > >> my.class.Name <http://my.class.name/>" and see the results.
> > >>
> > >> I want to simulate that and compare it to the same exact experience they
> > >> will get with other virtual machines for a variety of common scenarios
> > >> (number crunching, xml processing, http serving, database load, etc...)
> > >>
> > >> I will focus on the server because that's there the apache action (and
> > >> my personal interest) is.
> > >>
> > >> So, like I said, if there are 'compile time' switches that I can use to
> > >> turn 'release mode' on, please tell me and I'll re-do the tests.
> > >>
> > >> > 2006/11/17, Stefano Mazzocchi <stefano@apache.org>:
> > >> >> There are lies, damn lies and benchmarks.... which don't really
tell
> > >> you
> > >> >> if an implementation of a program is *faster* but at least it
tells
> > >> you
> > >> >> where you're at.
> > >> >>
> > >> >> So, as Geir managed to get the DSO linking problem go away in
DRLVM, I
> > >> >> was able to start running some benchmarks.
> > >> >>
> > >> >> The machine is the following:
> > >> >>
> > >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat
Sep 16
> > >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> > >> >>
> > >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> > >> >> bogomips 6410.31 (per CPU)
> > >> >>
> > >> >> There is nothing else running on the machine (load is 0.04 at
the time
> > >> >> of testing).
> > >> >>
> > >> >> The various virtual machines tested are:
> > >> >>
> > >> >> harmony
> > >> >> -------
> > >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache
Software
> > >> >> Foundation or its licensors, as applicable.
> > >> >> java version " 1.5.0"
> > >> >> pre-alpha : not complete or compatible
> > >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> > >> >>
> > >> >> sun5
> > >> >> ---
> > >> >> java version "1.5.0_09 "
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> > >> >>
> > >> >> sun6
> > >> >> ----
> > >> >> java version " 1.6.0-rc"
> > >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed
mode)
> > >> >>
> > >> >> ibm
> > >> >> ---
> > >> >> java version " 1.5.0"
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> > >> >> pxa64dev-20061002a (SR3) )
> > >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > >> >> j9vmxa6423-20061001 (JIT enabled)
> > >> >> J9VM - 20060915_08260_LHdSMr
> > >> >> JIT  - 20060908_1811_r8
> > >> >> GC   - 20060906_AA)
> > >> >> JCL  - 20061002
> > >> >>
> > >> >> bea
> > >> >> ---
> > >> >> java version "1.5.0_06 "
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > >> >> BEA JRockit(R) (build
> > >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> > >> >>
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> > >> >>
> > >> >> command: java jnt.scimark2.commandline
> > >> >>
> > >> >> NOTE: bigger number is better
> > >> >>
> > >> >> Sun6
> > >> >> Composite Score: 364.5832265230057
> > >> >> FFT (1024): 220.8458713892794
> > >> >> SOR (100x100):   696.1542342357722
> > >> >> Monte Carlo : 149.37978088875656
> > >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > >> >> LU (100x100): 430.1617273683819
> > >> >>
> > >> >> BEA
> > >> >> Composite Score: 359.13480378697835
> > >> >> FFT (1024): 303.8746880751562
> > >> >> SOR (100x100):   454.25628897202307
> > >> >> Monte Carlo : 93.23913192138497
> > >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> > >> >> LU (100x100): 413.8627835924175
> > >> >>
> > >> >> Sun5
> > >> >> Composite Score: 332.84987587548574
> > >> >> FFT (1024): 216.5144595799027
> > >> >> SOR (100x100):   689.429322146947
> > >> >> Monte Carlo : 25.791262124978065
> > >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > >> >> LU (100x100): 414.99493895566377
> > >> >>
> > >> >> IBM
> > >> >> Composite Score: 259.8249218693683
> > >> >> FFT (1024): 296.8415012789055
> > >> >> SOR (100x100):   428.974881649179
> > >> >> Monte Carlo : 89.15159857584082
> > >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > >> >> LU (100x100): 339.8042037225181
> > >> >>
> > >> >> Harmony
> > >> >> Composite Score: 113.65082278962575
> > >> >> FFT (1024): 203.76641991778123
> > >> >> SOR (100x100):   224.37761309236748
> > >> >> Monte Carlo : 9.063866256533116
> > >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > >> >> LU (100x100): 65.6410280487242
> > >> >>
> > >> >> In this test harmony is clearly lagging behind... at about 30%
> > >> >> performance of the best JVM, it's a little crappy. Please note
how
> > >> FFT's
> > >> >> performance is not so bad awhile monte carlo is pretty bad compared
to
> > >> >> BEA or IBM.
> > >> >>
> > >> >> Overall, it seems like there is some serious work to do here to
catch
> > >> up.
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test 2: Dhrystones
> > >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> > >> )
> > >> >>
> > >> >> command: java dhry 100000000
> > >> >>
> > >> >> NOTE: bigger is better
> > >> >>
> > >> >> NB: I modified the code to accept the count at input from the
command
> > >> >> line!
> > >> >>
> > >> >> sun6:     8552856 dhrystones/sec
> > >> >> sun5:     6605892
> > >> >> bea:      5678914
> > >> >> harmony:   669734
> > >> >> ibm:       501562
> > >> >>
> > >> >> The performance here is horrific but what's surprising is that
J9 is
> > >> >> even worse. No idea what's going on but it seems like something
is not
> > >> >> working as it should (in both harmony and J9)
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> > >> >>
> > >> >> command: java Sieve 30
> > >> >>
> > >> >> NB: I modified the test to run for a configurable amount of seconds.
> > >> >>
> > >> >> sun6     8545 sieves/sec
> > >> >> sun5     8364
> > >> >> bea      6174
> > >> >> harmony  1836
> > >> >> ibm       225
> > >> >>
> > >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> > >> >> lagging behind.
> > >> >>
> > >> >> Stay tuned for more tests.
> > >> >>
> > >> >> --
> > >> >> Stefano.
> > >> >>
> > >> >>
> > >>
> > >>
> > >> --
> > >> Stefano.
> > >>
> > >>
> > >
> >
> >
> > --
> > Stefano.
> >
> >
>
> --
> Egor Pasko
>
>

Mime
View raw message