harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Pasko <egor.pa...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Thu, 23 Nov 2006 14:02:32 GMT
On the 0x22A day of Apache Harmony Alexey Varlamov wrote:
> [snip]
> > We can try to tune DRLVM default mode for HelloWorld + Eclipse +
> > SciMark + DaCapo, would it be enough for users to be happy with
> > Harmony on their first glance? I dunno, actually. IMHO, almost all
> > java users taking care of performance know about client and server
> > modes and what applications are best for each mode.
> 
> Err, SciMark looks quite specific in this list, and I suspect it will
> provoke that "short blanket" scenario. Probably we should not include
> it for the default mode tuning.

What can we do:
* agree that SciMark is not for DRLVM default mode (=do nothing)
* take SciMark into account, make a weighted metric (with SciMark at
  low priority, of course)

We should be "reasonably" fast, is 4x slowdown on SciMark reasonable?
Tuning decisions are always not easy...

> As a side note, any user cares about performance but not all dare to
> tune it. 

Thank you for the side note)) There is going to be the "short blanket"
story forever or at least for a while. And options are all we can do
for those who care. I think, we all know that.

> This is the main motivation for measuring OOB at the first place...

Now I know, OOB=OutOfBox :)

> >
> > > > Currently we have in mind the following list:
> > > > - HWA (Hello World Application)
> > > > - SciMark
> > > > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > > > - Anything else?
> > > >
> > > > What do you think about this? Any additions to the list? Comments?
> > > > Questions?
> > >
> > > The problem I have with this is that I feel that each one of such
> > > scenario might require different tuning parameters... and if that is the
> > > case, you end up with the 'short blanket' problem: you improve here and
> > > you decrease there.
> > >
> > > An 'adaptive' scenario, on the other hand, would allow us to:
> > >
> > >  1) avoid trying to find the optimal defaults (since we can't possible
> > > test every scenario that will be useful in a way that is consistent with
> > > real world usage)
> > >
> > >  2) avoid the blanket problem, each VM can adapt to the scenario of use
> > >
> > >  3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> > > changes and 'retune' itself if the environment changes.
> > >
> > > Of course, there is a price to pay in such 'fedback variability' systems
> > > since they have to find the minima over and over again.
> > >
> > > So, another solution is to have a JVM "tuning parameters discovery mode"
> > > that you can run and you turn such "parameter finding" autoprofiling
> > > on... and the JVM dumps the tuning results for you on disk which you can
> > > later use to initialize the JVM on your own.
> > >
> > > Not sure how feasible or complicated to write this is, but wow does this
> > > sound on paper?
> > >
> > >
> > > > Thanks,
> > > > ---
> > > > Sergey Kuksenko
> > > > Intel Enterprise Solutions Software Division
> > > >
> > > >
> > > > On 11/17/06, Stefano Mazzocchi <stefano@apache.org> wrote:
> > > >>
> > > >> Alexey Varlamov wrote:
> > > >> > Stefano,
> > > >> >
> > > >> > It is a bit unfair to compare *debug* build of Harmony with other
> > > >> > release versions :)
> > > >>
> > > >> I'm simulating what a journalist with a developer could do.
> > > >>
> > > >> If there is a way to make it compile in 'release mode' (if such a
thing
> > > >> exists), I'll be very glad to redo the benchmarks.
> > > >>
> > > >> > I suppose all VMs where run in default mode (i.e. no special
cmd-line
> > > >> > switches)?
> > > >>
> > > >> Right. No switches. I'm simulating what users do when they get the
JVM:
> > > >> they run "java"... and if it's now fast enough they buy a new box.
> > > >>
> > > >> Having command line tuning parameters is mostly useless since most
> > > >> people don't know the internals of a JVM well enough to guess what
> > > >> parameters to tune anyway.
> > > >>
> > > >> So, what people will do once they get an harmony snapshot is "java
> > > >> my.class.Name <http://my.class.name/>" and see the results.
> > > >>
> > > >> I want to simulate that and compare it to the same exact experience
they
> > > >> will get with other virtual machines for a variety of common scenarios
> > > >> (number crunching, xml processing, http serving, database load, etc...)
> > > >>
> > > >> I will focus on the server because that's there the apache action
(and
> > > >> my personal interest) is.
> > > >>
> > > >> So, like I said, if there are 'compile time' switches that I can use
to
> > > >> turn 'release mode' on, please tell me and I'll re-do the tests.
> > > >>
> > > >> > 2006/11/17, Stefano Mazzocchi <stefano@apache.org>:
> > > >> >> There are lies, damn lies and benchmarks.... which don't
really tell
> > > >> you
> > > >> >> if an implementation of a program is *faster* but at least
it tells
> > > >> you
> > > >> >> where you're at.
> > > >> >>
> > > >> >> So, as Geir managed to get the DSO linking problem go away
in DRLVM, I
> > > >> >> was able to start running some benchmarks.
> > > >> >>
> > > >> >> The machine is the following:
> > > >> >>
> > > >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT
Sat Sep 16
> > > >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> > > >> >>
> > > >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> > > >> >> bogomips 6410.31 (per CPU)
> > > >> >>
> > > >> >> There is nothing else running on the machine (load is 0.04
at the time
> > > >> >> of testing).
> > > >> >>
> > > >> >> The various virtual machines tested are:
> > > >> >>
> > > >> >> harmony
> > > >> >> -------
> > > >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache
Software
> > > >> >> Foundation or its licensors, as applicable.
> > > >> >> java version " 1.5.0"
> > > >> >> pre-alpha : not complete or compatible
> > > >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug
build
> > > >> >>
> > > >> >> sun5
> > > >> >> ---
> > > >> >> java version "1.5.0_09 "
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed
mode)
> > > >> >>
> > > >> >> sun6
> > > >> >> ----
> > > >> >> java version " 1.6.0-rc"
> > > >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed
mode)
> > > >> >>
> > > >> >> ibm
> > > >> >> ---
> > > >> >> java version " 1.5.0"
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> > > >> >> pxa64dev-20061002a (SR3) )
> > > >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > > >> >> j9vmxa6423-20061001 (JIT enabled)
> > > >> >> J9VM - 20060915_08260_LHdSMr
> > > >> >> JIT  - 20060908_1811_r8
> > > >> >> GC   - 20060906_AA)
> > > >> >> JCL  - 20061002
> > > >> >>
> > > >> >> bea
> > > >> >> ---
> > > >> >> java version "1.5.0_06 "
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > > >> >> BEA JRockit(R) (build
> > > >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> > > >> >>
> > > >> >> command: java jnt.scimark2.commandline
> > > >> >>
> > > >> >> NOTE: bigger number is better
> > > >> >>
> > > >> >> Sun6
> > > >> >> Composite Score: 364.5832265230057
> > > >> >> FFT (1024): 220.8458713892794
> > > >> >> SOR (100x100):   696.1542342357722
> > > >> >> Monte Carlo : 149.37978088875656
> > > >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > > >> >> LU (100x100): 430.1617273683819
> > > >> >>
> > > >> >> BEA
> > > >> >> Composite Score: 359.13480378697835
> > > >> >> FFT (1024): 303.8746880751562
> > > >> >> SOR (100x100):   454.25628897202307
> > > >> >> Monte Carlo : 93.23913192138497
> > > >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> > > >> >> LU (100x100): 413.8627835924175
> > > >> >>
> > > >> >> Sun5
> > > >> >> Composite Score: 332.84987587548574
> > > >> >> FFT (1024): 216.5144595799027
> > > >> >> SOR (100x100):   689.429322146947
> > > >> >> Monte Carlo : 25.791262124978065
> > > >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > > >> >> LU (100x100): 414.99493895566377
> > > >> >>
> > > >> >> IBM
> > > >> >> Composite Score: 259.8249218693683
> > > >> >> FFT (1024): 296.8415012789055
> > > >> >> SOR (100x100):   428.974881649179
> > > >> >> Monte Carlo : 89.15159857584082
> > > >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > > >> >> LU (100x100): 339.8042037225181
> > > >> >>
> > > >> >> Harmony
> > > >> >> Composite Score: 113.65082278962575
> > > >> >> FFT (1024): 203.76641991778123
> > > >> >> SOR (100x100):   224.37761309236748
> > > >> >> Monte Carlo : 9.063866256533116
> > > >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > > >> >> LU (100x100): 65.6410280487242
> > > >> >>
> > > >> >> In this test harmony is clearly lagging behind... at about
30%
> > > >> >> performance of the best JVM, it's a little crappy. Please
note how
> > > >> FFT's
> > > >> >> performance is not so bad awhile monte carlo is pretty bad
compared to
> > > >> >> BEA or IBM.
> > > >> >>
> > > >> >> Overall, it seems like there is some serious work to do here
to catch
> > > >> up.
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test 2: Dhrystones
> > > >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> > > >> )
> > > >> >>
> > > >> >> command: java dhry 100000000
> > > >> >>
> > > >> >> NOTE: bigger is better
> > > >> >>
> > > >> >> NB: I modified the code to accept the count at input from
the command
> > > >> >> line!
> > > >> >>
> > > >> >> sun6:     8552856 dhrystones/sec
> > > >> >> sun5:     6605892
> > > >> >> bea:      5678914
> > > >> >> harmony:   669734
> > > >> >> ibm:       501562
> > > >> >>
> > > >> >> The performance here is horrific but what's surprising is
that J9 is
> > > >> >> even worse. No idea what's going on but it seems like something
is not
> > > >> >> working as it should (in both harmony and J9)
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> > > >> >>
> > > >> >> command: java Sieve 30
> > > >> >>
> > > >> >> NB: I modified the test to run for a configurable amount
of seconds.
> > > >> >>
> > > >> >> sun6     8545 sieves/sec
> > > >> >> sun5     8364
> > > >> >> bea      6174
> > > >> >> harmony  1836
> > > >> >> ibm       225
> > > >> >>
> > > >> >> IBM J9 clearly has something wrong on x86_64 but harmony
is clearly
> > > >> >> lagging behind.
> > > >> >>
> > > >> >> Stay tuned for more tests.
> > > >> >>
> > > >> >> --
> > > >> >> Stefano.
> > > >> >>
> > > >> >>
> > > >>
> > > >>
> > > >> --
> > > >> Stefano.
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Stefano.
> > >
> > >
> >
> > --
> > Egor Pasko
> >
> >
> 

-- 
Egor Pasko


Mime
View raw message