harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Pasko <egor.pa...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Mon, 20 Nov 2006 10:49:59 GMT
On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
> Egor Pasko wrote:
> > Stefano,
> > 
> > thank you for pointing peroformance issues out!
> 
> you're welcome, but I have to warn you, I've barely started... you might
> get sick of my performance regression charts in the future ;-)

I cannot wait to get the best from your charts :)

> > Although that was a debug build, it uncovers that Harmony is not
> > properly tuned for FP-intensive workloads as SciMark. One of the
> > possible reasons of this (known:) limitation is also raised in the
> > thread "[drlvm][jit][ia-32]register-based fast calling convention",
> > where Alex vounteers to help with more intelligent XMM register
> > usage. I consider this a quite-high priority, fully support Alex, hope
> > for the best, ready to help, etc etc.
> 
> Please understand that I'm not pointing fingers nor trying to insult
> people and their work (or pride).

I understand. I am just thinking in technical terms making assumptions
on the root causes, giving idea on the benchmark that should be
sensitive to what Alex is doing (so, he is free from writing a
benchmark on his own).

I am not trying to convince anyone that we are doing the good work. I
am thinking how to make the work even better. And your reports help here.

> Rather the opposite: there is nothing like hard facts and evolution
> charts to make developers focus. As much as japi makes it rewarding to
> work on the API because we approach the mythical 100%, by showing the
> performance comparisons to the state of the art, we understand how far
> behind (or ahead!) we are.
> 
> I fully believe in the power of open development over closed
> development. It's faster, more efficient, more innovative and overall,
> more effective.
> 
> Open development is the reason why Harmony exists, not open source.
> 
> Open source is a necessary condition, but hardly a sufficient one. Sun
> will realize that the hard way.

That's something to write in my blog :) AFAIR, Sun wants to have
comitters from the outside-sun-world. That's pretty good in terms of
openness.

> This means that Harmony has the full potential to blow all other JVMs
> out of the water in terms of reliability, portability and performance...
> just look at the development momentum and forecast what can happen in a
> a year or so.
> 
> So, no, don't take this the wrong way: it's not saying "bad guys, no
> donut", it's saying "keep it up and here's how fast you're going, just FYI".

Yes, thank you! I understand.

> Keep up the good work. I'm just building tools to give you (and others
> around us!) a condensed view of the status and evolution of the project
> in all of the aspects that matter (compatibility, reliability,
> performance, quality, and momentum)
>
> > On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
> >> There are lies, damn lies and benchmarks.... which don't really tell you
> >> if an implementation of a program is *faster* but at least it tells you
> >> where you're at.
> >>
> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> >> was able to start running some benchmarks.
> >>
> >> The machine is the following:
> >>
> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >>
> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> bogomips 6410.31 (per CPU)
> >>
> >> There is nothing else running on the machine (load is 0.04 at the time
> >> of testing).
> >>
> >> The various virtual machines tested are:
> >>
> >> harmony
> >> -------
> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> Foundation or its licensors, as applicable.
> >> java version "1.5.0"
> >> pre-alpha : not complete or compatible
> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >>
> >> sun5
> >> ---
> >> java version "1.5.0_09"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >>
> >> sun6
> >> ----
> >> java version "1.6.0-rc"
> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >>
> >> ibm
> >> ---
> >> java version "1.5.0"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> pxa64dev-20061002a (SR3) )
> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> j9vmxa6423-20061001 (JIT enabled)
> >> J9VM - 20060915_08260_LHdSMr
> >> JIT  - 20060908_1811_r8
> >> GC   - 20060906_AA)
> >> JCL  - 20061002
> >>
> >> bea
> >> ---
> >> java version "1.5.0_06"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> >> BEA JRockit(R) (build
> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >>
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >>
> >> command: java jnt.scimark2.commandline
> >>
> >> NOTE: bigger number is better
> >>
> >> Sun6
> >> Composite Score: 364.5832265230057
> >> FFT (1024): 220.8458713892794
> >> SOR (100x100):   696.1542342357722
> >> Monte Carlo : 149.37978088875656
> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> LU (100x100): 430.1617273683819
> >>
> >> BEA
> >> Composite Score: 359.13480378697835
> >> FFT (1024): 303.8746880751562
> >> SOR (100x100):   454.25628897202307
> >> Monte Carlo : 93.23913192138497
> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> LU (100x100): 413.8627835924175
> >>
> >> Sun5
> >> Composite Score: 332.84987587548574
> >> FFT (1024): 216.5144595799027
> >> SOR (100x100):   689.429322146947
> >> Monte Carlo : 25.791262124978065
> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> LU (100x100): 414.99493895566377
> >>
> >> IBM
> >> Composite Score: 259.8249218693683
> >> FFT (1024): 296.8415012789055
> >> SOR (100x100):   428.974881649179
> >> Monte Carlo : 89.15159857584082
> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> LU (100x100): 339.8042037225181
> >>
> >> Harmony
> >> Composite Score: 113.65082278962575
> >> FFT (1024): 203.76641991778123
> >> SOR (100x100):   224.37761309236748
> >> Monte Carlo : 9.063866256533116
> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> LU (100x100): 65.6410280487242
> >>
> >> In this test harmony is clearly lagging behind... at about 30%
> >> performance of the best JVM, it's a little crappy. Please note how FFT's
> >> performance is not so bad awhile monte carlo is pretty bad compared to
> >> BEA or IBM.
> >>
> >> Overall, it seems like there is some serious work to do here to catch up.
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
> >>
> >> command: java dhry 100000000
> >>
> >> NOTE: bigger is better
> >>
> >> NB: I modified the code to accept the count at input from the command line!
> >>
> >> sun6:     8552856 dhrystones/sec
> >> sun5:     6605892
> >> bea:      5678914
> >> harmony:   669734
> >> ibm:       501562
> >>
> >> The performance here is horrific but what's surprising is that J9 is
> >> even worse. No idea what's going on but it seems like something is not
> >> working as it should (in both harmony and J9)
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >>
> >> command: java Sieve 30
> >>
> >> NB: I modified the test to run for a configurable amount of seconds.
> >>
> >> sun6     8545 sieves/sec
> >> sun5     8364
> >> bea      6174
> >> harmony  1836
> >> ibm       225
> >>
> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> lagging behind.
> >>
> >> Stay tuned for more tests.
> >>
> >> -- 
> >> Stefano.
> >>
> >>
> > 
> 
> 
> -- 
> Stefano.
> 
> 

-- 
Egor Pasko


Mime
View raw message