Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
Message-ID: <455DE94F.8030006@apache.org>
Date: Fri, 17 Nov 2006 08:54:39 -0800
From: Stefano Mazzocchi <stefano@apache.org>
User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [performance] a few early benchmarks
References: <455D47F9.60606@apache.org>
 <f944d83d0611162134tb8a73f0p2b97595e7aca9c0@mail.gmail.com>
In-Reply-To: <f944d83d0611162134tb8a73f0p2b97595e7aca9c0@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Alexey Varlamov wrote:
> Stefano,
> 
> It is a bit unfair to compare *debug* build of Harmony with other
> release versions :)

I'm simulating what a journalist with a developer could do.

If there is a way to make it compile in 'release mode' (if such a thing
exists), I'll be very glad to redo the benchmarks.

> I suppose all VMs where run in default mode (i.e. no special cmd-line
> switches)?

Right. No switches. I'm simulating what users do when they get the JVM:
they run "java"... and if it's now fast enough they buy a new box.

Having command line tuning parameters is mostly useless since most
people don't know the internals of a JVM well enough to guess what
parameters to tune anyway.

So, what people will do once they get an harmony snapshot is "java
my.class.Name" and see the results.

I want to simulate that and compare it to the same exact experience they
will get with other virtual machines for a variety of common scenarios
(number crunching, xml processing, http serving, database load, etc...)

I will focus on the server because that's there the apache action (and
my personal interest) is.

So, like I said, if there are 'compile time' switches that I can use to
turn 'release mode' on, please tell me and I'll re-do the tests.

> 2006/11/17, Stefano Mazzocchi <stefano@apache.org>:
>> There are lies, damn lies and benchmarks.... which don't really tell you
>> if an implementation of a program is *faster* but at least it tells you
>> where you're at.
>>
>> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>> was able to start running some benchmarks.
>>
>> The machine is the following:
>>
>> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>> 01:50:50 UTC 2006 x86_64 GNU/Linux
>>
>> dual Intel(R) Pentium(R) D CPU 3.20GHz
>> bogomips 6410.31 (per CPU)
>>
>> There is nothing else running on the machine (load is 0.04 at the time
>> of testing).
>>
>> The various virtual machines tested are:
>>
>> harmony
>> -------
>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> Foundation or its licensors, as applicable.
>> java version "1.5.0"
>> pre-alpha : not complete or compatible
>> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>>
>> sun5
>> ---
>> java version "1.5.0_09"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>>
>> sun6
>> ----
>> java version "1.6.0-rc"
>> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>>
>> ibm
>> ---
>> java version "1.5.0"
>> Java(TM) 2 Runtime Environment, Standard Edition (build
>> pxa64dev-20061002a (SR3) )
>> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>> j9vmxa6423-20061001 (JIT enabled)
>> J9VM - 20060915_08260_LHdSMr
>> JIT  - 20060908_1811_r8
>> GC   - 20060906_AA)
>> JCL  - 20061002
>>
>> bea
>> ---
>> java version "1.5.0_06"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> BEA JRockit(R) (build
>> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>>
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>>
>> command: java jnt.scimark2.commandline
>>
>> NOTE: bigger number is better
>>
>> Sun6
>> Composite Score: 364.5832265230057
>> FFT (1024): 220.8458713892794
>> SOR (100x100):   696.1542342357722
>> Monte Carlo : 149.37978088875656
>> Sparse matmult (N=1000, nz=5000): 326.37451873283845
>> LU (100x100): 430.1617273683819
>>
>> BEA
>> Composite Score: 359.13480378697835
>> FFT (1024): 303.8746880751562
>> SOR (100x100):   454.25628897202307
>> Monte Carlo : 93.23913192138497
>> Sparse matmult (N=1000, nz=5000): 530.44112637391
>> LU (100x100): 413.8627835924175
>>
>> Sun5
>> Composite Score: 332.84987587548574
>> FFT (1024): 216.5144595799027
>> SOR (100x100):   689.429322146947
>> Monte Carlo : 25.791262124978065
>> Sparse matmult (N=1000, nz=5000): 317.5193965699373
>> LU (100x100): 414.99493895566377
>>
>> IBM
>> Composite Score: 259.8249218693683
>> FFT (1024): 296.8415012789055
>> SOR (100x100):   428.974881649179
>> Monte Carlo : 89.15159857584082
>> Sparse matmult (N=1000, nz=5000): 144.3524241203982
>> LU (100x100): 339.8042037225181
>>
>> Harmony
>> Composite Score: 113.65082278962575
>> FFT (1024): 203.76641991778123
>> SOR (100x100):   224.37761309236748
>> Monte Carlo : 9.063866256533116
>> Sparse matmult (N=1000, nz=5000): 65.4051866327227
>> LU (100x100): 65.6410280487242
>>
>> In this test harmony is clearly lagging behind... at about 30%
>> performance of the best JVM, it's a little crappy. Please note how FFT's
>> performance is not so bad awhile monte carlo is pretty bad compared to
>> BEA or IBM.
>>
>> Overall, it seems like there is some serious work to do here to catch up.
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>>
>> command: java dhry 100000000
>>
>> NOTE: bigger is better
>>
>> NB: I modified the code to accept the count at input from the command
>> line!
>>
>> sun6:     8552856 dhrystones/sec
>> sun5:     6605892
>> bea:      5678914
>> harmony:   669734
>> ibm:       501562
>>
>> The performance here is horrific but what's surprising is that J9 is
>> even worse. No idea what's going on but it seems like something is not
>> working as it should (in both harmony and J9)
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>>
>> command: java Sieve 30
>>
>> NB: I modified the test to run for a configurable amount of seconds.
>>
>> sun6     8545 sieves/sec
>> sun5     8364
>> bea      6174
>> harmony  1836
>> ibm       225
>>
>> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>> lagging behind.
>>
>> Stay tuned for more tests.
>>
>> -- 
>> Stefano.
>>
>>


-- 
Stefano.