harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Mazzocchi <stef...@apache.org>
Subject Re: [performance] a few early benchmarks
Date Wed, 22 Nov 2006 23:45:29 GMT
Robin Garner wrote:
>> Sergey Kuksenko wrote:
>>
>>> Lets do the simplest thing fist. :)
>>> We can do it. We only need to specify a set of workloads.
>> I've tried running dacapo with 10 warming stages and we are constantly
>> around 25% speed against the leading JVM (which is always sun5 or sun6),
>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>> results shortly.
> 
> In my graphs, I find that for drlvm iterations 2 and 3 have very similar
> performance - my assumption would be that by the third iteration, most of
> the benefit of recompilation has gone.
> 
>> Robin, any chance you guys can add 'skymark2' to the dacapo benchmarks?
>> I think that would be helpful.
> 
> Do you have a link for it ?  I've googled without success.  We are looking
> for benchmarks for the next major release, so I'd like to see if it meets
> our criteria.

sorry, that would be "scimark" :-) my bad.

>> Also, a great thing would be for dacapo to output an XML serialization
>> of the benchmark results.
> 
> The MMTk measurement harness does that, and it sounds like a good idea. 
> Any thoughts on what the output would look like ?  It's easy to write a
> new callback that will do this.

I would do something minimal yet complete like:

 <results version="" date="" os="" arch="" jvm="">
  <test name="antlr">
   <iteration type="warmup" time="3243"/>
   <iteration type="warmup" time="2433"/>
   <iteration type="warmup" time="2322"/>
   <iteration type="measured" time="2322"/>
  </test>
  <test name="...">
   ...
  </test>
  ...
 </results>

This would make it a lot easier to automate some sort of graphical results.

>> It also appears that we can't run some of the dacapo tests, most notably
>> antlr (jvm fails with no error and just terminates right during the
>> first warmup) and chart (because it requires a DISPLAY to be
>> available... can we run harmony headless yet?)
> 
> I've created a JIRA about the antlr issue.  I manage to run it in the
> regression tests by pulling out the DaCapo antlr classes and pre-pending
> them to the boot classpath.

hmmm, I wonder if this is due to version conflicts betwen various
versions of antlr because harmony does ship with it in the bootclasspath.

Which makes me wonder: have you guys thought about running dacapo in a
'reversed classloader' such as the one used by tomcat?

A 'reversed classloader' is one that loads classes first in its own
classpath and then delegates back to the system classloader (the JLS
suggests to do otherwise to avoid security problems, but if you know
what code you're running so this is not a big deal).

Here is such a thing that I've written a while ago for cocoon and kept
using ever since:

http://simile.mit.edu/repository/tools/loader/trunk/src/Loader.java

> Chart is easy to run in batch if you create a virtual framebuffer with
> Xvfb, but I've created a JIRA about this issue too.  Gnu Classpath has the
> same issue.

I see. Modern JVMs have the ability to run 'headless', are we planning
of having harmony do the same? it would *seriously* suck if one had to
have X11 installed on their servers just to generate static jpeg/png out
of charts (things that java publishing frameworks do all the time).

> From memory, installing xvfb requires the debian/ubuntu packages xvfb,
> x11-base and xfonts-base.

ok, I'll do that for now.

-- 
Stefano.


Mime
View raw message