harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ellison <t.p.elli...@gmail.com>
Subject Re: [general] SPECjbb2005 publication on Harmony
Date Tue, 18 Mar 2008 11:49:24 GMT
Sergey Nesterov wrote:
> I had tested SPECjbb2005 on Harmony and submitted the results for reviewing
> to SPEC.
> SPEC committee has some questions on my submission concerning Harmony
> support and testing of command-line options.

We had this discussion already on the private list!?

I believe that SPEC only want to publish results for code that people 
can and would run their applications on, not experimental or custom builds.

However, it would seem that SPEC are not used to dealing with open 
source projects on the same level as commercial implementations, so 
their evaluation criteria seem a bit strange.

> 1) Support of Harmony:
> The issue is section (1) of the SPECjbb2005 Run Rules [1], citing:
> "The implementation is generally available, documented and supported
> by the providing vendor."
> Though the support requirement for SPECjbb2005 is loosely defined, all
> current publications are on Java implementations that provide various
> levels of support. SPEC argues that this level of support cannot be compared
> to forum support provided by Harmony developers. SPEC argues that current
> production JVMs on the market today are fully supported by the companies
> that develop them and they need to clarify these issues before any result
> using Harmony can be accepted.
> What can you comment on this?

Our code is certainly generally available (more so than commercial 
builds), is documented (including design documentation on our website), 
and is supported by the Apache community who participate here.

However, it seems to me that SPEC want somewhere people can go and hand 
over money to support a particular build of the runtime.  They want to 
pay for somebody to fix the code in preference to the ability to fix the 
code themselves through open source.  I'm not sure why exchanging money 
is the key point for them.

There are people who will accept payment to 'support' Harmony by fixing 
bugs.  Independent consultants, like Andrew, will take paid work on 
Harmony.  Of course, their support terms are varied, like everyone else 
supporting a product.

> 2) Testing:
> SPEC also concerned about testing required for performance 
> optimizations to be considered as generally available, broad and
> suggested for production use. They want to know what checks are in
> place to prevent unsafe changes that benefit SPECjbb2005 only?

This is a joke right?  If SPEC try to tell you that commercial vendors 
do not test and optimize their code to benefit the SPEC scores 
specifically then they are deluding you.

We have no special command-line options that can only be used in running 
the SPEC benchmarks.  The command-line options that are typically used 
to get good performance runs are also valid (though not all appropriate) 
for running users' applications.

> I will be glad to see the input on these questions from Harmony developers.

SPEC run the risk of devaluing their benchmark and results if they are 
going to be selective about the body of people who contribute to the 
runtime's development.  I seriously doubt that potential adopters 
evaluate support levels based on the run rules.

Other performance benchmark suites are available...


> Thanks,
> Sergey Nesterov,
> CS Department of MSU
> [1] http://www.spec.org/jbb2005/docs/RunRules.html

View raw message