harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Mazzocchi <stef...@apache.org>
Subject Re: [performance] a few early benchmarks
Date Tue, 28 Nov 2006 02:40:42 GMT
Elford, Chris L wrote:
> Stefano Mazzocchi writes:
>>> ...
>>> Then you have to realize that not everybody has the $$$ or the
> intention
>>> to spend money to help us optimize Harmony. 
> 
> Your point that there is a $$$ barrier to entry for utilizing SPEC
> workloads is a good one.  I wonder if SPEC would consider some sort of
> grant to Apache to use their workloads with Harmony...  The SPEC site
> (http://www.spec.org/order.html) seems to read as if a single Apache
> license might cover all Harmony developers...
> 
>>> ...
>>> Our main and only objective is to please our users, and reducing the
>>> barrier for them to become developers as much as we possibly can. 
> 
> I agree wholeheartedly with the first part of this sentence regarding
> pleasing our users.  For the next 9 months or so, I'd also agree with
> the second part of this sentence in that lowering the barrier to
> developer entry is strongly desirable.  Your point is good in that for
> now, we need to do whatever we can to make sure that key workloads
> (including performance workloads) are available to our developers.  In
> the longer term, I would hope that we have more users than developers
> and that developers will pay attention to performance [and functional]
> bugs filed against Harmony on behalf of applications for which the
> source and/or all executable bits are unavailable.
> 
>>> ...
>>> But benchmarks have two objectives:
>>> show the current performance status *and* allow potential developers
> to
>>> test if their changes have made a positive impact before submitting
> them.
> 
> A benchmark is created to stress some combination of technologies.
> Benchmarks tend to be distilled from some set of real world
> requirements.  As someone who worked for years with industy consortia
> for standardizing benchmarks, I would argue that benchmarks have more
> than the two objectives you mention.  To name a few: attracting new
> users, showcasing your "product", providing a platform for level playing
> field comparisons with alternate implementations, allowing demonstration
> of technology readiness, driving improvements in key portions of the
> technology stack, ....
> 
> To showcase Harmony as a performance competitive JRE with technology
> readiness for client and enterprise class deployments, I believe that we
> will need to agree to underscore some combination of academic benchmarks
> (such as Dacapo memory management benchmarks), domain benchmarks (such
> as SciMark, XMLbench, etc.) and more general standardized industry
> benchmarks (such as SPEC and/or TPC).  Of course we all have our own
> biases but I personally don't think that Harmony will be successful
> without some due diligence applied to the latter and I believe that
> Harmony needs to find some way of working with some these benchmarks
> (e.g., SPECjbbXXXX, SPECjvmXXXX, SPECjAppServerXXXX, TPC-App).  
> 
> I assume that we will have contributors interested in different
> combinations of these benchmarks.  Harmony needs to create some guiding
> principles as to how the design/implementation will accommodate these
> varied performance interests while moving Harmony toward an ever better
> performing platform for our users.  Optimizing for performance tends to
> be at odds with optimizing for portability and/or for maintainability in
> that it tends to involve fast path exceptions to generalized code.  

Chris,

I don't claim to be an expert in performance evaluation nor I claim to
know which test is more useful than others for pure performance purpose.
It's not my competence and frankly not really my interest either.

My goal is to make Harmony successful and 'successful', to me, means a
healthy and diverse community of users and developers, the largest and
more diverse possible.

Harmony was started because we believed that the existing licensing
scenarios of 'FOSS java' was going to be a limitation for such a
community to be the largest and more diverse possible (ranging from
individual hobbyists to multinational corporations' engineers). Sun's
recent OpenJDK effort has not improved this scenario.

Harmony continues, IMO (and I'm obviously biased), to have the lead in
the 'potential' of community building in terms of maximum licensing
appeal, but decisions like using a non-open, for-pay workload as "the"
performance benchmark for this project goes against the very principle
of 'maximum community diversity' that Harmony was created to aspire to.

It's not only damaging, but entirely poisonous: it makes Harmony's own
reason to exist hypocritical.

And having SPEC donating a license to Apache won't solve anything: we
want a random student from a developing country to be able to contribute
as much as the top-notch performance engineer from a multinational
corporation, as we strongly believe that innovation happens in the
periphery more than it happens in the core.

Failure to execute in such direction will mean that Harmony will always
lag behind the communities that will manage to achieve that, a situation
that frankly I equate to 'social thermal death'.

Let me repeat for the third time: this is not to say that SPEC has no
reason to exist or that we should ignore it or ridicule it. I never said
that nor I think that's the case.

All I want is to lower the barriers to participation for anybody, for
anything, anywhere.

And with that in mind, DaCapo is far superior than SPEC, simply because
of their licensing model.

-- 
Stefano.


Mime
View raw message