harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Tue, 28 Nov 2006 06:44:07 GMT
It seems that we need to define first what "the" performance benchmark means.

Depending on that we'll be able to say whether for-pay workload will create
a barrier or not. Overall it's a question of how we organize the work.
When I do something for my employer it's not necessarily me, who runs
benchmarks on the code that I develop. If we have people committed
then they could run their favorite benchmarks on each JIRA patch, and report if
they want. That is similar to how we are going to support many platforms:
not every student has a variety of platforms Harmony supports.
But that is how community works and I don't see any problem here.


2006/11/28, Stefano Mazzocchi <stefano@apache.org>:
> Elford, Chris L wrote:
> > Stefano Mazzocchi writes:
> >>> ...
> >>> Then you have to realize that not everybody has the $$$ or the
> > intention
> >>> to spend money to help us optimize Harmony.
> >
> > Your point that there is a $$$ barrier to entry for utilizing SPEC
> > workloads is a good one.  I wonder if SPEC would consider some sort of
> > grant to Apache to use their workloads with Harmony...  The SPEC site
> > (http://www.spec.org/order.html) seems to read as if a single Apache
> > license might cover all Harmony developers...
> >
> >>> ...
> >>> Our main and only objective is to please our users, and reducing the
> >>> barrier for them to become developers as much as we possibly can.
> >
> > I agree wholeheartedly with the first part of this sentence regarding
> > pleasing our users.  For the next 9 months or so, I'd also agree with
> > the second part of this sentence in that lowering the barrier to
> > developer entry is strongly desirable.  Your point is good in that for
> > now, we need to do whatever we can to make sure that key workloads
> > (including performance workloads) are available to our developers.  In
> > the longer term, I would hope that we have more users than developers
> > and that developers will pay attention to performance [and functional]
> > bugs filed against Harmony on behalf of applications for which the
> > source and/or all executable bits are unavailable.
> >
> >>> ...
> >>> But benchmarks have two objectives:
> >>> show the current performance status *and* allow potential developers
> > to
> >>> test if their changes have made a positive impact before submitting
> > them.
> >
> > A benchmark is created to stress some combination of technologies.
> > Benchmarks tend to be distilled from some set of real world
> > requirements.  As someone who worked for years with industy consortia
> > for standardizing benchmarks, I would argue that benchmarks have more
> > than the two objectives you mention.  To name a few: attracting new
> > users, showcasing your "product", providing a platform for level playing
> > field comparisons with alternate implementations, allowing demonstration
> > of technology readiness, driving improvements in key portions of the
> > technology stack, ....
> >
> > To showcase Harmony as a performance competitive JRE with technology
> > readiness for client and enterprise class deployments, I believe that we
> > will need to agree to underscore some combination of academic benchmarks
> > (such as Dacapo memory management benchmarks), domain benchmarks (such
> > as SciMark, XMLbench, etc.) and more general standardized industry
> > benchmarks (such as SPEC and/or TPC).  Of course we all have our own
> > biases but I personally don't think that Harmony will be successful
> > without some due diligence applied to the latter and I believe that
> > Harmony needs to find some way of working with some these benchmarks
> > (e.g., SPECjbbXXXX, SPECjvmXXXX, SPECjAppServerXXXX, TPC-App).
> >
> > I assume that we will have contributors interested in different
> > combinations of these benchmarks.  Harmony needs to create some guiding
> > principles as to how the design/implementation will accommodate these
> > varied performance interests while moving Harmony toward an ever better
> > performing platform for our users.  Optimizing for performance tends to
> > be at odds with optimizing for portability and/or for maintainability in
> > that it tends to involve fast path exceptions to generalized code.
> Chris,
> I don't claim to be an expert in performance evaluation nor I claim to
> know which test is more useful than others for pure performance purpose.
> It's not my competence and frankly not really my interest either.
> My goal is to make Harmony successful and 'successful', to me, means a
> healthy and diverse community of users and developers, the largest and
> more diverse possible.
> Harmony was started because we believed that the existing licensing
> scenarios of 'FOSS java' was going to be a limitation for such a
> community to be the largest and more diverse possible (ranging from
> individual hobbyists to multinational corporations' engineers). Sun's
> recent OpenJDK effort has not improved this scenario.
> Harmony continues, IMO (and I'm obviously biased), to have the lead in
> the 'potential' of community building in terms of maximum licensing
> appeal, but decisions like using a non-open, for-pay workload as "the"
> performance benchmark for this project goes against the very principle
> of 'maximum community diversity' that Harmony was created to aspire to.
> It's not only damaging, but entirely poisonous: it makes Harmony's own
> reason to exist hypocritical.
> And having SPEC donating a license to Apache won't solve anything: we
> want a random student from a developing country to be able to contribute
> as much as the top-notch performance engineer from a multinational
> corporation, as we strongly believe that innovation happens in the
> periphery more than it happens in the core.
> Failure to execute in such direction will mean that Harmony will always
> lag behind the communities that will manage to achieve that, a situation
> that frankly I equate to 'social thermal death'.
> Let me repeat for the third time: this is not to say that SPEC has no
> reason to exist or that we should ignore it or ridicule it. I never said
> that nor I think that's the case.
> All I want is to lower the barriers to participation for anybody, for
> anything, anywhere.
> And with that in mind, DaCapo is far superior than SPEC, simply because
> of their licensing model.
> --
> Stefano.

View raw message