harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject Re: [performance] a few early benchmarks
Date Tue, 28 Nov 2006 06:55:42 GMT
And we need to make sure that a patch that improves DaCapo by 0.1%
does not lower down the SPEC by 10%

We need to agree on a target function to optimize, like some weighted mean.

Thanks,
Mikhail

2006/11/28, Geir Magnusson Jr. <geir@pobox.com>:
> The key is to not claim that SPEC is representative of "the" benchmark
> for Harmony.
>
> Sure, people can use it to test Harmony, but we need to be very careful
> here that someones drive for SPEC numbers doesn't have harmful effects
> on other benchmarks.  It's an interesting balancing act we'll have to do...
>
> geir
>
>
> Mikhail Loenko wrote:
> > It seems that we need to define first what "the" performance benchmark
> > means.
> >
> > Depending on that we'll be able to say whether for-pay workload will create
> > a barrier or not. Overall it's a question of how we organize the work.
> > When I do something for my employer it's not necessarily me, who runs
> > benchmarks on the code that I develop. If we have people committed
> > then they could run their favorite benchmarks on each JIRA patch, and
> > report if
> > they want. That is similar to how we are going to support many platforms:
> > not every student has a variety of platforms Harmony supports.
> > But that is how community works and I don't see any problem here.
> >
> > Thanks,
> > Mikhail
> >
> > 2006/11/28, Stefano Mazzocchi <stefano@apache.org>:
> >> Elford, Chris L wrote:
> >> > Stefano Mazzocchi writes:
> >> >>> ...
> >> >>> Then you have to realize that not everybody has the $$$ or the
> >> > intention
> >> >>> to spend money to help us optimize Harmony.
> >> >
> >> > Your point that there is a $$$ barrier to entry for utilizing SPEC
> >> > workloads is a good one.  I wonder if SPEC would consider some sort of
> >> > grant to Apache to use their workloads with Harmony...  The SPEC site
> >> > (http://www.spec.org/order.html) seems to read as if a single Apache
> >> > license might cover all Harmony developers...
> >> >
> >> >>> ...
> >> >>> Our main and only objective is to please our users, and reducing
the
> >> >>> barrier for them to become developers as much as we possibly can.
> >> >
> >> > I agree wholeheartedly with the first part of this sentence regarding
> >> > pleasing our users.  For the next 9 months or so, I'd also agree with
> >> > the second part of this sentence in that lowering the barrier to
> >> > developer entry is strongly desirable.  Your point is good in that for
> >> > now, we need to do whatever we can to make sure that key workloads
> >> > (including performance workloads) are available to our developers.  In
> >> > the longer term, I would hope that we have more users than developers
> >> > and that developers will pay attention to performance [and functional]
> >> > bugs filed against Harmony on behalf of applications for which the
> >> > source and/or all executable bits are unavailable.
> >> >
> >> >>> ...
> >> >>> But benchmarks have two objectives:
> >> >>> show the current performance status *and* allow potential developers
> >> > to
> >> >>> test if their changes have made a positive impact before submitting
> >> > them.
> >> >
> >> > A benchmark is created to stress some combination of technologies.
> >> > Benchmarks tend to be distilled from some set of real world
> >> > requirements.  As someone who worked for years with industy consortia
> >> > for standardizing benchmarks, I would argue that benchmarks have more
> >> > than the two objectives you mention.  To name a few: attracting new
> >> > users, showcasing your "product", providing a platform for level
> >> playing
> >> > field comparisons with alternate implementations, allowing
> >> demonstration
> >> > of technology readiness, driving improvements in key portions of the
> >> > technology stack, ....
> >> >
> >> > To showcase Harmony as a performance competitive JRE with technology
> >> > readiness for client and enterprise class deployments, I believe
> >> that we
> >> > will need to agree to underscore some combination of academic
> >> benchmarks
> >> > (such as Dacapo memory management benchmarks), domain benchmarks (such
> >> > as SciMark, XMLbench, etc.) and more general standardized industry
> >> > benchmarks (such as SPEC and/or TPC).  Of course we all have our own
> >> > biases but I personally don't think that Harmony will be successful
> >> > without some due diligence applied to the latter and I believe that
> >> > Harmony needs to find some way of working with some these benchmarks
> >> > (e.g., SPECjbbXXXX, SPECjvmXXXX, SPECjAppServerXXXX, TPC-App).
> >> >
> >> > I assume that we will have contributors interested in different
> >> > combinations of these benchmarks.  Harmony needs to create some guiding
> >> > principles as to how the design/implementation will accommodate these
> >> > varied performance interests while moving Harmony toward an ever better
> >> > performing platform for our users.  Optimizing for performance tends to
> >> > be at odds with optimizing for portability and/or for
> >> maintainability in
> >> > that it tends to involve fast path exceptions to generalized code.
> >>
> >> Chris,
> >>
> >> I don't claim to be an expert in performance evaluation nor I claim to
> >> know which test is more useful than others for pure performance purpose.
> >> It's not my competence and frankly not really my interest either.
> >>
> >> My goal is to make Harmony successful and 'successful', to me, means a
> >> healthy and diverse community of users and developers, the largest and
> >> more diverse possible.
> >>
> >> Harmony was started because we believed that the existing licensing
> >> scenarios of 'FOSS java' was going to be a limitation for such a
> >> community to be the largest and more diverse possible (ranging from
> >> individual hobbyists to multinational corporations' engineers). Sun's
> >> recent OpenJDK effort has not improved this scenario.
> >>
> >> Harmony continues, IMO (and I'm obviously biased), to have the lead in
> >> the 'potential' of community building in terms of maximum licensing
> >> appeal, but decisions like using a non-open, for-pay workload as "the"
> >> performance benchmark for this project goes against the very principle
> >> of 'maximum community diversity' that Harmony was created to aspire to.
> >>
> >> It's not only damaging, but entirely poisonous: it makes Harmony's own
> >> reason to exist hypocritical.
> >>
> >> And having SPEC donating a license to Apache won't solve anything: we
> >> want a random student from a developing country to be able to contribute
> >> as much as the top-notch performance engineer from a multinational
> >> corporation, as we strongly believe that innovation happens in the
> >> periphery more than it happens in the core.
> >>
> >> Failure to execute in such direction will mean that Harmony will always
> >> lag behind the communities that will manage to achieve that, a situation
> >> that frankly I equate to 'social thermal death'.
> >>
> >> Let me repeat for the third time: this is not to say that SPEC has no
> >> reason to exist or that we should ignore it or ridicule it. I never said
> >> that nor I think that's the case.
> >>
> >> All I want is to lower the barriers to participation for anybody, for
> >> anything, anywhere.
> >>
> >> And with that in mind, DaCapo is far superior than SPEC, simply because
> >> of their licensing model.
> >>
> >> --
> >> Stefano.
> >>
> >>
>

Mime
View raw message