harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Loenko" <mloe...@gmail.com>
Subject [build-test] regular runs of applications (was: Re: [performance] a few early benchmarks)
Date Tue, 28 Nov 2006 13:21:59 GMT
2006/11/28, Geir Magnusson Jr. <geir@pobox.com>:
>
>
> Mikhail Loenko wrote:
> > And we need to make sure that a patch that improves DaCapo by 0.1%
> > does not lower down the SPEC by 10%
>
> Absolutely :)
>
> >
> > We need to agree on a target function to optimize, like some weighted mean.
>
> It would be useful to understand what SPECjbb is actually testing - can
> someone describe it?
>
> Then, like a financial portfolio, we can weight based on those factors...

One more thing to work on is make sure that the workloads just run.
It's not that easy to achieve some performance on the workloades if they
just don't work :)

I suggest that we setup regular runs of some workloads and report
failures when regressions happen.

Comments?

Thanks,
Mikhail

>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/11/28, Geir Magnusson Jr. <geir@pobox.com>:
> >> The key is to not claim that SPEC is representative of "the" benchmark
> >> for Harmony.
> >>
> >> Sure, people can use it to test Harmony, but we need to be very careful
> >> here that someones drive for SPEC numbers doesn't have harmful effects
> >> on other benchmarks.  It's an interesting balancing act we'll have to
> >> do...
> >>
> >> geir
> >>
> >>
> >> Mikhail Loenko wrote:
> >> > It seems that we need to define first what "the" performance benchmark
> >> > means.
> >> >
> >> > Depending on that we'll be able to say whether for-pay workload will
> >> create
> >> > a barrier or not. Overall it's a question of how we organize the work.
> >> > When I do something for my employer it's not necessarily me, who runs
> >> > benchmarks on the code that I develop. If we have people committed
> >> > then they could run their favorite benchmarks on each JIRA patch, and
> >> > report if
> >> > they want. That is similar to how we are going to support many
> >> platforms:
> >> > not every student has a variety of platforms Harmony supports.
> >> > But that is how community works and I don't see any problem here.
> >> >
> >> > Thanks,
> >> > Mikhail
> >> >
> >> > 2006/11/28, Stefano Mazzocchi <stefano@apache.org>:
> >> >> Elford, Chris L wrote:
> >> >> > Stefano Mazzocchi writes:
> >> >> >>> ...
> >> >> >>> Then you have to realize that not everybody has the $$$
or the
> >> >> > intention
> >> >> >>> to spend money to help us optimize Harmony.
> >> >> >
> >> >> > Your point that there is a $$$ barrier to entry for utilizing
SPEC
> >> >> > workloads is a good one.  I wonder if SPEC would consider some
> >> sort of
> >> >> > grant to Apache to use their workloads with Harmony...  The SPEC
> >> site
> >> >> > (http://www.spec.org/order.html) seems to read as if a single
Apache
> >> >> > license might cover all Harmony developers...
> >> >> >
> >> >> >>> ...
> >> >> >>> Our main and only objective is to please our users, and
> >> reducing the
> >> >> >>> barrier for them to become developers as much as we possibly
can.
> >> >> >
> >> >> > I agree wholeheartedly with the first part of this sentence
> >> regarding
> >> >> > pleasing our users.  For the next 9 months or so, I'd also agree
> >> with
> >> >> > the second part of this sentence in that lowering the barrier
to
> >> >> > developer entry is strongly desirable.  Your point is good in
> >> that for
> >> >> > now, we need to do whatever we can to make sure that key workloads
> >> >> > (including performance workloads) are available to our
> >> developers.  In
> >> >> > the longer term, I would hope that we have more users than
> >> developers
> >> >> > and that developers will pay attention to performance [and
> >> functional]
> >> >> > bugs filed against Harmony on behalf of applications for which
the
> >> >> > source and/or all executable bits are unavailable.
> >> >> >
> >> >> >>> ...
> >> >> >>> But benchmarks have two objectives:
> >> >> >>> show the current performance status *and* allow potential
> >> developers
> >> >> > to
> >> >> >>> test if their changes have made a positive impact before
> >> submitting
> >> >> > them.
> >> >> >
> >> >> > A benchmark is created to stress some combination of technologies.
> >> >> > Benchmarks tend to be distilled from some set of real world
> >> >> > requirements.  As someone who worked for years with industy
> >> consortia
> >> >> > for standardizing benchmarks, I would argue that benchmarks have
> >> more
> >> >> > than the two objectives you mention.  To name a few: attracting
new
> >> >> > users, showcasing your "product", providing a platform for level
> >> >> playing
> >> >> > field comparisons with alternate implementations, allowing
> >> >> demonstration
> >> >> > of technology readiness, driving improvements in key portions
of the
> >> >> > technology stack, ....
> >> >> >
> >> >> > To showcase Harmony as a performance competitive JRE with technology
> >> >> > readiness for client and enterprise class deployments, I believe
> >> >> that we
> >> >> > will need to agree to underscore some combination of academic
> >> >> benchmarks
> >> >> > (such as Dacapo memory management benchmarks), domain benchmarks
> >> (such
> >> >> > as SciMark, XMLbench, etc.) and more general standardized industry
> >> >> > benchmarks (such as SPEC and/or TPC).  Of course we all have our
own
> >> >> > biases but I personally don't think that Harmony will be successful
> >> >> > without some due diligence applied to the latter and I believe
that
> >> >> > Harmony needs to find some way of working with some these benchmarks
> >> >> > (e.g., SPECjbbXXXX, SPECjvmXXXX, SPECjAppServerXXXX, TPC-App).
> >> >> >
> >> >> > I assume that we will have contributors interested in different
> >> >> > combinations of these benchmarks.  Harmony needs to create some
> >> guiding
> >> >> > principles as to how the design/implementation will accommodate
> >> these
> >> >> > varied performance interests while moving Harmony toward an ever
> >> better
> >> >> > performing platform for our users.  Optimizing for performance
> >> tends to
> >> >> > be at odds with optimizing for portability and/or for
> >> >> maintainability in
> >> >> > that it tends to involve fast path exceptions to generalized code.
> >> >>
> >> >> Chris,
> >> >>
> >> >> I don't claim to be an expert in performance evaluation nor I claim
to
> >> >> know which test is more useful than others for pure performance
> >> purpose.
> >> >> It's not my competence and frankly not really my interest either.
> >> >>
> >> >> My goal is to make Harmony successful and 'successful', to me, means
a
> >> >> healthy and diverse community of users and developers, the largest
and
> >> >> more diverse possible.
> >> >>
> >> >> Harmony was started because we believed that the existing licensing
> >> >> scenarios of 'FOSS java' was going to be a limitation for such a
> >> >> community to be the largest and more diverse possible (ranging from
> >> >> individual hobbyists to multinational corporations' engineers). Sun's
> >> >> recent OpenJDK effort has not improved this scenario.
> >> >>
> >> >> Harmony continues, IMO (and I'm obviously biased), to have the lead
in
> >> >> the 'potential' of community building in terms of maximum licensing
> >> >> appeal, but decisions like using a non-open, for-pay workload as "the"
> >> >> performance benchmark for this project goes against the very principle
> >> >> of 'maximum community diversity' that Harmony was created to aspire
> >> to.
> >> >>
> >> >> It's not only damaging, but entirely poisonous: it makes Harmony's
own
> >> >> reason to exist hypocritical.
> >> >>
> >> >> And having SPEC donating a license to Apache won't solve anything:
we
> >> >> want a random student from a developing country to be able to
> >> contribute
> >> >> as much as the top-notch performance engineer from a multinational
> >> >> corporation, as we strongly believe that innovation happens in the
> >> >> periphery more than it happens in the core.
> >> >>
> >> >> Failure to execute in such direction will mean that Harmony will
> >> always
> >> >> lag behind the communities that will manage to achieve that, a
> >> situation
> >> >> that frankly I equate to 'social thermal death'.
> >> >>
> >> >> Let me repeat for the third time: this is not to say that SPEC has
no
> >> >> reason to exist or that we should ignore it or ridicule it. I never
> >> said
> >> >> that nor I think that's the case.
> >> >>
> >> >> All I want is to lower the barriers to participation for anybody, for
> >> >> anything, anywhere.
> >> >>
> >> >> And with that in mind, DaCapo is far superior than SPEC, simply
> >> because
> >> >> of their licensing model.
> >> >>
> >> >> --
> >> >> Stefano.
> >> >>
> >> >>
> >>
>

Mime
View raw message