harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [performance] a few early benchmarks
Date Tue, 28 Nov 2006 12:48:31 GMT


Mikhail Loenko wrote:
> And we need to make sure that a patch that improves DaCapo by 0.1%
> does not lower down the SPEC by 10%

Absolutely :)

> 
> We need to agree on a target function to optimize, like some weighted mean.

It would be useful to understand what SPECjbb is actually testing - can 
someone describe it?

Then, like a financial portfolio, we can weight based on those factors...

geir

> 
> Thanks,
> Mikhail
> 
> 2006/11/28, Geir Magnusson Jr. <geir@pobox.com>:
>> The key is to not claim that SPEC is representative of "the" benchmark
>> for Harmony.
>>
>> Sure, people can use it to test Harmony, but we need to be very careful
>> here that someones drive for SPEC numbers doesn't have harmful effects
>> on other benchmarks.  It's an interesting balancing act we'll have to 
>> do...
>>
>> geir
>>
>>
>> Mikhail Loenko wrote:
>> > It seems that we need to define first what "the" performance benchmark
>> > means.
>> >
>> > Depending on that we'll be able to say whether for-pay workload will 
>> create
>> > a barrier or not. Overall it's a question of how we organize the work.
>> > When I do something for my employer it's not necessarily me, who runs
>> > benchmarks on the code that I develop. If we have people committed
>> > then they could run their favorite benchmarks on each JIRA patch, and
>> > report if
>> > they want. That is similar to how we are going to support many 
>> platforms:
>> > not every student has a variety of platforms Harmony supports.
>> > But that is how community works and I don't see any problem here.
>> >
>> > Thanks,
>> > Mikhail
>> >
>> > 2006/11/28, Stefano Mazzocchi <stefano@apache.org>:
>> >> Elford, Chris L wrote:
>> >> > Stefano Mazzocchi writes:
>> >> >>> ...
>> >> >>> Then you have to realize that not everybody has the $$$ or
the
>> >> > intention
>> >> >>> to spend money to help us optimize Harmony.
>> >> >
>> >> > Your point that there is a $$$ barrier to entry for utilizing SPEC
>> >> > workloads is a good one.  I wonder if SPEC would consider some 
>> sort of
>> >> > grant to Apache to use their workloads with Harmony...  The SPEC 
>> site
>> >> > (http://www.spec.org/order.html) seems to read as if a single Apache
>> >> > license might cover all Harmony developers...
>> >> >
>> >> >>> ...
>> >> >>> Our main and only objective is to please our users, and 
>> reducing the
>> >> >>> barrier for them to become developers as much as we possibly
can.
>> >> >
>> >> > I agree wholeheartedly with the first part of this sentence 
>> regarding
>> >> > pleasing our users.  For the next 9 months or so, I'd also agree 
>> with
>> >> > the second part of this sentence in that lowering the barrier to
>> >> > developer entry is strongly desirable.  Your point is good in 
>> that for
>> >> > now, we need to do whatever we can to make sure that key workloads
>> >> > (including performance workloads) are available to our 
>> developers.  In
>> >> > the longer term, I would hope that we have more users than 
>> developers
>> >> > and that developers will pay attention to performance [and 
>> functional]
>> >> > bugs filed against Harmony on behalf of applications for which the
>> >> > source and/or all executable bits are unavailable.
>> >> >
>> >> >>> ...
>> >> >>> But benchmarks have two objectives:
>> >> >>> show the current performance status *and* allow potential 
>> developers
>> >> > to
>> >> >>> test if their changes have made a positive impact before 
>> submitting
>> >> > them.
>> >> >
>> >> > A benchmark is created to stress some combination of technologies.
>> >> > Benchmarks tend to be distilled from some set of real world
>> >> > requirements.  As someone who worked for years with industy 
>> consortia
>> >> > for standardizing benchmarks, I would argue that benchmarks have 
>> more
>> >> > than the two objectives you mention.  To name a few: attracting new
>> >> > users, showcasing your "product", providing a platform for level
>> >> playing
>> >> > field comparisons with alternate implementations, allowing
>> >> demonstration
>> >> > of technology readiness, driving improvements in key portions of the
>> >> > technology stack, ....
>> >> >
>> >> > To showcase Harmony as a performance competitive JRE with technology
>> >> > readiness for client and enterprise class deployments, I believe
>> >> that we
>> >> > will need to agree to underscore some combination of academic
>> >> benchmarks
>> >> > (such as Dacapo memory management benchmarks), domain benchmarks 
>> (such
>> >> > as SciMark, XMLbench, etc.) and more general standardized industry
>> >> > benchmarks (such as SPEC and/or TPC).  Of course we all have our own
>> >> > biases but I personally don't think that Harmony will be successful
>> >> > without some due diligence applied to the latter and I believe that
>> >> > Harmony needs to find some way of working with some these benchmarks
>> >> > (e.g., SPECjbbXXXX, SPECjvmXXXX, SPECjAppServerXXXX, TPC-App).
>> >> >
>> >> > I assume that we will have contributors interested in different
>> >> > combinations of these benchmarks.  Harmony needs to create some 
>> guiding
>> >> > principles as to how the design/implementation will accommodate 
>> these
>> >> > varied performance interests while moving Harmony toward an ever 
>> better
>> >> > performing platform for our users.  Optimizing for performance 
>> tends to
>> >> > be at odds with optimizing for portability and/or for
>> >> maintainability in
>> >> > that it tends to involve fast path exceptions to generalized code.
>> >>
>> >> Chris,
>> >>
>> >> I don't claim to be an expert in performance evaluation nor I claim to
>> >> know which test is more useful than others for pure performance 
>> purpose.
>> >> It's not my competence and frankly not really my interest either.
>> >>
>> >> My goal is to make Harmony successful and 'successful', to me, means a
>> >> healthy and diverse community of users and developers, the largest and
>> >> more diverse possible.
>> >>
>> >> Harmony was started because we believed that the existing licensing
>> >> scenarios of 'FOSS java' was going to be a limitation for such a
>> >> community to be the largest and more diverse possible (ranging from
>> >> individual hobbyists to multinational corporations' engineers). Sun's
>> >> recent OpenJDK effort has not improved this scenario.
>> >>
>> >> Harmony continues, IMO (and I'm obviously biased), to have the lead in
>> >> the 'potential' of community building in terms of maximum licensing
>> >> appeal, but decisions like using a non-open, for-pay workload as "the"
>> >> performance benchmark for this project goes against the very principle
>> >> of 'maximum community diversity' that Harmony was created to aspire 
>> to.
>> >>
>> >> It's not only damaging, but entirely poisonous: it makes Harmony's own
>> >> reason to exist hypocritical.
>> >>
>> >> And having SPEC donating a license to Apache won't solve anything: we
>> >> want a random student from a developing country to be able to 
>> contribute
>> >> as much as the top-notch performance engineer from a multinational
>> >> corporation, as we strongly believe that innovation happens in the
>> >> periphery more than it happens in the core.
>> >>
>> >> Failure to execute in such direction will mean that Harmony will 
>> always
>> >> lag behind the communities that will manage to achieve that, a 
>> situation
>> >> that frankly I equate to 'social thermal death'.
>> >>
>> >> Let me repeat for the third time: this is not to say that SPEC has no
>> >> reason to exist or that we should ignore it or ridicule it. I never 
>> said
>> >> that nor I think that's the case.
>> >>
>> >> All I want is to lower the barriers to participation for anybody, for
>> >> anything, anywhere.
>> >>
>> >> And with that in mind, DaCapo is far superior than SPEC, simply 
>> because
>> >> of their licensing model.
>> >>
>> >> --
>> >> Stefano.
>> >>
>> >>
>>

Mime
View raw message