river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: Next steps after 2.2.1 release
Date Sun, 07 Apr 2013 04:24:34 GMT
On 4/6/2013 7:26 PM, Greg Trasuk wrote:
...
> Once we have a stable set of regression tests, then OK, we could
> think about improving performance or using Maven repositories as the
> codebase server.
...

I think there is something else you need before it would be a good idea 
to release any changes for the sake of performance - some examples of 
workloads whose performance you want to improve, and that are in fact 
improved by the changes.

I've worked on many performance campaigns, at several different 
companies including Cray Research and Sun Microsystems. Each campaign 
has been based on one or more benchmarks. The benchmarks could be 
industry-standard benchmarks, or could be sanitized versions of user 
workloads.

I've had many opportunities to compare measurements to expert 
predictions, including my own predictions, about what needs to be 
changed to improve performance. In no case have the expert predictions 
matched measurement.

Based on that experience I had serious concerns about working on River 
performance with no benchmarks. Are there any River users who care 
enough about performance to help with creating benchmarks, and running 
them in highly parallel environments? If not, maybe performance changes 
should not be a priority.

Without benchmarks, changes intended to improve performance may carry 
functional risk for no gain, or even for a performance regression.

Patricia

Mime
View raw message