Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: neutral (herse.apache.org: local policy)
Message-ID: <45759718.7080406@apache.org>
Date: Tue, 05 Dec 2006 07:58:16 -0800
From: Daniel John Debrunner <djd@apache.org>
User-Agent: Thunderbird 1.5.0.8 (Windows/20061025)
MIME-Version: 1.0
To: derby-dev@db.apache.org
Subject: Re: [jira] Commented: (DERBY-2118) Change some boundary checks in
 ArrayInputStream to ASSERTs to improve performance
References: <24091008.1165273582952.JavaMail.jira@brutus>
 <wjo8bqmienkr.fsf@sun.com>
In-Reply-To: <wjo8bqmienkr.fsf@sun.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Knut Anders Hatlen wrote:
> "Daniel John Debrunner (JIRA)" <jira@apache.org> writes:
> 
>> Daniel John Debrunner commented on DERBY-2118:
>> ----------------------------------------------
>>
>> The tests pass for the clearLimit() changes and I performance
>> changes for the select test ranging from -0.70% to +2.44%.
>> Sometimes I seem to have trouble getting consistent numbers out of
>> the test from DERBY-1961, even with an idle machine.
> 
> I am also seeing huge variations with the test from DERBY-1961. In
> fact, I always see huge variations when I'm running performance tests
> on Derby. My best guess so far is that the native code produced by the
> hotspot compiler varies between runs (it uses statistics to decide
> what to optimize and how to optimize it, so it might not be 100%
> predictable). To eliminate the variation, I usually look at the
> average of multiple runs.

True, but when the variations are multiples of the percent improvement 
I'm expecting for the change (ie. 5-10% for a 1% change) I have little 
confidence I can detect if the code is providing an improvement or not.

It would be good to convert this test to the harness Sunitha provided in 
DERBY-1970, then multiple runs within the same VM could be performance, 
these tend to become more stable as time the JIT settles down. Of course 
some of it could be due to garbage collection.

Also DERBY-1961 test has a minor problem in its TPS reporting though it 
does not seem to be significant cause in the inconsistent numbers. The 
number of operations is counted (N) while the main thread is sleeping 
for R seconds and then TPS = N / R. The issue is that the thread is not 
guaranteed to sleep for R seconds, thus the collection time is really (R 
+ delta), so it should be TPS = N / (R + delta). I modified the code to 
measure the actual sleep time and it seems with these tests delta was 
less than 0.01 seconds. I can imagine with some multi-client tests that 
delta could become significant. Moving to the DERBY-1970 harness would 
at least ensure we had a single place where we could work on consistent 
good reporting numbers.

Thanks,
Dan.