Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Wed, 08 Feb 2006 21:12:43 +0100
From: Dyre.Tjeldvoll@Sun.COM
Subject: Re: Regression Test Harness handling of "duration" field
In-reply-to: <43EA38FD.1050003@amberpoint.com>
To: derby-dev@db.apache.org
Message-id: <x1o33bithao4.fsf@sun.com>
MIME-version: 1.0
Content-type: text/plain; charset=us-ascii
User-Agent: Gnus/5.110003 (No Gnus v0.3) Emacs/21.2 (usg-unix-v)
References: <43E9ED78.6080405@Sun.COM> <43EA1B74.1030008@amberpoint.com>
 <43EA2780.9060407@MultiNet.no> <43EA2C22.3020304@amberpoint.com>
 <43EA36ED.3020200@sun.com> <43EA38FD.1050003@amberpoint.com>

>>>>> "BP" == Bryan Pendleton <bpendleton@amberpoint.com> writes:

    BP> David W. Van Couvering wrote:
    >> My understanding it's the wall clock time.

    >>> derbyall   630 6 624 0
    >>> Duration   45.6%

    BP> OK. So it's saying that this particular run of 'derbyall'
    BP> took 46.2% of the wall clock time that 'derbyall' took
    BP> on August 2, 2005.

    BP> That makes sense.

    BP> Given that this particular run failed badly, we probably
    BP> don't care about the Durations, then.

    BP> But in general, we'd probably want to keep our eyes open
    BP> for Duration values that started to get significantly
    BP> higher than 100%, because that would mean that we might
    BP> have accidentally introduced a performance regression.

    BP> Perhaps we could have some sort of trigger, so that if a
    BP> suite experienced a duration of, say, 150%, that was
    BP> treated as a regression failure, even if all the tests in
    BP> that suite passed?

Given that the "Derby way" requires all bug fixes to be accompanied
with a regression test that tests that the bug has not been
re-introduced, AND that all such regression tests should be a part of
derbyall (this is what I've been told, if it is not correct, please
let me know), it should only be a matter of time before your trigger
fires. Unless you create a new baseline regularly, of course...

-- 
dt