db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dyre.Tjeldv...@Sun.COM
Subject Re: Regression Test Harness handling of "duration" field
Date Wed, 08 Feb 2006 20:12:43 GMT
>>>>> "BP" == Bryan Pendleton <bpendleton@amberpoint.com> writes:

    BP> David W. Van Couvering wrote:
    >> My understanding it's the wall clock time.

    >>> derbyall   630 6 624 0
    >>> Duration   45.6%

    BP> OK. So it's saying that this particular run of 'derbyall'
    BP> took 46.2% of the wall clock time that 'derbyall' took
    BP> on August 2, 2005.

    BP> That makes sense.

    BP> Given that this particular run failed badly, we probably
    BP> don't care about the Durations, then.

    BP> But in general, we'd probably want to keep our eyes open
    BP> for Duration values that started to get significantly
    BP> higher than 100%, because that would mean that we might
    BP> have accidentally introduced a performance regression.

    BP> Perhaps we could have some sort of trigger, so that if a
    BP> suite experienced a duration of, say, 150%, that was
    BP> treated as a regression failure, even if all the tests in
    BP> that suite passed?

Given that the "Derby way" requires all bug fixes to be accompanied
with a regression test that tests that the bug has not been
re-introduced, AND that all such regression tests should be a part of
derbyall (this is what I've been told, if it is not correct, please
let me know), it should only be a matter of time before your trigger
fires. Unless you create a new baseline regularly, of course...

-- 
dt


Mime
View raw message