db-derby-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Db-derby Wiki] Update of "TestRegressions" by DavidVanCouvering
Date Fri, 20 Jan 2006 19:33:50 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The following page has been changed by DavidVanCouvering:

New page:
= Handling Test Regressions =

It is crucial that our regression test suite, derbyall, run with no failures.  This page describes
mechanisms and guidelines to help ensure that this is the case.

== Awareness ==

It's important for Derby developers to be aware of the state of the regression suite.  For
this reason, any time an automated regression test fails, an email will be sent to derby-dev
indicating that there were test failures and pointing to which tests failed.

We are also creating a separate mailing list called derby-test-results that posts the results
of each regression test run, regardless of whether it passed or failed.  A developer can subscribe
to this if they want to keep a constant pulse of test results.  Of one particular use is knowing
whether your recent checking caused tests to hang or fail to build so that test results don't
even get sent out.

Automated regression test results with lots of detail about the failure and showing the changes
that were applied for each revision can be obtained at http://www.multinet.no/~solberg/public/Apache/index.html.

== Managing Test Regressions ==

What follows are guidelines for how test regressions should be handled

* A committer is responsible for committing changes that do not regress the tests.  If they
notice that a commit they made appears to have caused a test failure, they should immediately
work to resolve it, working with the contributor if necessary.  Of course a committer can
not test on all platforms, and may not even be able to resolve a test failure on a platform
to which they do not have access, so in some cases, they may need to work with someone whose
itch is to have Derby work on that platform to fix the test

* If anyone notices that a test that is failing and it is not immediately resolved, you are
strongly encouraged to log a JIRA issue for it.  The issue should be given a high priority
(Critical for any test failure that crosses all platforms) and assigned to the JIRA category

* If necessary, a committer may take the matter into their own hands by directly contacting
the person they believe is responsible for the regression or even vetoing all code commits
that are not related to removing test regressions.

* For some test regressions, the issue may be minor and fixing the test may be significant.
 In this case the test should be removed from derbyall, or configured to not run in the configurations
which cause failures, and the JIRA issue category should be changed to Test.

View raw message