Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Date: Wed, 1 May 2013 18:34:15 +0000 (UTC)
From: "Kathey Marsden (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Message-ID: <JIRA.12645675.1367433216991.255960.1367433255822@arcas>
In-Reply-To: <JIRA.12645675.1367433216991@arcas>
References: <JIRA.12645675.1367433216991@arcas>
Subject: [jira] [Created] (DERBY-6210) Create a mechanism to exclude some
 testing of internal interfaces and likely to change behavior from
 compatibility testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Kathey Marsden created DERBY-6210:
-------------------------------------

             Summary: Create a mechanism to exclude some testing of internal interfaces and likely to change behavior from compatibility testing
                 Key: DERBY-6210
                 URL: https://issues.apache.org/jira/browse/DERBY-6210
             Project: Derby
          Issue Type: Bug
            Reporter: Kathey Marsden


For compatibility and upgrade testing, in order to expose bugs and omitted release notes I run the tests from an older branch against new releases.  The premise is that the functional tests are really just a JDBC application which should continue to pass with the new version.

The JUnit tests for Derby though are not pure functional tests which are backward compatible.  Often they make checks on things that are likely to change with new versions. This creates a large amount of noise in the testing: For example:

1) Checks for the exact contents or number of system tables.
2) Checks for "Not implemented" JDBC API calls which might become implemented.
3) Tests for client specific behavior which we expect might change to match embedded.
4) Unit tests which use internal interfaces that are likely to change.
5) Metadata tests which test the exact number of columns when 
6) Diff based tests which are likely to change with message changes.

It would be good to have a flag to omit these types of tests when doing compatibility testing.
My thought is to have a system property derby.tests.testCompat=true   and a method in BaseTestCase to test the property testCompat().


Then blocks of testing which should not be run or should have a different behavior with compatibility testing can be flagged as:
if (! testCompat())  {

// do testing  that we think might change, e.g. system table query.

}

This will allow the more detailed testing under normal circumstances but take some of the pain out of the compatibility testing moving forward.  It would require folks to think as they add new tests whether their testing as to whether they expect the tested behavior  to remain stable moving forward and put the block around it if they do not.

I welcome feedback on whether this or some other approach is preferable.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira