db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kathey Marsden (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DERBY-6210) Create a mechanism to exclude some testing of internal interfaces and likely to change behavior from compatibility testing
Date Wed, 01 May 2013 18:34:15 GMT
Kathey Marsden created DERBY-6210:
-------------------------------------

             Summary: Create a mechanism to exclude some testing of internal interfaces and
likely to change behavior from compatibility testing
                 Key: DERBY-6210
                 URL: https://issues.apache.org/jira/browse/DERBY-6210
             Project: Derby
          Issue Type: Bug
            Reporter: Kathey Marsden


For compatibility and upgrade testing, in order to expose bugs and omitted release notes I
run the tests from an older branch against new releases.  The premise is that the functional
tests are really just a JDBC application which should continue to pass with the new version.

The JUnit tests for Derby though are not pure functional tests which are backward compatible.
 Often they make checks on things that are likely to change with new versions. This creates
a large amount of noise in the testing: For example:

1) Checks for the exact contents or number of system tables.
2) Checks for "Not implemented" JDBC API calls which might become implemented.
3) Tests for client specific behavior which we expect might change to match embedded.
4) Unit tests which use internal interfaces that are likely to change.
5) Metadata tests which test the exact number of columns when 
6) Diff based tests which are likely to change with message changes.

It would be good to have a flag to omit these types of tests when doing compatibility testing.
My thought is to have a system property derby.tests.testCompat=true   and a method in BaseTestCase
to test the property testCompat().


Then blocks of testing which should not be run or should have a different behavior with compatibility
testing can be flagged as:
if (! testCompat())  {

// do testing  that we think might change, e.g. system table query.

}

This will allow the more detailed testing under normal circumstances but take some of the
pain out of the compatibility testing moving forward.  It would require folks to think as
they add new tests whether their testing as to whether they expect the tested behavior  to
remain stable moving forward and put the block around it if they do not.

I welcome feedback on whether this or some other approach is preferable.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message