db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kathey Marsden <kmarsdende...@sbcglobal.net>
Subject Re: Server and client compatibility test for 10.2 and 10.1
Date Fri, 20 Jan 2006 21:25:37 GMT
Myrna van Lunteren wrote:

>
>I think there's another change required in the test harness to support this
>type of testing. There'll be need for test skipping, for sed-ing, and
>(hopefully minimal) canons for diffing. Is that being implemented?
>
> I imagine something similar to what is currently in place for different JCC
>clients.
>
>However, with the JCC clients, we always use the derbyTesting.jar of the
>server level.
>
The more I think about it,  the JCC model for the test harness is not
right.  I think Raman has it right to test with the lower rev testing
jar.  The product should always work at least at the level provided by
Min(serverVersion,clientVersion) and that is what she is  testing.  As
you mention,  if you test with the newer testing jar and an older client
or server, the number of tests that can be run as new features  are
added and the tests change  will deteriorate rapidly.

This is what I propose as a  quick way to be able to run these  tests
and a  reasonable model moving forward.
Summary
-  Log a Jira entry for the likely regression.
-  Create a new test harness property runwithMixedVersions=<true |
false> default true
-  Create a BehaviourChecker class which tests can use to check expected
product behaviour
-  Update 10.1 tests as needed to use the runwithMixedVersions property
or BehaviourChecker methods

Details below:

1) File a high priority Jira issue for the one apparent regression
found.  <>(Stream of column value in result cannot be retrieved twice
error in connectionJdbc20.java  and LobTest.java)

2) Add a property to the test  harness,   runwithMixedVersions=<true |
false> default true
     This property would mean that the test would not run at all if the
server and client are different versions   This is a quick way to get
the suite  running cleanly.

3)In the functionalTests/util directory we make a BehaviourChecker class
(or better name) which tests can use to check the behaviour based on
combinations, drivers or whatever.  It will take a connection in the
constructor and will have methods like
closesForwardOnlyRSOnFetchOfLastRow() or
supportsAutogeneratedKeysForMultipleRowInserts() which check the
DatabaseMetaData to make the right decision.  For example,
closesForwardOnlyRSOnFetchOfLastRow()  will be true  for JCC and false 
if  the driverVersiion is greater than or equal to 10.2 (DERBY-213).

3) In the 10.1 branch, set the property for the failing tests. Hopefully
someone will start weekly runs with 10.1/10.2 combinations.

4) Java tests that need to handle different behaviour in order to pass 
can either do something generic to resolve the problem or use
BehaviourChecker to do the right thing.  Always the tests are being
changed in the lower level branch, right now that means 10.1.

Examples:
derbynet/NSinSameJVM.java
Test fails because network server no longer outputs connections to the
console by default.  The test can be enabled by changing the location of
the console output for the test, since it is not the console output
being tested.

jdbcapi/autoGeneratedJdbc30.java
Test fails because of different results with (10.1 client/10.2 server).  The 10.1 test could
be be enabled by.

1) Add a method to BehaviourChecker
public boolean suppportsAutogeneratedKeysOnMultipleRowInsert()
{
       // if database version from DatabaseMetaData
greatThanOrEqualTo(10,2,0)  return true
      // code left as excercise for the reader.
}

2) Change the test
    BehaviourChecker checker = new BehaviourChecker(conn);
     ....
    if (checker.suppportsAutogeneratedKeysOnMultipleRowInsert())
    {
          // expect one thing
    }
   else
    {
       // expect something else
    }


Pros as I see it:
-  We keep messy test changes in the old branches. 
-  We avoid excluding entire tests because of expected differences or a
single issue
-  BehaviourChecker has the beneficial side effect of documenting
expected behaviour.
-  BehaviourChecker  could be expanded to handle a lot of the  harness
property functionality as well as replace the various  framework
checking  logic in the individual tests.


Cons
- It is just not going to work for .sql tests.
- Those interested in maintaining this testing  will have to do work in
the maintenance branches, but I just don't see a way around that and it
keeps our trunk tests clean. 
-  Behaviour is spelled different ways by different people so probably a
different name would be better.

Kathey



Mime
View raw message