db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Myrna van Lunteren <m.v.lunte...@gmail.com>
Subject Re: Server and client compatibility test for 10.2 and 10.1
Date Sat, 21 Jan 2006 02:39:39 GMT
On 1/20/06, Kathey Marsden <kmarsdenderby@sbcglobal.net> wrote:
> Myrna van Lunteren wrote:
> >
> >I think there's another change required in the test harness to support
> this
> >type of testing. There'll be need for test skipping, for sed-ing, and
> >(hopefully minimal) canons for diffing. Is that being implemented?
> >
> > I imagine something similar to what is currently in place for different
> >clients.
> >
> >However, with the JCC clients, we always use the derbyTesting.jar of the
> >server level.
> >
> The more I think about it,  the JCC model for the test harness is not
> right.  I think Raman has it right to test with the lower rev testing
> jar.  The product should always work at least at the level provided by
> Min(serverVersion,clientVersion) and that is what she is  testing.  As
> you mention,  if you test with the newer testing jar and an older client
> or server, the number of tests that can be run as new features  are
> added and the tests change  will deteriorate rapidly.
> This is what I propose as a  quick way to be able to run these  tests
> and a  reasonable model moving forward.
> Summary
> -  Log a Jira entry for the likely regression.
> -  Create a new test harness property runwithMixedVersions=<true |
> false> default true
> -  Create a BehaviourChecker class which tests can use to check expected
> product behaviour
> -  Update 10.1 tests as needed to use the runwithMixedVersions property
> or BehaviourChecker methods
> Details below:
> 1) File a high priority Jira issue for the one apparent regression
> found.  <>(Stream of column value in result cannot be retrieved twice
> error in connectionJdbc20.java  and LobTest.java)
> 2) Add a property to the test  harness,   runwithMixedVersions=<true |
> false> default true
>     This property would mean that the test would not run at all if the
> server and client are different versions   This is a quick way to get
> the suite  running cleanly.
> 3)In the functionalTests/util directory we make a BehaviourChecker class
> (or better name) which tests can use to check the behaviour based on
> combinations, drivers or whatever.  It will take a connection in the
> constructor and will have methods like
> closesForwardOnlyRSOnFetchOfLastRow() or
> supportsAutogeneratedKeysForMultipleRowInserts() which check the
> DatabaseMetaData to make the right decision.  For example,
> closesForwardOnlyRSOnFetchOfLastRow()  will be true  for JCC and false
> if  the driverVersiion is greater than or equal to 10.2 (DERBY-213).
> 3) In the 10.1 branch, set the property for the failing tests. Hopefully
> someone will start weekly runs with 10.1/10.2 combinations.
> 4) Java tests that need to handle different behaviour in order to pass
> can either do something generic to resolve the problem or use
> BehaviourChecker to do the right thing.  Always the tests are being
> changed in the lower level branch, right now that means 10.1.
> Examples:
> derbynet/NSinSameJVM.java
> Test fails because network server no longer outputs connections to the
> console by default.  The test can be enabled by changing the location of
> the console output for the test, since it is not the console output
> being tested.

> Test fails because of different results with (10.1 client/10.2
> server).  The 10.1 test could be be enabled by.
> 1) Add a method to BehaviourChecker
> public boolean suppportsAutogeneratedKeysOnMultipleRowInsert()
> {
>       // if database version from DatabaseMetaData
> greatThanOrEqualTo(10,2,0)  return true
>      // code left as excercise for the reader.
> }
> 2) Change the test
>    BehaviourChecker checker = new BehaviourChecker(conn);
>     ....
>    if (checker.suppportsAutogeneratedKeysOnMultipleRowInsert())
>    {
>          // expect one thing
>    }
>   else
>    {
>       // expect something else
>    }
> Pros as I see it:
> -  We keep messy test changes in the old branches.
> -  We avoid excluding entire tests because of expected differences or a
> single issue
> -  BehaviourChecker has the beneficial side effect of documenting
> expected behaviour.
> -  BehaviourChecker  could be expanded to handle a lot of the  harness
> property functionality as well as replace the various  framework
> checking  logic in the individual tests.
> Cons
> - It is just not going to work for .sql tests.
> - Those interested in maintaining this testing  will have to do work in
> the maintenance branches, but I just don't see a way around that and it
> keeps our trunk tests clean.
> -  Behaviour is spelled different ways by different people so probably a
> different name would be better.
> Kathey

Makes sense.

re sql tests - I found, at least in main, while looking for the 'excludeJCC'
property we have in the test harness, (which apparently is currently not
used in any test properties file) that possibly the harness will look for
canons for clients in directories ver + major number + minor number, e.g. '
ver10.1'. or 'ver10.2'. It looked like it tries this whether the client is
DerbyNet or DerbyNetClient. Worthwhile checking into for whomever is going
to do this? This interesting bit of code is in harness/FileCompare.java.
I've not tested it.


View raw message