Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4E8C7653.1010106@oracle.com>
Date: Wed, 05 Oct 2011 17:22:59 +0200
From: Kristian Waagan <kristian.waagan@oracle.com>
User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US;
 rv:1.9.2.15) Gecko/20110412 Thunderbird/3.1.9
MIME-Version: 1.0
To: derby-dev@db.apache.org
Subject: Compatibility test (was: Re: More. Trouble with JVMInfo)
References: <204BEF24-0F42-40C4-A818-D10926780D34@sbcglobal.net>
 <4E7C9F8B.2090605@oracle.com> <1316809886.3626.17.camel@tecra>
 <4E807F5E.2020506@oracle.com> <4E80A938.9090302@sbcglobal.net>
 <4E80C0DB.30100@oracle.com> <4E80ED12.10906@sbcglobal.net>
 <4E81BFC9.7070802@oracle.com> <4E81CA2B.6010906@sbcglobal.net>
 <wjo8vcse12p6.fsf@oracle.com> <4E81FDE7.8040803@sbcglobal.net>
In-Reply-To: <4E81FDE7.8040803@sbcglobal.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 27.09.11 18:46, Kathey Marsden wrote:
> On 9/27/2011 7:26 AM, Knut Anders Hatlen wrote:
>> Another question is whether mixed versions is a configuration we need
>> to support (I don't think it's stated explicitly anywhere that we
>> actually do support it?).
> Years ago, I know this support was very important to several of our
> large consumers, but perhaps needs have changed. I am sure those that
> expressed the need for this to keep working are not up to 10.7 yet. I
> had thought CompatibiltyTest was doing the basic testing for mixed jars.
> I guess though, that test is just for basic protocol testing and must
> use separate class loaders or the basic testing would have failed.

--- Compatibility test rewrite ---

I recently spent time rewriting the compatibility test (DERBY-2076), and 
can confirm that it doesn't test mixed version jars. It uses separate 
processes for the client and the server.

However, while rewriting the test, I did find one or two problems 
related to mixing code from different releases. For instance, you cannot 
shut down a really old server using a newer NetworkServerControl. In 
this case I had to fork off a process to shut down the server using the 
old code.
The test is dependent on being able to run the newest derbyTesting.jar 
with the client and the server code (for instance to find the code for 
stored procedures).

I'd be interested in hearing what people believe the compatibility test 
should be able to do. The new version of the test only runs a test suite 
with a set of client versions against a set of server versions. It 
doesn't yet support running the client and the server with different JVM 
versions.


--- Compatibility test coverage ---

I have no metrics for what kind of coverage the current compatibility 
test gives, but I suspect it is rather low. Adding tests to the 
compatibility suite is in theory independent of the test framework, but 
running for instance suites.All with mixed versions of the client and 
the server will fail for several reasons:
  o existing network server setup decorators will interfere with the 
running server
  o features not existing in the client will be attempted tested if the 
newest derbyTesting.jar is used


--- Mixed jars testing ---
--- Run only client or embedded tests when invoking _Suite ---

As an experiment I added functionality to the test framework to run only 
client or embedded tests. This included minor changes in 
TestConfiguration and relatively small changes in a whole lot of 
suite()-methods. I only carried out the experiment for lang and jdbcapi.

When running lang._Suite with a 10.6 server and trunk client/testing, I 
got this from the swing testrunner:
589/618, 67 errors, 23 failures.
Many of the issues were related to BOOLEAN and truncate table (not 
supported by 10.6).

When running lang._Suite with a trunk server/testing and 10.6 client, I 
got this from the swing testrunner:
611/618, 4 errors, 26 failures.
Again many of the failures were related BOOLEAN.

I don't know if it is worth doing this kind of testing, and how to 
interpret the results, but the ability to run only embedded or client 
tests may be something people find valuable.

Mixed jars testing requires manual inspection of the results, since the 
test cases don't carry information about which versions of Derby support 
the features being tested.

To detect regressions it might be better to run the (old) tests against 
a newer server?


-- 
Kristian


[ snip ]