Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: pass (asf.osuosl.org: local policy)
Date: Fri, 01 Sep 2006 10:45:07 +0200
From: Vemund Ostgaard <Vemund.Ostgaard@Sun.COM>
Subject: Re: [junit] Top-level suites added
In-reply-to: <44F5D6EA.60607@apache.org>
Sender: Vemund.Ostgaard@Sun.COM
To: derby-dev@db.apache.org
Message-id: <44F7F313.4070508@sun.com>
MIME-version: 1.0
Content-type: text/plain; format=flowed; charset=us-ascii
Content-transfer-encoding: 7BIT
References: <44EF8416.5040503@apache.org> <44F58A22.7030702@sun.com>
 <44F5D6EA.60607@apache.org>
User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7.12) Gecko/20050927

Daniel John Debrunner wrote:

>The current approach has very little magic, one decorator for client and
>probably one decorator for DB2 client. I strongly agree that tests
>should not be overly influenced by the suite setup.
>  
>
I think this is a sound principle to keep in mind as work progresses on 
this new testsuite. It might not be possible to avoid suite influence 
totally, but hopefully it is kept at a level where it is not regarded as 
a problem with the testsuite.

>This is an interesting idea, some thoughts jump out at me:
>
> - A single test would potentially run in many configurations:
>     embedded, client, db2 client, encryption DES, encryption AES, ...
>  
>
If the number of configurations grows large, it might pay off to 
carefully review wether something should be a configuration of its own 
or rather be tested through a set of more specialized tests. When the 
master files are gone and we have a new assertion-based testsuite, the 
tests should only fail on things they are specifically looking for, 
possibly making them less suited for testing of a large number of 
configurations? I guess it will be a tradeoff, having many 
configurations or more tests.

> - The large number of decorators required per test could factor into
>large memory use for an entire suite, not sure of the overhead of each
>TestSetup.
>
> - Embedded tests would be running with the network server booted by
>default, not sure of the impact of this, starting the network server for
>each individual TestCase would slow an entire suite down too much
>  
>
Are you thinking of the impact of the network server running in the same 
vm as junit?

> - It does make running a single TestCase in all configurations easy
>
> - The potential requirement to modify all tests when a new
>configuration is added into the run, e.g. if I add a configuration that
>only works in 10% of the tests (that's my itch) then I have to modify
>the other 90% of tests to disable it from running. Assuming the default
>as you say is to run each test in all possible configurations, I assume
>this would be enforced in the base test class. Maybe that's the case
>today with the current approach where a test is ideally self describing.
>  
>
I guess that the default set of configurations that a TestCase supports 
could be something smaller than the set of all possible configurations. 
The default runtime configuration would be to run all tests in all 
possible configurations, but a TestCase with default settings would 
override this with its smaller (default) set. You would then have to fix 
your 10% tests by adding the new configuration to their supported set, 
instead of removing it from the other 90%. Still, that is work, but I 
would hope we don't end up with a jungle of configurations that anyhow 
would be a pain to maintain.

I do believe, though, that it makes sense that a TestCase itself 
maintains the information about what configurations/frameworks it 
supports or requires, and not suite classes located somewhere else.

> - Maybe there are some configurations that are best handled at the leaf
>level, e.g. embedded and client, but others are best handled at a suite
>level, e.g. db2 client, encryption. So a mix of the two approaches would
>be good.
>  
>
Yes, that may very well be. As I've mentioned above I also think it 
should be considered if all configurations are really needed or wether 
it can be tested as well or better with new tests instead.

>I'm going to continue my current setup approach because it's not opposed
>to what you approach, the majority of the work will be getting tests
>converted to JUnit and running in client & db2 client mode. Ie. most of
>the work overlaps anyway.
>  
>
Yes, that is true. The work you are doing now is making it a lot easier 
to imagine how things may fit together with a testsuite written and run 
only with junit, and to have opinions on that.

Vemund