db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel John Debrunner <...@apache.org>
Subject Re: [junit] Top-level suites added
Date Wed, 30 Aug 2006 18:20:26 GMT
Vemund Ostgaard wrote:


> I have a suggestion, based on the following view:
> 
> * I would avoid creating suites just to handle the various frameworks,
> or other configuration related things. This would mean a simpler
> structure of suites, easier to understand and maintain. If we want to
> have a  MATS suite for instance, we would not need an embedded_MATS
> suite and a client_MATS suite and so forth to build the structure.
> * It would be good if as little as possible (ideally no) magic was
> performed on a suite level. Tests should behave the same way when
> executed through the lowest level "leaf" suite as when executed as part
> of "derbyall", making it easier to reproduce problems.

The current approach has very little magic, one decorator for client and
probably one decorator for DB2 client. I strongly agree that tests
should not be overly influenced by the suite setup.

> How could this be accomplished?
> 
> Handle everything related to frameworks and configuration on a "leaf"
> level, I believe this could be done in the suite() method in each
> TestCase. Just add each Test once for each framework that this run is
> configured to use, using the correct decorator. If a test is not
> supposed to be run for a given framework, that information should be
> maintained in that TestCase in some way, so that test is not added for
> that framework. Default would be to run all Tests in all frameworks.

This is an interesting idea, some thoughts jump out at me:

 - A single test would potentially run in many configurations:
     embedded, client, db2 client, encryption DES, encryption AES, ...

 - The large number of decorators required per test could factor into
large memory use for an entire suite, not sure of the overhead of each
TestSetup.

 - Embedded tests would be running with the network server booted by
default, not sure of the impact of this, starting the network server for
each individual TestCase would slow an entire suite down too much

 - It does make running a single TestCase in all configurations easy

 - The potential requirement to modify all tests when a new
configuration is added into the run, e.g. if I add a configuration that
only works in 10% of the tests (that's my itch) then I have to modify
the other 90% of tests to disable it from running. Assuming the default
as you say is to run each test in all possible configurations, I assume
this would be enforced in the base test class. Maybe that's the case
today with the current approach where a test is ideally self describing.

 - Maybe there are some configurations that are best handled at the leaf
level, e.g. embedded and client, but others are best handled at a suite
level, e.g. db2 client, encryption. So a mix of the two approaches would
be good.

I'm going to continue my current setup approach because it's not opposed
to what you approach, the majority of the work will be getting tests
converted to JUnit and running in client & db2 client mode. Ie. most of
the work overlaps anyway.

Thanks for your thoughts,
Dan.










Mime
View raw message