harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject [classlib] Testing conventions - a proposal
Date Wed, 05 Jul 2006 12:45:05 GMT

Just seen Tim's note on test support classes and it really caught my 
attention as I have been mulling over this issue for a little while now. 
I think that it is a good time for us to return to the topic of class 
library test layouts.

The current proposal [1] sets out to segment our different types of test 
by placing them in different file locations. After looking at the recent 
changes to the LUNI module tests (where the layout guidelines were 
applied) I have a real concern that there are serious problems with this 
approach. We have started down a track of just continually growing the 
number of test source folders as new categories of test are identified 
and IMHO that is going to bring complexity and maintenance issues with 
these tests.

Consider the dimensions of tests that we have ...

Run on classpath
Run on bootclasspath
Behaves different between Harmony and RI
...and so on...

If you weigh up all of the different possible permutations and then 
consider that the above list is highly likely to be extended as things 
progress it is obvious that we are eventually heading for large amounts 
of related test code scattered or possibly duplicated across numerous 
"hard wired" source directories. How maintainable is that going to be ?

If we want to run different tests in different configurations then IMHO 
we need to be thinking a whole lot smarter. We need to be thinking about 
keeping tests for specific areas of functionality together (thus easing 
maintenance); we need something quick and simple to re-configure if 
necessary (pushing whole directories of files around the place does not 
seem a particularly lightweight approach); and something that is not 
going to potentially mess up contributed patches when the file they 
patch is found to have been recently pushed from source folder A to B.

To connect into another recent thread, there have been some posts lately 
about handling some test methods that fail on Harmony and have meant 
that entire test case classes have been excluded from our test runs. I 
have also been noticing some API test methods that pass fine on Harmony 
but fail when run against the RI. Are the different behaviours down to 
errors in the Harmony implementation ? An error in the RI implementation 
? A bug in the RI Javadoc ? Only after some investigation has been 
carried out do we know for sure. That takes time. What do we do with the 
test methods in the meantime ? Do we push them round the file system 
into yet another new source folder ? IMHO we need a testing strategy 
that enables such "problem" methods to be tracked easily without 
disruption to the rest of the other tests.

A couple of weeks ago I mentioned that the TestNG framework [2] seemed 
like a reasonably good way of allowing us to both group together 
different kinds of tests and permit the exclusion of individual 
tests/groups of tests [3]. I would like to strongly propose that we 
consider using TestNG as a means of providing the different test 
configurations required by Harmony. Using a combination of annotations 
and XML to capture the kinds of sophisticated test configurations that 
people need, and that allows us to specify down to the individual 
method, has got to be more scalable and flexible than where we are 
headed now.

Thanks for reading this far.

Best regards,

[2] http://testng.org

Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org

View raw message