harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oliver Deakin <oliver.dea...@googlemail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Fri, 07 Jul 2006 11:10:11 GMT
George Harley wrote:
> Hi,
>
> Just seen Tim's note on test support classes and it really caught my 
> attention as I have been mulling over this issue for a little while 
> now. I think that it is a good time for us to return to the topic of 
> class library test layouts.
>
> The current proposal [1] sets out to segment our different types of 
> test by placing them in different file locations. 

ok - on closer reading of this document, I have a few gripes...

First, what happened to the Maven layout we agreed on a while ago?
It seems that we have lost the java directory from underneath src/test. What
happens if we ever have native tests? Refactor the directory structure 
again?

Secondly, using the directory structure under src/test to separate API, impl
and stress tests seems wrong to me. At most I would say separate them
by package name - but really this is something that the test harness should
take care of, not something that we all need to do through laborious manual
transfer of files. Those tests that are marked as API get run on the 
classpath,
those that are marked as impl are run on the bootclasspath, and those that
are marked as stress tests dont get run as part of the standard test run -
the test harness should take care of all of this! Forgive me if Im wrong,
Im still a newbie with this harness, but isn't this all well within the 
capabilities
of TestNG? If I want a test to be a stress test, do I need to bury it down
in some directory called 
/src/test/stress/java/common/<package_name>/<class_name>
or do I just add an annotation to the test class marking it as a stress 
test that
runs on all platforms?

My final gripe might just be an error on the page - why are the common,
windows and linux directories at the same level as java and java.injected?
(Did I mention that platform specific tests can be handled by TestNG? ;)

> After looking at the recent changes to the LUNI module tests (where 
> the layout guidelines were applied) I have a real concern that there 
> are serious problems with this approach. We have started down a track 
> of just continually growing the number of test source folders as new 
> categories of test are identified and IMHO that is going to bring 
> complexity and maintenance issues with these tests.
>
> Consider the dimensions of tests that we have ...
>
> API
> Harmony-specific
> Platform-specific
> Run on classpath
> Run on bootclasspath
> Behaves different between Harmony and RI
> Stress
> ...and so on...
>
>
> If you weigh up all of the different possible permutations and then 
> consider that the above list is highly likely to be extended as things 
> progress it is obvious that we are eventually heading for large 
> amounts of related test code scattered or possibly duplicated across 
> numerous "hard wired" source directories. How maintainable is that 
> going to be ?

Not very - or was that a rhetorical question? ;)

>
> If we want to run different tests in different configurations then 
> IMHO we need to be thinking a whole lot smarter. We need to be 
> thinking about keeping tests for specific areas of functionality 
> together (thus easing maintenance); we need something quick and simple 
> to re-configure if necessary (pushing whole directories of files 
> around the place does not seem a particularly lightweight approach); 
> and something that is not going to potentially mess up contributed 
> patches when the file they patch is found to have been recently pushed 
> from source folder A to B.

Yes, this is a major drawback IMHO. Having files moved around from under 
the feet of other
patches (I know I am a culprit of this myself with the recent natives 
work) is very troublesome
and time consuming to repair. If we enforce the directory structure for 
tests that has been
described in previous threads, then I can see it causing us more 
problems rather than
simplifying the test process.

>
> To connect into another recent thread, there have been some posts 
> lately about handling some test methods that fail on Harmony and have 
> meant that entire test case classes have been excluded from our test 
> runs. I have also been noticing some API test methods that pass fine 
> on Harmony but fail when run against the RI. Are the different 
> behaviours down to errors in the Harmony implementation ? An error in 
> the RI implementation ? A bug in the RI Javadoc ? Only after some 
> investigation has been carried out do we know for sure. That takes 
> time. What do we do with the test methods in the meantime ? Do we push 
> them round the file system into yet another new source folder ? IMHO 
> we need a testing strategy that enables such "problem" methods to be 
> tracked easily without disruption to the rest of the other tests.
>
> A couple of weeks ago I mentioned that the TestNG framework [2] seemed 
> like a reasonably good way of allowing us to both group together 
> different kinds of tests and permit the exclusion of individual 
> tests/groups of tests [3]. I would like to strongly propose that we 
> consider using TestNG as a means of providing the different test 
> configurations required by Harmony. Using a combination of annotations 
> and XML to capture the kinds of sophisticated test configurations that 
> people need, and that allows us to specify down to the individual 
> method, has got to be more scalable and flexible than where we are 
> headed now.

Agreed - we should use the capabilities of existing harnesses to give us 
what we
need, not invent a complex and unmanageable directory structure that has no
real advantage over the alternatives.

>
> Thanks for reading this far.

My pleasure!

Regards,
Oliver

>
> Best regards,
> George
>
>
> [1] 
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>
> [2] http://testng.org
> [3] 
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message