harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Zhang" <zhanghuang...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Fri, 07 Jul 2006 03:10:55 GMT
On 7/5/06, George Harley <george.c.harley@googlemail.com> wrote:
> Hi,
> Just seen Tim's note on test support classes and it really caught my
> attention as I have been mulling over this issue for a little while now.
> I think that it is a good time for us to return to the topic of class
> library test layouts.
> The current proposal [1] sets out to segment our different types of test
> by placing them in different file locations. After looking at the recent
> changes to the LUNI module tests (where the layout guidelines were
> applied) I have a real concern that there are serious problems with this
> approach. We have started down a track of just continually growing the
> number of test source folders as new categories of test are identified
> and IMHO that is going to bring complexity and maintenance issues with
> these tests.
> Consider the dimensions of tests that we have ...
> Harmony-specific
> Platform-specific
> Run on classpath
> Run on bootclasspath
> Behaves different between Harmony and RI
> Stress
> ...and so on...

At least ... 3(common/win/linux) * 2 (classpath/bootclasspath) * 2(api/impl)
* ... = 12 * ...
Of course, for some module or package, there are only common folder *
classpath * api test, that the best thing.
The count of folders are really terrifc in worst situation.

If you weigh up all of the different possible permutations and then
> consider that the above list is highly likely to be extended as things
> progress it is obvious that we are eventually heading for large amounts
> of related test code scattered or possibly duplicated across numerous
> "hard wired" source directories. How maintainable is that going to be ?

Put my 0.02$ here.  Configuration is a better than physical folder layout,
not matter by which tool (Junit, TestNG, ....)
In fact, physical layout is also a configuration, which is controlled by
folder directory rather than annotations, xml, ...
I think storing test specific information (e.g. only win) in code is better
than in folder path (e.g **/win/...).
In most cases, annotation by professional test tool is more flexible, and

If we want to run different tests in different configurations then IMHO
> we need to be thinking a whole lot smarter. We need to be thinking about
> keeping tests for specific areas of functionality together (thus easing
> maintenance); we need something quick and simple to re-configure if
> necessary (pushing whole directories of files around the place does not
> seem a particularly lightweight approach); and something that is not
> going to potentially mess up contributed patches when the file they
> patch is found to have been recently pushed from source folder A to B.
> To connect into another recent thread, there have been some posts lately
> about handling some test methods that fail on Harmony and have meant
> that entire test case classes have been excluded from our test runs. I
> have also been noticing some API test methods that pass fine on Harmony
> but fail when run against the RI. Are the different behaviours down to
> errors in the Harmony implementation ? An error in the RI implementation
> ? A bug in the RI Javadoc ? Only after some investigation has been
> carried out do we know for sure. That takes time. What do we do with the
> test methods in the meantime ? Do we push them round the file system
> into yet another new source folder ? IMHO we need a testing strategy
> that enables such "problem" methods to be tracked easily without
> disruption to the rest of the other tests.
> A couple of weeks ago I mentioned that the TestNG framework [2] seemed
> like a reasonably good way of allowing us to both group together
> different kinds of tests and permit the exclusion of individual
> tests/groups of tests [3]. I would like to strongly propose that we
> consider using TestNG as a means of providing the different test
> configurations required by Harmony. Using a combination of annotations
> and XML to capture the kinds of sophisticated test configurations that
> people need, and that allows us to specify down to the individual
> method, has got to be more scalable and flexible than where we are
> headed now.

Maybe another two cents here after learning TestNG. :)

Thanks for reading this far.
> Best regards,
> George
> [1]
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> [2] http://testng.org
> [3]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org

Andrew Zhang
China Software Development Lab, IBM

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message