harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Zhang" <zhanghuang...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Sat, 15 Jul 2006 05:55:48 GMT
On 7/14/06, George Harley <george.c.harley@googlemail.com> wrote:
>
> Hi,
>
> If annotations were to be used to help us categorise tests in order to
> simplify the definition of test configurations - what's included and
> excluded etc - then a core set of annotations would need to be agreed by
> the project. Consider the possibilities that the TestNG "@Test"
> annotation offers us in this respect.
>
> First, if a test method was identified as being broken and needed to be
> excluded from all test runs while awaiting investigation then it would
> be a simple matter of setting its enabled field like this:
>
>    @Test(enabled=false)
>    public void myTest() {
>        ...
>    }
>
> Temporarily disabling a test method in this way means that it can be
> left in its original class and we do not have to refer to it in any
> suite configuration (e.g. in the suite xml file).
>
> If a test method was identified as being broken on a specific platform
> then we could make use of the groups field of the "@Test" type by making
> the method a member of a group that identifies its predicament.
> Something like this:
>
>    @Test(groups={"state.broken.win.IA32"})
>    public void myOtherTest() {
>        ...
>    }
>
> The configuration for running tests on Windows would then specifically
> exclude any test method (or class) that was a member of that group.
>
> Making a test method or type a member of a well-known group (well-known
> in the sense that the name and meaning has been agreed within the
> project) is essentially adding some descriptive attributes to the test.
> Like adjectives (the groups) and nouns (the tests) in the English
> language.


It's the same in the Chinese language. :)

To take another example, if there was a test class that
> contained methods only intended to be run on Windows and that were all
> specific to Harmony (i.e. not API tests) then  one could envisage the
> following kind of annotation:
>
>
> @Test(groups={"type.impl", "os.win.IA32"})
> public class MyTestClass {
>
>    public void testOne() {
>        ...
>    }
>
>    public void testTwo() {
>        ...
>    }
>
>    @Test(enabled=false)
>    public void brokenTest() {
>        ...
>    }
> }
>
> Here the annotation on MyTestClass applies to all of its test methods.
>
> So what are the well-known TestNG groups that we could define for use
> inside Harmony ? Here are some of my initial thoughts:
>
>
> * type.impl  --  tests that are specific to Harmony
>
> * state.broken.<platform id>  --  tests bust on a specific platform
>
> * state.broken  --  tests broken on every platform but we want to decide
> whether or not to run from our suite configuration
>
> * os.<platform id>  --  tests that are to be run only on the specified
> platform (a test could be member of more than one of these)


Hi George, I have a small question.
Is os.<platform id> duplicate of state.broken.<platform id>?
Does os.<platform.id> equals state.broken.<other platform ids>? ( os.<
platform.ids> = all platforms - state.broken.<platform ids>)


> What does everyone else think ? Does such a scheme sound reasonable ?


+0.02$.  I can't wait to see these annotations in Harmony. :)

I also have found some platform-dependent tests in NIO module.  Looking
forward to deal with them when TestNG is integrated. :)


Thanks for reading this far.
>
> Best regards,
> George
>
>
>
> George Harley wrote:
> > Hi,
> >
> > Just seen Tim's note on test support classes and it really caught my
> > attention as I have been mulling over this issue for a little while
> > now. I think that it is a good time for us to return to the topic of
> > class library test layouts.
> >
> > The current proposal [1] sets out to segment our different types of
> > test by placing them in different file locations. After looking at the
> > recent changes to the LUNI module tests (where the layout guidelines
> > were applied) I have a real concern that there are serious problems
> > with this approach. We have started down a track of just continually
> > growing the number of test source folders as new categories of test
> > are identified and IMHO that is going to bring complexity and
> > maintenance issues with these tests.
> >
> > Consider the dimensions of tests that we have ...
> >
> > API
> > Harmony-specific
> > Platform-specific
> > Run on classpath
> > Run on bootclasspath
> > Behaves different between Harmony and RI
> > Stress
> > ...and so on...
> >
> >
> > If you weigh up all of the different possible permutations and then
> > consider that the above list is highly likely to be extended as things
> > progress it is obvious that we are eventually heading for large
> > amounts of related test code scattered or possibly duplicated across
> > numerous "hard wired" source directories. How maintainable is that
> > going to be ?
> >
> > If we want to run different tests in different configurations then
> > IMHO we need to be thinking a whole lot smarter. We need to be
> > thinking about keeping tests for specific areas of functionality
> > together (thus easing maintenance); we need something quick and simple
> > to re-configure if necessary (pushing whole directories of files
> > around the place does not seem a particularly lightweight approach);
> > and something that is not going to potentially mess up contributed
> > patches when the file they patch is found to have been recently pushed
> > from source folder A to B.
> >
> > To connect into another recent thread, there have been some posts
> > lately about handling some test methods that fail on Harmony and have
> > meant that entire test case classes have been excluded from our test
> > runs. I have also been noticing some API test methods that pass fine
> > on Harmony but fail when run against the RI. Are the different
> > behaviours down to errors in the Harmony implementation ? An error in
> > the RI implementation ? A bug in the RI Javadoc ? Only after some
> > investigation has been carried out do we know for sure. That takes
> > time. What do we do with the test methods in the meantime ? Do we push
> > them round the file system into yet another new source folder ? IMHO
> > we need a testing strategy that enables such "problem" methods to be
> > tracked easily without disruption to the rest of the other tests.
> >
> > A couple of weeks ago I mentioned that the TestNG framework [2] seemed
> > like a reasonably good way of allowing us to both group together
> > different kinds of tests and permit the exclusion of individual
> > tests/groups of tests [3]. I would like to strongly propose that we
> > consider using TestNG as a means of providing the different test
> > configurations required by Harmony. Using a combination of annotations
> > and XML to capture the kinds of sophisticated test configurations that
> > people need, and that allows us to specify down to the individual
> > method, has got to be more scalable and flexible than where we are
> > headed now.
> >
> > Thanks for reading this far.
> >
> > Best regards,
> > George
> >
> >
> > [1]
> >
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> >
> > [2] http://testng.org
> > [3]
> >
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


-- 
Andrew Zhang
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message