harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Mon, 17 Jul 2006 15:09:45 GMT
Andrew Zhang wrote:
> On 7/14/06, George Harley <george.c.harley@googlemail.com> wrote:
>>
>> Hi,
>>
>> If annotations were to be used to help us categorise tests in order to
>> simplify the definition of test configurations - what's included and
>> excluded etc - then a core set of annotations would need to be agreed by
>> the project. Consider the possibilities that the TestNG "@Test"
>> annotation offers us in this respect.
>>
>> First, if a test method was identified as being broken and needed to be
>> excluded from all test runs while awaiting investigation then it would
>> be a simple matter of setting its enabled field like this:
>>
>>    @Test(enabled=false)
>>    public void myTest() {
>>        ...
>>    }
>>
>> Temporarily disabling a test method in this way means that it can be
>> left in its original class and we do not have to refer to it in any
>> suite configuration (e.g. in the suite xml file).
>>
>> If a test method was identified as being broken on a specific platform
>> then we could make use of the groups field of the "@Test" type by making
>> the method a member of a group that identifies its predicament.
>> Something like this:
>>
>>    @Test(groups={"state.broken.win.IA32"})
>>    public void myOtherTest() {
>>        ...
>>    }
>>
>> The configuration for running tests on Windows would then specifically
>> exclude any test method (or class) that was a member of that group.
>>
>> Making a test method or type a member of a well-known group (well-known
>> in the sense that the name and meaning has been agreed within the
>> project) is essentially adding some descriptive attributes to the test.
>> Like adjectives (the groups) and nouns (the tests) in the English
>> language.
>
>
> It's the same in the Chinese language. :)
>
> To take another example, if there was a test class that
>> contained methods only intended to be run on Windows and that were all
>> specific to Harmony (i.e. not API tests) then  one could envisage the
>> following kind of annotation:
>>
>>
>> @Test(groups={"type.impl", "os.win.IA32"})
>> public class MyTestClass {
>>
>>    public void testOne() {
>>        ...
>>    }
>>
>>    public void testTwo() {
>>        ...
>>    }
>>
>>    @Test(enabled=false)
>>    public void brokenTest() {
>>        ...
>>    }
>> }
>>
>> Here the annotation on MyTestClass applies to all of its test methods.
>>
>> So what are the well-known TestNG groups that we could define for use
>> inside Harmony ? Here are some of my initial thoughts:
>>
>>
>> * type.impl  --  tests that are specific to Harmony
>>
>> * state.broken.<platform id>  --  tests bust on a specific platform
>>
>> * state.broken  --  tests broken on every platform but we want to decide
>> whether or not to run from our suite configuration
>>
>> * os.<platform id>  --  tests that are to be run only on the specified
>> platform (a test could be member of more than one of these)
>
>
> Hi George, I have a small question.
> Is os.<platform id> duplicate of state.broken.<platform id>?
> Does os.<platform.id> equals state.broken.<other platform ids>? ( os.<
> platform.ids> = all platforms - state.broken.<platform ids>)

Hi Andrew,


The way I see it, membership of the group os.<platform id> (e.g. 
"os.win.IA32") means that the annotated type - class or method - is 
being flagged as specific to the identified platform. So, for instance 
the following annotation on a test class means it should only be run on 
the Win 32 platform:

@Test(groups={"os.win.IA32"})
public class Foo {
...various test methods...
}


The annotation state.broken.<platform id> (e.g. "state.broken.win.IA32") 
flags a test or entire test class as being broken *only* on the 
identified platform. The test is intended to be run everywhere but has 
been identified as failing on certain platforms. So, for instance, if 
there was a test in the NIO module that that should be run everywhere 
but failed on Linux then that test method could be annotated as:

@Test(groups={"state.broken.linux.IA32"})
public void testBaa() {
...
}

It is then simple to ensure that when the NIO tests are run on Windows 
that the above test *is* included but when run on Linux it is
specifically *excluded* because it is broken on that platform.

To my mind, there is a big distinction between having platform-specific 
tests (membership of a group os.*) and having tests that are intended to 
be run everywhere but are in fact broken on one or more platforms 
(membership of a group state.broken.*).


Best regards,
George


>
>
>> What does everyone else think ? Does such a scheme sound reasonable ?
>
>
> +0.02$.  I can't wait to see these annotations in Harmony. :)
>
> I also have found some platform-dependent tests in NIO module.  Looking
> forward to deal with them when TestNG is integrated. :)
>
>
> Thanks for reading this far.
>>
>> Best regards,
>> George
>>
>>
>>
>> George Harley wrote:
>> > Hi,
>> >
>> > Just seen Tim's note on test support classes and it really caught my
>> > attention as I have been mulling over this issue for a little while
>> > now. I think that it is a good time for us to return to the topic of
>> > class library test layouts.
>> >
>> > The current proposal [1] sets out to segment our different types of
>> > test by placing them in different file locations. After looking at the
>> > recent changes to the LUNI module tests (where the layout guidelines
>> > were applied) I have a real concern that there are serious problems
>> > with this approach. We have started down a track of just continually
>> > growing the number of test source folders as new categories of test
>> > are identified and IMHO that is going to bring complexity and
>> > maintenance issues with these tests.
>> >
>> > Consider the dimensions of tests that we have ...
>> >
>> > API
>> > Harmony-specific
>> > Platform-specific
>> > Run on classpath
>> > Run on bootclasspath
>> > Behaves different between Harmony and RI
>> > Stress
>> > ...and so on...
>> >
>> >
>> > If you weigh up all of the different possible permutations and then
>> > consider that the above list is highly likely to be extended as things
>> > progress it is obvious that we are eventually heading for large
>> > amounts of related test code scattered or possibly duplicated across
>> > numerous "hard wired" source directories. How maintainable is that
>> > going to be ?
>> >
>> > If we want to run different tests in different configurations then
>> > IMHO we need to be thinking a whole lot smarter. We need to be
>> > thinking about keeping tests for specific areas of functionality
>> > together (thus easing maintenance); we need something quick and simple
>> > to re-configure if necessary (pushing whole directories of files
>> > around the place does not seem a particularly lightweight approach);
>> > and something that is not going to potentially mess up contributed
>> > patches when the file they patch is found to have been recently pushed
>> > from source folder A to B.
>> >
>> > To connect into another recent thread, there have been some posts
>> > lately about handling some test methods that fail on Harmony and have
>> > meant that entire test case classes have been excluded from our test
>> > runs. I have also been noticing some API test methods that pass fine
>> > on Harmony but fail when run against the RI. Are the different
>> > behaviours down to errors in the Harmony implementation ? An error in
>> > the RI implementation ? A bug in the RI Javadoc ? Only after some
>> > investigation has been carried out do we know for sure. That takes
>> > time. What do we do with the test methods in the meantime ? Do we push
>> > them round the file system into yet another new source folder ? IMHO
>> > we need a testing strategy that enables such "problem" methods to be
>> > tracked easily without disruption to the rest of the other tests.
>> >
>> > A couple of weeks ago I mentioned that the TestNG framework [2] seemed
>> > like a reasonably good way of allowing us to both group together
>> > different kinds of tests and permit the exclusion of individual
>> > tests/groups of tests [3]. I would like to strongly propose that we
>> > consider using TestNG as a means of providing the different test
>> > configurations required by Harmony. Using a combination of annotations
>> > and XML to capture the kinds of sophisticated test configurations that
>> > people need, and that allows us to specify down to the individual
>> > method, has got to be more scalable and flexible than where we are
>> > headed now.
>> >
>> > Thanks for reading this far.
>> >
>> > Best regards,
>> > George
>> >
>> >
>> > [1]
>> >
>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>
>> >
>> > [2] http://testng.org
>> > [3]
>> >
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>
>> >
>> >
>> >
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message