harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Wed, 30 Aug 2006 03:35:13 GMT
I'm just catching up after a few days of travel.  Have you begun 
sketching this out on the wiki?


Richard Liang wrote:
> Hello All,
> Now let's talk about the TestNG groups. I have read the related threads 
> which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them 
> are good discussion about TestNG groups.
> IMHO, we may define Harmony test groups according the following 4 
> dimensions:
> 1) [Platform] os.any, os.<platform id>
> *os.any* - group of tests which pass on any platform. IMHO, most of our 
> tests should be in this group.
> *os.<platform id>* - group of tests which are designed for one specific 
> platform. A test may be in more than one of the groups. e.g., 
> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>    ** os.any and os.<platform id> are mutually exclusive, that is, tests 
> in os.any group should not be in os.win.IA32.
> 2) [Test state] state.broken, state.broken.<platform id>
> *state.broken* - group of tests which fail on every platform, because of 
> bugs of tests or implementation. We need to fix the bugs of tests or 
> implementation to make them pass.
> *state.broken.<platform id>* - groups of test which only fail on one 
> specific platform. A test may be in more than one of the groups. e.g., 
> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>     **state.broken.<platform id> group may be used as a convenient way 
> to indicate that a test is platform-specific. e.g., If we support 10 
> platforms, and one test are designed for 9 platforms except for MacOS, 
> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
> 3) [Test type] type.api, type.impl
> *type.api* - group of tests which are tests for APIs in the Java 
> Specification
> *type.impl* - groups of tests which are tests for Harmony-specific 
> implementation
>     ** type.api and type.impl are also mutually exclusive.
> 4) [Test Level] level.unit, level.integration, level.system, 
> level.stress, etc. (Levels of Test refer to the increase in complexity 
> as moving through test cycle. )
>    ** A test may be in more than one of the groups.
>    ** In fact, some tests such as System tests are the verification of 
> the entire system.  Maybe we'll put them into a separate project. e.g., 
> harmony/enhanced/SVT (System Verification Test).
> If we want to run all the unit test for APIs on windows, we may use 
> TestNG groups to select the tests:
>        <groups>
>            <run>
>                <include name="os.any" />
>                <include name="type.api" />
>                <include name="os.win.IA32" />
>                <exclude name="state.broken" />
>                <exclude name="state.broken.win.IA32" />
>            </run>
>        </groups>
> Well, I think our most of existing tests are in the groups of {"os.any", 
> "type.api", "level.unit"}, and I have asked TestNG to add a new option 
> "-groups" for its JUnitConverter which allow us to specify the test 
> groups when migrate from JUnit test to TestNG test.
> Thanks for reading so far, and I will highly appreciate your comments or 
> suggestion.  ;-)

Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org

View raw message