harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Sat, 02 Sep 2006 16:28:49 GMT


Stepan Mishura wrote:
> On 8/25/06, Richard Liang wrote:
>>
>> Hello All,
>>
>> Now let's talk about the TestNG groups. I have read the related threads
>> which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
>> are good discussion about TestNG groups.
>>
>> IMHO, we may define Harmony test groups according the following 4
>> dimensions:
>>
>> 1) [Platform] os.any, os.<platform id>
>> *os.any* - group of tests which pass on any platform. IMHO, most of our
>> tests should be in this group.
>> *os.<platform id>* - group of tests which are designed for one specific
>> platform. A test may be in more than one of the groups. e.g.,
>> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>
>>    ** os.any and os.<platform id> are mutually exclusive, that is,
>> tests in os.any group should not be in os.win.IA32.
>>
>> 2) [Test state] state.broken, state.broken.<platform id>
>> *state.broken* - group of tests which fail on every platform, because
>> of bugs of tests or implementation. We need to fix the bugs of tests or
>> implementation to make them pass.
>> *state.broken.<platform id>* - groups of test which only fail on one
>> specific platform. A test may be in more than one of the groups. e.g.,
>> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>
>>     **state.broken.<platform id> group may be used as a convenient way
>> to indicate that a test is platform-specific. e.g., If we support 10
>> platforms, and one test are designed for 9 platforms except for MacOS,
>> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>
>
> If a test is marked as *state.broken.MacOS* then it sounds like the
> test/implementation should be fixed. IMO we should use tag 
> os.<platform id>
> to define explicitly valid platforms for the test so in this 
> particular case
> we should use 9 os.<platform id>s
>

Thank you,  Stepan.  This sounds reasonable, we should explicitly list 
the valid platforms.

Best regards,
Richard

>
> 3) [Test type] type.api, type.impl
>> *type.api* - group of tests which are tests for APIs in the Java
>> Specification
>> *type.impl* - groups of tests which are tests for Harmony-specific
>> implementation
>>
>>     ** type.api and type.impl are also mutually exclusive.
>>
>> 4) [Test Level] level.unit, level.integration, level.system,
>> level.stress, etc. (Levels of Test refer to the increase in complexity
>> as moving through test cycle. )
>>    ** A test may be in more than one of the groups.
>>    ** In fact, some tests such as System tests are the verification of
>> the entire system.  Maybe we'll put them into a separate project. e.g.,
>> harmony/enhanced/SVT (System Verification Test).
>
>
> Mixing different types of testing into one test-file doesn't look good 
> for
> me. I'd separate such tests by placing into different 
> directories/packages.
>
> Thanks,
> Stepan.
>
> If we want to run all the unit test for APIs on windows, we may use
>> TestNG groups to select the tests:
>>        <groups>
>>            <run>
>>                <include name="os.any" />
>>                <include name="type.api" />
>>                <include name="os.win.IA32" />
>>                <exclude name="state.broken" />
>>                <exclude name="state.broken.win.IA32" />
>>            </run>
>>        </groups>
>>
>>
>> Well, I think our most of existing tests are in the groups of {"os.any",
>> "type.api", "level.unit"}, and I have asked TestNG to add a new option
>> "-groups" for its JUnitConverter which allow us to specify the test
>> groups when migrate from JUnit test to TestNG test.
>>
>> Thanks for reading so far, and I will highly appreciate your comments or
>> suggestion.  ;-)
>>
>> -- 
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>
> ------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message