harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivanov, Alexey A" <alexey.a.iva...@intel.com>
Subject RE: [classlib][TestNG] groups of Harmony test
Date Mon, 28 Aug 2006 09:24:03 GMT

>-----Original Message-----
>From: Andrew Zhang [mailto:zhanghuangzhu@gmail.com]
>Sent: Monday, August 28, 2006 1:06 PM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib][TestNG] groups of Harmony test
>
>On 8/28/06, Ivanov, Alexey A <alexey.a.ivanov@intel.com> wrote:
>>
>>
>> >-----Original Message-----
>> >From: Richard Liang [mailto:richard.liangyx@gmail.com]
>> >Sent: Friday, August 25, 2006 7:18 PM
>> >To: harmony-dev@incubator.apache.org
>> >Subject: [classlib][TestNG] groups of Harmony test
>> >
>> >Hello All,
>> >
>> >Now let's talk about the TestNG groups. I have read the related
threads
>> >which posted by George, Vladimir Ivanov and Alexei Zakharov. All of
>> them
>> >are good discussion about TestNG groups.
>> >
>> >IMHO, we may define Harmony test groups according the following 4
>> >dimensions:
>> >
>> >1) [Platform] os.any, os.<platform id>
>> > *os.any* - group of tests which pass on any platform. IMHO, most of
>> our
>> >tests should be in this group.
>> > *os.<platform id>* - group of tests which are designed for one
>> specific
>> >platform. A test may be in more than one of the groups. e.g.,
>> >@Test(groups={"os.win.IA32", "os.linux.IA32"})
>> >
>> >    ** os.any and os.<platform id> are mutually exclusive, that is,
>> >tests in os.any group should not be in os.win.IA32.
>> >
>> >2) [Test state] state.broken, state.broken.<platform id>
>> > *state.broken* - group of tests which fail on every platform,
because
>> >of bugs of tests or implementation. We need to fix the bugs of tests
or
>> >implementation to make them pass.
>> > *state.broken.<platform id>* - groups of test which only fail on
one
>> >specific platform. A test may be in more than one of the groups.
e.g.,
>> >@Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>> >
>> >     **state.broken.<platform id> group may be used as a convenient
way
>> >to indicate that a test is platform-specific. e.g., If we support 10
>> >platforms, and one test are designed for 9 platforms except for
MacOS,
>> >instead of list 9 os.<platform id>, we can just use
state.broken.MacOS
>> >
>> >3) [Test type] type.api, type.impl
>> > *type.api* - group of tests which are tests for APIs in the Java
>> >Specification
>> > *type.impl* - groups of tests which are tests for Harmony-specific
>> >implementation
>> >
>> >     ** type.api and type.impl are also mutually exclusive.
>> >
>> >4) [Test Level] level.unit, level.integration, level.system,
>> >level.stress, etc. (Levels of Test refer to the increase in
complexity
>> >as moving through test cycle. )
>> >    ** A test may be in more than one of the groups.
>> >    ** In fact, some tests such as System tests are the verification
of
>> >the entire system.  Maybe we'll put them into a separate project.
e.g.,
>> >harmony/enhanced/SVT (System Verification Test).
>>
>> 5) [Environment] env.display, env.headless
>>   To distinguish AWT and Swing tests which need a display to run, and
>> those which don't, as Mark proposed [1].
>
>
>Will display option be passed manually as an argument to TestNG, or
>detected automatically when running test?

I think it should be passed as an argument to TestNG.
Ant script can be setup to detect the state by default, with the ability
to override it from the command line.


Regards,
Alexey.

>
>Regards,
>> Alexey.
>>
>> [1]
>>
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb
>> ox/%3c200608241441.k7OEfuBR018998@d06av02.portsmouth.uk.ibm.com%3e
>>
>> >
>> >If we want to run all the unit test for APIs on windows, we may use
>> >TestNG groups to select the tests:
>> >        <groups>
>> >            <run>
>> >                <include name="os.any" />
>> >                <include name="type.api" />
>> >                <include name="os.win.IA32" />
>> >                <exclude name="state.broken" />
>> >                <exclude name="state.broken.win.IA32" />
>> >            </run>
>> >        </groups>
>> >
>> >
>> >Well, I think our most of existing tests are in the groups of
>> {"os.any",
>> >"type.api", "level.unit"}, and I have asked TestNG to add a new
option
>> >"-groups" for its JUnitConverter which allow us to specify the test
>> >groups when migrate from JUnit test to TestNG test.
>> >
>> >Thanks for reading so far, and I will highly appreciate your
comments
>> or
>> >suggestion.  ;-)
>> >
>> >--
>> >Richard Liang
>> >China Software Development Lab, IBM
>> >
>> >
>> >
>>
>---------------------------------------------------------------------
>> >Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> >For additional commands, e-mail:
harmony-dev-help@incubator.apache.org
>>
>> --
>> Alexey A. Ivanov
>> Intel Middleware Product Division
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail:
harmony-dev-help@incubator.apache.org
>>
>>
>
>
>--
>Andrew Zhang
>China Software Development Lab, IBM

--
Alexey A. Ivanov
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message