harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Mon, 04 Sep 2006 07:43:07 GMT


Vladimir Ivanov wrote:
> On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>>
>>
>>
>> Mikhail Loenko wrote:
>> > Hi Vladimir
>> >
>> > Could you please decribe for what purpose it will be used?
>> >
>> > I mean why one might have to either exclude or run only regression
>> tests?
>>
>> If running all tests takes up much time, running all regression test
>> (for one particular version) may be a convenient way to verify the
>> correctness of bug-fixing.
>
>
> Thanks Richard. It is exactly what I want to say :)
> On the other hand may be all proposed groups need similar explanation 
> (smt.
> like use-case for group)?
>

Good idea ;-)

I will add some use-cases for the groups. And *everyone* is welcome to 
edit the wiki page [1]

[1]http://wiki.apache.org/harmony/Testing_Convention

Best regards,
Richard

> thanks, Vladimir
>
> Best regards,
>> Richard.
>>
>> >
>> > Thanks,
>> > Mikhail
>> >
>> > 2006/8/30, Vladimir Ivanov <ivavladimir@gmail.com>:
>> >> On 8/30/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> > Vladimir Ivanov wrote:
>> >> > > Also some tag for regression tests should be added.
>> >> > Yes. Do you think we could annotate regression test as
>> >> > *level.regression*? Thanks a lot.
>> >>
>> >>
>> >> Yes, I do. While tests can have more than one group it will enough.
>> >>  thanks, Vladimir
>> >>
>> >>
>> >> Richard
>> >> > > thanks, Vladimir
>> >> > >
>> >> > >
>> >> > > On 8/28/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> Richard Liang wrote:
>> >> > >> > Hello All,
>> >> > >> >
>> >> > >> > Now let's talk about the TestNG groups. I have read the

>> related
>> >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
>> >> Zakharov.
>> >> > >> > All of them are good discussion about TestNG groups.
>> >> > >> >
>> >> > >> > IMHO, we may define Harmony test groups according the

>> following
>> 4
>> >> > >> > dimensions:
>> >> > >> >
>> >> > >> > 1) [Platform] os.any, os.<platform id>
>> >> > >> > *os.any* - group of tests which pass on any platform.
IMHO,
>> >> most of
>> >> > >> > our tests should be in this group.
>> >> > >> > *os.<platform id>* - group of tests which are designed
for one
>> >> > >> > specific platform. A test may be in more than one of
the
>> >> groups. e.g
>> >> > .,
>> >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> >> > >> >
>> >> > >> >    ** os.any and os.<platform id> are mutually
exclusive, that
>> >> is,
>> >> > >> > tests in os.any group should not be in os.win.IA32.
>> >> > >> >
>> >> > >> > 2) [Test state] state.broken, state.broken.<platform
id>
>> >> > >> > *state.broken* - group of tests which fail on every platform,
>> >> because
>> >> > >> > of bugs of tests or implementation. We need to fix the
bugs of
>> >> tests
>> >> > >> > or implementation to make them pass.
>> >> > >> > *state.broken.<platform id>* - groups of test which
only fail
>> >> on one
>> >> > >> > specific platform. A test may be in more than one of
the
>> >> groups. e.g
>> >> > .,
>> >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
>> "})
>> >> > >> >
>> >> > >> >     **state.broken.<platform id> group may be used
as a
>> >> convenient
>> >> > way
>> >> > >> > to indicate that a test is platform-specific. e.g., If
we
>> >> support 10
>> >> > >> > platforms, and one test are designed for 9 platforms
except 
>> for
>> >> > MacOS,
>> >> > >> > instead of list 9 os.<platform id>, we can just
use
>> >> > state.broken.MacOS
>> >> > >> >
>> >> > >> > 3) [Test type] type.api , type.impl
>> >> > >> > *type.api* - group of tests which are tests for APIs
in the 
>> Java
>> >> > >> > Specification
>> >> > >> > *type.impl* - groups of tests which are tests for
>> >> Harmony-specific
>> >> > >> > implementation
>> >> > >> >
>> >> > >> >     ** type.api and type.impl are also mutually exclusive.
>> >> > >> >
>> >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
>> >> > >> > level.stress, etc. (Levels of Test refer to the increase
in
>> >> > complexity
>> >> > >> > as moving through test cycle. )
>> >> > >> >    ** A test may be in more than one of the groups.
>> >> > >> >    ** In fact, some tests such as System tests are the
>> >> verification
>> >> > of
>> >> > >> > the entire system.  Maybe we'll put them into a separate
>> project.
>> >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>> >> > >> >
>> >> > >> > If we want to run all the unit test for APIs on windows,
we
>> >> may use
>> >> > >> > TestNG groups to select the tests:
>> >> > >> >        <groups>
>> >> > >> >            <run>
>> >> > >> >                <include name="os.any" />
>> >> > >> >                <include name="type.api" />
>> >> > >> >                <include name="os.win.IA32" />
>> >> > >> >                <exclude name=" state.broken" />
>> >> > >> >                <exclude name="state.broken.win.IA32"
/>
>> >> > >> >            </run>
>> >> > >> >        </groups>
>> >> > >> >
>> >> > >> Hello All,
>> >> > >>
>> >> > >> I'm sorry. It seems that the example does not work. I will

>> try to
>> >> > figure
>> >> > >> another example soon. ;-)
>> >> > >>
>> >> > >> Best regards,
>> >> > >> Richard
>> >> > >> >
>> >> > >> > Well, I think our most of existing tests are in the groups
of
>> >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
TestNG
>> >> to add
>> >> > a
>> >> > >> > new option "-groups" for its JUnitConverter which allow
us to
>> >> specify
>> >> > >> > the test groups when migrate from JUnit test to TestNG
test.
>> >> > >> >
>> >> > >> > Thanks for reading so far, and I will highly appreciate
your
>> >> comments
>> >> > >> > or suggestion.  ;-)
>> >> > >> >
>> >> > >>
>> >> > >> --
>> >> > >> Richard Liang
>> >> > >> China Software Development Lab, IBM
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> ---------------------------------------------------------------------
>> >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> > >> To unsubscribe, e-mail:
>> >> harmony-dev-unsubscribe@incubator.apache.org
>> >> > >> For additional commands, e-mail:
>> >> harmony-dev-help@incubator.apache.org
>> >> > >>
>> >> > >>
>> >> > >
>> >> >
>> >> > --
>> >> > Richard Liang
>> >> > China Software Development Lab, IBM
>> >> >
>> >> >
>> >> >
>> >> > 
>> ---------------------------------------------------------------------
>> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> > To unsubscribe, e-mail: 
>> harmony-dev-unsubscribe@incubator.apache.org
>> >> > For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>> -- 
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message