harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Zhang" <zhanghuang...@gmail.com>
Subject Re: Re: [classlib][TestNG] groups of Harmony test
Date Mon, 04 Sep 2006 13:56:41 GMT
On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>
> I sent my reply this afternoon, but I have not received it. (it seems
> there
> is something wrong with my smtp server). So I send it again. :-)
>
> On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com> wrote:
> >
> > IMNSO it doesn't make sense to arbitrarily partition the tests based
> > on a moniker, such as 'integration test', 'unit test', 'regression
> > test' etc. For one thing, developers are generally not good at
> > agreeing on the difference between them :-)
>
>
>
> This is really a problem, however it might be simpler than we imagine. We
> are open to any discussion. ;-) Anyway, developers are required to write
> unit tests.
>
>
> If you've got fast and slow tests, then have a group for fast and slow
> > tests. Then you can choose to just run the fast tests, and any
> > automated build system can handle running the slow tests.
>
>
> IMHO, "fast or slow" may not be the key point. The question is whether we
> have any requirements to run only the regression tests.


I think your first proposal looks fine. Platform + os + state + api/impl
information are enough for all tests. We have to assure all tests pass on
every platform.

For quick run, developers can use their own short approach, like run single
test in IDE... It's not business of TestNG. :-)


Best regards,
> Richard
>
> Alex.
> >
> > On 04/09/06, Vladimir Ivanov <ivavladimir@gmail.com> wrote:
> > > On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > Mikhail Loenko wrote:
> > > > > Hi Vladimir
> > > > >
> > > > > Could you please decribe for what purpose it will be used?
> > > > >
> > > > > I mean why one might have to either exclude or run only regression
> > > > tests?
> > > >
> > > > If running all tests takes up much time, running all regression test
> > > > (for one particular version) may be a convenient way to verify the
> > > > correctness of bug-fixing.
> > >
> > >
> > > Thanks Richard. It is exactly what I want to say :)
> > > On the other hand may be all proposed groups need similar explanation
> > (smt.
> > > like use-case for group)?
> > >
> > >  thanks, Vladimir
> > >
> > > Best regards,
> > > > Richard.
> > > >
> > > > >
> > > > > Thanks,
> > > > > Mikhail
> > > > >
> > > > > 2006/8/30, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > > >> On 8/30/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Vladimir Ivanov wrote:
> > > > >> > > Also some tag for regression tests should be added.
> > > > >> > Yes. Do you think we could annotate regression test as
> > > > >> > *level.regression*? Thanks a lot.
> > > > >>
> > > > >>
> > > > >> Yes, I do. While tests can have more than one group it will
> enough.
> > > > >>  thanks, Vladimir
> > > > >>
> > > > >>
> > > > >> Richard
> > > > >> > > thanks, Vladimir
> > > > >> > >
> > > > >> > >
> > > > >> > > On 8/28/06, Richard Liang <richard.liangyx@gmail.com>
wrote:
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Richard Liang wrote:
> > > > >> > >> > Hello All,
> > > > >> > >> >
> > > > >> > >> > Now let's talk about the TestNG groups. I
have read the
> > related
> > > > >> > >> > threads which posted by George, Vladimir Ivanov
and Alexei
> > > > >> Zakharov.
> > > > >> > >> > All of them are good discussion about TestNG
groups.
> > > > >> > >> >
> > > > >> > >> > IMHO, we may define Harmony test groups according
the
> > following
> > > > 4
> > > > >> > >> > dimensions:
> > > > >> > >> >
> > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > >> > >> > *os.any* - group of tests which pass on any
platform.
> IMHO,
> > > > >> most of
> > > > >> > >> > our tests should be in this group.
> > > > >> > >> > *os.<platform id>* - group of tests
which are designed for
> > one
> > > > >> > >> > specific platform. A test may be in more than
one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > >> > >> >
> > > > >> > >> >    ** os.any and os.<platform id> are
mutually exclusive,
> > that
> > > > >> is,
> > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > >> > >> >
> > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform
id>
> > > > >> > >> > *state.broken* - group of tests which fail
on every
> > platform,
> > > > >> because
> > > > >> > >> > of bugs of tests or implementation. We need
to fix the
> bugs
> > of
> > > > >> tests
> > > > >> > >> > or implementation to make them pass.
> > > > >> > >> > *state.broken.<platform id>* - groups
of test which only
> > fail
> > > > >> on one
> > > > >> > >> > specific platform. A test may be in more than
one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > os.broken.linux.IA64
> > > > "})
> > > > >> > >> >
> > > > >> > >> >     **state.broken.<platform id> group
may be used as a
> > > > >> convenient
> > > > >> > way
> > > > >> > >> > to indicate that a test is platform-specific.
e.g., If we
> > > > >> support 10
> > > > >> > >> > platforms, and one test are designed for 9
platforms
> except
> > for
> > > > >> > MacOS,
> > > > >> > >> > instead of list 9 os.<platform id>,
we can just use
> > > > >> > state.broken.MacOS
> > > > >> > >> >
> > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > >> > >> > *type.api* - group of tests which are tests
for APIs in
> the
> > Java
> > > > >> > >> > Specification
> > > > >> > >> > *type.impl* - groups of tests which are tests
for
> > > > >> Harmony-specific
> > > > >> > >> > implementation
> > > > >> > >> >
> > > > >> > >> >     ** type.api and type.impl are also mutually
exclusive.
> > > > >> > >> >
> > > > >> > >> > 4) [Test Level] level.unit, level.integration,
> level.system,
> > > > >> > >> > level.stress, etc. (Levels of Test refer to
the increase
> in
> > > > >> > complexity
> > > > >> > >> > as moving through test cycle. )
> > > > >> > >> >    ** A test may be in more than one of the
groups.
> > > > >> > >> >    ** In fact, some tests such as System tests
are the
> > > > >> verification
> > > > >> > of
> > > > >> > >> > the entire system.  Maybe we'll put them into
a separate
> > > > project.
> > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification
Test).
> > > > >> > >> >
> > > > >> > >> > If we want to run all the unit test for APIs
on windows,
> we
> > > > >> may use
> > > > >> > >> > TestNG groups to select the tests:
> > > > >> > >> >        <groups>
> > > > >> > >> >            <run>
> > > > >> > >> >                <include name="os.any" />
> > > > >> > >> >                <include name="type.api"
/>
> > > > >> > >> >                <include name="os.win.IA32"
/>
> > > > >> > >> >                <exclude name=" state.broken"
/>
> > > > >> > >> >                <exclude name="state.broken.win.IA32"
/>
> > > > >> > >> >            </run>
> > > > >> > >> >        </groups>
> > > > >> > >> >
> > > > >> > >> Hello All,
> > > > >> > >>
> > > > >> > >> I'm sorry. It seems that the example does not work.
I will
> try
> > to
> > > > >> > figure
> > > > >> > >> another example soon. ;-)
> > > > >> > >>
> > > > >> > >> Best regards,
> > > > >> > >> Richard
> > > > >> > >> >
> > > > >> > >> > Well, I think our most of existing tests are
in the groups
> > of
> > > > >> > >> > {"os.any", "type.api", "level.unit"}, and
I have asked
> > TestNG
> > > > >> to add
> > > > >> > a
> > > > >> > >> > new option "-groups" for its JUnitConverter
which allow us
> > to
> > > > >> specify
> > > > >> > >> > the test groups when migrate from JUnit test
to TestNG
> test.
> > > > >> > >> >
> > > > >> > >> > Thanks for reading so far, and I will highly
appreciate
> your
> > > > >> comments
> > > > >> > >> > or suggestion.  ;-)
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Richard Liang
> > > > >> > >> China Software Development Lab, IBM
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > >> Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > >> > >> To unsubscribe, e-mail:
> > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > >> For additional commands, e-mail:
> > > > >> harmony-dev-help@incubator.apache.org
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Richard Liang
> > > > >> > China Software Development Lab, IBM
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > ---------------------------------------------------------------------
> > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > >> > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > For additional commands, e-mail:
> > > > harmony-dev-help@incubator.apache.org
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > > --
> > > > Richard Liang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>


-- 
Andrew Zhang
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message