harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Zhang" <zhanghuang...@gmail.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Mon, 04 Sep 2006 14:17:38 GMT
On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>
> On 9/4/06, Mikhail Loenko <mloenko@gmail.com> wrote:
> > Well, my question was for what particular reason? for example?
> >
> > Tio verify correctness of bug-fixing IMHO all the unit, intergration,
> api, and
> > regression tests should be run
>
> Running all tests are always good to verify our code quality. And I
> think this is what we have been doing. But what will we do if it takes
> us 24 hours to run all Harmony tests? Anyway, running regression tests
> could provide some confidence to our bug-fixing. We may consider it as
> another option. :-)


I prefer to run all tests in one module instead. :) Although it can not
guarentee all tests would pass, it's less likey to break build system
frequently.  If the fix causes other module fails, it shows the lack of
tests in its module. And we should add the corresponding test in the module.

Currently, Harmony regression test is a test for certain Harmony jira issue.
So IMHO, running all regression tests for certain issue doesn't make sense.

But whether using annotation to mark regression test is another story. At
least, it doesn't have any disadvantages compared with comment way, does it?


Best regards,
> Richard
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/9/4, Richard Liang <richard.liangyx@gmail.com>:
> > >
> > >
> > > Mikhail Loenko wrote:
> > > > Hi Vladimir
> > > >
> > > > Could you please decribe for what purpose it will be used?
> > > >
> > > > I mean why one might have to either exclude or run only regression
> tests?
> > >
> > > If running all tests takes up much time, running all regression test
> > > (for one particular version) may be a convenient way to verify the
> > > correctness of bug-fixing.
> > >
> > > Best regards,
> > > Richard.
> > >
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > 2006/8/30, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > >> On 8/30/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > Vladimir Ivanov wrote:
> > > >> > > Also some tag for regression tests should be added.
> > > >> > Yes. Do you think we could annotate regression test as
> > > >> > *level.regression*? Thanks a lot.
> > > >>
> > > >>
> > > >> Yes, I do. While tests can have more than one group it will enough.
> > > >>  thanks, Vladimir
> > > >>
> > > >>
> > > >> Richard
> > > >> > > thanks, Vladimir
> > > >> > >
> > > >> > >
> > > >> > > On 8/28/06, Richard Liang <richard.liangyx@gmail.com>
wrote:
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> Richard Liang wrote:
> > > >> > >> > Hello All,
> > > >> > >> >
> > > >> > >> > Now let's talk about the TestNG groups. I have
read the
> related
> > > >> > >> > threads which posted by George, Vladimir Ivanov
and Alexei
> > > >> Zakharov.
> > > >> > >> > All of them are good discussion about TestNG groups.
> > > >> > >> >
> > > >> > >> > IMHO, we may define Harmony test groups according
the
> following 4
> > > >> > >> > dimensions:
> > > >> > >> >
> > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > >> > >> > *os.any* - group of tests which pass on any platform.
IMHO,
> > > >> most of
> > > >> > >> > our tests should be in this group.
> > > >> > >> > *os.<platform id>* - group of tests which
are designed for
> one
> > > >> > >> > specific platform. A test may be in more than one
of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > >> > >> >
> > > >> > >> >    ** os.any and os.<platform id> are mutually
exclusive,
> that
> > > >> is,
> > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > >> > >> >
> > > >> > >> > 2) [Test state] state.broken, state.broken.<platform
id>
> > > >> > >> > *state.broken* - group of tests which fail on every
> platform,
> > > >> because
> > > >> > >> > of bugs of tests or implementation. We need to
fix the bugs
> of
> > > >> tests
> > > >> > >> > or implementation to make them pass.
> > > >> > >> > *state.broken.<platform id>* - groups of
test which only
> fail
> > > >> on one
> > > >> > >> > specific platform. A test may be in more than one
of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> os.broken.linux.IA64"})
> > > >> > >> >
> > > >> > >> >     **state.broken.<platform id> group may
be used as a
> > > >> convenient
> > > >> > way
> > > >> > >> > to indicate that a test is platform-specific. e.g.,
If we
> > > >> support 10
> > > >> > >> > platforms, and one test are designed for 9 platforms
except
> for
> > > >> > MacOS,
> > > >> > >> > instead of list 9 os.<platform id>, we can
just use
> > > >> > state.broken.MacOS
> > > >> > >> >
> > > >> > >> > 3) [Test type] type.api , type.impl
> > > >> > >> > *type.api* - group of tests which are tests for
APIs in the
> Java
> > > >> > >> > Specification
> > > >> > >> > *type.impl* - groups of tests which are tests for
> > > >> Harmony-specific
> > > >> > >> > implementation
> > > >> > >> >
> > > >> > >> >     ** type.api and type.impl are also mutually
exclusive.
> > > >> > >> >
> > > >> > >> > 4) [Test Level] level.unit, level.integration,
level.system,
> > > >> > >> > level.stress, etc. (Levels of Test refer to the
increase in
> > > >> > complexity
> > > >> > >> > as moving through test cycle. )
> > > >> > >> >    ** A test may be in more than one of the groups.
> > > >> > >> >    ** In fact, some tests such as System tests
are the
> > > >> verification
> > > >> > of
> > > >> > >> > the entire system.  Maybe we'll put them into a
separate
> project.
> > > >> > >> > e.g., harmony/enhanced/SVT (System Verification
Test).
> > > >> > >> >
> > > >> > >> > If we want to run all the unit test for APIs on
windows, we
> > > >> may use
> > > >> > >> > TestNG groups to select the tests:
> > > >> > >> >        <groups>
> > > >> > >> >            <run>
> > > >> > >> >                <include name="os.any" />
> > > >> > >> >                <include name="type.api" />
> > > >> > >> >                <include name="os.win.IA32" />
> > > >> > >> >                <exclude name=" state.broken"
/>
> > > >> > >> >                <exclude name="state.broken.win.IA32"
/>
> > > >> > >> >            </run>
> > > >> > >> >        </groups>
> > > >> > >> >
> > > >> > >> Hello All,
> > > >> > >>
> > > >> > >> I'm sorry. It seems that the example does not work.
I will try
> to
> > > >> > figure
> > > >> > >> another example soon. ;-)
> > > >> > >>
> > > >> > >> Best regards,
> > > >> > >> Richard
> > > >> > >> >
> > > >> > >> > Well, I think our most of existing tests are in
the groups
> of
> > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have
asked
> TestNG
> > > >> to add
> > > >> > a
> > > >> > >> > new option "-groups" for its JUnitConverter which
allow us
> to
> > > >> specify
> > > >> > >> > the test groups when migrate from JUnit test to
TestNG test.
> > > >> > >> >
> > > >> > >> > Thanks for reading so far, and I will highly appreciate
your
> > > >> comments
> > > >> > >> > or suggestion.  ;-)
> > > >> > >> >
> > > >> > >>
> > > >> > >> --
> > > >> > >> Richard Liang
> > > >> > >> China Software Development Lab, IBM
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> > >> Terms of use :
> http://incubator.apache.org/harmony/mailing.html
> > > >> > >> To unsubscribe, e-mail:
> > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > >> For additional commands, e-mail:
> > > >> harmony-dev-help@incubator.apache.org
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Richard Liang
> > > >> > China Software Development Lab, IBM
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> ---------------------------------------------------------------------
> > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > >> > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > --
> > > Richard Liang
> > > China Software Development Lab, IBM
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


-- 
Andrew Zhang
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message