harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Liang" <richard.lian...@gmail.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Mon, 04 Sep 2006 15:05:55 GMT
On 9/4/06, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> >
> > On 9/4/06, Mikhail Loenko <mloenko@gmail.com> wrote:
> > > Well, my question was for what particular reason? for example?
> > >
> > > Tio verify correctness of bug-fixing IMHO all the unit, intergration,
> > api, and
> > > regression tests should be run
> >
> > Running all tests are always good to verify our code quality. And I
> > think this is what we have been doing. But what will we do if it takes
> > us 24 hours to run all Harmony tests? Anyway, running regression tests
> > could provide some confidence to our bug-fixing. We may consider it as
> > another option. :-)
>
>
> I prefer to run all tests in one module instead. :) Although it can not
> guarentee all tests would pass, it's less likey to break build system
> frequently.  If the fix causes other module fails, it shows the lack of
> tests in its module. And we should add the corresponding test in the module.
>

Running all tests for one module is practical for most cases (except luni).


> Currently, Harmony regression test is a test for certain Harmony jira issue.
> So IMHO, running all regression tests for certain issue doesn't make sense.

Yes, Harmony regression test is not the so-called "regression test"
which aim to verify that no unwanted changes were introduced to one
part of the system as a
result of making changes to another part of the system.

I agree that we may not want to run all the "regression test". If
there is no objection I will remove "level.regression"

>
> But whether using annotation to mark regression test is another story. At
> least, it doesn't have any disadvantages compared with comment way, does it?
>

We may use @Test(description="Regression: Harmony-XXXX").

>
> Best regards,
> > Richard
> >
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2006/9/4, Richard Liang <richard.liangyx@gmail.com>:
> > > >
> > > >
> > > > Mikhail Loenko wrote:
> > > > > Hi Vladimir
> > > > >
> > > > > Could you please decribe for what purpose it will be used?
> > > > >
> > > > > I mean why one might have to either exclude or run only regression
> > tests?
> > > >
> > > > If running all tests takes up much time, running all regression test
> > > > (for one particular version) may be a convenient way to verify the
> > > > correctness of bug-fixing.
> > > >
> > > > Best regards,
> > > > Richard.
> > > >
> > > > >
> > > > > Thanks,
> > > > > Mikhail
> > > > >
> > > > > 2006/8/30, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > > >> On 8/30/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Vladimir Ivanov wrote:
> > > > >> > > Also some tag for regression tests should be added.
> > > > >> > Yes. Do you think we could annotate regression test as
> > > > >> > *level.regression*? Thanks a lot.
> > > > >>
> > > > >>
> > > > >> Yes, I do. While tests can have more than one group it will enough.
> > > > >>  thanks, Vladimir
> > > > >>
> > > > >>
> > > > >> Richard
> > > > >> > > thanks, Vladimir
> > > > >> > >
> > > > >> > >
> > > > >> > > On 8/28/06, Richard Liang <richard.liangyx@gmail.com>
wrote:
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Richard Liang wrote:
> > > > >> > >> > Hello All,
> > > > >> > >> >
> > > > >> > >> > Now let's talk about the TestNG groups. I
have read the
> > related
> > > > >> > >> > threads which posted by George, Vladimir Ivanov
and Alexei
> > > > >> Zakharov.
> > > > >> > >> > All of them are good discussion about TestNG
groups.
> > > > >> > >> >
> > > > >> > >> > IMHO, we may define Harmony test groups according
the
> > following 4
> > > > >> > >> > dimensions:
> > > > >> > >> >
> > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > >> > >> > *os.any* - group of tests which pass on any
platform. IMHO,
> > > > >> most of
> > > > >> > >> > our tests should be in this group.
> > > > >> > >> > *os.<platform id>* - group of tests
which are designed for
> > one
> > > > >> > >> > specific platform. A test may be in more than
one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > >> > >> >
> > > > >> > >> >    ** os.any and os.<platform id> are
mutually exclusive,
> > that
> > > > >> is,
> > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > >> > >> >
> > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform
id>
> > > > >> > >> > *state.broken* - group of tests which fail
on every
> > platform,
> > > > >> because
> > > > >> > >> > of bugs of tests or implementation. We need
to fix the bugs
> > of
> > > > >> tests
> > > > >> > >> > or implementation to make them pass.
> > > > >> > >> > *state.broken.<platform id>* - groups
of test which only
> > fail
> > > > >> on one
> > > > >> > >> > specific platform. A test may be in more than
one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > os.broken.linux.IA64"})
> > > > >> > >> >
> > > > >> > >> >     **state.broken.<platform id> group
may be used as a
> > > > >> convenient
> > > > >> > way
> > > > >> > >> > to indicate that a test is platform-specific.
e.g., If we
> > > > >> support 10
> > > > >> > >> > platforms, and one test are designed for 9
platforms except
> > for
> > > > >> > MacOS,
> > > > >> > >> > instead of list 9 os.<platform id>,
we can just use
> > > > >> > state.broken.MacOS
> > > > >> > >> >
> > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > >> > >> > *type.api* - group of tests which are tests
for APIs in the
> > Java
> > > > >> > >> > Specification
> > > > >> > >> > *type.impl* - groups of tests which are tests
for
> > > > >> Harmony-specific
> > > > >> > >> > implementation
> > > > >> > >> >
> > > > >> > >> >     ** type.api and type.impl are also mutually
exclusive.
> > > > >> > >> >
> > > > >> > >> > 4) [Test Level] level.unit, level.integration,
level.system,
> > > > >> > >> > level.stress, etc. (Levels of Test refer to
the increase in
> > > > >> > complexity
> > > > >> > >> > as moving through test cycle. )
> > > > >> > >> >    ** A test may be in more than one of the
groups.
> > > > >> > >> >    ** In fact, some tests such as System tests
are the
> > > > >> verification
> > > > >> > of
> > > > >> > >> > the entire system.  Maybe we'll put them into
a separate
> > project.
> > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification
Test).
> > > > >> > >> >
> > > > >> > >> > If we want to run all the unit test for APIs
on windows, we
> > > > >> may use
> > > > >> > >> > TestNG groups to select the tests:
> > > > >> > >> >        <groups>
> > > > >> > >> >            <run>
> > > > >> > >> >                <include name="os.any" />
> > > > >> > >> >                <include name="type.api"
/>
> > > > >> > >> >                <include name="os.win.IA32"
/>
> > > > >> > >> >                <exclude name=" state.broken"
/>
> > > > >> > >> >                <exclude name="state.broken.win.IA32"
/>
> > > > >> > >> >            </run>
> > > > >> > >> >        </groups>
> > > > >> > >> >
> > > > >> > >> Hello All,
> > > > >> > >>
> > > > >> > >> I'm sorry. It seems that the example does not work.
I will try
> > to
> > > > >> > figure
> > > > >> > >> another example soon. ;-)
> > > > >> > >>
> > > > >> > >> Best regards,
> > > > >> > >> Richard
> > > > >> > >> >
> > > > >> > >> > Well, I think our most of existing tests are
in the groups
> > of
> > > > >> > >> > {"os.any", "type.api", "level.unit"}, and
I have asked
> > TestNG
> > > > >> to add
> > > > >> > a
> > > > >> > >> > new option "-groups" for its JUnitConverter
which allow us
> > to
> > > > >> specify
> > > > >> > >> > the test groups when migrate from JUnit test
to TestNG test.
> > > > >> > >> >
> > > > >> > >> > Thanks for reading so far, and I will highly
appreciate your
> > > > >> comments
> > > > >> > >> > or suggestion.  ;-)
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Richard Liang
> > > > >> > >> China Software Development Lab, IBM
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > >> Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > >> > >> To unsubscribe, e-mail:
> > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > >> For additional commands, e-mail:
> > > > >> harmony-dev-help@incubator.apache.org
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Richard Liang
> > > > >> > China Software Development Lab, IBM
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > ---------------------------------------------------------------------
> > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > >> > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > > --
> > > > Richard Liang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>


-- 
Richard Liang
China Software Development Lab, IBM

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message