Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of richard.liangyx@gmail.com
 designates 64.233.184.235 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=VsLUNfwlAIcvVDs3U0CQYVK/8cZVodroBniqeUj0AmLBHCnRAgyp0vHNkBx6je3deGBFh66uHU+ORTiOCSTi8rajzxdc8S05esbdptM6mukVT70pvZ/nDn8WfGppiiq0Z/8edcMNjgq84L1ogGZZBE5nvHyDrCMM3ij99nRAdOE=
Message-ID: <ea7a13d60609040723h16c37d22jdf4a36eb9d546dfc@mail.gmail.com>
Date: Mon, 4 Sep 2006 22:23:12 +0800
From: "Richard Liang" <richard.liangyx@gmail.com>
To: harmony-dev@incubator.apache.org
Subject: Re: Re: [classlib][TestNG] groups of Harmony test
In-Reply-To: <4d0b24970609040656v3a961368n48618364923f8e06@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <44EF149D.4030609@gmail.com>
	 <7273946b0608290224w2baed84akd0bffea135e72667@mail.gmail.com>
	 <44F4FEDA.8030501@gmail.com>
	 <7273946b0608292308v7b3b1925id04ee5793a8e6e8d@mail.gmail.com>
	 <906dd82e0609032210p7fa41731qa70a9367d82b033f@mail.gmail.com>
	 <44FBCDF6.9070001@gmail.com>
	 <7273946b0609040026t106e4438j22186417d3dff324@mail.gmail.com>
	 <636fd28e0609040041i72afe846u37b8794973de6f1f@mail.gmail.com>
	 <ea7a13d60609040632x3dba13fdj1377354eab3dd596@mail.gmail.com>
	 <4d0b24970609040656v3a961368n48618364923f8e06@mail.gmail.com>

On 9/4/06, Andrew Zhang <zhanghuangzhu@gmail.com> wrote:
> On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> >
> > I sent my reply this afternoon, but I have not received it. (it seems
> > there
> > is something wrong with my smtp server). So I send it again. :-)
> >
> > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com> wrote:
> > >
> > > IMNSO it doesn't make sense to arbitrarily partition the tests based
> > > on a moniker, such as 'integration test', 'unit test', 'regression
> > > test' etc. For one thing, developers are generally not good at
> > > agreeing on the difference between them :-)
> >
> >
> >
> > This is really a problem, however it might be simpler than we imagine. We
> > are open to any discussion. ;-) Anyway, developers are required to write
> > unit tests.
> >
> >
> > If you've got fast and slow tests, then have a group for fast and slow
> > > tests. Then you can choose to just run the fast tests, and any
> > > automated build system can handle running the slow tests.
> >
> >
> > IMHO, "fast or slow" may not be the key point. The question is whether we
> > have any requirements to run only the regression tests.
>
>
> I think your first proposal looks fine. Platform + os + state + api/impl
> information are enough for all tests. We have to assure all tests pass on
> every platform.
>

Thanks a lot. Well, I agree :-). TestNG annotations are most helpful
to us in the management of unit tests that are prone to change. For
example, some unit tests suddenly break on a particular platform. or
we may want to exclude some tests temporarily.

Here I just try to list "all" the possible facets, so that we could
have more thorough discussion.

Best regards,
Richard

> For quick run, developers can use their own short approach, like run single
> test in IDE... It's not business of TestNG. :-)
>
>
> Best regards,
> > Richard
> >
> > Alex.
> > >
> > > On 04/09/06, Vladimir Ivanov <ivavladimir@gmail.com> wrote:
> > > > On 9/4/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > Mikhail Loenko wrote:
> > > > > > Hi Vladimir
> > > > > >
> > > > > > Could you please decribe for what purpose it will be used?
> > > > > >
> > > > > > I mean why one might have to either exclude or run only regression
> > > > > tests?
> > > > >
> > > > > If running all tests takes up much time, running all regression test
> > > > > (for one particular version) may be a convenient way to verify the
> > > > > correctness of bug-fixing.
> > > >
> > > >
> > > > Thanks Richard. It is exactly what I want to say :)
> > > > On the other hand may be all proposed groups need similar explanation
> > > (smt.
> > > > like use-case for group)?
> > > >
> > > >  thanks, Vladimir
> > > >
> > > > Best regards,
> > > > > Richard.
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Mikhail
> > > > > >
> > > > > > 2006/8/30, Vladimir Ivanov <ivavladimir@gmail.com>:
> > > > > >> On 8/30/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > Vladimir Ivanov wrote:
> > > > > >> > > Also some tag for regression tests should be added.
> > > > > >> > Yes. Do you think we could annotate regression test as
> > > > > >> > *level.regression*? Thanks a lot.
> > > > > >>
> > > > > >>
> > > > > >> Yes, I do. While tests can have more than one group it will
> > enough.
> > > > > >>  thanks, Vladimir
> > > > > >>
> > > > > >>
> > > > > >> Richard
> > > > > >> > > thanks, Vladimir
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On 8/28/06, Richard Liang <richard.liangyx@gmail.com> wrote:
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> Richard Liang wrote:
> > > > > >> > >> > Hello All,
> > > > > >> > >> >
> > > > > >> > >> > Now let's talk about the TestNG groups. I have read the
> > > related
> > > > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > > > >> Zakharov.
> > > > > >> > >> > All of them are good discussion about TestNG groups.
> > > > > >> > >> >
> > > > > >> > >> > IMHO, we may define Harmony test groups according the
> > > following
> > > > > 4
> > > > > >> > >> > dimensions:
> > > > > >> > >> >
> > > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > > >> > >> > *os.any* - group of tests which pass on any platform.
> > IMHO,
> > > > > >> most of
> > > > > >> > >> > our tests should be in this group.
> > > > > >> > >> > *os.<platform id>* - group of tests which are designed for
> > > one
> > > > > >> > >> > specific platform. A test may be in more than one of the
> > > > > >> groups. e.g
> > > > > >> > .,
> > > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > > >> > >> >
> > > > > >> > >> >    ** os.any and os.<platform id> are mutually exclusive,
> > > that
> > > > > >> is,
> > > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > > >> > >> >
> > > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > > > >> > >> > *state.broken* - group of tests which fail on every
> > > platform,
> > > > > >> because
> > > > > >> > >> > of bugs of tests or implementation. We need to fix the
> > bugs
> > > of
> > > > > >> tests
> > > > > >> > >> > or implementation to make them pass.
> > > > > >> > >> > *state.broken.<platform id>* - groups of test which only
> > > fail
> > > > > >> on one
> > > > > >> > >> > specific platform. A test may be in more than one of the
> > > > > >> groups. e.g
> > > > > >> > .,
> > > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > > os.broken.linux.IA64
> > > > > "})
> > > > > >> > >> >
> > > > > >> > >> >     **state.broken.<platform id> group may be used as a
> > > > > >> convenient
> > > > > >> > way
> > > > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > > > >> support 10
> > > > > >> > >> > platforms, and one test are designed for 9 platforms
> > except
> > > for
> > > > > >> > MacOS,
> > > > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > > > >> > state.broken.MacOS
> > > > > >> > >> >
> > > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > > >> > >> > *type.api* - group of tests which are tests for APIs in
> > the
> > > Java
> > > > > >> > >> > Specification
> > > > > >> > >> > *type.impl* - groups of tests which are tests for
> > > > > >> Harmony-specific
> > > > > >> > >> > implementation
> > > > > >> > >> >
> > > > > >> > >> >     ** type.api and type.impl are also mutually exclusive.
> > > > > >> > >> >
> > > > > >> > >> > 4) [Test Level] level.unit, level.integration,
> > level.system,
> > > > > >> > >> > level.stress, etc. (Levels of Test refer to the increase
> > in
> > > > > >> > complexity
> > > > > >> > >> > as moving through test cycle. )
> > > > > >> > >> >    ** A test may be in more than one of the groups.
> > > > > >> > >> >    ** In fact, some tests such as System tests are the
> > > > > >> verification
> > > > > >> > of
> > > > > >> > >> > the entire system.  Maybe we'll put them into a separate
> > > > > project.
> > > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > > > >> > >> >
> > > > > >> > >> > If we want to run all the unit test for APIs on windows,
> > we
> > > > > >> may use
> > > > > >> > >> > TestNG groups to select the tests:
> > > > > >> > >> >        <groups>
> > > > > >> > >> >            <run>
> > > > > >> > >> >                <include name="os.any" />
> > > > > >> > >> >                <include name="type.api" />
> > > > > >> > >> >                <include name="os.win.IA32" />
> > > > > >> > >> >                <exclude name=" state.broken" />
> > > > > >> > >> >                <exclude name="state.broken.win.IA32" />
> > > > > >> > >> >            </run>
> > > > > >> > >> >        </groups>
> > > > > >> > >> >
> > > > > >> > >> Hello All,
> > > > > >> > >>
> > > > > >> > >> I'm sorry. It seems that the example does not work. I will
> > try
> > > to
> > > > > >> > figure
> > > > > >> > >> another example soon. ;-)
> > > > > >> > >>
> > > > > >> > >> Best regards,
> > > > > >> > >> Richard
> > > > > >> > >> >
> > > > > >> > >> > Well, I think our most of existing tests are in the groups
> > > of
> > > > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> > > TestNG
> > > > > >> to add
> > > > > >> > a
> > > > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> > > to
> > > > > >> specify
> > > > > >> > >> > the test groups when migrate from JUnit test to TestNG
> > test.
> > > > > >> > >> >
> > > > > >> > >> > Thanks for reading so far, and I will highly appreciate
> > your
> > > > > >> comments
> > > > > >> > >> > or suggestion.  ;-)
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Richard Liang
> > > > > >> > >> China Software Development Lab, IBM
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > >> Terms of use :
> > > http://incubator.apache.org/harmony/mailing.html
> > > > > >> > >> To unsubscribe, e-mail:
> > > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > > >> > >> For additional commands, e-mail:
> > > > > >> harmony-dev-help@incubator.apache.org
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Richard Liang
> > > > > >> > China Software Development Lab, IBM
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > ---------------------------------------------------------------------
> > > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > >> > To unsubscribe, e-mail:
> > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > >> > For additional commands, e-mail:
> > > > > harmony-dev-help@incubator.apache.org
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Richard Liang
> > > > > China Software Development Lab, IBM
> > > > >
> > > > >
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>


-- 
Richard Liang
China Software Development Lab, IBM

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org