Return-Path: 
 <harmony-dev-return-12558-apmail-incubator-harmony-dev-archive=incubator.apache.org@incubator.apache.org>
Delivered-To: apmail-incubator-harmony-dev-archive@www.apache.org
Received: (qmail 32397 invoked from network); 28 Aug 2006 09:06:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 28 Aug 2006 09:06:14 -0000
Received: (qmail 8649 invoked by uid 500); 28 Aug 2006 09:06:08 -0000
Delivered-To: apmail-incubator-harmony-dev-archive@incubator.apache.org
Received: (qmail 8038 invoked by uid 500); 28 Aug 2006 09:06:07 -0000
Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:harmony-dev-help@incubator.apache.org>
List-Unsubscribe: <mailto:harmony-dev-unsubscribe@incubator.apache.org>
List-Post: <mailto:harmony-dev@incubator.apache.org>
List-Id: <harmony-dev.incubator.apache.org>
Reply-To: harmony-dev@incubator.apache.org
Delivered-To: mailing list harmony-dev@incubator.apache.org
Received: (qmail 8027 invoked by uid 99); 28 Aug 2006 09:06:07 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Aug 2006 02:06:07 -0700
X-ASF-Spam-Status: No, hits=0.5 required=10.0
	tests=DNS_FROM_RFC_ABUSE,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (asf.osuosl.org: domain of zhanghuangzhu@gmail.com
 designates 66.249.82.238 as permitted sender)
Received: from [66.249.82.238] (HELO wx-out-0506.google.com) (66.249.82.238)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Aug 2006 02:06:06 -0700
Received: by wx-out-0506.google.com with SMTP id s13so1578389wxc
        for <harmony-dev@incubator.apache.org>;
 Mon, 28 Aug 2006 02:05:45 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=sFY/FVk/+k9JyJkLSDTDNGWpSqTARLOBhD0GuKDkHhEOMSU9nPHXhdcLW6CxIDLhU/LkIdC2tQgD3s38HePdOdC+aRrCWbWneDa7N1SJHtQZ7Y8bi3WUx+9aA8TY9PfE8Uewv13vCdp18xEMugHYcjMiD9pPTM7A8wu5SD4DDsk=
Received: by 10.70.18.11 with SMTP id 11mr8963305wxr;
        Mon, 28 Aug 2006 02:05:45 -0700 (PDT)
Received: by 10.70.123.4 with HTTP; Mon, 28 Aug 2006 02:05:45 -0700 (PDT)
Message-ID: <4d0b24970608280205x798e094ak4a68d05ded40ed5d@mail.gmail.com>
Date: Mon, 28 Aug 2006 17:05:45 +0800
From: "Andrew Zhang" <zhanghuangzhu@gmail.com>
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][TestNG] groups of Harmony test
In-Reply-To: <8E389A5F2FEABA4CB1DEC35A25CB39CE30F42E@mssmsx411>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_161016_27101829.1156755945567"
References: <8E389A5F2FEABA4CB1DEC35A25CB39CE30F42E@mssmsx411>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

------=_Part_161016_27101829.1156755945567
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On 8/28/06, Ivanov, Alexey A <alexey.a.ivanov@intel.com> wrote:
>
>
> >-----Original Message-----
> >From: Richard Liang [mailto:richard.liangyx@gmail.com]
> >Sent: Friday, August 25, 2006 7:18 PM
> >To: harmony-dev@incubator.apache.org
> >Subject: [classlib][TestNG] groups of Harmony test
> >
> >Hello All,
> >
> >Now let's talk about the TestNG groups. I have read the related threads
> >which posted by George, Vladimir Ivanov and Alexei Zakharov. All of
> them
> >are good discussion about TestNG groups.
> >
> >IMHO, we may define Harmony test groups according the following 4
> >dimensions:
> >
> >1) [Platform] os.any, os.<platform id>
> > *os.any* - group of tests which pass on any platform. IMHO, most of
> our
> >tests should be in this group.
> > *os.<platform id>* - group of tests which are designed for one
> specific
> >platform. A test may be in more than one of the groups. e.g.,
> >@Test(groups={"os.win.IA32", "os.linux.IA32"})
> >
> >    ** os.any and os.<platform id> are mutually exclusive, that is,
> >tests in os.any group should not be in os.win.IA32.
> >
> >2) [Test state] state.broken, state.broken.<platform id>
> > *state.broken* - group of tests which fail on every platform, because
> >of bugs of tests or implementation. We need to fix the bugs of tests or
> >implementation to make them pass.
> > *state.broken.<platform id>* - groups of test which only fail on one
> >specific platform. A test may be in more than one of the groups. e.g.,
> >@Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> >
> >     **state.broken.<platform id> group may be used as a convenient way
> >to indicate that a test is platform-specific. e.g., If we support 10
> >platforms, and one test are designed for 9 platforms except for MacOS,
> >instead of list 9 os.<platform id>, we can just use state.broken.MacOS
> >
> >3) [Test type] type.api, type.impl
> > *type.api* - group of tests which are tests for APIs in the Java
> >Specification
> > *type.impl* - groups of tests which are tests for Harmony-specific
> >implementation
> >
> >     ** type.api and type.impl are also mutually exclusive.
> >
> >4) [Test Level] level.unit, level.integration, level.system,
> >level.stress, etc. (Levels of Test refer to the increase in complexity
> >as moving through test cycle. )
> >    ** A test may be in more than one of the groups.
> >    ** In fact, some tests such as System tests are the verification of
> >the entire system.  Maybe we'll put them into a separate project. e.g.,
> >harmony/enhanced/SVT (System Verification Test).
>
> 5) [Environment] env.display, env.headless
>   To distinguish AWT and Swing tests which need a display to run, and
> those which don't, as Mark proposed [1].


Will display option be passed manually as an argument to TestNG, or detected
automatically  when running test?

Regards,
> Alexey.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb
> ox/%3c200608241441.k7OEfuBR018998@d06av02.portsmouth.uk.ibm.com%3e
>
> >
> >If we want to run all the unit test for APIs on windows, we may use
> >TestNG groups to select the tests:
> >        <groups>
> >            <run>
> >                <include name="os.any" />
> >                <include name="type.api" />
> >                <include name="os.win.IA32" />
> >                <exclude name="state.broken" />
> >                <exclude name="state.broken.win.IA32" />
> >            </run>
> >        </groups>
> >
> >
> >Well, I think our most of existing tests are in the groups of
> {"os.any",
> >"type.api", "level.unit"}, and I have asked TestNG to add a new option
> >"-groups" for its JUnitConverter which allow us to specify the test
> >groups when migrate from JUnit test to TestNG test.
> >
> >Thanks for reading so far, and I will highly appreciate your comments
> or
> >suggestion.  ;-)
> >
> >--
> >Richard Liang
> >China Software Development Lab, IBM
> >
> >
> >
> >---------------------------------------------------------------------
> >Terms of use : http://incubator.apache.org/harmony/mailing.html
> >To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
> --
> Alexey A. Ivanov
> Intel Middleware Product Division
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


-- 
Andrew Zhang
China Software Development Lab, IBM

------=_Part_161016_27101829.1156755945567--