From harmony-dev-return-12837-apmail-incubator-harmony-dev-archive=incubator.apache.org@incubator.apache.org Tue Sep 05 04:10:25 2006 Return-Path: Delivered-To: apmail-incubator-harmony-dev-archive@www.apache.org Received: (qmail 57200 invoked from network); 5 Sep 2006 04:10:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 5 Sep 2006 04:10:20 -0000 Received: (qmail 52311 invoked by uid 500); 5 Sep 2006 04:10:13 -0000 Delivered-To: apmail-incubator-harmony-dev-archive@incubator.apache.org Received: (qmail 52258 invoked by uid 500); 5 Sep 2006 04:10:13 -0000 Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: harmony-dev@incubator.apache.org Delivered-To: mailing list harmony-dev@incubator.apache.org Received: (qmail 52246 invoked by uid 99); 5 Sep 2006 04:10:13 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Sep 2006 21:10:13 -0700 X-ASF-Spam-Status: No, hits=1.9 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: 202.81.18.154 is neither permitted nor denied by domain of richard.liangyx@gmail.com) Received: from [202.81.18.154] (HELO ausmtp05.au.ibm.com) (202.81.18.154) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Sep 2006 21:10:12 -0700 Received: from sd0208e0.au.ibm.com (d23rh904.au.ibm.com [202.81.18.202]) by ausmtp05.au.ibm.com (8.13.6/8.13.6) with ESMTP id k854CIAK7528634 for ; Tue, 5 Sep 2006 14:12:18 +1000 Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.250.237]) by sd0208e0.au.ibm.com (8.13.6/8.13.6/NCO v8.1.1) with ESMTP id k854D9Tt244140 for ; Tue, 5 Sep 2006 14:13:10 +1000 Received: from d23av04.au.ibm.com (loopback [127.0.0.1]) by d23av04.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id k85431Ek006587 for ; Tue, 5 Sep 2006 14:09:33 +1000 Received: from d23m0011.cn.ibm.com (d23m0011.cn.ibm.com [9.181.32.74]) by d23av04.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id k848sDwP008036 for ; Mon, 4 Sep 2006 18:54:58 +1000 Received: from [127.0.0.1] ([9.181.106.104]) by d23m0011.cn.ibm.com (Lotus Domino Release 6.5.5HF262) with ESMTP id 2006090416480677-4254 ; Mon, 4 Sep 2006 16:48:06 +0800 Message-ID: <44FBE844.8040101@gmail.com> Date: Mon, 04 Sep 2006 16:48:04 +0800 From: Richard Liang User-Agent: Thunderbird 1.5.0.5 (Windows/20060719) MIME-Version: 1.0 To: harmony-dev@incubator.apache.org Subject: Re: [classlib][TestNG] groups of Harmony test References: <44EF149D.4030609@gmail.com> <44F2EE16.8020509@gmail.com> <7273946b0608290224w2baed84akd0bffea135e72667@mail.gmail.com> <44F4FEDA.8030501@gmail.com> <7273946b0608292308v7b3b1925id04ee5793a8e6e8d@mail.gmail.com> <906dd82e0609032210p7fa41731qa70a9367d82b033f@mail.gmail.com> <44FBCDF6.9070001@gmail.com> <7273946b0609040026t106e4438j22186417d3dff324@mail.gmail.com> <636fd28e0609040041i72afe846u37b8794973de6f1f@mail.gmail.com> In-Reply-To: <636fd28e0609040041i72afe846u37b8794973de6f1f@mail.gmail.com> X-MIMETrack: Itemize by SMTP Server on D23M0011/23/M/IBM(Release 6.5.5HF262 | April 5, 2006) at 04/09/2006 16:48:06, Serialize by Router on D23M0011/23/M/IBM(Release 6.5.5HF262 | April 5, 2006) at 04/09/2006 16:54:58, Serialize complete at 04/09/2006 16:54:58 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8; format=flowed X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Alex Blewitt wrote: > IMNSO it doesn't make sense to arbitrarily partition the tests based > on a moniker, such as 'integration test', 'unit test', 'regression > test' etc. For one thing, developers are generally not good at > agreeing on the difference between them :-) This is really a problem, however it might be simpler than we imagine. We are open to any discussion. ;-) Anyway, developers are required to write unit tests. > > If you've got fast and slow tests, then have a group for fast and slow > tests. Then you can choose to just run the fast tests, and any > automated build system can handle running the slow tests. IMHO, "fast or slow" may not be the key point. The question is whether we have any requirements to run only the regression tests. > > Alex. > > On 04/09/06, Vladimir Ivanov wrote: >> On 9/4/06, Richard Liang wrote: >> > >> > >> > >> > Mikhail Loenko wrote: >> > > Hi Vladimir >> > > >> > > Could you please decribe for what purpose it will be used? >> > > >> > > I mean why one might have to either exclude or run only regression >> > tests? >> > >> > If running all tests takes up much time, running all regression test >> > (for one particular version) may be a convenient way to verify the >> > correctness of bug-fixing. >> >> >> Thanks Richard. It is exactly what I want to say :) >> On the other hand may be all proposed groups need similar explanation >> (smt. >> like use-case for group)? >> >> thanks, Vladimir >> >> Best regards, >> > Richard. >> > >> > > >> > > Thanks, >> > > Mikhail >> > > >> > > 2006/8/30, Vladimir Ivanov : >> > >> On 8/30/06, Richard Liang wrote: >> > >> > >> > >> > >> > >> > >> > >> > Vladimir Ivanov wrote: >> > >> > > Also some tag for regression tests should be added. >> > >> > Yes. Do you think we could annotate regression test as >> > >> > *level.regression*? Thanks a lot. >> > >> >> > >> >> > >> Yes, I do. While tests can have more than one group it will enough. >> > >> thanks, Vladimir >> > >> >> > >> >> > >> Richard >> > >> > > thanks, Vladimir >> > >> > > >> > >> > > >> > >> > > On 8/28/06, Richard Liang wrote: >> > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> Richard Liang wrote: >> > >> > >> > Hello All, >> > >> > >> > >> > >> > >> > Now let's talk about the TestNG groups. I have read the >> related >> > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei >> > >> Zakharov. >> > >> > >> > All of them are good discussion about TestNG groups. >> > >> > >> > >> > >> > >> > IMHO, we may define Harmony test groups according the >> following >> > 4 >> > >> > >> > dimensions: >> > >> > >> > >> > >> > >> > 1) [Platform] os.any, os. >> > >> > >> > *os.any* - group of tests which pass on any platform. IMHO, >> > >> most of >> > >> > >> > our tests should be in this group. >> > >> > >> > *os.* - group of tests which are designed >> for one >> > >> > >> > specific platform. A test may be in more than one of the >> > >> groups. e.g >> > >> > ., >> > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"}) >> > >> > >> > >> > >> > >> > ** os.any and os. are mutually exclusive, >> that >> > >> is, >> > >> > >> > tests in os.any group should not be in os.win.IA32. >> > >> > >> > >> > >> > >> > 2) [Test state] state.broken, state.broken. >> > >> > >> > *state.broken* - group of tests which fail on every >> platform, >> > >> because >> > >> > >> > of bugs of tests or implementation. We need to fix the >> bugs of >> > >> tests >> > >> > >> > or implementation to make them pass. >> > >> > >> > *state.broken.* - groups of test which only >> fail >> > >> on one >> > >> > >> > specific platform. A test may be in more than one of the >> > >> groups. e.g >> > >> > ., >> > >> > >> > @Test(groups={"state.broken.linux.IA32", >> "os.broken.linux.IA64 >> > "}) >> > >> > >> > >> > >> > >> > **state.broken. group may be used as a >> > >> convenient >> > >> > way >> > >> > >> > to indicate that a test is platform-specific. e.g., If we >> > >> support 10 >> > >> > >> > platforms, and one test are designed for 9 platforms >> except for >> > >> > MacOS, >> > >> > >> > instead of list 9 os., we can just use >> > >> > state.broken.MacOS >> > >> > >> > >> > >> > >> > 3) [Test type] type.api , type.impl >> > >> > >> > *type.api* - group of tests which are tests for APIs in >> the Java >> > >> > >> > Specification >> > >> > >> > *type.impl* - groups of tests which are tests for >> > >> Harmony-specific >> > >> > >> > implementation >> > >> > >> > >> > >> > >> > ** type.api and type.impl are also mutually exclusive. >> > >> > >> > >> > >> > >> > 4) [Test Level] level.unit, level.integration, level.system, >> > >> > >> > level.stress, etc. (Levels of Test refer to the increase in >> > >> > complexity >> > >> > >> > as moving through test cycle. ) >> > >> > >> > ** A test may be in more than one of the groups. >> > >> > >> > ** In fact, some tests such as System tests are the >> > >> verification >> > >> > of >> > >> > >> > the entire system. Maybe we'll put them into a separate >> > project. >> > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test). >> > >> > >> > >> > >> > >> > If we want to run all the unit test for APIs on windows, we >> > >> may use >> > >> > >> > TestNG groups to select the tests: >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> Hello All, >> > >> > >> >> > >> > >> I'm sorry. It seems that the example does not work. I will >> try to >> > >> > figure >> > >> > >> another example soon. ;-) >> > >> > >> >> > >> > >> Best regards, >> > >> > >> Richard >> > >> > >> > >> > >> > >> > Well, I think our most of existing tests are in the >> groups of >> > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked >> TestNG >> > >> to add >> > >> > a >> > >> > >> > new option "-groups" for its JUnitConverter which allow >> us to >> > >> specify >> > >> > >> > the test groups when migrate from JUnit test to TestNG test. >> > >> > >> > >> > >> > >> > Thanks for reading so far, and I will highly appreciate your >> > >> comments >> > >> > >> > or suggestion. ;-) >> > >> > >> > >> > >> > >> >> > >> > >> -- >> > >> > >> Richard Liang >> > >> > >> China Software Development Lab, IBM >> > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> >> --------------------------------------------------------------------- >> > >> > >> Terms of use : >> http://incubator.apache.org/harmony/mailing.html >> > >> > >> To unsubscribe, e-mail: >> > >> harmony-dev-unsubscribe@incubator.apache.org >> > >> > >> For additional commands, e-mail: >> > >> harmony-dev-help@incubator.apache.org >> > >> > >> >> > >> > >> >> > >> > > >> > >> > >> > >> > -- >> > >> > Richard Liang >> > >> > China Software Development Lab, IBM >> > >> > >> > >> > >> > >> > >> > >> > >> --------------------------------------------------------------------- >> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html >> > >> > To unsubscribe, e-mail: >> harmony-dev-unsubscribe@incubator.apache.org >> > >> > For additional commands, e-mail: >> > harmony-dev-help@incubator.apache.org >> > >> > >> > >> > >> > >> >> > >> >> > > >> > > >> --------------------------------------------------------------------- >> > > Terms of use : http://incubator.apache.org/harmony/mailing.html >> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org >> > > For additional commands, e-mail: >> harmony-dev-help@incubator.apache.org >> > > >> > > >> > >> > -- >> > Richard Liang >> > China Software Development Lab, IBM >> > >> > >> > >> > --------------------------------------------------------------------- >> > Terms of use : http://incubator.apache.org/harmony/mailing.html >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org >> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org >> > >> > >> >> > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org > For additional commands, e-mail: harmony-dev-help@incubator.apache.org > > -- Richard Liang China Software Development Lab, IBM --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org For additional commands, e-mail: harmony-dev-help@incubator.apache.org