Return-Path: 
 <river-dev-return-5318-apmail-incubator-river-dev-archive=incubator.apache.org@incubator.apache.org>
Delivered-To: apmail-incubator-river-dev-archive@minotaur.apache.org
Received: (qmail 16319 invoked from network); 12 Nov 2010 18:04:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 18:04:26 -0000
Received: (qmail 93028 invoked by uid 500); 12 Nov 2010 18:04:57 -0000
Delivered-To: apmail-incubator-river-dev-archive@incubator.apache.org
Received: (qmail 92964 invoked by uid 500); 12 Nov 2010 18:04:56 -0000
Mailing-List: contact river-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:river-dev-help@incubator.apache.org>
List-Unsubscribe: <mailto:river-dev-unsubscribe@incubator.apache.org>
List-Post: <mailto:river-dev@incubator.apache.org>
List-Id: <river-dev.incubator.apache.org>
Reply-To: river-dev@incubator.apache.org
Delivered-To: mailing list river-dev@incubator.apache.org
Received: (qmail 92956 invoked by uid 99); 12 Nov 2010 18:04:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 18:04:56 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.18.0.24] (HELO exprod5og112.obsmtp.com) (64.18.0.24)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Nov 2010 18:04:49 +0000
Received: from source ([209.202.150.67]) by exprod5ob112.postini.com
 ([64.18.4.12]) with SMTP
	ID DSNKTN2Bqx/3UVJTw67hZREIPIFJ/EQTcSMi@postini.com;
 Fri, 12 Nov 2010 10:04:29 PST
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: test framework migration - was: Re: Hudson build is back to
 normal
Date: Fri, 12 Nov 2010 13:03:49 -0500
Message-ID: 
 <77F1E32F67C8D5479858C0C7E93EB46503AA4370@WAL-MAIL.global.avidww.com>
In-Reply-To: <4CDD117F.6030501@zeus.net.au>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: test framework migration - was: Re: Hudson build is back to
 normal
Thread-Index: AcuCUhcHA2mNTpBCS2Cn2C5acaQWHwAP1vpQ
References: <2101938159.1941288776689346.JavaMail.hudson@aegis>
	<1987111689.2571288839742596.JavaMail.hudson@aegis>
	<AANLkTinLUtHaTbjjb5HxRBtVCFk+-WkogxUuYCis9K67@mail.gmail.com>
	<4CD3B6C4.4080903@zeus.net.au>
	<AANLkTi=CNA96bRLRrDpH3zQ1ajweOASrrGnQf73yfBMb@mail.gmail.com>
	<4CD4EEB9.1030300@zeus.net.au>
	<AANLkTi=ko5Nt491Mbir5A8rY7MX3wFRCnjpYJodwzGk8@mail.gmail.com>
	<20101107115412.22d31db1@brain>	<4CD6AB27.1020409@acm.org>
	<4CDC9294.2070100@zeus.net.au>
 <AANLkTin_JfC6uiu9MTt-BiMYPLdbTtO4tBeKmiEOUWoK@mail.gmail.com>
 <4CDD117F.6030501@zeus.net.au>
From: "Christopher Dolan" <christopher.dolan@avid.com>
To: <river-dev@incubator.apache.org>

Last year my project switched all of our tests from Junit 3 to TestNG.  =
My big win with TestNG vs. Junit 4 was the ability to classify the tests =
and decide at runtime which to run.  Most of my tests are annotated with =
@Test(groups=3D"build") which are the tests executed by my continuous =
integration process.  Only a subset of my code is Mac-specific, so those =
tests are marked as @Test(groups=3D{"build","macos"}).  My really slow =
tests are marked as @Test(groups=3D"all") and are only executed on =
specific machines.

>From Ant, I choose which groups to run via <testng groups=3D"build" ... =
/> or <testng groups=3D"macos" ... /> etc.

TestNG also has an @DataProvider notation that dramatically simplifies =
groups of tests that have the exact same code but different initial =
conditions.  Here's a too-simple example just to show the technique:


   @DataProvider(name=3D"testcases")
   public Object[][] getTestcases() {
        return new Object[][] {
            { 1, "1" },
            { -0, "0" },
        };
   }

   @Test(groups=3D"build", dataProvider=3D"testcases")
   public void testNumification(int num, String expected) {
       Assert.assertEquals("" + num, expected);
   }

Another useful annotation regards exceptions, which says to fail the =
test unless an exception class (or subclass) is thrown

   @Test(groups=3D"build", =
expectedException=3DIllegalArgumentException.class)
   public void testParseIntFailure() {
       Integer.parseInt("not a number");
   }

TestNG also lets you throw org.testng.SkipException at runtime to =
neither pass nor fail the test.  For example, if I want to run a test on =
Win7 but skip it on WinXP.


But, I agree with Peter that it has to be all TestNG or all JUnit.  You =
can make them play nice, but it's not worth the hassle.  The conversion =
was a little tedious (do NOT trust the auto-convert tools!  Convert by =
hand instead!) but bearable.  TestNG has their own org.testng.Assert =
that has a different API from org.junit.Assert, but both throw =
AssertionError in the end so it doesn't actually matter which you use. I =
like TestNG's API a little better (methods arguments are ordered "got, =
expected, message" vs. Junit's "message, expected, got").



Chris


-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]=20
Sent: Friday, November 12, 2010 4:06 AM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration - was: Re: Hudson build is back to =
normal

Thanks Tom, I agree, if someone has a good reason...

Cheers,

Peter.

Tom Hobbs wrote:
> I've written a couple of JUnit tests.
>
> I've not used TestNG for many years so can't really comment on it's
> comparison with JUnit.  I don't object to converting my tests, however
> I'd rather convert them because "TestNG does X which we need to do Y"
> rather than "We should swap to TestNG because it's better."
>
> I agree, this should be an either/or decision.  We should definitely
> *not* be using both.
>
> On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <jini@zeus.net.au> =
wrote:
>  =20
>> Patricia Shanahan wrote:
>>    =20
>>> Zsolt K=FAti wrote:
>>>      =20
>>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>>> Jonathan Costers <jonathan.costers@googlemail.com> wrote:
>>>>
>>>> ...
>>>>        =20
>>>>> Not all of them use/need a multi VM setup. Those are candidates =
for
>>>>> JUnit. The others would be QA candidates.
>>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>>> requires knowledge of how the jtreg framework operates, as well as
>>>>> the proposed target framework (JUnit, QA).
>>>>>
>>>>>
>>>>>          =20
>>>>>> JUnit's good when we're only testing a single object
>>>>>> implementation, we can document and expect people to utilse the =
qa
>>>>>> suite for more complex tests.
>>>>>>            =20
>>>>> Agreed.
>>>>>          =20
>>>> Hello hard workers,
>>>>
>>>> It would be worth considering the use of TestNG instead of JUnit.
>>>> I have no experience in their comparison, so relied on other
>>>> sources when I was to decid what framework to use (like this:
>>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>>> TestNG features that are missing from JUnit can be useful in a =
complex
>>>> test environment like that of River.
>>>>        =20
>>> If we were starting cold, with no existing tests, I might be open to =
this
>>> suggestion. As it is, we already have a QA framework that can do all =
the
>>> complex, multi-JVM tests, and we have over 1000 existing tests using =
it.
>>>
>>> I think the objective in converting jtreg tests would be to reduce =
the
>>> number of frameworks, and the amount of software we need installed, =
in order
>>> to run a full test. Switching them to TestNG, or anything else other =
than
>>> JUnit or the River QA framework, would not achieve that.
>>>
>>> Patricia
>>>
>>>      =20
>> Has anyone else written any junit tests other than myself? If TestNG =
is
>> justifiably better than Junit, I'd be prepared to convert my tests, I
>> believe many of the annotations are common?
>>
>> But it would have to be Junit OR TestNG, not both.
>>
>> Cheers,
>>
>> Peter.
>>
>>    =20
>
>  =20