Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of george.c.harley@googlemail.com
 designates 64.233.182.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=googlemail.com;
        h=received:message-id:date:from:reply-to:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
        b=dWUwKvRhs7ZB86/6EOtj3eSLClOGMTLisxQaMbl1bbMkxEFQp3TtWh27Pzueas8ySFHK/ZmgVRk6JiFDy0zlAl8AcXpER2LZrBwA3ztLSau7yJNG45lbiWTtK5XQw/z8FOBNzORlJyJCmB73+TdXm/9wTVBL8pwunWJJ0c0+EYw=
Message-ID: <44B26673.2060001@googlemail.com>
Date: Mon, 10 Jul 2006 15:38:43 +0100
From: George Harley <george.c.harley@googlemail.com>
Reply-To: harmony-dev@incubator.apache.org
User-Agent: Thunderbird 1.5.0.4 (Windows/20060516)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib] Testing conventions - a proposal
References: <003f01c6a218$1a474130$0f01a8c0@LITTLEGUY>
	 <44AFA5D5.7010200@pobox.com>
	 <636fd28e0607080726v6b17efb8v7ab7eb2a01ca6355@mail.gmail.com>
 <2c9597b90607100718o5bc10f68s9ee870fcbc33584f@mail.gmail.com>
In-Reply-To: <2c9597b90607100718o5bc10f68s9ee870fcbc33584f@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Alexei Zakharov wrote:
>> Actually, there's a very valid benefit for using TestNG markers (=
>> annotations/JavaDoc) for grouping tests; the directory structure is a
>> tree, whereas the markers can form any slice of tests, and the sets
>
> Concerning TestNG vs JUnit. I just like to pay your attention on the
> fact what it is possible to achieve the same level of test
> grouping/slicing with JUnit TestSuites. You may define any number of
> intersecting suites - XXXAPIFailingSuite, XXXHYSpecificSuite,
> XXXWinSpecificSuite or whatever. Without necessity of migrating to new
> (probably unstable) test harness.
> Just my two cents.
>
>

Hi Alexei,

You are quite correct that JUnit test suites are another alternative 
here. If I recall correctly, their use was discussed in the very early 
days of this project but it came to nothing and we instead went down the 
route of using exclusion filters in the Ant JUnit task. That approach 
does not offer much in the way of fine grain control and relies on us 
pushing stuff around the repository. Hence the kicking off of this thread.

For the purposes of this discussion it would be fascinating to find out 
why you refer to TestNG as being an "unstable" test harness. What is 
that statement based on ?

Best regards,
George


> 2006/7/8, Alex Blewitt <alex.blewitt@gmail.com>:
>> On 08/07/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>> >
>> > So while I like the annotations, and expect we can use them 
>> effectively,
>> > I have an instinctive skepticism of annotations right now because in
>> > general (in general in Java), I'm not convinced we've used them enough
>> > to grok good design patterns.
>>
>> There's really no reason to get hung up on the annotations. TestNG
>> works just as well with JavaDoc source comments; annotations are only
>> another means to that end. (They're probably a better one for the
>> future, but it's just an implementation detail.)
>>
>> > Now since I still haven't read the thread fully, I'm jumping to
>> > conclusions, taking it to the extreme, etc etc, but my thinking in
>> > writing the above is that if we bury everything about our test
>> > 'parameter space' in annotations, some of the visible organization we
>> > have now w/ on-disk layout becomes invisible, and the readable
>> > "summaries" of aspects of testing that we'd have in an XML metadata
>> > document (or whatever) also are hard because you need to scan the
>> > sources to find all instances of annotation "X".
>>
>> I'm hoping that this would be just as applicable to using JavaDoc
>> variants, and that the problem's not with annotations per se.
>>
>> In either case, both are grokkable with tools -- either
>> annotation-savy readers or a JavaDoc tag processor, and it wouldn't be
>> hard to configure one of those to periodically scan the codebase to
>> generate reports. Furthermore, as long as the annotation X is well
>> defined, *you* don't have to scan it -- you leave it up to TestNG to
>> figure it out.
>>
>> Actually, there's a very valid benefit for using TestNG markers (=
>> annotations/JavaDoc) for grouping tests; the directory structure is a
>> tree, whereas the markers can form any slice of tests, and the sets
>> don't need to be strict subsets (with a tree, everything has to be a
>> strict subset of its parents). That means that it's possible to define
>> a marker IO to run all the IO tests, or a marker Win32 to run all the
>> Win32 tests, and both of those will contain IO-specific Win32 tests.
>> You can't do that in a tree structure without duplicating content
>> somewhere along the line (e.g. /win/io or /io/win). Neither of these
>> scale well, and every time you add a new dimension, you're doubling
>> the structure of the directory, but merely adding a new marker with
>> TestNG. So if you wanted to have (say) boot classpath tests vs api
>> tests, then you'd ahve to have /api/win/io and /boot/win/io (or
>> various permutations as applicable).
>>
>> Most of the directory-based arguments seem to be along the lines of
>> "/api/win/io is better! No, /win/io/api is better!". Just have an
>> 'api', 'win', 'io' TestNG marker, and then let TestNG figure out which
>> ones to run. You can then even get specific, and only run the Windows
>> IO API tests, if you really want -- but if you don't, you get the
>> benefit of being able to run all IO tests (both API and boot).
>>
>> There doesn't seem to be any benefit to having a strict tree-like
>> structure to the tests when it's possible to have a multi-dimensional
>> matrix of all possible combinations that's managed by the tool.
>>
>> Alex.
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org