harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Mon, 10 Jul 2006 15:30:36 GMT
Hi George,

> For the purposes of this discussion it would be fascinating to find out
> why you refer to TestNG as being an "unstable" test harness. What is
> that statement based on ?

My exact statement was referring to TestNG as "probably unstable"
rather than simply "unstable". ;)  This statement was based on posts
from Richard Liang about the bug in the TestNG migration tool and on
common sense. If the project has such an obvious bug in one place it
may probably have other bugs in other places. JUnit is quite famous
and widely used toolkit that proved to be stable enough. TestNG is
neither famous nor widely used. And IMHO it makes sense to be careful
with new exciting tools until we *really* need their innovative
functionality.


2006/7/10, George Harley <george.c.harley@googlemail.com>:
> Alexei Zakharov wrote:
> >> Actually, there's a very valid benefit for using TestNG markers (=
> >> annotations/JavaDoc) for grouping tests; the directory structure is a
> >> tree, whereas the markers can form any slice of tests, and the sets
> >
> > Concerning TestNG vs JUnit. I just like to pay your attention on the
> > fact what it is possible to achieve the same level of test
> > grouping/slicing with JUnit TestSuites. You may define any number of
> > intersecting suites - XXXAPIFailingSuite, XXXHYSpecificSuite,
> > XXXWinSpecificSuite or whatever. Without necessity of migrating to new
> > (probably unstable) test harness.
> > Just my two cents.
> >
> >
>
> Hi Alexei,
>
> You are quite correct that JUnit test suites are another alternative
> here. If I recall correctly, their use was discussed in the very early
> days of this project but it came to nothing and we instead went down the
> route of using exclusion filters in the Ant JUnit task. That approach
> does not offer much in the way of fine grain control and relies on us
> pushing stuff around the repository. Hence the kicking off of this thread.
>
> For the purposes of this discussion it would be fascinating to find out
> why you refer to TestNG as being an "unstable" test harness. What is
> that statement based on ?
>
> Best regards,
> George
>
>
> > 2006/7/8, Alex Blewitt <alex.blewitt@gmail.com>:
> >> On 08/07/06, Geir Magnusson Jr <geir@pobox.com> wrote:
> >> >
> >> > So while I like the annotations, and expect we can use them
> >> effectively,
> >> > I have an instinctive skepticism of annotations right now because in
> >> > general (in general in Java), I'm not convinced we've used them enough
> >> > to grok good design patterns.
> >>
> >> There's really no reason to get hung up on the annotations. TestNG
> >> works just as well with JavaDoc source comments; annotations are only
> >> another means to that end. (They're probably a better one for the
> >> future, but it's just an implementation detail.)
> >>
> >> > Now since I still haven't read the thread fully, I'm jumping to
> >> > conclusions, taking it to the extreme, etc etc, but my thinking in
> >> > writing the above is that if we bury everything about our test
> >> > 'parameter space' in annotations, some of the visible organization we
> >> > have now w/ on-disk layout becomes invisible, and the readable
> >> > "summaries" of aspects of testing that we'd have in an XML metadata
> >> > document (or whatever) also are hard because you need to scan the
> >> > sources to find all instances of annotation "X".
> >>
> >> I'm hoping that this would be just as applicable to using JavaDoc
> >> variants, and that the problem's not with annotations per se.
> >>
> >> In either case, both are grokkable with tools -- either
> >> annotation-savy readers or a JavaDoc tag processor, and it wouldn't be
> >> hard to configure one of those to periodically scan the codebase to
> >> generate reports. Furthermore, as long as the annotation X is well
> >> defined, *you* don't have to scan it -- you leave it up to TestNG to
> >> figure it out.
> >>
> >> Actually, there's a very valid benefit for using TestNG markers (=
> >> annotations/JavaDoc) for grouping tests; the directory structure is a
> >> tree, whereas the markers can form any slice of tests, and the sets
> >> don't need to be strict subsets (with a tree, everything has to be a
> >> strict subset of its parents). That means that it's possible to define
> >> a marker IO to run all the IO tests, or a marker Win32 to run all the
> >> Win32 tests, and both of those will contain IO-specific Win32 tests.
> >> You can't do that in a tree structure without duplicating content
> >> somewhere along the line (e.g. /win/io or /io/win). Neither of these
> >> scale well, and every time you add a new dimension, you're doubling
> >> the structure of the directory, but merely adding a new marker with
> >> TestNG. So if you wanted to have (say) boot classpath tests vs api
> >> tests, then you'd ahve to have /api/win/io and /boot/win/io (or
> >> various permutations as applicable).
> >>
> >> Most of the directory-based arguments seem to be along the lines of
> >> "/api/win/io is better! No, /win/io/api is better!". Just have an
> >> 'api', 'win', 'io' TestNG marker, and then let TestNG figure out which
> >> ones to run. You can then even get specific, and only run the Windows
> >> IO API tests, if you really want -- but if you don't, you get the
> >> benefit of being able to run all IO tests (both API and boot).
> >>
> >> There doesn't seem to be any benefit to having a strict tree-like
> >> structure to the tests when it's possible to have a multi-dimensional
> >> matrix of all possible combinations that's managed by the tool.
> >>
> >> Alex.


-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message