harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: [classlib] TestNG v. JUnit
Date Tue, 11 Jul 2006 08:47:50 GMT
Hi Richard,

Thank you for that great summary. Another thing that may be of interest 
to people is that TestNG has been around for a couple of years now while 
JUnit 4 was released approximately .... four months ago.

Best regards,
George



Richard Liang wrote:
>
>
> Nathan Beyer wrote:
>> Not to add another fire to this topic, but with all things being 
>> relative,
>> so far this topic has been comparison of the TestNG and JUnit v3.8. From
>> what I understand, the latest JUnit v4.1 provides many of the same
>> annotation features that TestNG does, as well guaranteed 
>> compatibility with
>> JUnit v3-based tests.
>> If we were to compare moving to TestNG with upgrading to JUnit 4.1, 
>> would
>> there still be as much value in the proposition to move to TestNG?
>>
>>   
> It's hard to give an exhaustive comparison of JUnit4 and TestNG. There 
> is an existing presentation "Comparison TestNG / JUnit 4"[1], however, 
> it's in German. Not sure if there are German here. ;-)
> I just try to list some items (Correct me if I'm wrong)
>
> Both JUit4 and TestNG:
> 1)  Test classes do not have to extend from junit.framework.TestCase.
> 2) Test methods do not have to be prefixed with 'test'.
> 3) Use @Test annotations to mark a method as a test case.
> 4) Use @Before and @After annotations to identify set up and tear 
> down. (TestNG uses @Configuration.beforeTestMethod and 
> @Configuration.afterTestMethod )
> 5) Use @BeforeClass and @AfterClass annotations to identify one time 
> set up and one time tear down. (TestNG uses 
> @Configuration.beforeTestClass and @Configuration.afterTestClass )
> 6) @Test.timeout to specify the maximum time to execute
> 7) @Test.expected to specify the expected exception to be thrown 
> (TestNG uses @ExpectedExceptions)
> 8) Can execute Junit 3.8 test cases.
>
> *Differences*:
> 1) JUnit4 requires Java 5.0. while TestNG can work with Java 1.4 and 
> Java 5.0
> 2) TestNG provides more annotations to facilitate testing 
> configuration[2]
> 3) TestNG "groups" is more sophisticated than JUnit test suite[3]
> 4) TestNG make it easy to rerun failed tests[4]
> ....
>
> 1. http://www.qaware.de/downloads/to1-adersberger.pdf
> 2. http://testng.org/doc/documentation-main.html#annotations
> 3. http://testng.org/doc/documentation-main.html#test-groups
> 4. http://testng.org/doc/documentation-main.html#rerunning
>
> Best regards,
> Richard.
>
>> -Nathan
>>
>>  
>>> -----Original Message-----
>>> From: George Harley [mailto:george.c.harley@googlemail.com]
>>> Sent: Monday, July 10, 2006 3:57 PM
>>> To: harmony-dev@incubator.apache.org
>>> Subject: Re: [classlib] Testing conventions - a proposal
>>>
>>> Alexei Zakharov wrote:
>>>    
>>>> Hi George,
>>>>
>>>>      
>>>>> For the purposes of this discussion it would be fascinating to 
>>>>> find out
>>>>> why you refer to TestNG as being an "unstable" test harness. What is
>>>>> that statement based on ?
>>>>>         
>>>> My exact statement was referring to TestNG as "probably unstable"
>>>> rather than simply "unstable". ;)  This statement was based on posts
>>>> from Richard Liang about the bug in the TestNG migration tool and on
>>>> common sense. If the project has such an obvious bug in one place it
>>>> may probably have other bugs in other places. JUnit is quite famous
>>>> and widely used toolkit that proved to be stable enough. TestNG is
>>>> neither famous nor widely used. And IMHO it makes sense to be careful
>>>> with new exciting tools until we *really* need their innovative
>>>> functionality.
>>>>
>>>>       
>>> Hi Alexei,
>>>
>>> Last I heard, Richard posted saying that there was no bug in the
>>> migration tool [1]. The command line tool is designed to locate JUnit
>>> tests under a specified location and add the TestNG annotations to 
>>> them.
>>> That's what it does.
>>>
>>> You are right to say that it makes sense to be careful in this matter.
>>> Nobody wants to do anything that affects Harmony in an adverse way.
>>>
>>> Best regards,
>>> George
>>>
>>> [1]
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-
>>> dev/200607.mbox/%3c44B1C084.3020408@gmail.com%3e
>>>
>>>
>>>    
>>>> 2006/7/10, George Harley <george.c.harley@googlemail.com>:
>>>>      
>>>>> Alexei Zakharov wrote:
>>>>>        
>>>>>>> Actually, there's a very valid benefit for using TestNG markers
(=
>>>>>>> annotations/JavaDoc) for grouping tests; the directory structure
is
>>>>>>>             
>>> a
>>>    
>>>>>>> tree, whereas the markers can form any slice of tests, and the
sets
>>>>>>>             
>>>>>> Concerning TestNG vs JUnit. I just like to pay your attention on
the
>>>>>> fact what it is possible to achieve the same level of test
>>>>>> grouping/slicing with JUnit TestSuites. You may define any number
of
>>>>>> intersecting suites - XXXAPIFailingSuite, XXXHYSpecificSuite,
>>>>>> XXXWinSpecificSuite or whatever. Without necessity of migrating to
>>>>>>           
>>> new
>>>    
>>>>>> (probably unstable) test harness.
>>>>>> Just my two cents.
>>>>>>
>>>>>>
>>>>>>           
>>>>> Hi Alexei,
>>>>>
>>>>> You are quite correct that JUnit test suites are another alternative
>>>>> here. If I recall correctly, their use was discussed in the very 
>>>>> early
>>>>> days of this project but it came to nothing and we instead went down
>>>>>         
>>> the
>>>    
>>>>> route of using exclusion filters in the Ant JUnit task. That approach
>>>>> does not offer much in the way of fine grain control and relies on us
>>>>> pushing stuff around the repository. Hence the kicking off of this
>>>>> thread.
>>>>>
>>>>> For the purposes of this discussion it would be fascinating to 
>>>>> find out
>>>>> why you refer to TestNG as being an "unstable" test harness. What is
>>>>> that statement based on ?
>>>>>
>>>>> Best regards,
>>>>> George
>>>>>
>>>>>
>>>>>        
>>>>>> 2006/7/8, Alex Blewitt <alex.blewitt@gmail.com>:
>>>>>>          
>>>>>>> On 08/07/06, Geir Magnusson Jr <geir@pobox.com> wrote:
>>>>>>>            
>>>>>>>> So while I like the annotations, and expect we can use them
>>>>>>>>               
>>>>>>> effectively,
>>>>>>>            
>>>>>>>> I have an instinctive skepticism of annotations right now
>>>>>>>>               
>>>>> because in
>>>>>        
>>>>>>>> general (in general in Java), I'm not convinced we've used
them
>>>>>>>>               
>>>>> enough
>>>>>        
>>>>>>>> to grok good design patterns.
>>>>>>>>               
>>>>>>> There's really no reason to get hung up on the annotations. TestNG
>>>>>>> works just as well with JavaDoc source comments; annotations
are
>>>>>>>             
>>> only
>>>    
>>>>>>> another means to that end. (They're probably a better one for
the
>>>>>>> future, but it's just an implementation detail.)
>>>>>>>
>>>>>>>            
>>>>>>>> Now since I still haven't read the thread fully, I'm jumping
to
>>>>>>>> conclusions, taking it to the extreme, etc etc, but my thinking
in
>>>>>>>> writing the above is that if we bury everything about our
test
>>>>>>>> 'parameter space' in annotations, some of the visible
>>>>>>>>               
>>>>> organization we
>>>>>        
>>>>>>>> have now w/ on-disk layout becomes invisible, and the readable
>>>>>>>> "summaries" of aspects of testing that we'd have in an XML
>>>>>>>>               
>>> metadata
>>>    
>>>>>>>> document (or whatever) also are hard because you need to
scan the
>>>>>>>> sources to find all instances of annotation "X".
>>>>>>>>               
>>>>>>> I'm hoping that this would be just as applicable to using JavaDoc
>>>>>>> variants, and that the problem's not with annotations per se.
>>>>>>>
>>>>>>> In either case, both are grokkable with tools -- either
>>>>>>> annotation-savy readers or a JavaDoc tag processor, and it
>>>>>>>             
>>>>> wouldn't be
>>>>>        
>>>>>>> hard to configure one of those to periodically scan the codebase
to
>>>>>>> generate reports. Furthermore, as long as the annotation X is
well
>>>>>>> defined, *you* don't have to scan it -- you leave it up to 
>>>>>>> TestNG to
>>>>>>> figure it out.
>>>>>>>
>>>>>>> Actually, there's a very valid benefit for using TestNG markers
(=
>>>>>>> annotations/JavaDoc) for grouping tests; the directory structure
is
>>>>>>>             
>>> a
>>>    
>>>>>>> tree, whereas the markers can form any slice of tests, and the
sets
>>>>>>> don't need to be strict subsets (with a tree, everything has
to 
>>>>>>> be a
>>>>>>> strict subset of its parents). That means that it's possible
to
>>>>>>>             
>>>>> define
>>>>>        
>>>>>>> a marker IO to run all the IO tests, or a marker Win32 to run
all
>>>>>>>             
>>> the
>>>    
>>>>>>> Win32 tests, and both of those will contain IO-specific Win32

>>>>>>> tests.
>>>>>>> You can't do that in a tree structure without duplicating content
>>>>>>> somewhere along the line (e.g. /win/io or /io/win). Neither of

>>>>>>> these
>>>>>>> scale well, and every time you add a new dimension, you're doubling
>>>>>>> the structure of the directory, but merely adding a new marker
with
>>>>>>> TestNG. So if you wanted to have (say) boot classpath tests vs
api
>>>>>>> tests, then you'd ahve to have /api/win/io and /boot/win/io (or
>>>>>>> various permutations as applicable).
>>>>>>>
>>>>>>> Most of the directory-based arguments seem to be along the lines
of
>>>>>>> "/api/win/io is better! No, /win/io/api is better!". Just have
an
>>>>>>> 'api', 'win', 'io' TestNG marker, and then let TestNG figure
out
>>>>>>>             
>>>>> which
>>>>>        
>>>>>>> ones to run. You can then even get specific, and only run the
>>>>>>>             
>>> Windows
>>>    
>>>>>>> IO API tests, if you really want -- but if you don't, you get
the
>>>>>>> benefit of being able to run all IO tests (both API and boot).
>>>>>>>
>>>>>>> There doesn't seem to be any benefit to having a strict tree-like
>>>>>>> structure to the tests when it's possible to have a multi-
>>>>>>>             
>>> dimensional
>>>    
>>>>>>> matrix of all possible combinations that's managed by the tool.
>>>>>>>
>>>>>>> Alex.
>>>>>>>             
>>>>       
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>     
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>   
>


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message