harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geir Magnusson Jr <g...@pobox.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 17:09:07 GMT


George Harley wrote:
> Hi Geir,
> 
> As you may recall, a while back I floated the idea and supplied some
> seed code to define all known test failing test methods in an XML file
> (an "exclusions list") that could be used by JUnit at test run time to
> skip over them while allowing the rest of the test methods in a class to
> run [1]. Obviously I thought about that when catching up with this
> thread but, more importantly, your comment about being reluctant to have
> more dependencies on JUnit also motivated me to go off and read some
> more about TestNG [2].
> 
> It was news to me that TestNG provides out-of-the-box support for
> excluding specific test methods as well as groups of methods (where the
> groups are declared in source file annotations or Javadoc comments).
> Even better, it can do this on existing JUnit test code provided that
> the necessary meta-data (annotations if compiling to a 1.5 target;
> Javadoc comments if targeting 1.4 like we currently are). There is a
> utility available in the TestNG download and also in the Eclipse support
> plug-in that helps migrate directories of existing JUnit tests to TestNG
> by adding in the basic meta-data (although for me the Eclipse version
> also tried to break the test class inheritance from
> junit.framework.TestCase which was definitely not what was required).
> 
> Perhaps ... just perhaps ... we should be looking at something like
> TestNG (or my wonderful "exclusions list" :-) ) to provide the
> granularity of test configuration that we need.
> 
> Just a thought.

How 'bout that ;)

geir

> 
> Best regards,
> George
> 
> [1] http://issues.apache.org/jira/browse/HARMONY-263
> [2] http://testng.org
> 
> 
> 
> Geir Magnusson Jr wrote:
>> Alexei Zakharov wrote:
>>  
>>> Hi,
>>> +1 for (3), but I think it will be better to define suite() method and
>>> enumerate passing tests there rather than to comment out the code.
>>>     
>>
>> I'm reluctant to see more dependencies on JUnit when we could control at
>> a level higher in the build system.
>>
>> Hard to explain, I guess, but if our exclusions are buried in .java, I
>> would think that reporting and tracking over time is going to be much
>> harder.
>>
>> geir
>>
>>  
>>> 2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
>>>    
>>>> Hello Vladimir,
>>>>
>>>> +1 to option 3) . We shall comment the failed test cases out and add
>>>> FIXME to remind us to diagnose the problems later. ;-)
>>>>
>>>> Vladimir Ivanov wrote:
>>>>      
>>>>> I see your point.
>>>>> But I feel that we can miss regression in non-tested code if we
>>>>> exclude
>>>>> TestCases.
>>>>> Now, for example we miss testing of
>>>>>         
>>>> java.lang.Class/Process/Thread/String
>>>>      
>>>>> and some other classes.
>>>>>
>>>>> While we have failing tests and don't want to pay attention to these
>>>>> failures we can:
>>>>> 1) Leave things as is – do not run TestCases with failing tests.
>>>>> 2) Split passing/failing TestCase into separate "failing TestCase" and
>>>>> "passing TestCase" and exclude "failing TestCases". When test or
>>>>> implementation is fixed we move tests from failing TestCase to passing
>>>>> TestCase.
>>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests
>>>>> instead
>>>>> of 0 for String.
>>>>> 4) Run all TestCases, then, compare test run results with the 'list of
>>>>> known
>>>>> failures' and see whether new failures appeared. This, I think, is
>>>>>         
>>>> better
>>>>      
>>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list of
>>>>>         
>>>> known
>>>>      
>>>>> failing tests and exclude list where we put crashing tests.
>>>>>
>>>>> Thanks, Vladimir
>>>>> On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>>>>>        
>>>>>> Mikhail Loenko wrote:
>>>>>>          
>>>>>>> Hi Vladimir,
>>>>>>>
>>>>>>> IMHO the tests are to verify that an update does not introduce
any
>>>>>>> regression. So there are two options: remember which exactly
>>>>>>>             
>>>> tests may
>>>>      
>>>>>> fail
>>>>>>          
>>>>>>> and remember that all tests must pass. I believe the latter one
is
>>>>>>>             
>>>>>> a bit
>>>>>>          
>>>>>>> easier and safer.
>>>>>>>             
>>>>>> +1
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>>          
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
>>>>>>>            
>>>>>>>> Hi,
>>>>>>>> Working with tests I noticed that we are excluding some tests
just
>>>>>>>> because
>>>>>>>> several tests from single TestCase fail.
>>>>>>>>
>>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest'
has 60
>>>>>>>> tests and
>>>>>>>> only 2 of them fails. But the build excludes the whole TestCase
>>>>>>>>               
>>>>>> and we
>>>>>>          
>>>>>>>> just
>>>>>>>> miss testing of java.lang.String implementation.
>>>>>>>>
>>>>>>>> Do we really need to exclude TestCases in 'ant test' target?
>>>>>>>>
>>>>>>>> My suggestion is: do not exclude any tests until it crashes
VM.
>>>>>>>> If somebody needs a list of tests that always passed a separated
>>>>>>>> target can
>>>>>>>> be added to build.
>>>>>>>>
>>>>>>>> Do you think we should add target 'test-all' to the build?
>>>>>>>>  Thanks, Vladimir
>>>>>>>>
>>>>>>>>
>>>>>>>>               
>>>>>>>             
>>>> ---------------------------------------------------------------------
>>>>      
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail:
>>>>>>>             
>>>> harmony-dev-help@incubator.apache.org
>>>>      
>>>>>>>             
>>>>>> -- 
>>>>>>
>>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>>> IBM Java technology centre, UK.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>           
>>>> -- 
>>>> Richard Liang
>>>> China Software Development Lab, IBM
>>>>       
>>>
>>>     
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>   
> 
> 
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> 
> 
> 

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message