harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 16:58:27 GMT
Hi Geir,

As you may recall, a while back I floated the idea and supplied some 
seed code to define all known test failing test methods in an XML file 
(an "exclusions list") that could be used by JUnit at test run time to 
skip over them while allowing the rest of the test methods in a class to 
run [1]. Obviously I thought about that when catching up with this 
thread but, more importantly, your comment about being reluctant to have 
more dependencies on JUnit also motivated me to go off and read some 
more about TestNG [2].

It was news to me that TestNG provides out-of-the-box support for 
excluding specific test methods as well as groups of methods (where the 
groups are declared in source file annotations or Javadoc comments). 
Even better, it can do this on existing JUnit test code provided that 
the necessary meta-data (annotations if compiling to a 1.5 target; 
Javadoc comments if targeting 1.4 like we currently are). There is a 
utility available in the TestNG download and also in the Eclipse support 
plug-in that helps migrate directories of existing JUnit tests to TestNG 
by adding in the basic meta-data (although for me the Eclipse version 
also tried to break the test class inheritance from 
junit.framework.TestCase which was definitely not what was required).

Perhaps ... just perhaps ... we should be looking at something like 
TestNG (or my wonderful "exclusions list" :-) ) to provide the 
granularity of test configuration that we need.

Just a thought.

Best regards,
George

[1] http://issues.apache.org/jira/browse/HARMONY-263
[2] http://testng.org



Geir Magnusson Jr wrote:
> Alexei Zakharov wrote:
>   
>> Hi,
>> +1 for (3), but I think it will be better to define suite() method and
>> enumerate passing tests there rather than to comment out the code.
>>     
>
> I'm reluctant to see more dependencies on JUnit when we could control at
> a level higher in the build system.
>
> Hard to explain, I guess, but if our exclusions are buried in .java, I
> would think that reporting and tracking over time is going to be much
> harder.
>
> geir
>
>   
>> 2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
>>     
>>> Hello Vladimir,
>>>
>>> +1 to option 3) . We shall comment the failed test cases out and add
>>> FIXME to remind us to diagnose the problems later. ;-)
>>>
>>> Vladimir Ivanov wrote:
>>>       
>>>> I see your point.
>>>> But I feel that we can miss regression in non-tested code if we exclude
>>>> TestCases.
>>>> Now, for example we miss testing of
>>>>         
>>> java.lang.Class/Process/Thread/String
>>>       
>>>> and some other classes.
>>>>
>>>> While we have failing tests and don't want to pay attention to these
>>>> failures we can:
>>>> 1) Leave things as is – do not run TestCases with failing tests.
>>>> 2) Split passing/failing TestCase into separate "failing TestCase" and
>>>> "passing TestCase" and exclude "failing TestCases". When test or
>>>> implementation is fixed we move tests from failing TestCase to passing
>>>> TestCase.
>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests
>>>> instead
>>>> of 0 for String.
>>>> 4) Run all TestCases, then, compare test run results with the 'list of
>>>> known
>>>> failures' and see whether new failures appeared. This, I think, is
>>>>         
>>> better
>>>       
>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list of
>>>>         
>>> known
>>>       
>>>> failing tests and exclude list where we put crashing tests.
>>>>
>>>> Thanks, Vladimir
>>>> On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>>>>         
>>>>> Mikhail Loenko wrote:
>>>>>           
>>>>>> Hi Vladimir,
>>>>>>
>>>>>> IMHO the tests are to verify that an update does not introduce any
>>>>>> regression. So there are two options: remember which exactly
>>>>>>             
>>> tests may
>>>       
>>>>> fail
>>>>>           
>>>>>> and remember that all tests must pass. I believe the latter one is
>>>>>>             
>>>>> a bit
>>>>>           
>>>>>> easier and safer.
>>>>>>             
>>>>> +1
>>>>>
>>>>> Tim
>>>>>
>>>>>           
>>>>>> Thanks,
>>>>>> Mikhail
>>>>>>
>>>>>> 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
>>>>>>             
>>>>>>> Hi,
>>>>>>> Working with tests I noticed that we are excluding some tests
just
>>>>>>> because
>>>>>>> several tests from single TestCase fail.
>>>>>>>
>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest' has
60
>>>>>>> tests and
>>>>>>> only 2 of them fails. But the build excludes the whole TestCase
>>>>>>>               
>>>>> and we
>>>>>           
>>>>>>> just
>>>>>>> miss testing of java.lang.String implementation.
>>>>>>>
>>>>>>> Do we really need to exclude TestCases in 'ant test' target?
>>>>>>>
>>>>>>> My suggestion is: do not exclude any tests until it crashes VM.
>>>>>>> If somebody needs a list of tests that always passed a separated
>>>>>>> target can
>>>>>>> be added to build.
>>>>>>>
>>>>>>> Do you think we should add target 'test-all' to the build?
>>>>>>>  Thanks, Vladimir
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>             
>>> ---------------------------------------------------------------------
>>>       
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail:
>>>>>>             
>>> harmony-dev-help@incubator.apache.org
>>>       
>>>>>>             
>>>>> --
>>>>>
>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>> IBM Java technology centre, UK.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>           
>>> -- 
>>> Richard Liang
>>> China Software Development Lab, IBM
>>>       
>>
>>     
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>   


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message