harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Harley <george.c.har...@googlemail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Thu, 29 Jun 2006 08:37:54 GMT
Nathan Beyer wrote:
> Two suggestions:
> 1. Approve the testing strategy [1] and implement/rework the modules
> appropriately.
> 2. Fix the tests!
>
> -Nathan
>
> [1]
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
>   

Hi Nathan,

What are your thoughts on running or not running test cases containing 
problematic test methods while those methods are being investigated and 
fixed up ?


Best regards,
George


>
>   
>> -----Original Message-----
>> From: Geir Magnusson Jr [mailto:geir@pobox.com]
>> Sent: Tuesday, June 27, 2006 12:09 PM
>> To: harmony-dev@incubator.apache.org
>> Subject: Re: [classlib][testing] excluding the failed tests
>>
>>
>>
>> George Harley wrote:
>>     
>>> Hi Geir,
>>>
>>> As you may recall, a while back I floated the idea and supplied some
>>> seed code to define all known test failing test methods in an XML file
>>> (an "exclusions list") that could be used by JUnit at test run time to
>>> skip over them while allowing the rest of the test methods in a class to
>>> run [1]. Obviously I thought about that when catching up with this
>>> thread but, more importantly, your comment about being reluctant to have
>>> more dependencies on JUnit also motivated me to go off and read some
>>> more about TestNG [2].
>>>
>>> It was news to me that TestNG provides out-of-the-box support for
>>> excluding specific test methods as well as groups of methods (where the
>>> groups are declared in source file annotations or Javadoc comments).
>>> Even better, it can do this on existing JUnit test code provided that
>>> the necessary meta-data (annotations if compiling to a 1.5 target;
>>> Javadoc comments if targeting 1.4 like we currently are). There is a
>>> utility available in the TestNG download and also in the Eclipse support
>>> plug-in that helps migrate directories of existing JUnit tests to TestNG
>>> by adding in the basic meta-data (although for me the Eclipse version
>>> also tried to break the test class inheritance from
>>> junit.framework.TestCase which was definitely not what was required).
>>>
>>> Perhaps ... just perhaps ... we should be looking at something like
>>> TestNG (or my wonderful "exclusions list" :-) ) to provide the
>>> granularity of test configuration that we need.
>>>
>>> Just a thought.
>>>       
>> How 'bout that ;)
>>
>> geir
>>
>>     
>>> Best regards,
>>> George
>>>
>>> [1] http://issues.apache.org/jira/browse/HARMONY-263
>>> [2] http://testng.org
>>>
>>>
>>>
>>> Geir Magnusson Jr wrote:
>>>       
>>>> Alexei Zakharov wrote:
>>>>
>>>>         
>>>>> Hi,
>>>>> +1 for (3), but I think it will be better to define suite() method and
>>>>> enumerate passing tests there rather than to comment out the code.
>>>>>
>>>>>           
>>>> I'm reluctant to see more dependencies on JUnit when we could control
>>>>         
>> at
>>     
>>>> a level higher in the build system.
>>>>
>>>> Hard to explain, I guess, but if our exclusions are buried in .java, I
>>>> would think that reporting and tracking over time is going to be much
>>>> harder.
>>>>
>>>> geir
>>>>
>>>>
>>>>         
>>>>> 2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
>>>>>
>>>>>           
>>>>>> Hello Vladimir,
>>>>>>
>>>>>> +1 to option 3) . We shall comment the failed test cases out and
add
>>>>>> FIXME to remind us to diagnose the problems later. ;-)
>>>>>>
>>>>>> Vladimir Ivanov wrote:
>>>>>>
>>>>>>             
>>>>>>> I see your point.
>>>>>>> But I feel that we can miss regression in non-tested code if
we
>>>>>>> exclude
>>>>>>> TestCases.
>>>>>>> Now, for example we miss testing of
>>>>>>>
>>>>>>>               
>>>>>> java.lang.Class/Process/Thread/String
>>>>>>
>>>>>>             
>>>>>>> and some other classes.
>>>>>>>
>>>>>>> While we have failing tests and don't want to pay attention to
these
>>>>>>> failures we can:
>>>>>>> 1) Leave things as is - do not run TestCases with failing tests.
>>>>>>> 2) Split passing/failing TestCase into separate "failing TestCase"
>>>>>>>               
>> and
>>     
>>>>>>> "passing TestCase" and exclude "failing TestCases". When test
or
>>>>>>> implementation is fixed we move tests from failing TestCase to
>>>>>>>               
>> passing
>>     
>>>>>>> TestCase.
>>>>>>> 3) Comment failing tests in TestCases. It is better to run 58
tests
>>>>>>> instead
>>>>>>> of 0 for String.
>>>>>>> 4) Run all TestCases, then, compare test run results with the
'list
>>>>>>>               
>> of
>>     
>>>>>>> known
>>>>>>> failures' and see whether new failures appeared. This, I think,
is
>>>>>>>
>>>>>>>               
>>>>>> better
>>>>>>
>>>>>>             
>>>>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list
of
>>>>>>>
>>>>>>>               
>>>>>> known
>>>>>>
>>>>>>             
>>>>>>> failing tests and exclude list where we put crashing tests.
>>>>>>>
>>>>>>> Thanks, Vladimir
>>>>>>> On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Hi Vladimir,
>>>>>>>>>
>>>>>>>>> IMHO the tests are to verify that an update does not
introduce any
>>>>>>>>> regression. So there are two options: remember which
exactly
>>>>>>>>>
>>>>>>>>>                   
>>>>>> tests may
>>>>>>
>>>>>>             
>>>>>>>> fail
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> and remember that all tests must pass. I believe the
latter one is
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> a bit
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> easier and safer.
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> +1
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>> 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>> Hi,
>>>>>>>>>> Working with tests I noticed that we are excluding
some tests
>>>>>>>>>>                     
>> just
>>     
>>>>>>>>>> because
>>>>>>>>>> several tests from single TestCase fail.
>>>>>>>>>>
>>>>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest'
has 60
>>>>>>>>>> tests and
>>>>>>>>>> only 2 of them fails. But the build excludes the
whole TestCase
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>> and we
>>>>>>>>
>>>>>>>>                 
>>>>>>>>>> just
>>>>>>>>>> miss testing of java.lang.String implementation.
>>>>>>>>>>
>>>>>>>>>> Do we really need to exclude TestCases in 'ant test'
target?
>>>>>>>>>>
>>>>>>>>>> My suggestion is: do not exclude any tests until
it crashes VM.
>>>>>>>>>> If somebody needs a list of tests that always passed
a separated
>>>>>>>>>> target can
>>>>>>>>>> be added to build.
>>>>>>>>>>
>>>>>>>>>> Do you think we should add target 'test-all' to the
build?
>>>>>>>>>>  Thanks, Vladimir
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>>>>> IBM Java technology centre, UK.
>>>>>>>>
>>>>>>>>                 
>>>>>> --
>>>>>> Richard Liang
>>>>>> China Software Development Lab, IBM
>>>>>>             
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>   


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message