harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Beyer" <nbe...@kc.rr.com>
Subject RE: [classlib][testing] excluding the failed tests
Date Thu, 29 Jun 2006 01:17:02 GMT
Two suggestions:
1. Approve the testing strategy [1] and implement/rework the modules
appropriately.
2. Fix the tests!

-Nathan

[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html


> -----Original Message-----
> From: Geir Magnusson Jr [mailto:geir@pobox.com]
> Sent: Tuesday, June 27, 2006 12:09 PM
> To: harmony-dev@incubator.apache.org
> Subject: Re: [classlib][testing] excluding the failed tests
> 
> 
> 
> George Harley wrote:
> > Hi Geir,
> >
> > As you may recall, a while back I floated the idea and supplied some
> > seed code to define all known test failing test methods in an XML file
> > (an "exclusions list") that could be used by JUnit at test run time to
> > skip over them while allowing the rest of the test methods in a class to
> > run [1]. Obviously I thought about that when catching up with this
> > thread but, more importantly, your comment about being reluctant to have
> > more dependencies on JUnit also motivated me to go off and read some
> > more about TestNG [2].
> >
> > It was news to me that TestNG provides out-of-the-box support for
> > excluding specific test methods as well as groups of methods (where the
> > groups are declared in source file annotations or Javadoc comments).
> > Even better, it can do this on existing JUnit test code provided that
> > the necessary meta-data (annotations if compiling to a 1.5 target;
> > Javadoc comments if targeting 1.4 like we currently are). There is a
> > utility available in the TestNG download and also in the Eclipse support
> > plug-in that helps migrate directories of existing JUnit tests to TestNG
> > by adding in the basic meta-data (although for me the Eclipse version
> > also tried to break the test class inheritance from
> > junit.framework.TestCase which was definitely not what was required).
> >
> > Perhaps ... just perhaps ... we should be looking at something like
> > TestNG (or my wonderful "exclusions list" :-) ) to provide the
> > granularity of test configuration that we need.
> >
> > Just a thought.
> 
> How 'bout that ;)
> 
> geir
> 
> >
> > Best regards,
> > George
> >
> > [1] http://issues.apache.org/jira/browse/HARMONY-263
> > [2] http://testng.org
> >
> >
> >
> > Geir Magnusson Jr wrote:
> >> Alexei Zakharov wrote:
> >>
> >>> Hi,
> >>> +1 for (3), but I think it will be better to define suite() method and
> >>> enumerate passing tests there rather than to comment out the code.
> >>>
> >>
> >> I'm reluctant to see more dependencies on JUnit when we could control
> at
> >> a level higher in the build system.
> >>
> >> Hard to explain, I guess, but if our exclusions are buried in .java, I
> >> would think that reporting and tracking over time is going to be much
> >> harder.
> >>
> >> geir
> >>
> >>
> >>> 2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
> >>>
> >>>> Hello Vladimir,
> >>>>
> >>>> +1 to option 3) . We shall comment the failed test cases out and add
> >>>> FIXME to remind us to diagnose the problems later. ;-)
> >>>>
> >>>> Vladimir Ivanov wrote:
> >>>>
> >>>>> I see your point.
> >>>>> But I feel that we can miss regression in non-tested code if we
> >>>>> exclude
> >>>>> TestCases.
> >>>>> Now, for example we miss testing of
> >>>>>
> >>>> java.lang.Class/Process/Thread/String
> >>>>
> >>>>> and some other classes.
> >>>>>
> >>>>> While we have failing tests and don't want to pay attention to these
> >>>>> failures we can:
> >>>>> 1) Leave things as is - do not run TestCases with failing tests.
> >>>>> 2) Split passing/failing TestCase into separate "failing TestCase"
> and
> >>>>> "passing TestCase" and exclude "failing TestCases". When test or
> >>>>> implementation is fixed we move tests from failing TestCase to
> passing
> >>>>> TestCase.
> >>>>> 3) Comment failing tests in TestCases. It is better to run 58 tests
> >>>>> instead
> >>>>> of 0 for String.
> >>>>> 4) Run all TestCases, then, compare test run results with the 'list
> of
> >>>>> known
> >>>>> failures' and see whether new failures appeared. This, I think,
is
> >>>>>
> >>>> better
> >>>>
> >>>>> then 1, 2 and 3, but, overhead is that we support 2 lists - list
of
> >>>>>
> >>>> known
> >>>>
> >>>>> failing tests and exclude list where we put crashing tests.
> >>>>>
> >>>>> Thanks, Vladimir
> >>>>> On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>> Hi Vladimir,
> >>>>>>>
> >>>>>>> IMHO the tests are to verify that an update does not introduce
any
> >>>>>>> regression. So there are two options: remember which exactly
> >>>>>>>
> >>>> tests may
> >>>>
> >>>>>> fail
> >>>>>>
> >>>>>>> and remember that all tests must pass. I believe the latter
one is
> >>>>>>>
> >>>>>> a bit
> >>>>>>
> >>>>>>> easier and safer.
> >>>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> Tim
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>> 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>> Working with tests I noticed that we are excluding some
tests
> just
> >>>>>>>> because
> >>>>>>>> several tests from single TestCase fail.
> >>>>>>>>
> >>>>>>>> For example, the TestCase 'tests.api.java.lang.StringTest'
has 60
> >>>>>>>> tests and
> >>>>>>>> only 2 of them fails. But the build excludes the whole
TestCase
> >>>>>>>>
> >>>>>> and we
> >>>>>>
> >>>>>>>> just
> >>>>>>>> miss testing of java.lang.String implementation.
> >>>>>>>>
> >>>>>>>> Do we really need to exclude TestCases in 'ant test'
target?
> >>>>>>>>
> >>>>>>>> My suggestion is: do not exclude any tests until it
crashes VM.
> >>>>>>>> If somebody needs a list of tests that always passed
a separated
> >>>>>>>> target can
> >>>>>>>> be added to build.
> >>>>>>>>
> >>>>>>>> Do you think we should add target 'test-all' to the
build?
> >>>>>>>>  Thanks, Vladimir
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> Tim Ellison (t.p.ellison@gmail.com)
> >>>>>> IBM Java technology centre, UK.
> >>>>>>
> >>>> --
> >>>> Richard Liang
> >>>> China Software Development Lab, IBM



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message