harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 08:11:27 GMT
Hi,
+1 for (3), but I think it will be better to define suite() method and
enumerate passing tests there rather than to comment out the code.

2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
> Hello Vladimir,
>
> +1 to option 3) . We shall comment the failed test cases out and add
> FIXME to remind us to diagnose the problems later. ;-)
>
> Vladimir Ivanov wrote:
> > I see your point.
> > But I feel that we can miss regression in non-tested code if we exclude
> > TestCases.
> > Now, for example we miss testing of java.lang.Class/Process/Thread/String
> > and some other classes.
> >
> > While we have failing tests and don't want to pay attention to these
> > failures we can:
> > 1) Leave things as is – do not run TestCases with failing tests.
> > 2) Split passing/failing TestCase into separate "failing TestCase" and
> > "passing TestCase" and exclude "failing TestCases". When test or
> > implementation is fixed we move tests from failing TestCase to passing
> > TestCase.
> > 3) Comment failing tests in TestCases. It is better to run 58 tests
> > instead
> > of 0 for String.
> > 4) Run all TestCases, then, compare test run results with the 'list of
> > known
> > failures' and see whether new failures appeared. This, I think, is better
> > then 1, 2 and 3, but, overhead is that we support 2 lists - list of known
> > failing tests and exclude list where we put crashing tests.
> >
> > Thanks, Vladimir
> > On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
> >>
> >> Mikhail Loenko wrote:
> >> > Hi Vladimir,
> >> >
> >> > IMHO the tests are to verify that an update does not introduce any
> >> > regression. So there are two options: remember which exactly tests may
> >> fail
> >> > and remember that all tests must pass. I believe the latter one is
> >> a bit
> >> > easier and safer.
> >>
> >> +1
> >>
> >> Tim
> >>
> >> > Thanks,
> >> > Mikhail
> >> >
> >> > 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
> >> >> Hi,
> >> >> Working with tests I noticed that we are excluding some tests just
> >> >> because
> >> >> several tests from single TestCase fail.
> >> >>
> >> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60
> >> >> tests and
> >> >> only 2 of them fails. But the build excludes the whole TestCase
> >> and we
> >> >> just
> >> >> miss testing of java.lang.String implementation.
> >> >>
> >> >> Do we really need to exclude TestCases in 'ant test' target?
> >> >>
> >> >> My suggestion is: do not exclude any tests until it crashes VM.
> >> >> If somebody needs a list of tests that always passed a separated
> >> >> target can
> >> >> be added to build.
> >> >>
> >> >> Do you think we should add target 'test-all' to the build?
> >> >>  Thanks, Vladimir
> >> >>
> >> >>
> >> >
> >> > ---------------------------------------------------------------------
> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >> >
> >> >
> >>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM



-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message