harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ivanov" <ivavladi...@gmail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 04:18:57 GMT
I see your point.
But I feel that we can miss regression in non-tested code if we exclude
TestCases.
Now, for example we miss testing of java.lang.Class/Process/Thread/String
and some other classes.

While we have failing tests and don't want to pay attention to these
failures we can:
1) Leave things as is – do not run TestCases with failing tests.
2) Split passing/failing TestCase into separate "failing TestCase" and
"passing TestCase" and exclude "failing TestCases". When test or
implementation is fixed we move tests from failing TestCase to passing
TestCase.
3) Comment failing tests in TestCases. It is better to run 58 tests instead
of 0 for String.
4) Run all TestCases, then, compare test run results with the 'list of known
failures' and see whether new failures appeared. This, I think, is better
then 1, 2 and 3, but, overhead is that we support 2 lists - list of known
failing tests and exclude list where we put crashing tests.

 Thanks, Vladimir
On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>
> Mikhail Loenko wrote:
> > Hi Vladimir,
> >
> > IMHO the tests are to verify that an update does not introduce any
> > regression. So there are two options: remember which exactly tests may
> fail
> > and remember that all tests must pass. I believe the latter one is a bit
> > easier and safer.
>
> +1
>
> Tim
>
> > Thanks,
> > Mikhail
> >
> > 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
> >> Hi,
> >> Working with tests I noticed that we are excluding some tests just
> >> because
> >> several tests from single TestCase fail.
> >>
> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60
> >> tests and
> >> only 2 of them fails. But the build excludes the whole TestCase and we
> >> just
> >> miss testing of java.lang.String implementation.
> >>
> >> Do we really need to exclude TestCases in 'ant test' target?
> >>
> >> My suggestion is: do not exclude any tests until it crashes VM.
> >> If somebody needs a list of tests that always passed a separated
> >> target can
> >> be added to build.
> >>
> >> Do you think we should add target 'test-all' to the build?
> >>  Thanks, Vladimir
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message