harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ellison <t.p.elli...@gmail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 09:16:38 GMT
There was a submission that enabled finer control of failing tests (even
by platform etc.)

I may be wrong but commenting out tests usually means that they never
get fixed; even putting them into exclude clauses in the ant script is
too hidden for me -- I prefer to see the exclusions and failures made
available very clearly.

Regards,
Tim

Alexei Zakharov wrote:
> Hi,
> +1 for (3), but I think it will be better to define suite() method and
> enumerate passing tests there rather than to comment out the code.
> 
> 2006/6/27, Richard Liang <richard.liangyx@gmail.com>:
>> Hello Vladimir,
>>
>> +1 to option 3) . We shall comment the failed test cases out and add
>> FIXME to remind us to diagnose the problems later. ;-)
>>
>> Vladimir Ivanov wrote:
>> > I see your point.
>> > But I feel that we can miss regression in non-tested code if we exclude
>> > TestCases.
>> > Now, for example we miss testing of
>> java.lang.Class/Process/Thread/String
>> > and some other classes.
>> >
>> > While we have failing tests and don't want to pay attention to these
>> > failures we can:
>> > 1) Leave things as is – do not run TestCases with failing tests.
>> > 2) Split passing/failing TestCase into separate "failing TestCase" and
>> > "passing TestCase" and exclude "failing TestCases". When test or
>> > implementation is fixed we move tests from failing TestCase to passing
>> > TestCase.
>> > 3) Comment failing tests in TestCases. It is better to run 58 tests
>> > instead
>> > of 0 for String.
>> > 4) Run all TestCases, then, compare test run results with the 'list of
>> > known
>> > failures' and see whether new failures appeared. This, I think, is
>> better
>> > then 1, 2 and 3, but, overhead is that we support 2 lists - list of
>> known
>> > failing tests and exclude list where we put crashing tests.
>> >
>> > Thanks, Vladimir
>> > On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>> >>
>> >> Mikhail Loenko wrote:
>> >> > Hi Vladimir,
>> >> >
>> >> > IMHO the tests are to verify that an update does not introduce any
>> >> > regression. So there are two options: remember which exactly
>> tests may
>> >> fail
>> >> > and remember that all tests must pass. I believe the latter one is
>> >> a bit
>> >> > easier and safer.
>> >>
>> >> +1
>> >>
>> >> Tim
>> >>
>> >> > Thanks,
>> >> > Mikhail
>> >> >
>> >> > 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
>> >> >> Hi,
>> >> >> Working with tests I noticed that we are excluding some tests just
>> >> >> because
>> >> >> several tests from single TestCase fail.
>> >> >>
>> >> >> For example, the TestCase 'tests.api.java.lang.StringTest' has
60
>> >> >> tests and
>> >> >> only 2 of them fails. But the build excludes the whole TestCase
>> >> and we
>> >> >> just
>> >> >> miss testing of java.lang.String implementation.
>> >> >>
>> >> >> Do we really need to exclude TestCases in 'ant test' target?
>> >> >>
>> >> >> My suggestion is: do not exclude any tests until it crashes VM.
>> >> >> If somebody needs a list of tests that always passed a separated
>> >> >> target can
>> >> >> be added to build.
>> >> >>
>> >> >> Do you think we should add target 'test-all' to the build?
>> >> >>  Thanks, Vladimir
>> >> >>
>> >> >>
>> >> >
>> >> >
>> ---------------------------------------------------------------------
>> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> >> > For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>> >> >
>> >> >
>> >>
>> >> --
>> >>
>> >> Tim Ellison (t.p.ellison@gmail.com)
>> >> IBM Java technology centre, UK.
>> >>
>> >> ---------------------------------------------------------------------
>> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >>
>> >>
>> >
>>
>> -- 
>> Richard Liang
>> China Software Development Lab, IBM
> 
> 
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message