harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib][testing] excluding the failed tests
Date Tue, 27 Jun 2006 05:13:15 GMT
Hello Vladimir,

+1 to option 3) . We shall comment the failed test cases out and add 
FIXME to remind us to diagnose the problems later. ;-)

Vladimir Ivanov wrote:
> I see your point.
> But I feel that we can miss regression in non-tested code if we exclude
> TestCases.
> Now, for example we miss testing of java.lang.Class/Process/Thread/String
> and some other classes.
>
> While we have failing tests and don't want to pay attention to these
> failures we can:
> 1) Leave things as is – do not run TestCases with failing tests.
> 2) Split passing/failing TestCase into separate "failing TestCase" and
> "passing TestCase" and exclude "failing TestCases". When test or
> implementation is fixed we move tests from failing TestCase to passing
> TestCase.
> 3) Comment failing tests in TestCases. It is better to run 58 tests 
> instead
> of 0 for String.
> 4) Run all TestCases, then, compare test run results with the 'list of 
> known
> failures' and see whether new failures appeared. This, I think, is better
> then 1, 2 and 3, but, overhead is that we support 2 lists - list of known
> failing tests and exclude list where we put crashing tests.
>
> Thanks, Vladimir
> On 6/26/06, Tim Ellison <t.p.ellison@gmail.com> wrote:
>>
>> Mikhail Loenko wrote:
>> > Hi Vladimir,
>> >
>> > IMHO the tests are to verify that an update does not introduce any
>> > regression. So there are two options: remember which exactly tests may
>> fail
>> > and remember that all tests must pass. I believe the latter one is 
>> a bit
>> > easier and safer.
>>
>> +1
>>
>> Tim
>>
>> > Thanks,
>> > Mikhail
>> >
>> > 2006/6/26, Vladimir Ivanov <ivavladimir@gmail.com>:
>> >> Hi,
>> >> Working with tests I noticed that we are excluding some tests just
>> >> because
>> >> several tests from single TestCase fail.
>> >>
>> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60
>> >> tests and
>> >> only 2 of them fails. But the build excludes the whole TestCase 
>> and we
>> >> just
>> >> miss testing of java.lang.String implementation.
>> >>
>> >> Do we really need to exclude TestCases in 'ant test' target?
>> >>
>> >> My suggestion is: do not exclude any tests until it crashes VM.
>> >> If somebody needs a list of tests that always passed a separated
>> >> target can
>> >> be added to build.
>> >>
>> >> Do you think we should add target 'test-all' to the build?
>> >>  Thanks, Vladimir
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>> -- 
>>
>> Tim Ellison (t.p.ellison@gmail.com)
>> IBM Java technology centre, UK.
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message