harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sian January" <sianjanu...@googlemail.com>
Subject Re: API tests failing on the RI
Date Fri, 10 Oct 2008 09:44:38 GMT
I think it would be good to have a way of running

1. All API tests
2. All API tests except non-bug differences (TCK-style)

on any VM.

So where one method tests both common behaviour and non-bug
differences it should be split in two so that we don't lose the tests
for common behaviour when running the TCK-style suite.  I don't really
have a strong opinion about whether we then identify non-bug
difference test cases by annotations or by moving to a different class
or otherwise, but HARMONY-263 looks like a huge amount of XML and a
lot of work to implement for the whole of Harmony compared to doing
something like having one "NonBugDifferencesTest.java" class per
module.

I also think there's another advantage in clearly stating which tests
cases are for non-bug differences, which is that if someone makes a
change that causes one of these tests to fail it's easy to find the
corresponding JIRA or mailing list discussion, rather than just assume
the test is wrong  (and vice-versa we may currently have some tests
that are wrong that we're assuming are non-bug differences).


2008/10/10 Regis <xu.regis@gmail.com>:
>
> Mark Hindess wrote:
>>
>> In message <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com>,
>> "Alexey Petrenko" writes:
>>>
>>> 2008/9/30 Mark Hindess <mark.hindess@googlemail.com>:
>>>>
>>>> In order to fix this bug I had to fix a number of invalid API tests.
>>>> I think it would be a good idea to:
>>>>
>>>> 1) Run the API tests against the RI
>>>>
>>>> 2) Create exclude lists - with references to the relevant JIRA - for
>>>> non-bug differences so the tests can be regularly run on the RI and
>>>> expected to pass cleanly
>>>>
>>>> 3) Fix the non-non-bug (!) differences.
>>>
>>> This job really needs to be done...
>>
>> I had a quick look at how much work this might but immediately hit an
>> issue that I think is best discussed first.  The luni test in:
>>
>>  api/common/org/apache/harmony/luni/tests/java/io/FileTest.java
>>
>> has 52 asserts.  One (on line 2153) fails on the RI (because of a fix in
>> HARMONY-3207 for which no non-bug difference jira was created AFAIK).
>>
>> Does the exclude list need to exclude the entire test - which would seem
>
> FileTest could pass on Harmony, I always think the exclude list is a place
> to hold the tests which just only break on Harmony.
>>
>> to be a waste of potentially useful tests?  Or is there a better way
>> with junit 4?  Or do we just start splitting out tests into separate
>> source files - like FileNonBugDifferenceTest.java - for reference in
>> exclude lists?
>
> I agree it waste useful tests, and I don't think splitting out tests is the
> best way. FileTest just a simple example, requirements may become complex,
> we have to split out more test files and they may have overlaps, it make
> these files hard to manage and maintain. I thinks we need a way to group and
> control tests on method level. And I found[1]
> [1] https://issues.apache.org/jira/browse/HARMONY-263
> which extend the junit to support to exclude tests in method level in XML,
> it maybe be helpful.
>>
>> I know we've discussed this many times before (along with repeatedly
>> discounting testng) but I'd like to resolve this once and for all so we
>> can use the tests to their full potential.
>>
>> This is a concrete example.  How should we resolve this?
>>
>> I should stress that I have no strong opinion about testng or junit, but
>> I do have a strong opinion about the need to understand the differences
>> between the behaviour of our code and the RI particularly given the
>> continuing absence of a TCK.  To me this means running as many tests as
>> possible on the RI to confirm that the tests are valid and documenting
>> (close to the code if possible) or fixing every case where our behaviour
>> doesn't match the RI.
>>
>> Regards,
>>  Mark.
>>
>>
>>
>
> Best Regards,
> Regis.
>



-- 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Mime
View raw message