harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Beyer" <ndbe...@apache.org>
Subject Re: API tests failing on the RI
Date Fri, 10 Oct 2008 01:06:51 GMT
Perhaps we should have an isolated TCK-like test set, which can be run
without any special setup or build - just a JRE. Perhaps this would
create some dual-maintenance, but I agree with Mark's sentiment and
concern and think some amount of sacrifice may be necessary. A couple
of benefits to this isolation would including eliminating the complex
per-project API/impl tests and just make everything impl tests,
failing tests would never be excluded, they are either correct or not,
failures mean a Harmony bug.

-Nathan

On Thu, Oct 9, 2008 at 9:55 AM, Mark Hindess
<mark.hindess@googlemail.com> wrote:
>
> In message <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com>,
> "Alexey Petrenko" writes:
>>
>> 2008/9/30 Mark Hindess <mark.hindess@googlemail.com>:
>> >
>> > In order to fix this bug I had to fix a number of invalid API tests.
>> > I think it would be a good idea to:
>> >
>> > 1) Run the API tests against the RI
>> >
>> > 2) Create exclude lists - with references to the relevant JIRA - for
>> > non-bug differences so the tests can be regularly run on the RI and
>> > expected to pass cleanly
>> >
>> > 3) Fix the non-non-bug (!) differences.
>>
>> This job really needs to be done...
>
> I had a quick look at how much work this might but immediately hit an
> issue that I think is best discussed first.  The luni test in:
>
>  api/common/org/apache/harmony/luni/tests/java/io/FileTest.java
>
> has 52 asserts.  One (on line 2153) fails on the RI (because of a fix in
> HARMONY-3207 for which no non-bug difference jira was created AFAIK).
>
> Does the exclude list need to exclude the entire test - which would seem
> to be a waste of potentially useful tests?  Or is there a better way
> with junit 4?  Or do we just start splitting out tests into separate
> source files - like FileNonBugDifferenceTest.java - for reference in
> exclude lists?
>
> I know we've discussed this many times before (along with repeatedly
> discounting testng) but I'd like to resolve this once and for all so we
> can use the tests to their full potential.
>
> This is a concrete example.  How should we resolve this?
>
> I should stress that I have no strong opinion about testng or junit, but
> I do have a strong opinion about the need to understand the differences
> between the behaviour of our code and the RI particularly given the
> continuing absence of a TCK.  To me this means running as many tests as
> possible on the RI to confirm that the tests are valid and documenting
> (close to the code if possible) or fixing every case where our behaviour
> doesn't match the RI.
>
> Regards,
>  Mark.
>
>
>

Mime
View raw message