harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Regis <xu.re...@gmail.com>
Subject Re: API tests failing on the RI
Date Fri, 10 Oct 2008 03:39:18 GMT

Mark Hindess wrote:
> In message <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com>,
> "Alexey Petrenko" writes:
>> 2008/9/30 Mark Hindess <mark.hindess@googlemail.com>:
>>> In order to fix this bug I had to fix a number of invalid API tests.
>>> I think it would be a good idea to:
>>> 1) Run the API tests against the RI
>>> 2) Create exclude lists - with references to the relevant JIRA - for
>>> non-bug differences so the tests can be regularly run on the RI and
>>> expected to pass cleanly
>>> 3) Fix the non-non-bug (!) differences.
>> This job really needs to be done...
> I had a quick look at how much work this might but immediately hit an
> issue that I think is best discussed first.  The luni test in:
>   api/common/org/apache/harmony/luni/tests/java/io/FileTest.java
> has 52 asserts.  One (on line 2153) fails on the RI (because of a fix in
> HARMONY-3207 for which no non-bug difference jira was created AFAIK).
> Does the exclude list need to exclude the entire test - which would seem
FileTest could pass on Harmony, I always think the exclude list is a 
place to hold the tests which just only break on Harmony.
> to be a waste of potentially useful tests?  Or is there a better way
> with junit 4?  Or do we just start splitting out tests into separate
> source files - like FileNonBugDifferenceTest.java - for reference in
> exclude lists?
I agree it waste useful tests, and I don't think splitting out tests is 
the best way. FileTest just a simple example, requirements may become 
complex, we have to split out more test files and they may have 
overlaps, it make these files hard to manage and maintain. I thinks we 
need a way to group and control tests on method level. And I found[1]
[1] https://issues.apache.org/jira/browse/HARMONY-263
which extend the junit to support to exclude tests in method level in 
XML, it maybe be helpful.
> I know we've discussed this many times before (along with repeatedly
> discounting testng) but I'd like to resolve this once and for all so we
> can use the tests to their full potential.
> This is a concrete example.  How should we resolve this?
> I should stress that I have no strong opinion about testng or junit, but
> I do have a strong opinion about the need to understand the differences
> between the behaviour of our code and the RI particularly given the
> continuing absence of a TCK.  To me this means running as many tests as
> possible on the RI to confirm that the tests are valid and documenting
> (close to the code if possible) or fixing every case where our behaviour
> doesn't match the RI.
> Regards,
>  Mark.

Best Regards,

View raw message