Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: neutral (athena.apache.org: 194.196.100.162 is neither permitted
 nor denied by domain of mark.hindess@googlemail.com)
Message-Id: <200810091455.m99EtSsd011644@d06av03.portsmouth.uk.ibm.com>
In-reply-to: <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com> 
References: <1367711648.1222235264363.JavaMail.jira@brutus>
 <48D9E27C.20400@gmail.com> <48E1E68E.5070102@gmail.com>
 <200809301240.m8UCe1t8010297@d12av04.megacenter.de.ibm.com>
 <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com>
Comments: In-reply-to "Alexey Petrenko" <alexey.a.petrenko@gmail.com>
   message dated "Tue, 30 Sep 2008 16:47:48 +0400."
From: Mark Hindess <mark.hindess@googlemail.com>
To: dev@harmony.apache.org
Subject: Re: API tests failing on the RI
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Date: Thu, 09 Oct 2008 15:55:28 +0100


In message <c3755b3a0809300547y3d96afb1sd7cb99511e74202@mail.gmail.com>,
"Alexey Petrenko" writes:
>
> 2008/9/30 Mark Hindess <mark.hindess@googlemail.com>:
> >
> > In order to fix this bug I had to fix a number of invalid API tests.
> > I think it would be a good idea to:
> >
> > 1) Run the API tests against the RI
> >
> > 2) Create exclude lists - with references to the relevant JIRA - for
> > non-bug differences so the tests can be regularly run on the RI and
> > expected to pass cleanly
> >
> > 3) Fix the non-non-bug (!) differences.
>
> This job really needs to be done...

I had a quick look at how much work this might but immediately hit an
issue that I think is best discussed first.  The luni test in:

  api/common/org/apache/harmony/luni/tests/java/io/FileTest.java

has 52 asserts.  One (on line 2153) fails on the RI (because of a fix in
HARMONY-3207 for which no non-bug difference jira was created AFAIK).

Does the exclude list need to exclude the entire test - which would seem
to be a waste of potentially useful tests?  Or is there a better way
with junit 4?  Or do we just start splitting out tests into separate
source files - like FileNonBugDifferenceTest.java - for reference in
exclude lists?

I know we've discussed this many times before (along with repeatedly
discounting testng) but I'd like to resolve this once and for all so we
can use the tests to their full potential.

This is a concrete example.  How should we resolve this?

I should stress that I have no strong opinion about testng or junit, but
I do have a strong opinion about the need to understand the differences
between the behaviour of our code and the RI particularly given the
continuing absence of a TCK.  To me this means running as many tests as
possible on the RI to confirm that the tests are valid and documenting
(close to the code if possible) or fixing every case where our behaviour
doesn't match the RI.

Regards,
 Mark.