Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Subject: RE: [general][M3] release candidate status
Date: Tue, 2 Oct 2007 19:39:23 +0400
Message-ID: 
 <3D8E84095C6A524A985B787423094E40015E96FC@mssmsx411.ccr.corp.intel.com>
In-Reply-To: <af4b1ec30710020551m67e8b6a4x8dc1d7779ebd3b@mail.gmail.com>
Thread-Topic: [general][M3] release candidate status
Thread-Index: AcgE80voUG2LrvB3Thm8uHPZQdYG7gAFAckQ
From: "Udovichenko, Nellya" <nellya.udovichenko@intel.com>
To: <dev@harmony.apache.org>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

Some about GUT test failures. Please sorry for delay, I was on vacation.

I've reproduced the bugs. These 2 failures are very similar and I think=20
it is the same problem. May be the bug is in
java.net.HttpURLConnection.getInputStream() method and it's the
regression after the patch HARMONY-617 at July. Now I try to write the
reproducible test.

It seems the failures aren't critical for M3 release but it would=20
be better to exclude the tests for further investigation. What do you
think?

Regards,
Nellya.


-----Original Message-----
From: Pavlenko, Andrey A [mailto:andrey.a.pavlenko@googlemail.com]=20
Sent: Tuesday, October 02, 2007 4:52 PM
To: dev@harmony.apache.org
Subject: Re: [general][M3] release candidate status

Stepan, I agree with you that the functional test suite have to be
improved.
I'm going to take care about it.

At the first step I propose to exclude some "heavy" tests intermittently
failing because of timeout until they are redesigned.  I've  attached
the
patch for exclude list to HARMONY-4888.
Also I've already fixed several tests:
HARMONY-4859
HARMONY-4864
HARMONY-4889 - there no fix for the tests mentioned in this issue, but I
propose to exclude them until they are fixed. The patch for the exclude
list
is attached.

Could someone apply the above patches, please?

On 10/2/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
>
> Hi,
>
> Currently, the next milestone candidate (r580985) is under testing.
>
> There were a number of investigation reports for r579330 and I'd like
> to update its summary to track the progress. All
>
> * Class library:
>     - Windows: 1 crash (hard to reproduce)
>     - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test -
> HARMONY-4872)
> * Eclipse 3.2 unit tests:
>     - Windows:
>         4 failures - 1 is expected, 3 hard to reproduce (exclude list
> was updated)
>     - Linux:
>         issue with
org.eclipse.jdt.core.tests.compiler.regression.TestAllsuite
>         timeout still remains.
> * Eclipse 3.3 unit tests:
>     Success rate on Windows - 99.77% and Linux 99.46%.
>     This results will be taken as baseline for tracking regressions.
> * Eclipse Geronimo Application (EGA) scenario x 48h
>     - Windows:
>        failed after 8 running hours, most probably the problem
>       with scenario automation - no crashes/exceptions in the output
> * Eclipse TPTP Tests:
>        - Linux: 1 intermittent failure
> * Functional Test Suite:
>     4 regressions were found - HARMONY-4866, HARMONY-4874,
> HARMONY-4857, HARMONY-4861
>     6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1
> case), JIRA for others??
>     HARMONY-4867 - should be defined whether this is regression or not
>     Also there are many failures due to timeout or intermittent
failures.
>    The suite definitely must be improved.
> * Geronimo Unit Tests
>     2 regressions on Windows/Linux - no evaluation yet
> * JDKTools tests:
>     Windows: 1 failure - reproducible only if the whole suite is run.
> (HARMONY-4868)
> * Reliability Test Suite:
>      - Windows: 13 failures - according to the evaluation report most
> of them are intermittent and additional investigation is required
>     - Linux: 14 failures - the same as for Windows
> * Stress Test Suite:
>     - Windows: 3 failures
>     - Linux: 11 failures
>        Most of test failures due to timeout or OutOfMemoryError - the
> suite must be improved. There is HARMONY-4114 with 1Mb patch for the
> suite and harness but it is too risky to apply it during code freeze -
> will be applied in M4.
> * VTS VM Test Suite
>     - Linux: 1 intermittent failure - problems in DRLVM shutdown code
> (HARMONY-????).
>
> Thanks,
> Stepan.
>
> On 9/28/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > Hi all,
> >
> > OK. Here is testing status for r579330 M3 milestone candidate.
> > It is based on testing status page[1] and investigation reports for
> > suites on dev-list.
> > It is provided to decide whether r579330 is OK to be released or we
> > should fix found issues and build next release candidate.
> >
> > Status reporter please correct me if I miss something.
> >
> > I'd like to say that this in this milestone we used BTI for the
first
> > time to get 100% of testing results. Some issues in BTI were fixed
but
> > there are some still to be resolved, for example instability of
> > several suites, mostly Eclipse unit tests on Linux.
> >
> >
> > The following suites passed on Linux/Windows ia32 platform: Ant
> > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> > Tomcat scenario
> >
> > The the next list of issues we have (per suite):
> > - Classlibrary unit tests:
> >    1 crash on Windows that is hard to reproduce:
> >    1 crash on Linux that is reproducible and it is bug in DRLVM
> (HARMONY-????):
> >    1 test failure on Linux (incorrect test HARMONY-????)
> >
> > - Eclipse unit tests 3.2
> >    Windows: 4 failures. 1 failure is expected (exclude list should
be
> > updated). 3 failure are not reproducible
> >    Linux: crash of
> > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes
due
> > timeout
> >
> > - Eclipse unit tests 3.3
> >   Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> > on M2 so it is a good baseline for M3.
> >   Linux: the same problem as for 3.2. I've setup another CC Linux
> > host for the suite and see no crashes, pass rate is 99.45%
> >
> >  So I'm going to publish these reports for M3 and run 3.2 suite on
> > these lucky host to obtain results without the crash.
> >
> > - Eclipse Geronimo Application (EGA) scenario x 48h
> >   Windows: there may be and issue with the scenario automation - the
> > scenario run more then 8 hours. As I understand several iterations
> > were done successfully. And it failed because of missing window,
there
> > are no crash/exceptions in the log.
> >
> >  Also the scenario passed on Linux.
> >
> > - Eclipse TPTP Tests
> >  1 failure on Linux - most probably intermittent
> >
> > - Functional Test Suite
> >   21 failures on Windows most of then failure due to timeout. 1
> > regression (HARMONY-4857)
> >   Uploading Linux failed - I'm going to check why. From the first
> > glance the results are same as for Windows.
> >
> > - Geronimo Unit Tests
> >  2 possible(I haven't seen analysis report) regressions on
> > Windows/Linux. These test passed on M2:
> >  org.apache.geronimo.tomcat.ContainerTest
> >  org.apache.geronimo.tomcat.JAASSecurityTest
> >
> > - JDKTools Tests
> >  1 failure on Windows. The test stably fails when it runs in a bunch
> > with others and may fail if run standalone :
> >
> > - Reliability Test Suite
> >  The status is a bit unclear to me. There is 1 test is incorrect
> > test. And some tests failed because of timeout but its increase
didn't
> > help.
> >  Also some tests were removed from exclude list for r579330 but some
> > of them failed.
> >
> > - Stress Test Suite
> >  3 failures on Windows and 11 on Linux - I haven't seen analysis
report
> >
> > - VTS VM Test Suite
> >  1 failure on Linux - the test passed on previous snapshot should be
> analysed.
> >
> > Any comments are welcome.
> >
> > [1]
>
http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.
html
> >
> > Thanks,
> > Stepan Mishura
> > Intel Enterprise Solutions Software Division
>
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: 125252, Moscow, Russian Federation,=20
Chapayevsky Per, 14.

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.