Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of mloenko@gmail.com designates
 64.233.182.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=BqaLDhWPn+Kc0GYkKjDkwyjrzlF0Pd1uK61bUJQ/RjIHqHGpFfxV+F3SAZFMdAwCMomPg6c9JaT+H1/WKT93sh41Q9hsBgXtrm8EpgqiP7Iuyc1ZKboAl7Ifi0hD+yeh08lZlQVrlbcb0eJ0v1o6SXZB0Fnk6a/AAnVpQMRaRjo=
Message-ID: <906dd82e0710020054w58690fe8y180fbb7738f86919@mail.gmail.com>
Date: Tue, 2 Oct 2007 11:54:04 +0400
From: "Mikhail Loenko" <mloenko@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [general][M3] release candidate status
In-Reply-To: <6e47b64f0710020046g598cc2d8o1c5ba90a92fb2d3e@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <6e47b64f0709280102t5fe06a2fid47fa8b1ebb7b3f3@mail.gmail.com>
	 <6e47b64f0710020046g598cc2d8o1c5ba90a92fb2d3e@mail.gmail.com>

Do we see any M3 blockers so far?

Thanks,
Mikhail

2007/10/2, Stepan Mishura <stepan.mishura@gmail.com>:
> Hi,
>
> Currently, the next milestone candidate (r580985) is under testing.
>
> There were a number of investigation reports for r579330 and I'd like
> to update its summary to track the progress. All
>
> * Class library:
>    - Windows: 1 crash (hard to reproduce)
>    - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - HARMONY-4872)
> * Eclipse 3.2 unit tests:
>    - Windows:
>        4 failures - 1 is expected, 3 hard to reproduce (exclude list
> was updated)
>    - Linux:
>        issue with org.eclipse.jdt.core.tests.compiler.regression.TestAll suite
>        timeout still remains.
> * Eclipse 3.3 unit tests:
>    Success rate on Windows - 99.77% and Linux 99.46%.
>    This results will be taken as baseline for tracking regressions.
> * Eclipse Geronimo Application (EGA) scenario x 48h
>    - Windows:
>       failed after 8 running hours, most probably the problem
>      with scenario automation - no crashes/exceptions in the output
> * Eclipse TPTP Tests:
>       - Linux: 1 intermittent failure
> * Functional Test Suite:
>    4 regressions were found - HARMONY-4866, HARMONY-4874,
> HARMONY-4857, HARMONY-4861
>    6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1
> case), JIRA for others??
>    HARMONY-4867 - should be defined whether this is regression or not
>    Also there are many failures due to timeout or intermittent failures.
>   The suite definitely must be improved.
> * Geronimo Unit Tests
>    2 regressions on Windows/Linux - no evaluation yet
> * JDKTools tests:
>    Windows: 1 failure - reproducible only if the whole suite is run.
> (HARMONY-4868)
> * Reliability Test Suite:
>     - Windows: 13 failures - according to the evaluation report most
> of them are intermittent and additional investigation is required
>    - Linux: 14 failures - the same as for Windows
> * Stress Test Suite:
>    - Windows: 3 failures
>    - Linux: 11 failures
>       Most of test failures due to timeout or OutOfMemoryError - the
> suite must be improved. There is HARMONY-4114 with 1Mb patch for the
> suite and harness but it is too risky to apply it during code freeze -
> will be applied in M4.
> * VTS VM Test Suite
>    - Linux: 1 intermittent failure - problems in DRLVM shutdown code
> (HARMONY-????).
>
> Thanks,
> Stepan.
>
> On 9/28/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > Hi all,
> >
> > OK. Here is testing status for r579330 M3 milestone candidate.
> > It is based on testing status page[1] and investigation reports for
> > suites on dev-list.
> > It is provided to decide whether r579330 is OK to be released or we
> > should fix found issues and build next release candidate.
> >
> > Status reporter please correct me if I miss something.
> >
> > I'd like to say that this in this milestone we used BTI for the first
> > time to get 100% of testing results. Some issues in BTI were fixed but
> > there are some still to be resolved, for example instability of
> > several suites, mostly Eclipse unit tests on Linux.
> >
> >
> > The following suites passed on Linux/Windows ia32 platform: Ant
> > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> > Tomcat scenario
> >
> > The the next list of issues we have (per suite):
> > - Classlibrary unit tests:
> >    1 crash on Windows that is hard to reproduce:
> >    1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
> >    1 test failure on Linux (incorrect test HARMONY-????)
> >
> > - Eclipse unit tests 3.2
> >    Windows: 4 failures. 1 failure is expected (exclude list should be
> > updated). 3 failure are not reproducible
> >    Linux: crash of
> > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> > timeout
> >
> > - Eclipse unit tests 3.3
> >   Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> > on M2 so it is a good baseline for M3.
> >   Linux: the same problem as for 3.2. I've setup another CC Linux
> > host for the suite and see no crashes, pass rate is 99.45%
> >
> >  So I'm going to publish these reports for M3 and run 3.2 suite on
> > these lucky host to obtain results without the crash.
> >
> > - Eclipse Geronimo Application (EGA) scenario x 48h
> >   Windows: there may be and issue with the scenario automation - the
> > scenario run more then 8 hours. As I understand several iterations
> > were done successfully. And it failed because of missing window, there
> > are no crash/exceptions in the log.
> >
> >  Also the scenario passed on Linux.
> >
> > - Eclipse TPTP Tests
> >  1 failure on Linux - most probably intermittent
> >
> > - Functional Test Suite
> >   21 failures on Windows most of then failure due to timeout. 1
> > regression (HARMONY-4857)
> >   Uploading Linux failed - I'm going to check why. From the first
> > glance the results are same as for Windows.
> >
> > - Geronimo Unit Tests
> >  2 possible(I haven't seen analysis report) regressions on
> > Windows/Linux. These test passed on M2:
> >  org.apache.geronimo.tomcat.ContainerTest
> >  org.apache.geronimo.tomcat.JAASSecurityTest
> >
> > - JDKTools Tests
> >  1 failure on Windows. The test stably fails when it runs in a bunch
> > with others and may fail if run standalone :
> >
> > - Reliability Test Suite
> >  The status is a bit unclear to me. There is 1 test is incorrect
> > test. And some tests failed because of timeout but its increase didn't
> > help.
> >  Also some tests were removed from exclude list for r579330 but some
> > of them failed.
> >
> > - Stress Test Suite
> >  3 failures on Windows and 11 on Linux - I haven't seen analysis report
> >
> > - VTS VM Test Suite
> >  1 failure on Linux - the test passed on previous snapshot should be analysed.
> >
> > Any comments are welcome.
> >
> > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
> >
> > Thanks,
> > Stepan Mishura
> > Intel Enterprise Solutions Software Division
>