harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: [general][M3] release candidate status
Date Tue, 02 Oct 2007 07:46:39 GMT
Hi,

Currently, the next milestone candidate (r580985) is under testing.

There were a number of investigation reports for r579330 and I'd like
to update its summary to track the progress. All

* Class library:
    - Windows: 1 crash (hard to reproduce)
    - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - HARMONY-4872)
* Eclipse 3.2 unit tests:
    - Windows:
        4 failures - 1 is expected, 3 hard to reproduce (exclude list
was updated)
    - Linux:
        issue with org.eclipse.jdt.core.tests.compiler.regression.TestAll suite
        timeout still remains.
* Eclipse 3.3 unit tests:
    Success rate on Windows - 99.77% and Linux 99.46%.
    This results will be taken as baseline for tracking regressions.
* Eclipse Geronimo Application (EGA) scenario x 48h
    - Windows:
       failed after 8 running hours, most probably the problem
      with scenario automation - no crashes/exceptions in the output
* Eclipse TPTP Tests:
       - Linux: 1 intermittent failure
* Functional Test Suite:
    4 regressions were found - HARMONY-4866, HARMONY-4874,
HARMONY-4857, HARMONY-4861
    6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1
case), JIRA for others??
    HARMONY-4867 - should be defined whether this is regression or not
    Also there are many failures due to timeout or intermittent failures.
   The suite definitely must be improved.
* Geronimo Unit Tests
    2 regressions on Windows/Linux - no evaluation yet
* JDKTools tests:
    Windows: 1 failure - reproducible only if the whole suite is run.
(HARMONY-4868)
* Reliability Test Suite:
     - Windows: 13 failures - according to the evaluation report most
of them are intermittent and additional investigation is required
    - Linux: 14 failures - the same as for Windows
* Stress Test Suite:
    - Windows: 3 failures
    - Linux: 11 failures
       Most of test failures due to timeout or OutOfMemoryError - the
suite must be improved. There is HARMONY-4114 with 1Mb patch for the
suite and harness but it is too risky to apply it during code freeze -
will be applied in M4.
* VTS VM Test Suite
    - Linux: 1 intermittent failure - problems in DRLVM shutdown code
(HARMONY-????).

Thanks,
Stepan.

On 9/28/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> Hi all,
>
> OK. Here is testing status for r579330 M3 milestone candidate.
> It is based on testing status page[1] and investigation reports for
> suites on dev-list.
> It is provided to decide whether r579330 is OK to be released or we
> should fix found issues and build next release candidate.
>
> Status reporter please correct me if I miss something.
>
> I'd like to say that this in this milestone we used BTI for the first
> time to get 100% of testing results. Some issues in BTI were fixed but
> there are some still to be resolved, for example instability of
> several suites, mostly Eclipse unit tests on Linux.
>
>
> The following suites passed on Linux/Windows ia32 platform: Ant
> Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> Tomcat scenario
>
> The the next list of issues we have (per suite):
> - Classlibrary unit tests:
>    1 crash on Windows that is hard to reproduce:
>    1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
>    1 test failure on Linux (incorrect test HARMONY-????)
>
> - Eclipse unit tests 3.2
>    Windows: 4 failures. 1 failure is expected (exclude list should be
> updated). 3 failure are not reproducible
>    Linux: crash of
> org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> timeout
>
> - Eclipse unit tests 3.3
>   Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> on M2 so it is a good baseline for M3.
>   Linux: the same problem as for 3.2. I've setup another CC Linux
> host for the suite and see no crashes, pass rate is 99.45%
>
>  So I'm going to publish these reports for M3 and run 3.2 suite on
> these lucky host to obtain results without the crash.
>
> - Eclipse Geronimo Application (EGA) scenario x 48h
>   Windows: there may be and issue with the scenario automation - the
> scenario run more then 8 hours. As I understand several iterations
> were done successfully. And it failed because of missing window, there
> are no crash/exceptions in the log.
>
>  Also the scenario passed on Linux.
>
> - Eclipse TPTP Tests
>  1 failure on Linux - most probably intermittent
>
> - Functional Test Suite
>   21 failures on Windows most of then failure due to timeout. 1
> regression (HARMONY-4857)
>   Uploading Linux failed - I'm going to check why. From the first
> glance the results are same as for Windows.
>
> - Geronimo Unit Tests
>  2 possible(I haven't seen analysis report) regressions on
> Windows/Linux. These test passed on M2:
>  org.apache.geronimo.tomcat.ContainerTest
>  org.apache.geronimo.tomcat.JAASSecurityTest
>
> - JDKTools Tests
>  1 failure on Windows. The test stably fails when it runs in a bunch
> with others and may fail if run standalone :
>
> - Reliability Test Suite
>  The status is a bit unclear to me. There is 1 test is incorrect
> test. And some tests failed because of timeout but its increase didn't
> help.
>  Also some tests were removed from exclude list for r579330 but some
> of them failed.
>
> - Stress Test Suite
>  3 failures on Windows and 11 on Linux - I haven't seen analysis report
>
> - VTS VM Test Suite
>  1 failure on Linux - the test passed on previous snapshot should be analysed.
>
> Any comments are welcome.
>
> [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
>
> Thanks,
> Stepan Mishura
> Intel Enterprise Solutions Software Division

Mime
View raw message