harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ellison <t.p.elli...@gmail.com>
Subject Re: [general][M3] release candidate status
Date Fri, 28 Sep 2007 09:23:00 GMT
This is a wonderful summary, thanks Stepan.

Comments in-lined below.

Stepan Mishura wrote:
> Hi all,
> 
> OK. Here is testing status for r579330 M3 milestone candidate.
> It is based on testing status page[1] and investigation reports for
> suites on dev-list.
> It is provided to decide whether r579330 is OK to be released or we
> should fix found issues and build next release candidate.
> 
> Status reporter please correct me if I miss something.
> 
> I'd like to say that this in this milestone we used BTI for the first
> time to get 100% of testing results. Some issues in BTI were fixed but
> there are some still to be resolved, for example instability of
> several suites, mostly Eclipse unit tests on Linux.

Yes, this is getting a lot more testing, which is great to see.

> The following suites passed on Linux/Windows ia32 platform: Ant
> Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> Tomcat scenario

Cool.

> The the next list of issues we have (per suite):
> - Classlibrary unit tests:
>     1 crash on Windows that is hard to reproduce:
>     1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
>     1 test failure on Linux (incorrect test HARMONY-????)

I looked into that one failing test in java.net and I maintain that the
test is prone to failure because it is using the deprecated
Support_PortManager.  I created HARMONY-4872 to track it, but believe it
is not a blocking issue for M3.

> - Eclipse unit tests 3.2
>     Windows: 4 failures. 1 failure is expected (exclude list should be
> updated). 3 failure are not reproducible
>     Linux: crash of
> org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> timeout

It crashes rather than fails when timing out?  Is there a JIRA for this
one too that you know about?

> - Eclipse unit tests 3.3
>    Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> on M2 so it is a good baseline for M3.
>    Linux: the same problem as for 3.2. I've setup another CC Linux
> host for the suite and see no crashes, pass rate is 99.45%

Looks good enough for where we are today.

>   So I'm going to publish these reports for M3 and run 3.2 suite on
> these lucky host to obtain results without the crash.
> 
> - Eclipse Geronimo Application (EGA) scenario x 48h
>    Windows: there may be and issue with the scenario automation - the
> scenario run more then 8 hours. As I understand several iterations
> were done successfully. And it failed because of missing window, there
> are no crash/exceptions in the log.
> 
>   Also the scenario passed on Linux.

That is looking good too then.

> - Eclipse TPTP Tests
>   1 failure on Linux - most probably intermittent
> 
> - Functional Test Suite
>    21 failures on Windows most of then failure due to timeout. 1
> regression (HARMONY-4857)
>    Uploading Linux failed - I'm going to check why. From the first
> glance the results are same as for Windows.

Would be good to figure out what is happening with HARMONY-4857.  How
about we mark this as an M3 blocker until we understand it better?

> - Geronimo Unit Tests
>   2 possible(I haven't seen analysis report) regressions on
> Windows/Linux. These test passed on M2:
>   org.apache.geronimo.tomcat.ContainerTest
>   org.apache.geronimo.tomcat.JAASSecurityTest

JIRA issues for these too?

> - JDKTools Tests
>   1 failure on Windows. The test stably fails when it runs in a bunch
> with others and may fail if run standalone :
> 
> - Reliability Test Suite
>   The status is a bit unclear to me. There is 1 test is incorrect
> test. And some tests failed because of timeout but its increase didn't
> help.
>   Also some tests were removed from exclude list for r579330 but some
> of them failed.
> 
> - Stress Test Suite
>   3 failures on Windows and 11 on Linux - I haven't seen analysis report

Is anybody familiar enough with the history of the reliability and
stress test results to offer an opinion here?

> - VTS VM Test Suite
>   1 failure on Linux - the test passed on previous snapshot should be analysed.
> 
> Any comments are welcome.

Good job! I think that although there are many failures reported, there
are many more tests and testing scenarios being exercised.  We should
chase down the critical bugs for M3, but be prepared to triage the
others into the M4 task list.

Regards,
Tim

> [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
> 
> Thanks,
> Stepan Mishura
> Intel Enterprise Solutions Software Division
> 

Mime
View raw message