harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Yakushev" <andrey.yakus...@gmail.com>
Subject Re: [general][M3] release candidate status
Date Fri, 28 Sep 2007 15:15:37 GMT
> > - Reliability Test Suite
> >   The status is a bit unclear to me. There is 1 test is incorrect
> > test. And some tests failed because of timeout but its increase didn't
> > help.

Really it helped for 5 tests of 9, but 9 different tests failed.
Almost all reliability tests failures are intermittent. And my local
run for r579492 showed much lower failure ratio: 6 valid fails on
Windows and 3 on Linux x86. So cruise control hosts could catch
additional race conditions or similar problems somehow.

> >   Also some tests were removed from exclude list for r579330 but some
> > of them failed.

Also strange, they never failed on my test runs. Need additional investigation.

>
> Is anybody familiar enough with the history of the reliability and
> stress test results to offer an opinion here?

I watched reliability results, and they improved from 25-30 usual
fails in August to 10-15 on the latest builds. More precisely:

Date Windows-x86 Linux-x86 failures
Sept/25 13 14
Sept/22 9 13
Sept/20 ?? 10
Sept/18 15 15
Sept/16 18 15
Sept/14 15 18
Sept/11 10 9
Sept/7 ?? 12


Thanks,
Andrey


On 9/28/07, Tim Ellison <t.p.ellison@gmail.com> wrote:
> This is a wonderful summary, thanks Stepan.
>
> Comments in-lined below.
>
> Stepan Mishura wrote:
> > Hi all,
> >
> > OK. Here is testing status for r579330 M3 milestone candidate.
> > It is based on testing status page[1] and investigation reports for
> > suites on dev-list.
> > It is provided to decide whether r579330 is OK to be released or we
> > should fix found issues and build next release candidate.
> >
> > Status reporter please correct me if I miss something.
> >
> > I'd like to say that this in this milestone we used BTI for the first
> > time to get 100% of testing results. Some issues in BTI were fixed but
> > there are some still to be resolved, for example instability of
> > several suites, mostly Eclipse unit tests on Linux.
>
> Yes, this is getting a lot more testing, which is great to see.
>
> > The following suites passed on Linux/Windows ia32 platform: Ant
> > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> > Tomcat scenario
>
> Cool.
>
> > The the next list of issues we have (per suite):
> > - Classlibrary unit tests:
> >     1 crash on Windows that is hard to reproduce:
> >     1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
> >     1 test failure on Linux (incorrect test HARMONY-????)
>
> I looked into that one failing test in java.net and I maintain that the
> test is prone to failure because it is using the deprecated
> Support_PortManager.  I created HARMONY-4872 to track it, but believe it
> is not a blocking issue for M3.
>
> > - Eclipse unit tests 3.2
> >     Windows: 4 failures. 1 failure is expected (exclude list should be
> > updated). 3 failure are not reproducible
> >     Linux: crash of
> > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> > timeout
>
> It crashes rather than fails when timing out?  Is there a JIRA for this
> one too that you know about?
>
> > - Eclipse unit tests 3.3
> >    Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> > on M2 so it is a good baseline for M3.
> >    Linux: the same problem as for 3.2. I've setup another CC Linux
> > host for the suite and see no crashes, pass rate is 99.45%
>
> Looks good enough for where we are today.
>
> >   So I'm going to publish these reports for M3 and run 3.2 suite on
> > these lucky host to obtain results without the crash.
> >
> > - Eclipse Geronimo Application (EGA) scenario x 48h
> >    Windows: there may be and issue with the scenario automation - the
> > scenario run more then 8 hours. As I understand several iterations
> > were done successfully. And it failed because of missing window, there
> > are no crash/exceptions in the log.
> >
> >   Also the scenario passed on Linux.
>
> That is looking good too then.
>
> > - Eclipse TPTP Tests
> >   1 failure on Linux - most probably intermittent
> >
> > - Functional Test Suite
> >    21 failures on Windows most of then failure due to timeout. 1
> > regression (HARMONY-4857)
> >    Uploading Linux failed - I'm going to check why. From the first
> > glance the results are same as for Windows.
>
> Would be good to figure out what is happening with HARMONY-4857.  How
> about we mark this as an M3 blocker until we understand it better?
>
> > - Geronimo Unit Tests
> >   2 possible(I haven't seen analysis report) regressions on
> > Windows/Linux. These test passed on M2:
> >   org.apache.geronimo.tomcat.ContainerTest
> >   org.apache.geronimo.tomcat.JAASSecurityTest
>
> JIRA issues for these too?
>
> > - JDKTools Tests
> >   1 failure on Windows. The test stably fails when it runs in a bunch
> > with others and may fail if run standalone :
> >
> > - Reliability Test Suite
> >   The status is a bit unclear to me. There is 1 test is incorrect
> > test. And some tests failed because of timeout but its increase didn't
> > help.
> >   Also some tests were removed from exclude list for r579330 but some
> > of them failed.
> >
> > - Stress Test Suite
> >   3 failures on Windows and 11 on Linux - I haven't seen analysis report
>
> Is anybody familiar enough with the history of the reliability and
> stress test results to offer an opinion here?
>
> > - VTS VM Test Suite
> >   1 failure on Linux - the test passed on previous snapshot should be analysed.
> >
> > Any comments are welcome.
>
> Good job! I think that although there are many failures reported, there
> are many more tests and testing scenarios being exercised.  We should
> chase down the critical bugs for M3, but be prepared to triage the
> others into the M4 task list.
>
> Regards,
> Tim
>
> > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
> >
> > Thanks,
> > Stepan Mishura
> > Intel Enterprise Solutions Software Division
> >
>


-- 
Thanks,
Andrey

Mime
View raw message