harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: [general][M3] release candidate status
Date Wed, 03 Oct 2007 06:07:41 GMT
On 10/2/07, Tim Ellison <t.p.ellison@gmail.com> wrote:
> Stepan Mishura wrote:
> > On 10/2/07, Mikhail Loenko <mloenko@gmail.com> wrote:
> >> Do we see any M3 blockers so far?
> >
> > As I understand there are no critical/blocker issues for r579330.
>
> ? there were a number of regression on that revision that were blockers
> and have been fixed.
>

All fixes for issues raised on dev-list were included to the next
milestone candidate. Did I miss any?

> > Roughly,
> > - several regressions were found (most of them were fixed and included
> > into the new milestone candidate r580985), BWT, I encourage everyone
> > to use M3 umbrella for tracking [1]
>
> Agreed.  I see two still open against that umbrella task.  I think we
> can unlink HARMONY-4865 (tptp failures), but HARMONY-4184 looks like a
> candidate to fix, right?
>

Alexey (Varlamov) did a quick fix for HARMONY-4184.

Alexey, could you update its status? Did your quick fix(r580932)
resolve the issue?

> > - there are couple of suites to be improved (too many timeouts and
> > intermittent failures): Reliability/Stress/Functional
> > - automation for some suites should to be fixed: EGA/EUT
> > - few reports are missed: Geronimo Unit Tests (looks like regressions)
>
> This is all good stuff to capture in our list of things to do for M4.
>

Yes, I hope to fix all these issues for M4 to make auto-testing more reliable.

Thanks,
Stepan.

> Hopefully we can declare M3 soonest and unfreeze the code.
>
> Regards,
> Tim
>
> > [1] https://issues.apache.org/jira/browse/HARMONY-4843
> >
> > Thanks,
> > Stepan.
> >
> >> Thanks,
> >> Mikhail
> >>
> >> 2007/10/2, Stepan Mishura <stepan.mishura@gmail.com>:
> >>> Hi,
> >>>
> >>> Currently, the next milestone candidate (r580985) is under testing.
> >>>
> >>> There were a number of investigation reports for r579330 and I'd like
> >>> to update its summary to track the progress. All
> >>>
> >>> * Class library:
> >>>    - Windows: 1 crash (hard to reproduce)
> >>>    - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - HARMONY-4872)
> >>> * Eclipse 3.2 unit tests:
> >>>    - Windows:
> >>>        4 failures - 1 is expected, 3 hard to reproduce (exclude list
> >>> was updated)
> >>>    - Linux:
> >>>        issue with org.eclipse.jdt.core.tests.compiler.regression.TestAll
suite
> >>>        timeout still remains.
> >>> * Eclipse 3.3 unit tests:
> >>>    Success rate on Windows - 99.77% and Linux 99.46%.
> >>>    This results will be taken as baseline for tracking regressions.
> >>> * Eclipse Geronimo Application (EGA) scenario x 48h
> >>>    - Windows:
> >>>       failed after 8 running hours, most probably the problem
> >>>      with scenario automation - no crashes/exceptions in the output
> >>> * Eclipse TPTP Tests:
> >>>       - Linux: 1 intermittent failure
> >>> * Functional Test Suite:
> >>>    4 regressions were found - HARMONY-4866, HARMONY-4874,
> >>> HARMONY-4857, HARMONY-4861
> >>>    6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1
> >>> case), JIRA for others??
> >>>    HARMONY-4867 - should be defined whether this is regression or not
> >>>    Also there are many failures due to timeout or intermittent failures.
> >>>   The suite definitely must be improved.
> >>> * Geronimo Unit Tests
> >>>    2 regressions on Windows/Linux - no evaluation yet
> >>> * JDKTools tests:
> >>>    Windows: 1 failure - reproducible only if the whole suite is run.
> >>> (HARMONY-4868)
> >>> * Reliability Test Suite:
> >>>     - Windows: 13 failures - according to the evaluation report most
> >>> of them are intermittent and additional investigation is required
> >>>    - Linux: 14 failures - the same as for Windows
> >>> * Stress Test Suite:
> >>>    - Windows: 3 failures
> >>>    - Linux: 11 failures
> >>>       Most of test failures due to timeout or OutOfMemoryError - the
> >>> suite must be improved. There is HARMONY-4114 with 1Mb patch for the
> >>> suite and harness but it is too risky to apply it during code freeze -
> >>> will be applied in M4.
> >>> * VTS VM Test Suite
> >>>    - Linux: 1 intermittent failure - problems in DRLVM shutdown code
> >>> (HARMONY-????).
> >>>
> >>> Thanks,
> >>> Stepan.
> >>>
> >>> On 9/28/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> >>>> Hi all,
> >>>>
> >>>> OK. Here is testing status for r579330 M3 milestone candidate.
> >>>> It is based on testing status page[1] and investigation reports for
> >>>> suites on dev-list.
> >>>> It is provided to decide whether r579330 is OK to be released or we
> >>>> should fix found issues and build next release candidate.
> >>>>
> >>>> Status reporter please correct me if I miss something.
> >>>>
> >>>> I'd like to say that this in this milestone we used BTI for the first
> >>>> time to get 100% of testing results. Some issues in BTI were fixed but
> >>>> there are some still to be resolved, for example instability of
> >>>> several suites, mostly Eclipse unit tests on Linux.
> >>>>
> >>>>
> >>>> The following suites passed on Linux/Windows ia32 platform: Ant
> >>>> Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests,
> >>>> DRLVM regression tests, Jetty scenario, Scimark, Struts scenario,
> >>>> Tomcat scenario
> >>>>
> >>>> The the next list of issues we have (per suite):
> >>>> - Classlibrary unit tests:
> >>>>    1 crash on Windows that is hard to reproduce:
> >>>>    1 crash on Linux that is reproducible and it is bug in DRLVM (HARMONY-????):
> >>>>    1 test failure on Linux (incorrect test HARMONY-????)
> >>>>
> >>>> - Eclipse unit tests 3.2
> >>>>    Windows: 4 failures. 1 failure is expected (exclude list should be
> >>>> updated). 3 failure are not reproducible
> >>>>    Linux: crash of
> >>>> org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due
> >>>> timeout
> >>>>
> >>>> - Eclipse unit tests 3.3
> >>>>   Windows: 49 failures, pass rate 99.77%. We haven't tried the suite
> >>>> on M2 so it is a good baseline for M3.
> >>>>   Linux: the same problem as for 3.2. I've setup another CC Linux
> >>>> host for the suite and see no crashes, pass rate is 99.45%
> >>>>
> >>>>  So I'm going to publish these reports for M3 and run 3.2 suite on
> >>>> these lucky host to obtain results without the crash.
> >>>>
> >>>> - Eclipse Geronimo Application (EGA) scenario x 48h
> >>>>   Windows: there may be and issue with the scenario automation - the
> >>>> scenario run more then 8 hours. As I understand several iterations
> >>>> were done successfully. And it failed because of missing window, there
> >>>> are no crash/exceptions in the log.
> >>>>
> >>>>  Also the scenario passed on Linux.
> >>>>
> >>>> - Eclipse TPTP Tests
> >>>>  1 failure on Linux - most probably intermittent
> >>>>
> >>>> - Functional Test Suite
> >>>>   21 failures on Windows most of then failure due to timeout. 1
> >>>> regression (HARMONY-4857)
> >>>>   Uploading Linux failed - I'm going to check why. From the first
> >>>> glance the results are same as for Windows.
> >>>>
> >>>> - Geronimo Unit Tests
> >>>>  2 possible(I haven't seen analysis report) regressions on
> >>>> Windows/Linux. These test passed on M2:
> >>>>  org.apache.geronimo.tomcat.ContainerTest
> >>>>  org.apache.geronimo.tomcat.JAASSecurityTest
> >>>>
> >>>> - JDKTools Tests
> >>>>  1 failure on Windows. The test stably fails when it runs in a bunch
> >>>> with others and may fail if run standalone :
> >>>>
> >>>> - Reliability Test Suite
> >>>>  The status is a bit unclear to me. There is 1 test is incorrect
> >>>> test. And some tests failed because of timeout but its increase didn't
> >>>> help.
> >>>>  Also some tests were removed from exclude list for r579330 but some
> >>>> of them failed.
> >>>>
> >>>> - Stress Test Suite
> >>>>  3 failures on Windows and 11 on Linux - I haven't seen analysis report
> >>>>
> >>>> - VTS VM Test Suite
> >>>>  1 failure on Linux - the test passed on previous snapshot should be
analysed.
> >>>>
> >>>> Any comments are welcome.
> >>>>
> >>>> [1] http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html
> >>>>
> >>>> Thanks,
> >>>> Stepan Mishura
> >>>> Intel Enterprise Solutions Software Division

Mime
View raw message