harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Ostrovsky" <eugene.s.ostrov...@gmail.com>
Subject Re: [general] M3 - code frozen
Date Wed, 26 Sep 2007 09:44:36 GMT
Stepan,

I've run the failed TPTP test 1000 times on harmony M2 and M3 snapshots:

M2:  35 failures (1000 runs)
M3:  36 failures (1000 runs)

This results prove that:
1. This failure is intermittent (~3.5% failures)
2. This is not a regression relative to M2

The similar results I've got for JDKTools failed test
org.apache.harmony.jpda.tests.jdwp.MultiSession.RefTypeIDTest:

M2:  14 failures (1000 runs)
M3:  12 failures (1000 runs)

Thanks,
Eugene.

On 9/25/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> On 9/25/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > On 9/25/07, Eugene Ostrovsky wrote:
> > > > Ops, I missed TPTP tests - 1 failure on Linux. Please pick up.
> > >
> > > TPTP Linux x86 - 1 test failed:
> > > org.eclipse.tptp.ac.testautomation.junits.AggStandaloneProfiling.agg_threadPowerWorkload_1_F
> > > We've seen this test failed before. It is a known intermittent failure
> > > due to TPTP bug.
> > >
> >
> > Yes, looks like intermittent failure - the test passed on two previous
> > snapshots (r577501 and r576550). I'm OK with excluding it if there are
> > no objections.
> >
>
> Well, I take back my words about two previous snapshots - it is not
> indication of intermittent failure. I can rerun TPTP suite and if the
> test pass then it can be excluded as intermittent failure otherwise
> the failure should be investigated. Agree?
> Thanks,
> Stepan.
>
> > I assume there should be corresponding JIRA filled if the TPTP bug is know.
> > Please update the JIRA with the patch that excludes the test.
> >
> > Thanks,
> > Stepan.
> >
> > > Probably it would be better to exclude this test.
> > >
> > > Thanks,
> > > Eugene.
> > >
> > >
> > > On 9/25/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > > > On 9/25/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > > > > On 9/24/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > > > > > On 9/22/07, Stepan Mishura wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > According to the plan - Sept. 22 is code freeze date for
M3 so let's
> > > > > > > follow policy:
> > > > > > > "no more commits please without agreement from two committers
on the dev list."
> > > > > > >
> > > > > > > Let's do more testing and analyze if there any critical/blocker
issues
> > > > > > > for consideration.
> > > > > > > Please raise here issues that you think MUST be fixed for
M3.
> > > > > >
> > > > > > Here is [1] the status of the last snapshot (r578410) that includes
> > > > > > last classlib updates. Unfortunately, not everything is green
there.
> > > > > > I'm going to inspect all results to make sure that there are
no
> > > > > > failures caused by CC configuration issues.
> > > > > >
> > > > >
> > > > > I've compared testing status for the last snapshot r578410 [1] with
M2 [2][3].
> > > > >
> > > > > We have the following short status for failed suites:
> > > > >
> > > > > Windows_x86:
> > > > >  * classlib: 2 tests from pack200 module fail on snapshot (but pass
> > > > > on debug build)
> > > > >  * Eclipse unit tests 3.2: there is no tests report like for M3.
The
> > > > > pass rate was improved from 99.32%[3] to 99.7%[1]
> > > > >  * Eclipse unit tests 3.3 are new and the pass rate is 99.77%. I
> > > > > think is acceptable
> > > > >  * EGA x48 hours scenario fails. According to [4] it passed on M2.
> > > > >  * Functional: need more analysis, currently I see that 2 tests were
> > > > > enabled and new 15 regressions.
> > > > >  * Geronimo: 2 regressions
> > > > >  * JDK tools: 1 test failed. It might be intermediate failure - the
> > > > > test failed due to timeout
> > > > >  * Reliability: 65 tests passed for M2 and 64 for M3. Investigation
> > > > > is required.
> > > > >  * Stress: 190 tests passed for M2 and 189 for M3. Investigation
is required.
> > > > >
> > > > > Linux_x86:
> > > > >  * classlib: 2 tests from pack200 (as for Windows),  1 luni tests
> > > > > failed and 1 crash of security test
> > > > >  * Eclipse unit tests 3.2: 2 suites crashed so pass rate is 69.60%.
I
> > > > > assume this may be CC host configuration issue. Going to investigate.
> > > > >  * Eclipse unit tests 3.3 are new and the pass rate is 96.47%. I
> > > > > think is acceptable
> > > > >  * EGA x48 hours scenario: the same as for Windows (scenario fails
on M3)
> > > > >  * Functional: need more analysis, similar to Windows - some test
are
> > > > > passing now but there are new failures.
> > > > >  * Geronimo: 2 regressions (the same as for Windows)
> > > > >  * Reliability: need more analysis
> > > > >  * Stress: need more analysis
> > > > >
> > > >
> > > > Ops, I missed TPTP tests - 1 failure on Linux. Please pick up.
> > > >
> > > > > As I remember Sean took pack200 tests. And Alexei Zakharov took
> > > > > security test crash.
> > > > > I'm going to sort things out with Eclipse unit tests 3.2 crash on
> > > > > Linux. And to look info failed jdktools test.
> > > > >
> > > >
> > > > There are 2 JIRA for improving EUTs reports.
> > > > 1) HARMONY-4835: if fixes reporter to take into account expected crashes.
> > > > 2) HARMONY-4847: renames index.htm to index.html
> > > >
> > > > I'm going to apply patches them to the infra. Objections?
> > > >
> > > > Thanks,
> > > > Stepan.
> > > >
> > > > > So volunteers are required for: EGAx48, Geronimo, functional,
> > > > > reliability and stress suites.
> > > > >
> > > > > Also we have 2 JIRAs to be resolved for M3 under milstone unmblella[5]:
> > > > > https://issues.apache.org/jira/browse/HARMONY-4844
> > > > > https://issues.apache.org/jira/browse/HARMONY-4810
> > > > >
> > > > > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r578410/index.html
> > > > > [2] http://people.apache.org/~mloenko/snapshot_testing/script/r551077/index.html
> > > > > [3] http://wiki.apache.org/harmony/milestones/M2
> > > > > [4] http://mail-archives.apache.org/mod_mbox/harmony-dev/200706.mbox/%3c678b3f320706290508l554079dau6af1a82b17d62b54@mail.gmail.com%3e
> > > > > [5] https://issues.apache.org/jira/browse/HARMONY-4843
> > > > >
> > > > > Thanks,
> > > > > Stepan.
> > > > >
> > > > >
> > > > > > Please feel free to pick up any issue for investigation. For
example,
> > > > > > 2 pack200 classlib test failed. Also there is a crash of
> > > > > > org.apache.harmony.security.tests.java.security.cert.serialization.CertificateTest
> > > > > > on Linux x86
> > > > > >
> > > > > > BTW, should we consider BTI's workspace (that we use for M3
testing)
> > > > > > frozen too?
> > > > > >
> > > > > > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r578410/index.html
> > > > > >
> > > > > > Thanks,
> > > > > > Stepan.
>

Mime
View raw message