Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of sianjanuary@googlemail.com
 designates 209.85.146.178 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=rDbCtpHkmyLX244jQeCQMui2Ty79wt6SdPsyiflp7hVzm/h39+dKbJtpdldnCe3olX
         ld0Wf4j6OHaCOKjTOCzDiVDVm6fmvIfVfQ2Ty6b91eNOHbC5c9P3Z2Rmbm6LOrBWHI3e
         6A6aQJzfNy7FRUi8N31jovhWbScHuHaC2b/qc=
Message-ID: <fc2fc95c0808140523j3fef2986w1360fe7c8bee2f46@mail.gmail.com>
Date: Thu, 14 Aug 2008 13:23:34 +0100
From: "Sian January" <sianjanuary@googlemail.com>
To: dev@harmony.apache.org
Subject: Re: [testing] first M7 candidate (r681495) testing status
In-Reply-To: <f9761b910808111026l28811b1bo708ea7896b2c965d@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <f9761b910808111026l28811b1bo708ea7896b2c965d@mail.gmail.com>

Does anyone have a feel for whether any of these failures should be
blockers for M7?  In terms of the Harmony test suite I think the pass
rate is fine because we're doing better than M6, and EUTs look ok too,
but I'm not really sure if we need to be concerned about any of:

4) Functional
5) JDKTools Tests
6) JettyScenario
7) Reliability
8) Stress

or

10) Eclipse Hello World Application.

Any thoughts?

If none of these are blockers then it would be good to start a vote in
the next day or two because the code has been frozen for nearly 3
weeks now.

Thanks,

Sian


On 11/08/2008, chunrong lai <chunronglai@gmail.com> wrote:
> Hi, all:
>
>  Here is r681495 snapshot testing status:
> http://people.apache.org/~chunrong/snapshots/r681495/index.html. I am using
> two platforms (Linux x86, windows x86_64) at the moment. Hopefully we will
> have other two platforms in future for M8. Well, although we are testing
> only two platforms for M7, my experience is that the status for another two
> platforms should be not worse or just include some extra intermittent errors
> which can be investigated in some later stages.
>
>  The following suites passed on Linux x86/Windows x86_64 platforms: Ant
> Scenario (or self-hosting), Axis application, Dacapo, DRLVM regression
> tests, Geronimo Unit Tests, Scimark, Tomcat scenario, VTS VM Test Suite.
>
>  Most of the failures are known issues (for M6). Although we can observe
> 15~20 new issues, those issues happen only in 1 platform and they look more
> like the intermittent/timeout issues (less reproducible) to me. Overall I
> myself would like to think r681495 is more stable than M6.
>
>  Please add your comments and clarifications (please also see M6 testing
> results http://people.apache.org/~smishura/r653525/,
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/%3c6e47b64f0805070304l38845ee0se01fb93fbfc05586@mail.gmail.com%3eand
> integrity testing results
> http://people.apache.org/~chunrong/harmony-integrity/ as a comparison).
>
>  1) Classlib:
>    1.1) Since r644719 (which committed patch for HARMONY-5684)
>
>        org.apache.harmony.lang.management.MemoryPoolMXBeanImplTest
>
> org.apache.harmony.lang.management.tests.java.lang.management.MemoryPoolMXBeanTest
>
>         failed in both platforms
>
>    1.2) Two failures
>
>        org.apache.harmony.luni.tests.java.net.MulticastSocketTest (Failed
> in windows_x86 running of M6)
>        tests.api.java.security.PermissionCollectionTest           (Failed
> in M6)
>
>         are observed in Linux x86.
>
>  2) DRLVM tests:
>    2.1) One failure
>
>         java.lang.ClassGenericsTest.test_2
>
>         is observed in Linux x86 snapshot testing.
>         I can see some old discussion in the mailing list about that but I
> am not sure the expected status here.
>         They should be intemittent errors since the integrity testing just
> run well mostly.
>
>  3) EUTs:
>
>    3.1) Linux x86: 99.36%
>         A recorded JIRA for this suite is HARMONY-2914 which wastes file
> handlers and makes system unstable.
>
>  4) Functional:
>    4.1) Old regressions on both platforms:
>         api.java.text.MessageFormat (HARMONY-5430)
>         api.java.util.jar.Manifest  (HARMONY-5473)
>         api.java.beans.beancontext.BeanContextTest (also in M6, recorded
> as  regression caused by changes in locale data)
>         api.java.beans.persistence.EncoderTest  (also in M6, recorded as
> regression in the beans module)
>         api.java.beans.persistence.EncoderDecoderTest (also in M6,
> regression in the beans module)
>         reg.vm.btest5625 (also in M6, recorded as intermittent and not
> reproducible manually)
>
>    4.2) Old regressions on 1 platform
>         api.java.rmi.basicexception (ERROR in Linux x86, HARMONY-5823)
>         api.java.rmi.basicregistry.RemoteServerTest (ERROR in Linux x86,
> HARMONY-5823)
>         jpda.jdwp.scenario.ST07.ST07Test (ERROR in windows x86_64, in M6 it
> is recorded as regression since M4)
>         java.math.F_BigIntegerMatrixMultiplyTest_01 (ERROR on Linux x86,
> recorded as Timeout, not reproducible
>  in M6)
>         reg.vm.btest5717 (ERROR on Windows X86_64, recorded as "timeout,
> the test is too heavy" in M6)
>         jit.HLO.inline.ControlFlow.IfElse.IfElse1.IfElseTest1 (FAILED in
> windows x86_64, recorded as "looks like
> issue in test" in M6)
>         jit.HLO.devirt.Runtime.RuntimeExtend1 (FAILED on windows x86_64, in
> M6 it is recorded as not regression and started to fail on M5)
>         reg.vm.btest6353.Btest6353 (ERROR on Windows x86_64, recorded also
> failed on M3 and might be issue with the test)
>
>    4.3) New regressions on 1 platform (looks intermittent)
>         reg.jit.btest8029.Btest8029 (FAILED in Linux x86)
>         func.reg.jit.btest5710.Btest5710 (FAILED in Linux x86)
>         api.java.security.cert.F_CertPathTest_06.F_CertPathTest_06 (ERROR
> in Linux x86)
>         api.java.security.cert.F_CertPathTest_05.F_CertPathTest_05 (ERROR
> in Linux x86)
>
>  5) JDKTools Tests:
>    Several timeouts are observed in Linux x86 snapshot running. They are:
>
> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch001
>
> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch002
>
> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch003
>
> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch004
>
> org.apache.harmony.jpda.tests.jdwp.Events.CombinedEventsTest.testCombinedEvents_04
>
> org.apache.harmony.jpda.tests.jdwp.MultiSession.AttachConnectorTest.testAttachConnector001
>
> org.apache.harmony.jpda.tests.jdwp.MultiSession.MethodEntryExitTest.testMethodEvent001
>
> org.apache.harmony.jpda.tests.jdwp.MultiSession.ResumeTest.testResume
>
> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadEndTest.testThreadEnd001
>
> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadStartTest.testThreadStart001
>    The Linux-only timeouts are also observed in the integrity testing
> results.
>    JIRA HARMONY-5833 has been filed for one of them.
>
>  6) JettyScenario:
>    The Linux x86 running failed because of the unresolved HARMONY-5219.
>
>  7) Reliability:
>    Several failures are observed in windows x86_64 running.
>    7.1) Old regressions
>         api.net.DatagramTest (HARMONY-5531)
>         api.text.DecimalFormat_Locales - (in M6 it is recorded as also
> intermittent in M5)
>
>    7.2) New/intemittent regressions
>         api.kernel.thread.ThreadLocalTest.ThreadLocalTest
>         api.kernel.exec.RunExec
>
>  8) Stress
>    Different test cases failed on different platforms.
>    8.1) Timeouts on Linux x86.
>
> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD003.ThreadTest003
>
> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD007.ThreadTest007
>
> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD009.ThreadTest009
>
> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD011.ThreadTest011
>
>    8.2) Failed cases on Windows x86_64 with unknown reasons.
>
> stress.org.apache.harmony.test.stress.classloader.MixThreads.TreeClasses.testTreeClasses2
>
> stress.org.apache.harmony.test.stress.classloader.NotSynchThreads.TreeClasses.testTreeClasses
>
>    I have not found records for them.
>
>  9) Strut_test
>    Broken with same error report as M6.
>
>  10) Eclipse Hello World Application.
>    Although the testing framework just reports EUT-API status as "PASSED".
> A fresh workspace running just failed in configuration stage (
> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/%3c6e47b64f0805120106o387a49f1rfb2c33d1042d2f41@mail.gmail.com%3e)
> since SVN commit 641928 (which committed patch for HARMONY-4569).
>
> thanks,
> chunrong
>


-- 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU