Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of nbeyer@gmail.com designates
 64.233.166.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=fE5V792jjqbuQjqBoTSfyaprX+cCOUqmyf1kZL9gVzeFRx7ce8YVuMLy8zfhURa2gE
         iNKIOVFEsxs+phSYiIcAPdDhpjh1Jel/F9wbMAWjglyFf7tzbxBTSgQIpVh+UMu8CCx7
         icTNEnP63SygJRvig21AwcoGtquaWLh8nJx3g=
Message-ID: <3b3f27c60808131738l40184b49n135b86e50f436ee5@mail.gmail.com>
Date: Wed, 13 Aug 2008 19:38:56 -0500
From: "Nathan Beyer" <ndbeyer@apache.org>
Sender: nbeyer@gmail.com
To: dev@harmony.apache.org
Subject: Re: [testing] first M7 candidate (r681495) testing status
In-Reply-To: <f9761b910808121940j467b15faj7b331789e4f5cb7@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <f9761b910808111026l28811b1bo708ea7896b2c965d@mail.gmail.com>
	 <3b3f27c60808111840q6a4a9a18i23ccd202d9a00f33@mail.gmail.com>
	 <f9761b910808111905j5b4c4e57y15a8db335ce0d5db@mail.gmail.com>
	 <3b3f27c60808111943h2a4b959ahf31b8d29fd8f0f28@mail.gmail.com>
	 <f9761b910808112016k60ab12f2m91cb0952e31fa7a1@mail.gmail.com>
	 <3b3f27c60808121757n1277a2fcn3aceb271418934ea@mail.gmail.com>
	 <f9761b910808121940j467b15faj7b331789e4f5cb7@mail.gmail.com>

We should get this integrated into buildtest directly and documented.
This after all, is the whole point of buildtest.

-Nathan

On Tue, Aug 12, 2008 at 9:40 PM, chunrong lai <chunronglai@gmail.com> wrote:
> Thanks.
> (1) Setting for e-mail notifications are in framework.local.properties as
> http://harmony.apache.org/subcomponents/buildtest/howto.html.
>
> # Setup for mail notifications
> framework.parameters.usemail=...
> framework.parameters.usemail.to=alerts@harmony.apache.org
> framework.parameters.usemail.from=nomail@intel.com
>
> One may need to copy mail.jar and activation.jar from
> infra/build/libs/cruisecontrol-bin-2.6.2/lib/ to $ANT_HOME/lib.
>
> (2) After running integrity testing the direcory set in
> framework.local.properties
>
> # Directory to store archived results
> framework.parameters.optional.upload.dir=*<results_dir>*
>
> will take effect thus an archived results will be stored there. The file
> name is like
>
> [laichunrong@MRObj-lnx32-03 infra]$ ls ~/ccresult/build/to_upload/
> autouploading                        linux_x86.drlvm-test.status.zip
> linux_x86.jdktools-test.status.zip
> linux_x86.classlib-test.status.zip   linux_x86.ehwa-api.status.zip
> linux_x86.make-snapshot.status.zip
> linux_x86.drlvm-reg-test.status.zip  linux_x86.hdk.status.zip
> uploadresults.sh
>
> (3)
> http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/tools/integrity/status-table/
>
> include the table and perl scripts to generate the integrity testing
> result page from above results.
>
> One needs to specify the result file/web page location and can configure the
> included platforms/test suites there to enable it.
>
> So having a web page to look at every evening can be achieved by
> uploading/extracting the results files and running the perl scripts
> everyday. I see all required settings have been documented.
> (4) We also added local contents to cc-exec-adaptor.xml in BTI for the
> auto-uploading/web-page generating. It has not been committed/documented.
>
>             update="true" />
>
> +        <available property="autouploads"
> file="${upload.dir}/autouploading"/>
> +        <antcall target="-auto-upload"/>
>    </target>
>
> +    <target name="-auto-upload" if="autouploads">
> +        <exec executable="${upload.dir}/uploadresults.sh"
> dir="${upload.dir}">
> +        </exec>
> +    </target>
> </project>
> The scripts use scp to (auto)upload results and ssh to (auto)run above perl
> scripts remotely (required keys setting).
>
> Thanks.
>
> On 8/13/08, Nathan Beyer <nbeyer@gmail.com> wrote:
>>
>> On Mon, Aug 11, 2008 at 10:16 PM, chunrong lai <chunronglai@gmail.com
>> >wrote:
>>
>> >   Thanks.
>> >   I think the standard scripts produce that site although I always added
>> > publish/e-mail notification setup before install/run the test suites.
>>
>>
>> Where is it at, I'm not able to find it and I would really like to recreate
>> this? How do you get it published to people.apache.org? I don't want an
>> email, I just want a web page to look at every evening.
>>
>> Can you document how you set it up? We need to have this integrity thing
>> documented, so anyone can recreate it for the next milestone.
>>
>> -Nathan
>>
>>
>> >
>> >   It is understandable that some tests in classlib-test and drlvm-test
>> get
>> > failing report. We just need to understand the failures.
>> >   As reported in M6,
>> > org.apache.harmony.lang.management.MemoryPoolMXBeanImplTest,
>> >
>> >
>> org.apache.harmony.lang.management.tests.java.lang.management.MemoryPoolMXBeanTest
>> > etc fail classlib-test.
>> >   Also we observed that java.lang.ClassGenericsTest.test_2 sometimes fail
>> > drlvm-test.
>> >   Another timeout issue of drlvm-test comes from
>> > thread.SmallStackThreadTest_jit, as discussed in HARMONY-4601, the test
>> > case
>> > may run very slow when system resource is tight and finally introduce a
>> > timeout. Re-running the suite generally fixes the timeout.
>> >
>> >
>> >
>> >
>> >
>> > On 8/12/08, Nathan Beyer <ndbeyer@apache.org> wrote:
>> > >
>> > > It's the tests that are failing - both classlib-test and drlvm-test.
>> > >
>> > > What's the specific integrity setup that's being used for that site?
>> Does
>> > > the standard out-of-the-box buildtest scripts produce that site?
>> > >
>> > > -Nathan
>> > >
>> > > On Mon, Aug 11, 2008 at 9:05 PM, chunrong lai <chunronglai@gmail.com>
>> > > wrote:
>> > >
>> > > > hi, Nathan:
>> > > >     Steps to run testing cycle are listed in
>> > > > http://harmony.apache.org/subcomponents/buildtest/index.html. One
>> > > > integrity
>> > > > testing example is
>> > > >
>> http://harmony.apache.org/subcomponents/buildtest/howto.html#Extended.
>> > > >      What is the error message you met (in what step)? What is the
>> > > content
>> > > > of the framework.local.properties file? I think that will help to
>> > figure
>> > > > out
>> > > > the problem.
>> > > >
>> > > >
>> > > > On 8/12/08, Nathan Beyer <ndbeyer@apache.org> wrote:
>> > > > >
>> > > > > What's being used to produce the integrity tests? I have an x86_64
>> > > Linux
>> > > > > box
>> > > > > that I can dedicate to testing. I've tried just using buildtest
>> with
>> > > > > 'classlib,drlvm,classib-test,drlvm-test', but I haven't gotten a
>> > clean
>> > > > pass
>> > > > > yet.
>> > > > >
>> > > > > -Nathan
>> > > > >
>> > > > > On Mon, Aug 11, 2008 at 12:26 PM, chunrong lai <
>> > chunronglai@gmail.com
>> > > > > >wrote:
>> > > > >
>> > > > > > Hi, all:
>> > > > > >
>> > > > > >  Here is r681495 snapshot testing status:
>> > > > > > http://people.apache.org/~chunrong/snapshots/r681495/index.html<
>> http://people.apache.org/%7Echunrong/snapshots/r681495/index.html>
>> > <
>> > > http://people.apache.org/%7Echunrong/snapshots/r681495/index.html>
>> > > > <
>> > > > > http://people.apache.org/%7Echunrong/snapshots/r681495/index.html
>> >.
>> > > > > > I am using
>> > > > > > two platforms (Linux x86, windows x86_64) at the moment.
>> Hopefully
>> > we
>> > > > > will
>> > > > > > have other two platforms in future for M8. Well, although we are
>> > > > testing
>> > > > > > only two platforms for M7, my experience is that the status for
>> > > another
>> > > > > two
>> > > > > > platforms should be not worse or just include some extra
>> > intermittent
>> > > > > > errors
>> > > > > > which can be investigated in some later stages.
>> > > > > >
>> > > > > >  The following suites passed on Linux x86/Windows x86_64
>> platforms:
>> > > Ant
>> > > > > > Scenario (or self-hosting), Axis application, Dacapo, DRLVM
>> > > regression
>> > > > > > tests, Geronimo Unit Tests, Scimark, Tomcat scenario, VTS VM Test
>> > > > Suite.
>> > > > > >
>> > > > > >  Most of the failures are known issues (for M6). Although we can
>> > > > observe
>> > > > > > 15~20 new issues, those issues happen only in 1 platform and they
>> > > look
>> > > > > more
>> > > > > > like the intermittent/timeout issues (less reproducible) to me.
>> > > Overall
>> > > > I
>> > > > > > myself would like to think r681495 is more stable than M6.
>> > > > > >
>> > > > > >  Please add your comments and clarifications (please also see M6
>> > > > testing
>> > > > > > results http://people.apache.org/~smishura/r653525/<
>> http://people.apache.org/%7Esmishura/r653525/>
>> > <
>> > > http://people.apache.org/%7Esmishura/r653525/>
>> > > > <
>> > > > > http://people.apache.org/%7Esmishura/r653525/>
>> > > > > > ,
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/%3c6e47b64f0805070304l38845ee0se01fb93fbfc05586@mail.gmail.com%3eand
>> > > > > > integrity testing results
>> > > > > > http://people.apache.org/~chunrong/harmony-integrity/<
>> http://people.apache.org/%7Echunrong/harmony-integrity/>
>> > <
>> > > http://people.apache.org/%7Echunrong/harmony-integrity/>
>> > > > <
>> > > > > http://people.apache.org/%7Echunrong/harmony-integrity/>as a
>> > > > comparison).
>> > > > > >
>> > > > > >  1) Classlib:
>> > > > > >    1.1) Since r644719 (which committed patch for HARMONY-5684)
>> > > > > >
>> > > > >
>> >        org.apache.harmony.lang.management.MemoryPoolMXBeanImplTest
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.lang.management.tests.java.lang.management.MemoryPoolMXBeanTest
>> > > > > >
>> > > > > >         failed in both platforms
>> > > > > >
>> > > > > >    1.2) Two failures
>> > > > > >
>> > > > > >        org.apache.harmony.luni.tests.java.net.MulticastSocketTest
>> > > > (Failed
>> > > > > > in windows_x86 running of M6)
>> > > > > >        tests.api.java.security.PermissionCollectionTest
>> > > > (Failed
>> > > > > > in M6)
>> > > > > >
>> > > > > >         are observed in Linux x86.
>> > > > > >
>> > > > > >  2) DRLVM tests:
>> > > > > >    2.1) One failure
>> > > > > >
>> > > > > >         java.lang.ClassGenericsTest.test_2
>> > > > > >
>> > > > > >         is observed in Linux x86 snapshot testing.
>> > > > > >         I can see some old discussion in the mailing list about
>> > that
>> > > > but
>> > > > > I
>> > > > > > am not sure the expected status here.
>> > > > > >         They should be intemittent errors since the integrity
>> > testing
>> > > > > just
>> > > > > > run well mostly.
>> > > > > >
>> > > > > >  3) EUTs:
>> > > > > >
>> > > > > >    3.1) Linux x86: 99.36%
>> > > > > >         A recorded JIRA for this suite is HARMONY-2914 which
>> wastes
>> > > > file
>> > > > > > handlers and makes system unstable.
>> > > > > >
>> > > > > >  4) Functional:
>> > > > > >    4.1) Old regressions on both platforms:
>> > > > > >         api.java.text.MessageFormat (HARMONY-5430)
>> > > > > >         api.java.util.jar.Manifest  (HARMONY-5473)
>> > > > > >         api.java.beans.beancontext.BeanContextTest (also in M6,
>> > > > recorded
>> > > > > > as  regression caused by changes in locale data)
>> > > > > >         api.java.beans.persistence.EncoderTest  (also in M6,
>> > recorded
>> > > > as
>> > > > > > regression in the beans module)
>> > > > > >         api.java.beans.persistence.EncoderDecoderTest (also in
>> M6,
>> > > > > > regression in the beans module)
>> > > > > >         reg.vm.btest5625 (also in M6, recorded as intermittent
>> and
>> > > not
>> > > > > > reproducible manually)
>> > > > > >
>> > > > > >    4.2) Old regressions on 1 platform
>> > > > > >         api.java.rmi.basicexception (ERROR in Linux x86,
>> > > HARMONY-5823)
>> > > > > >         api.java.rmi.basicregistry.RemoteServerTest (ERROR in
>> Linux
>> > > > x86,
>> > > > > > HARMONY-5823)
>> > > > > >         jpda.jdwp.scenario.ST07.ST07Test (ERROR in windows
>> x86_64,
>> > in
>> > > > M6
>> > > > > it
>> > > > > > is recorded as regression since M4)
>> > > > > >         java.math.F_BigIntegerMatrixMultiplyTest_01 (ERROR on
>> Linux
>> > > > x86,
>> > > > > > recorded as Timeout, not reproducible
>> > > > > >  in M6)
>> > > > > >         reg.vm.btest5717 (ERROR on Windows X86_64, recorded as
>> > > > "timeout,
>> > > > > > the test is too heavy" in M6)
>> > > > > >         jit.HLO.inline.ControlFlow.IfElse.IfElse1.IfElseTest1
>> > (FAILED
>> > > > in
>> > > > > > windows x86_64, recorded as "looks like
>> > > > > > issue in test" in M6)
>> > > > > >         jit.HLO.devirt.Runtime.RuntimeExtend1 (FAILED on windows
>> > > > x86_64,
>> > > > > in
>> > > > > > M6 it is recorded as not regression and started to fail on M5)
>> > > > > >         reg.vm.btest6353.Btest6353 (ERROR on Windows x86_64,
>> > recorded
>> > > > > also
>> > > > > > failed on M3 and might be issue with the test)
>> > > > > >
>> > > > > >    4.3) New regressions on 1 platform (looks intermittent)
>> > > > > >         reg.jit.btest8029.Btest8029 (FAILED in Linux x86)
>> > > > > >         func.reg.jit.btest5710.Btest5710 (FAILED in Linux x86)
>> > > > > >
>> api.java.security.cert.F_CertPathTest_06.F_CertPathTest_06
>> > > > (ERROR
>> > > > > > in Linux x86)
>> > > > > >
>> api.java.security.cert.F_CertPathTest_05.F_CertPathTest_05
>> > > > (ERROR
>> > > > > > in Linux x86)
>> > > > > >
>> > > > > >  5) JDKTools Tests:
>> > > > > >    Several timeouts are observed in Linux x86 snapshot running.
>> > They
>> > > > are:
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch001
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch002
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch003
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.DebuggerOnDemand.OnthrowDebuggerLaunchTest.testDebuggerLaunch004
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.Events.CombinedEventsTest.testCombinedEvents_04
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.MultiSession.AttachConnectorTest.testAttachConnector001
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.MultiSession.MethodEntryExitTest.testMethodEvent001
>> > > > > >
>> > > > > >
>> > org.apache.harmony.jpda.tests.jdwp.MultiSession.ResumeTest.testResume
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadEndTest.testThreadEnd001
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.harmony.jpda.tests.jdwp.MultiSession.ThreadStartTest.testThreadStart001
>> > > > > >    The Linux-only timeouts are also observed in the integrity
>> > testing
>> > > > > > results.
>> > > > > >    JIRA HARMONY-5833 has been filed for one of them.
>> > > > > >
>> > > > > >  6) JettyScenario:
>> > > > > >    The Linux x86 running failed because of the unresolved
>> > > HARMONY-5219.
>> > > > > >
>> > > > > >  7) Reliability:
>> > > > > >    Several failures are observed in windows x86_64 running.
>> > > > > >    7.1) Old regressions
>> > > > > >         api.net.DatagramTest (HARMONY-5531)
>> > > > > >         api.text.DecimalFormat_Locales - (in M6 it is recorded as
>> > > also
>> > > > > > intermittent in M5)
>> > > > > >
>> > > > > >    7.2) New/intemittent regressions
>> > > > > >         api.kernel.thread.ThreadLocalTest.ThreadLocalTest
>> > > > > >         api.kernel.exec.RunExec
>> > > > > >
>> > > > > >  8) Stress
>> > > > > >    Different test cases failed on different platforms.
>> > > > > >    8.1) Timeouts on Linux x86.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD003.ThreadTest003
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD007.ThreadTest007
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD009.ThreadTest009
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.jpda.jdwp.scenario.THREAD011.ThreadTest011
>> > > > > >
>> > > > > >    8.2) Failed cases on Windows x86_64 with unknown reasons.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.classloader.MixThreads.TreeClasses.testTreeClasses2
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> stress.org.apache.harmony.test.stress.classloader.NotSynchThreads.TreeClasses.testTreeClasses
>> > > > > >
>> > > > > >    I have not found records for them.
>> > > > > >
>> > > > > >  9) Strut_test
>> > > > > >    Broken with same error report as M6.
>> > > > > >
>> > > > > >  10) Eclipse Hello World Application.
>> > > > > >    Although the testing framework just reports EUT-API status as
>> > > > > "PASSED".
>> > > > > > A fresh workspace running just failed in configuration stage (
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://mail-archives.apache.org/mod_mbox/harmony-dev/200805.mbox/%3c6e47b64f0805120106o387a49f1rfb2c33d1042d2f41@mail.gmail.com%3e
>> > > > > > )
>> > > > > > since SVN commit 641928 (which committed patch for HARMONY-4569).
>> > > > > >
>> > > > > > thanks,
>> > > > > > chunrong
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>