Return-Path: Delivered-To: apmail-harmony-dev-archive@www.apache.org Received: (qmail 92694 invoked from network); 21 Nov 2006 03:19:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 21 Nov 2006 03:19:20 -0000 Received: (qmail 60456 invoked by uid 500); 21 Nov 2006 03:19:27 -0000 Delivered-To: apmail-harmony-dev-archive@harmony.apache.org Received: (qmail 60422 invoked by uid 500); 21 Nov 2006 03:19:26 -0000 Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@harmony.apache.org Delivered-To: mailing list dev@harmony.apache.org Received: (qmail 60407 invoked by uid 99); 21 Nov 2006 03:19:26 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Nov 2006 19:19:26 -0800 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: 216.86.168.178 is neither permitted nor denied by domain of geir@pobox.com) Received: from [216.86.168.178] (HELO mxout-03.mxes.net) (216.86.168.178) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Nov 2006 19:19:13 -0800 Received: from [192.168.1.106] (unknown [67.86.14.213]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTP id B35F051935 for ; Mon, 20 Nov 2006 22:18:48 -0500 (EST) Message-ID: <45627027.6050508@pobox.com> Date: Mon, 20 Nov 2006 22:19:03 -0500 From: "Geir Magnusson Jr." Reply-To: geir@pobox.com User-Agent: Thunderbird 1.5.0.8 (Macintosh/20061025) MIME-Version: 1.0 To: dev@harmony.apache.org Subject: Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties References: <20061120112741.888EF1A9846@eris.apache.org> <45620457.9060405@pobox.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Alexey Varlamov wrote: > 2006/11/21, Geir Magnusson Jr. : >> >> >> Alexey Varlamov wrote: >> > Folks, >> > >> > I've just did a little step in improving pre-commit testing for DRLVM; >> > most important change is a move from "perTest" forking mode to "once" >> > (aka sameVM mode). >> > This reduces testing time drastically (~50%), but may introduce some >> > extra instability (like new intermittent failures or timeouts). >> >> Plus the inability to figure out what's screwing things up. >> >> I can't decide if I like this. On one hand, I like it because it's >> actually better to find side-effects - some test can pass, but still >> leave the VM in a broken state that another test will show. >> >> OTOH, it does remove the clarity of each test being a single, separate >> test. It conceptually mixes integration testing with unit testing. > > Well, initially purpose for the kernel tests *is* unit testing and > AFAIU they were written with this in mind (with some exceptions > maybe). For conceptual purity we should re-group tests in the first > place - drop some trivial smoke tests (or move to kernel if > appropriate) and extract some complex cases from kernel (e.g. from > RuntimeTest or SystemExtensionTest). > So, I see no point to soothe ourselves with successful "perTest" runs > if we know the same simple tests may fail in sameVM mode. I don't _know_ that. What I'm pointing out is that we simply need a way to switch between modes, because it's useful to help figure out what's causing the problem. > >> >> Can you please just add a switch? That way in the event of a failure, >> we can re-run with forking on, and therefore can tell if the crash is >> specifically due to the test that is crashing, or a side effect caused >> by something that came before. > > There is the switch: kernel.test.forkmode=[once|perTest]. > I just assumed the keys are easily dediced from contents of the > test.properties, but probably should have announced them explicitly. > OK, here they are with current defaults: > > kernel.test.mode=jet,opt,int // list of EEs to run, as configured > in the test.properties > kernel.test.failfast=off // allows to abort testing > immediately after the first failure > kernel.test.forkmode=once // [once|perTest] > kernel.test.timeout=360000 Great stuff - is this documented anywhere (besides here) ? geir > >> geir >> >> >> >> > Nevertheless I believe we are ready to accept this challenge and it is >> > important to take it earlier. >> > Other enhancements, are described below; hopefully will make us more >> > productive. Have fun! >> > >> > >> > 2006/11/20, varlax@apache.org : >> >> Author: varlax >> >> Date: Mon Nov 20 03:27:39 2006 >> >> New Revision: 477149 >> >> >> >> URL: http://svn.apache.org/viewvc?view=rev&rev=477149 >> >> Log: >> >> Refactored kernel.test.xml, made it more "ant-minded" and added some >> >> features: >> >> [+] "kernel.test.forkmode" support, "once" by default; >> >> [+] single "test.case" support; >> >> [+] "kernel.test.failfast" behaviour support; >> >> [!] Improved results reporting; >> >> [+] easy customization via property file; >> >> >> >> Some intermittent failures appeared on Win32 due to forking only >> >> "once", but we must meet them sooner or later - why not now... >> >> >> >> Added: >> >> harmony/enhanced/drlvm/trunk/build/make/test.properties >> >> Modified: >> >> harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml >> >> >> >> Modified: harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml >> >> URL: >> >> http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml?view=diff&rev=477149&r1=477148&r2=477149 >> >> >> >> ============================================================================== >> >> >> >> --- harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml >> >> (original) >> >> +++ harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml >> >> Mon Nov 20 03:27:39 2006 >> >> @@ -1,17 +1,18 @@ >> >> >> >> >> >> >> >> @@ -20,7 +21,9 @@ >> >> > >> location="${build.semi.dir}/kernel.tests"/> >> >> > >> location="${kernel.test.dir}/resources"/> >> >> > >> location="${kernel.test.dir}/classes"/> >> >> - >> >> + >> >> + >> >> + >> >> >> >> >> >> >> >> @@ -36,68 +39,58 @@ >> >> >> >> >> >> >> >> - >> >> - >> >> - >> >> >> >> - >> >> - >> >> - >> >> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> - return="batch.status"> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> - return="jit.jet.status"> >> >> - > >> value="jitrino.jet" /> >> >> - > >> value="-Dem.properties=jet" /> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> - return="jit.opt.status"> >> >> - > >> value="jitrino.opt" /> >> >> - > >> value="-Dem.properties=opt" /> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> - return="interpreter.status"> >> >> - > >> value="interpreter" /> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> substring="FAILED" /> >> >> - > >> substring="FAILED" /> >> >> - > >> substring="FAILED" /> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> + > >> + depends="compile-kernel-test, run-kernel-test, >> >> report-kernel-test, check-kernel-test"/> >> >> >> >> - >> >> + >> >> + > >> else="false"> >> >> + >> >> + >> >> + > >> + trim="true" keepgoing="${kernel.test.continue}"> >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + > >> + trim="true" keepgoing="true"> >> >> + >> >> + > >> todir="${kernel.test.dir}/reports/@{mode}.mode"> >> >> + > >> dir="${kernel.test.dir}/reports/@{mode}.mode"> >> >> + >> >> + >> >> + > >> + >> >> todir="${kernel.test.dir}/reports/@{mode}.mode/html" /> >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + > >> + srcFile="${kernel.test.dir}/reports/summary"/> >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + >> >> + Please find detailed results under >> >> ${kernel.test.dir}/reports directory. >> >> + >> >> + >> >> + > >> substring="FAILED" /> >> >> + >> >> + >> >> + >> >> + >> >> + >> >> >> >> > >> destdir="${test.resource.path}" >> >> @@ -111,177 +104,61 @@ >> >> >> >> >> >> >> >> - >> >> + >> >> + >> >> + >> >> + >> >> >> >> >> >> >> >> - > >> location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> + > >> + >> >> location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/> >> >> + >> >> + >> >> ============================================== >> >> Please set the classpath of junit as follows: >> >> build.bat -Djunit.jar=%JUNIT_HOME% test >> >> ============================================== >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> + >> >> >> >> >> >> >> >> + > >> from="${kernel.mode}.name" override="on"/> >> >> + > >> from="${kernel.mode}.switch" override="on"/> >> >> + >> >> + ================================== >> >> + Run kernel tests using ${kernel.mode.name} >> >> + ================================== >> >> + >> >> > >> - location="${kernel.test.dir}/reports/batch.mode"/> >> >> + location="${kernel.test.dir}/reports/${kernel.mode}.mode"/> >> >> >> >> >> >> - > >> timeout="${kernel.test.timeout}" >> >> - haltonfailure="no" failureproperty="batch.failed" >> >> - filtertrace="no" printsummary="on" >> >> - jvm="${build.deploy.dir}/bin/java"> >> >> - >> >> - >> >> + > >> + filtertrace="no" printsummary="on" showoutput="off" >> >> + haltonfailure="${kernel.test.failfast}" >> >> + forkmode="${kernel.test.forkmode}" >> >> + timeout="${kernel.test.timeout}" >> >> + jvm="${test.jvm.exe}"> >> >> >> >> - >> >> >> >> + >> >> + >> >> > >> value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}" >> >> /> >> >> - >> >> - >> >> + >> >> + >> >> >> >> >> >> >> >> >> >> + > >> if="test.case" /> >> >> >> >> >> >> - >> >> - >> >> + > >> else="PASSED"> >> >> + >> >> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - ================================== >> >> - Run kernel tests using ${jit.or.interpreter} >> >> - ================================== >> >> - >> >> - >> >> - > >> - >> >> location="${kernel.test.dir}/reports/${jit.or.interpreter}"/> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> file="${kernel.test.javasrc}" /> >> >> - > >> input="@{test}" regexp="(.)+${last.dir}\${file.separator}" replace="" >> >> defaultValue="@{test}" /> >> >> - > >> input="${class.name}" regexp="\.java" replace="" >> >> defaultValue="${class.name}" /> >> >> - > >> input="${class.name}" regexp="\${file.separator}" replace="." >> >> global="true" defaultValue="${class.name}" /> >> >> - >> >> - >> >> - > >> - timeout="${kernel.test.timeout}" >> >> newenvironment="yes" filtertrace="no" >> >> - jvm="${build.deploy.dir}/bin/java"> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> value="-Dtest.resource.path=${test.resource.path}"/> >> >> - > >> value="-Demma.coverage.out.file=${emma.coverage.out}" /> >> >> - >> >> - > >> value="-Xbootclasspath/p:${emma.coverage.ip}${path.separator}${emma.jar}" >> >> /> >> >> - > >> value="-Dvm.use_verifier=${vm.use_verifier}" /> >> >> - >> >> - > >> value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}" >> >> /> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> value="FAILED"> >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - >> >> - > >> todir="${kernel.test.dir}/ref-reports/html" /> >> >> - >> >> - >> >> - > >> from="${jit.or.interpreter}.status"/> >> >> - >> >> - > >> message="==============================================" /> >> >> - >> >> - > >> message="${report.dir}${file.separator}html${file.separator}index.html" >> >> /> >> >> - > >> message="==============================================" /> >> >> >> >> + > >> destfile="${kernel.test.dir}/reports/summary" >> >> + > ${kernel.mode.name} >> >> ${kernel.status}${line.separator} >> >> >> >> >> >> >> >> Added: harmony/enhanced/drlvm/trunk/build/make/test.properties >> >> URL: >> >> http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/test.properties?view=auto&rev=477149 >> >> >> >> ============================================================================== >> >> >> >> --- harmony/enhanced/drlvm/trunk/build/make/test.properties (added) >> >> +++ harmony/enhanced/drlvm/trunk/build/make/test.properties Mon Nov 20 >> >> 03:27:39 2006 >> >> @@ -0,0 +1,19 @@ >> >> +kernel.test.mode=jet,opt,int >> >> +kernel.test.failfast=off >> >> +kernel.test.forkmode=once >> >> +kernel.test.timeout=360000 >> >> + >> >> + >> >> +jit.name=Client mode JIT (default) >> >> +jit.switch=-Dbogus >> >> + >> >> +jet.name=jitrino.JET >> >> +jet.switch=-Xem:jet >> >> + >> >> +opt.name=jitrino.OPT >> >> +opt.switch=-Xem:opt >> >> + >> >> +int.name=interpreter >> >> +int.switch=-Xint >> >> + >> >> + >> >> >> >> >> >> >> > >> >