harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Varlamov" <alexey.v.varla...@gmail.com>
Subject Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Date Tue, 21 Nov 2006 03:02:15 GMT
2006/11/21, Geir Magnusson Jr. <geir@pobox.com>:
>
>
> Alexey Varlamov wrote:
> > Folks,
> >
> > I've just did a little step in improving pre-commit testing for DRLVM;
> > most important change is a move from "perTest" forking mode to "once"
> > (aka sameVM mode).
> > This reduces testing time drastically (~50%), but may introduce some
> > extra instability (like new intermittent failures or timeouts).
>
> Plus the inability to figure out what's screwing things up.
>
> I can't decide if I like this.  On one hand, I like it because it's
> actually better to find side-effects - some test can pass, but still
> leave the VM in a broken state that another test will show.
>
> OTOH, it does remove the clarity of each test being a single, separate
> test.  It conceptually mixes integration testing with unit testing.

Well, initially purpose for the kernel tests *is* unit testing and
AFAIU they were written with this in mind (with some exceptions
maybe). For conceptual purity we should re-group tests in the first
place - drop some trivial smoke tests (or move to kernel if
appropriate) and extract some complex cases from kernel (e.g. from
RuntimeTest or SystemExtensionTest).
So, I see no point to soothe ourselves with successful "perTest" runs
if we know the same simple tests may fail in sameVM mode.

>
> Can you please just add a switch?  That way in the event of a failure,
> we can re-run with forking on, and therefore can tell if the crash is
> specifically due to the test that is crashing, or a side effect caused
> by something that came before.

There is the switch: kernel.test.forkmode=[once|perTest].
I just assumed the keys are easily dediced from contents of the
test.properties, but probably should have announced them explicitly.
OK, here they are with current defaults:

kernel.test.mode=jet,opt,int     // list of EEs to run, as configured
in the test.properties
kernel.test.failfast=off              // allows to abort testing
immediately after the first failure
kernel.test.forkmode=once     // [once|perTest]
kernel.test.timeout=360000

> geir
>
>
>
> > Nevertheless I believe we are ready to accept this challenge and it is
> > important to take it earlier.
> > Other enhancements, are described below; hopefully will make us more
> > productive. Have fun!
> >
> >
> > 2006/11/20, varlax@apache.org <varlax@apache.org>:
> >> Author: varlax
> >> Date: Mon Nov 20 03:27:39 2006
> >> New Revision: 477149
> >>
> >> URL: http://svn.apache.org/viewvc?view=rev&rev=477149
> >> Log:
> >> Refactored kernel.test.xml, made it more "ant-minded" and added some
> >> features:
> >> [+] "kernel.test.forkmode" support, "once" by default;
> >> [+] single "test.case" support;
> >> [+] "kernel.test.failfast" behaviour support;
> >> [!] Improved results reporting;
> >> [+] easy customization via property file;
> >>
> >> Some intermittent failures appeared on Win32 due to forking only
> >> "once", but we must meet them sooner or later - why not now...
> >>
> >> Added:
> >>    harmony/enhanced/drlvm/trunk/build/make/test.properties
> >> Modified:
> >>    harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml
> >>
> >> Modified: harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml
> >> URL:
> >> http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml?view=diff&rev=477149&r1=477148&r2=477149
> >>
> >> ==============================================================================
> >>
> >> --- harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml
> >> (original)
> >> +++ harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml
> >> Mon Nov 20 03:27:39 2006
> >> @@ -1,17 +1,18 @@
> >>  <!--
> >> -    Copyright 2006 The Apache Software Foundation or its licensors,
> >> as applicable.
> >> -
> >> -    Licensed under the Apache License, Version 2.0 (the "License");
> >> -    you may not use this file except in compliance with the License.
> >> -    You may obtain a copy of the License at
> >> -
> >> -       http://www.apache.org/licenses/LICENSE-2.0
> >> -
> >> -    Unless required by applicable law or agreed to in writing, software
> >> -    distributed under the License is distributed on an "AS IS" BASIS,
> >> -    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >> implied.
> >> -    See the License for the specific language governing permissions and
> >> -    limitations under the License.
> >> +     Licensed to the Apache Software Foundation (ASF) under one or more
> >> +     contributor license agreements.  See the NOTICE file distributed
> >> with
> >> +     this work for additional information regarding copyright ownership.
> >> +     The ASF licenses this file to You under the Apache License,
> >> Version 2.0
> >> +     (the "License"); you may not use this file except in compliance
> >> with
> >> +     the License.  You may obtain a copy of the License at
> >> +
> >> +         http://www.apache.org/licenses/LICENSE-2.0
> >> +
> >> +     Unless required by applicable law or agreed to in writing, software
> >> +     distributed under the License is distributed on an "AS IS" BASIS,
> >> +     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >> implied.
> >> +     See the License for the specific language governing permissions and
> >> +     limitations under the License.
> >>  -->
> >>
> >>  <project name="KERNEL TESTS">
> >> @@ -20,7 +21,9 @@
> >>     <property name="kernel.test.dir"
> >> location="${build.semi.dir}/kernel.tests"/>
> >>     <property name="test.resource.path"
> >> location="${kernel.test.dir}/resources"/>
> >>     <property name="test.class.path"
> >> location="${kernel.test.dir}/classes"/>
> >> -
> >> +
> >> +    <property name="test.jvm.exe" value="${build.deploy.dir}/bin/java"/>
> >> +    <property file="test.properties"/>
> >>
> >>     <patternset id="kernel.test.java.pattern">
> >>         <include name="org/apache/harmony/**/*Test.java" />
> >> @@ -36,68 +39,58 @@
> >>         <exclude name="java/lang/RuntimeAdditionalTest40.java" />
> >>         <exclude name="java/lang/RuntimeAdditionalTest41.java" />
> >>         <exclude name="java/lang/RuntimeAdditionalTest42.java" />
> >> -
> >> -       <!-- This test fails -->
> >> -       <!--exclude name="java/lang/ThreadTest.java" /-->
> >>     </patternset>
> >> -
> >> -    <property name="vm.use_verifier" value="1" />
> >> -    <property name="kernel.test.timeout" value="360000" />
> >>
> >> -    <target name="kernel.test" depends="-pre-kernel-test">
> >> -        <property name="test.mode" value="jet+opt+int" />
> >> -        <if>
> >> -            <contains string="${test.mode}" substring="batch" />
> >> -            <then>
> >> -                <antcallback target="-run-kernel-test-batch"
> >> -                    return="batch.status">
> >> -                </antcallback>
> >> -            </then>
> >> -        </if>
> >> -        <if>
> >> -            <contains string="${test.mode}" substring="jet" />
> >> -            <then>
> >> -                <antcallback target="-run-kernel-test"
> >> -                    return="jit.jet.status">
> >> -                    <param name="jit.or.interpreter"
> >> value="jitrino.jet" />
> >> -                    <param name="int.or.jit.arg"
> >> value="-Dem.properties=jet" />
> >> -                </antcallback>
> >> -            </then>
> >> -        </if>
> >> -        <if>
> >> -            <contains string="${test.mode}" substring="opt" />
> >> -            <then>
> >> -                <antcallback target="-run-kernel-test"
> >> -                    return="jit.opt.status">
> >> -                    <param name="jit.or.interpreter"
> >> value="jitrino.opt" />
> >> -                    <param name="int.or.jit.arg"
> >> value="-Dem.properties=opt" />
> >> -                </antcallback>
> >> -            </then>
> >> -        </if>
> >> -        <if>
> >> -            <contains string="${test.mode}" substring="int" />
> >> -            <then>
> >> -                <antcallback target="-run-kernel-test"
> >> -                    return="interpreter.status">
> >> -                    <param name="jit.or.interpreter"
> >> value="interpreter" />
> >> -                    <param name="int.or.jit.arg" value="-Xint" />
> >> -                </antcallback>
> >> -            </then>
> >> -        </if>
> >> -        <if>
> >> -            <or>
> >> -                <contains string="${jit.jet.status}"
> >> substring="FAILED" />
> >> -                <contains string="${jit.opt.status}"
> >> substring="FAILED" />
> >> -                <contains string="${interpreter.status}"
> >> substring="FAILED" />
> >> -                <contains string="${batch.status}" substring="FAILED" />
> >> -            </or>
> >> -            <then>
> >> -                <fail message="Kernel tests FAILED" />
> >> -            </then>
> >> -        </if>
> >> -    </target>
> >> +    <target name="kernel.test"
> >> +        depends="compile-kernel-test, run-kernel-test,
> >> report-kernel-test, check-kernel-test"/>
> >>
> >> -    <target name="compile-kernel-tests" depends="">
> >> +    <target name="run-kernel-test" depends="-prepare-kernel-test">
> >> +        <condition property="kernel.test.continue" value="true"
> >> else="false">
> >> +            <isfalse value="${kernel.test.failfast}"/>
> >> +        </condition>
> >> +        <for list="${kernel.test.mode}" param="mode"
> >> +            trim="true" keepgoing="${kernel.test.continue}">
> >> +            <sequential>
> >> +                <antcall target="-run-kernel-test-batch">
> >> +                    <param name="kernel.mode" value="@{mode}" />
> >> +                </antcall>
> >> +            </sequential>
> >> +        </for>
> >> +    </target>
> >> +
> >> +    <target name="report-kernel-test">
> >> +        <for list="${kernel.test.mode}" param="mode"
> >> +            trim="true" keepgoing="true">
> >> +            <sequential>
> >> +                <junitreport
> >> todir="${kernel.test.dir}/reports/@{mode}.mode">
> >> +                    <fileset
> >> dir="${kernel.test.dir}/reports/@{mode}.mode">
> >> +                        <include name="*.xml" />
> >> +                    </fileset>
> >> +                    <report format="frames"
> >> +
> >> todir="${kernel.test.dir}/reports/@{mode}.mode/html" />
> >> +                </junitreport>
> >> +            </sequential>
> >> +        </for>
> >> +    </target>
> >> +
> >> +    <target name="check-kernel-test">
> >> +        <loadfile property="kernel.test.summary"
> >> +            srcFile="${kernel.test.dir}/reports/summary"/>
> >> +        <echo message="  ***************************${line.separator}"/>
> >> +        <echo message="  *** Kernel tests status ***${line.separator}"/>
> >> +        <echo message="  ***************************${line.separator}"/>
> >> +        <echo message="${line.separator}"/>
> >> +        <echo message="${kernel.test.summary}"/>
> >> +        <echo message="${line.separator}"/>
> >> +        <echo>Please find detailed results under
> >> ${kernel.test.dir}/reports directory.</echo>
> >> +
> >> +        <condition property="kernel.failures">
> >> +            <contains string="${kernel.test.summary}"
> >> substring="FAILED" />
> >> +        </condition>
> >> +        <fail if="kernel.failures" message="There were test
> >> failures." />
> >> +    </target>
> >> +
> >> +    <target name="compile-kernel-test">
> >>         <mkdir dir="${test.resource.path}" />
> >>         <javac srcdir="${kernel.test.javasrc}/../kernel_resources"
> >>             destdir="${test.resource.path}"
> >> @@ -111,177 +104,61 @@
> >>         <delete quiet="true" dir="${test.class.path}/notfound" />
> >>     </target>
> >>
> >> -    <target name="-pre-kernel-test" depends="compile-kernel-tests">
> >> +    <target name="-prepare-kernel-test">
> >> +        <delete quiet="true" dir="${kernel.test.dir}/reports" />
> >> +        <mkdir dir="${kernel.test.dir}/reports" />
> >> +
> >>         <condition property="junit.jar" value="${junit.home}/junit.jar">
> >>             <isset property="junit.home" />
> >>         </condition>
> >> -        <property name="junit.jar"
> >> location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/>
> >> -        <if>
> >> -            <not>
> >> -                <available file="${junit.jar}"/>
> >> -            </not>
> >> -            <then>
> >> -                <echo>
> >> +        <property name="junit.jar"
> >> +
> >> location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/>
> >> +        <available file="${junit.jar}" property="junit.found"/>
> >> +        <fail unless="junit.found">
> >>                 ==============================================
> >>                 Please set the classpath of junit as follows:
> >>                 build.bat -Djunit.jar=%JUNIT_HOME% test
> >>                 ==============================================
> >> -                </echo>
> >> -                <fail message="Kernel tests misconfigured" />
> >> -            </then>
> >> -        </if>
> >> -
> >> -        <delete quiet="true" dir="${kernel.test.dir}/reports" />
> >> -        <delete quiet="true" dir="${kernel.test.dir}/ref-reports" />
> >> -        <mkdir dir="${kernel.test.dir}/reports" />
> >> -        <mkdir dir="${kernel.test.dir}/ref-reports" />
> >> -
> >> +        </fail>
> >>     </target>
> >>
> >>     <target name="-run-kernel-test-batch">
> >> +        <propertycopy property="kernel.mode.name"
> >> from="${kernel.mode}.name" override="on"/>
> >> +        <propertycopy property="kernel.mode.switch"
> >> from="${kernel.mode}.switch" override="on"/>
> >> +        <echo>
> >> +        ==================================
> >> +        Run kernel tests using ${kernel.mode.name}
> >> +        ==================================
> >> +        </echo>
> >>         <property name="report.dir"
> >> -            location="${kernel.test.dir}/reports/batch.mode"/>
> >> +            location="${kernel.test.dir}/reports/${kernel.mode}.mode"/>
> >>         <mkdir dir="${report.dir}" />
> >>
> >> -        <junit fork="yes" forkmode="once"
> >> timeout="${kernel.test.timeout}"
> >> -            haltonfailure="no" failureproperty="batch.failed"
> >> -            filtertrace="no" printsummary="on"
> >> -            jvm="${build.deploy.dir}/bin/java">
> >> -
> >> -            <env key="JAVA_HOME" value="${build.deploy.dir}"/>
> >> +        <junit fork="yes" failureproperty="kernel.failed"
> >> +            filtertrace="no" printsummary="on" showoutput="off"
> >> +            haltonfailure="${kernel.test.failfast}"
> >> +            forkmode="${kernel.test.forkmode}"
> >> +            timeout="${kernel.test.timeout}"
> >> +            jvm="${test.jvm.exe}">
> >>
> >> -            <jvmarg value="-Dtest.resource.path=${test.resource.path}"/>
> >>             <jvmarg value="-Dvm.assert_dialog=0" />
> >> +            <jvmarg value="-Dtest.resource.path=${test.resource.path}"/>
> >> +            <jvmarg value="${kernel.mode.switch}" />
> >>             <jvmarg
> >> value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}"
> >> />
> >> -            <formatter type="brief"/>
> >> -            <batchtest todir="${report.dir}">
> >> +            <formatter type="xml"/>
> >> +            <batchtest todir="${report.dir}" unless="test.case">
> >>                 <fileset dir="${kernel.test.javasrc}">
> >>                     <patternset refid="kernel.test.java.pattern"/>
> >>                 </fileset>
> >>             </batchtest>
> >> +            <test name="${test.case}" todir="${report.dir}"
> >> if="test.case" />
> >>         </junit>
> >>
> >> -        <condition property="batch.status" value="FAILED" else="PASSED">
> >> -            <isset property="batch.failed" />
> >> +        <condition property="kernel.status" value="FAILED"
> >> else="PASSED">
> >> +            <isset property="kernel.failed" />
> >>         </condition>
> >> -    </target>
> >> -
> >> -
> >> -    <target name="-run-kernel-test">
> >> -        <echo>
> >> -               ==================================
> >> -               Run kernel tests using ${jit.or.interpreter}
> >> -               ==================================
> >> -               </echo>
> >> -
> >> -        <property name="report.dir"
> >> -
> >> location="${kernel.test.dir}/reports/${jit.or.interpreter}"/>
> >> -        <mkdir dir="${report.dir}" />
> >> -
> >> -        <for param="test">
> >> -            <fileset dir="${kernel.test.javasrc}">
> >> -                <patternset refid="kernel.test.java.pattern" />
> >> -            </fileset>
> >> -            <sequential>
> >> -
> >> -                <basename property="last.dir"
> >> file="${kernel.test.javasrc}" />
> >> -                <propertyregex property="class.name" override="true"
> >> input="@{test}" regexp="(.)+${last.dir}\${file.separator}" replace=""
> >> defaultValue="@{test}" />
> >> -                <propertyregex property="class.name" override="true"
> >> input="${class.name}" regexp="\.java" replace=""
> >> defaultValue="${class.name}" />
> >> -                <propertyregex property="class.name" override="true"
> >> input="${class.name}" regexp="\${file.separator}" replace="."
> >> global="true" defaultValue="${class.name}" />
> >> -                <echo message="RUNNING : ${class.name}" />
> >> -
> >> -                <junit fork="yes" failureproperty="@{test}.failed"
> >> -                    timeout="${kernel.test.timeout}"
> >> newenvironment="yes" filtertrace="no"
> >> -                    jvm="${build.deploy.dir}/bin/java">
> >> -
> >> -                    <env key="JAVA_HOME" value="${build.deploy.dir}"/>
> >> -
> >> -                    <!--env key="LD_LIBRARY_PATH"
> >> value="${build.deploy.dir}/bin:${env.LD_LIBRARY_PATH}" /-->
> >> -                    <env key="SYSTEMDRIVE" value="${env.SystemDrive}" />
> >> -                    <env key="HOME" value="${env.HOME}" />
> >> -                    <env key="PATH" value="${env.PATH}" />
> >> -                    <jvmarg
> >> value="-Dtest.resource.path=${test.resource.path}"/>
> >> -                    <jvmarg
> >> value="-Demma.coverage.out.file=${emma.coverage.out}" />
> >> -                    <jvmarg value="-Demma.coverage.out.merge=true" />
> >> -                    <jvmarg
> >> value="-Xbootclasspath/p:${emma.coverage.ip}${path.separator}${emma.jar}"
> >> />
> >> -                    <jvmarg
> >> value="-Dvm.use_verifier=${vm.use_verifier}" />
> >> -                    <jvmarg value="-Dvm.assert_dialog=0" />
> >> -                    <jvmarg
> >> value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}"
> >> />
> >> -                    <jvmarg value="${int.or.jit.arg}" />
> >> -                    <jvmarg value="-ea:java.lang.reflect..." />
> >> -                    <formatter type="brief"/>
> >> -                    <formatter type="xml" />
> >> -                    <test name="${class.name}" todir="${report.dir}"/>
> >> -                </junit>
> >> -                <condition property="${jit.or.interpreter}.status"
> >> value="FAILED">
> >> -                    <isset property="@{test}.failed" />
> >> -                </condition>
> >> -
> >> -                <!--
> >> -                Gregory
> >> -                It appears that some kernel unit tests for drlvm
> >> classes are
> >> -                implementation specific and may fail on RI. So it
> >> makes no
> >> -                sense to check tests on RI. Also running test on RI
> >> -                right after it failed on drlvm creates wrong test run
> >> report.
> >> -
> >> -                I am commenting this block for now in case someone
> >> wants it
> >> -                back, possibly under some condition. My POV is that this
> >> -                block should be removed eventually.
> >> -                <if>
> >> -                    <or>
> >> -                        <isset property="@{test}.failed" />
> >> -                        <isset property="kernel.test.reference" />
> >> -                    </or>
> >> -                    <then>
> >> -                        <junit fork="yes"
> >> failureproperty="@{test}.ref.failed"
> >> -                            timeout="${kernel.test.timeout}"
> >> -                            newenvironment="yes" filtertrace="no">
> >> -
> >> -
> >> -                            <env key="JAVA_HOME"
> >> value="${build.deploy.dir}"/>
> >> -
> >> -                            <!- -env key="LD_LIBRARY_PATH"
> >> value="${build.deploy.dir}/bin:${env.LD_LIBRARY_PATH}" /- ->
> >> -                            <env key="SYSTEMDRIVE"
> >> value="${env.SystemDrive}" />
> >> -                            <env key="HOME" value="${env.HOME}" />
> >> -                            <env key="PATH" value="${env.PATH}" />
> >> -                            <jvmarg
> >> value="-Dtest.resource.path=${test.resource.path}"/>
> >> -                            <jvmarg
> >> value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}"
> >> />
> >> -                            <formatter type="brief"/>
> >> -                            <formatter type="xml" />
> >> -                            <test name="${class.name}"
> >> todir="${kernel.test.dir}/ref-reports"/>
> >> -                        </junit>
> >> -
> >> -                        <if>
> >> -                            <isset property="@{test}.ref.failed" />
> >> -                            <then>
> >> -                                <echo message="   **** FAILED on
> >> reference JRE ****" />
> >> -                            </then>
> >> -                        </if>
> >> -                    </then>
> >> -                </if>
> >> -                /-->
> >> -            </sequential>
> >> -        </for>
> >> -
> >> -        <junitreport todir="${report.dir}">
> >> -            <fileset dir="${report.dir}">
> >> -                <include name="TEST-*.xml" />
> >> -            </fileset>
> >> -            <report format="frames" todir="${report.dir}/html" />
> >> -        </junitreport>
> >> -        <junitreport todir="${kernel.test.dir}/ref-reports">
> >> -            <fileset dir="${kernel.test.dir}/ref-reports">
> >> -                <include name="TEST-*.xml" />
> >> -            </fileset>
> >> -            <report format="frames"
> >> todir="${kernel.test.dir}/ref-reports/html" />
> >> -        </junitreport>
> >> -        <property name="${jit.or.interpreter}.status" value="PASSED"/>
> >> -        <propertycopy name="status"
> >> from="${jit.or.interpreter}.status"/>
> >> -        <echo message="" />
> >> -        <echo
> >> message="==============================================" />
> >> -        <echo message="Kernel tests ${status} using
> >> ${jit.or.interpreter}. Please find the detailed results here:" />
> >> -        <echo
> >> message="${report.dir}${file.separator}html${file.separator}index.html"
> >> />
> >> -        <echo
> >> message="==============================================" />
> >>
> >> +        <concat append="on"
> >> destfile="${kernel.test.dir}/reports/summary"
> >> +            >      ${kernel.mode.name}
> >> ${kernel.status}${line.separator}</concat>
> >>     </target>
> >>  </project>
> >>
> >> Added: harmony/enhanced/drlvm/trunk/build/make/test.properties
> >> URL:
> >> http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/test.properties?view=auto&rev=477149
> >>
> >> ==============================================================================
> >>
> >> --- harmony/enhanced/drlvm/trunk/build/make/test.properties (added)
> >> +++ harmony/enhanced/drlvm/trunk/build/make/test.properties Mon Nov 20
> >> 03:27:39 2006
> >> @@ -0,0 +1,19 @@
> >> +kernel.test.mode=jet,opt,int
> >> +kernel.test.failfast=off
> >> +kernel.test.forkmode=once
> >> +kernel.test.timeout=360000
> >> +
> >> +
> >> +jit.name=Client mode JIT (default)
> >> +jit.switch=-Dbogus
> >> +
> >> +jet.name=jitrino.JET
> >> +jet.switch=-Xem:jet
> >> +
> >> +opt.name=jitrino.OPT
> >> +opt.switch=-Xem:opt
> >> +
> >> +int.name=interpreter
> >> +int.switch=-Xint
> >> +
> >> +
> >>
> >>
> >>
> >
>

Mime
View raw message