harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Fedotov" <alexei.fedo...@gmail.com>
Subject Re: [drlvm][classlib unit tests] iterative runs
Date Sat, 25 Nov 2006 02:21:41 GMT
Tatiana,
I cannot stop myself from thinking about your wonderful scripts.

1.
Looking at http://issues.apache.org/jira/browse/HARMONY-2282 I
realized that in case of absence of DRLVM crash it would be quite
useful to know if the bug is reproducible with J9.

Failures on two different VMs convinces test authors to look in their
tests once again to decide how valid they are. So if it wasn't
difficult to launch your script for J9, this would be quite useful.

2. The second thing which comes into my mind is speeding up test runs
by hacking build.xml files:

perl -i -e 'undef $/; $_ = <>; s/(<batchtest.*?<\/batchtest>)/$1 x
50/es; print' modules/*/build.xml

You can revert build files later using svn revert modules/*/build.xml.
I wonder if we can set forkmode to vary a reliability load from
running all tests in the same JVM to running each test separately.

-- 
Thank you,
Alexei




On 11/23/06, tatyana doubtsova <tatyanadoubtsova@gmail.com> wrote:
> I created lists of intermittent unit tests failures on Win XP in perTest
> mode:
> http://issues.apache.org/jira/browse/HARMONY-2282 - swing module
> http://issues.apache.org/jira/browse/HARMONY-2284 - security module
>
> Could anybody take a look at it?
>
> I did not observe intermittent failures in other modules on Win XP so far.
> I'm going to continue iterative classlib tests runs.
>
> Thanks,
> Tanya
>
> On 11/22/06, Alexei Fedotov <alexei.fedotov@gmail.com> wrote:
>
> > Tatiana,
> > That's great! Feel free to file JIRA issues about new problems.
> >
> > This is quite interesting that you haven't discovered
> > org.apache.harmony.logging.tests.java.util.logging.SocketHandlerTest
> > failure using effectively the same revision as I.  This makes me think
> > that my problems can be local to my computer.
> >
> > --
> > Thank you,
> > Alexei
> >
> > On 11/21/06, tatyana doubtsova <tatyanadoubtsova@gmail.com> wrote:
> > > Hello,
> > >
> > > I've run classlib unit tests on Win XP in Other VM mode 50 times for
> > each
> > > module in order to reveal sporadic failures.
> > >
> > >
> > >
> > > The script is
> > >
> > >
> > >
> > > #!sh
> > >
> > > cat modulesList | while read moduleName; do
> > >
> > >            A=0
> > >
> > >            while [ "$A" -lt "50" ]; do
> > >
> > >            echo ITERATION $A
> > >
> > >            ant -
> > >
> > Dtest.jre.home=c:/users/tdoubtso/harmony/drlvm/trunk/build/win_ia32_msvc_debug/deploy/jre\
> > >
> > >                        -Dbuild.module=$moduleName -
> > > Dhy.test.forkmode=perTest \
> > >
> > >                        test 2>&1 | tee log_${moduleName}_${A}.txt
> > >
> > >            cp -rf build/test_report build/test_report.${moduleName}_${A}
> > >
> > >            A=`expr $A + 1`
> > >
> > >            done
> > >
> > > done
> > >
> > >
> > >
> > >
> > >
> > > $ cat moduleList
> > >
> > >
> > >
> > > accessibility
> > >
> > > archive
> > >
> > > auth
> > >
> > > awt
> > >
> > > beans
> > >
> > > crypto
> > >
> > > instrument
> > >
> > > jndi
> > >
> > > lang-management
> > >
> > > logging
> > >
> > > luni
> > >
> > > math
> > >
> > > nio
> > >
> > > nio_char
> > >
> > > prefs
> > >
> > > print
> > >
> > > regex
> > >
> > > rmi
> > >
> > > security
> > >
> > > sound
> > >
> > > sql
> > >
> > > swing
> > >
> > > text
> > >
> > > tools
> > >
> > > x-net
> > >
> > >
> > >
> > >
> > >
> > > I've observed the following failures on svn=r476124
> > >
> > >
> > >
> > >
> > >
> > >  *Module*
> > >
> > > *Test*
> > >
> > > *Number of failures*
> > >
> > > *issue*
> > >
> > > luni
> > >
> > >
> > org.apache.harmony.tests.internal.net.www.protocol.http.HttpURLConnectionTest
> > >
> > > 50 out of 50
> > >
> > > 1879 <http://issues.apache.org/jira/browse/HARMONY-1879>, updated
> > >
> > > luni
> > >
> > > tests.api.java.net.InetAddressTest
> > >
> > > 50 out of 50
> > >
> > > Test was eliminated from repository after Novenber,17
> > >
> > > prefs
> > >
> > > org.apache.harmony.prefs.tests.java.util.prefs.PreferencesTest
> > >
> > > 50 out of 50
> > >
> > > 1751 <http://issues.apache.org/jira/browse/HARMONY-1751>
> > >
> > > swing
> > >
> > > javax.swing.text.WrappedPlainView_SimpleTest
> > >
> > > 1 time out of 50
> > >
> > > 1)
> > >
> > > swing
> > >
> > > javax.swing.SpinnerDateModelTest
> > >
> > > 35 times out of 50
> > >
> > > 2)
> > >
> > >
> > >
> > >
> > > 1). Eror
> > >
> > > Timeout occurred
> > >
> > > junit.framework.AssertionFailedError: Timeout occurred
> > >
> > >
> > >
> > > 2). Failure
> > >
> > > expected:<0> but was:<7>
> > >
> > > junit.framework.AssertionFailedError: expected:<0> but was:<7>
at
> > > javax.swing.SpinnerDateModelTest.testSpinnerDateModel(
> > > SpinnerDateModelTest.java:59) at
> > > java.lang.reflect.VMReflection.invokeMethod(Native
> > > Method) at javax.swing.BasicSwingTestCase.runBareSuper(
> > > BasicSwingTestCase.java) at javax.swing.BasicSwingTestCase.runBareImpl(
> > > BasicSwingTestCase.java) at javax.swing.BasicSwingTestCase$1.run(
> > > BasicSwingTestCase.java) at java.lang.Thread.run(Unknown Source)
> > >
> > >
> > >
> > > 2070 <http://issues.apache.org/jira/browse/HARMONY-2070> was not
> > reproduced
> > > during this test run
> > >
> > >
> > >
> > > I can't reproduce 1) and 2) today on svn = r477507
> > >
> > >
> > >
> > > I've excluded failed tests and restarted multiple test run on Win XP in
> > > order to make sure there is no other sporadic failures.
> > >
> > > Also I'm going to start the multiple classlib tests run on SUSE 9
> > >
> > >
> > >
> > >
> > >
> > > To extract the results I've used commands:
> > >
> > > cd classlib/trunk/buid
> > >
> > > # find errors
> > >
> > > find . -name *Test.xml -exec grep -l "<testsuite errors=\"[1-9]*\" " {}
> > \; >
> > > ERRORS
> > >
> > > # find failures
> > >
> > > find . -name *Test.xml -exec grep -l "<testsuite errors=\"*\"
> > > failures=\"[1-9]*\"" {} \; > FAILURES
> > >
> > > find . -type f -size 0 | grep *.xml > ZERO_LENGTH
> > >
> > >
> > >
> > > Also I've compared the number of files in test_report directories:
> > >
> > > #!sh
> > >
> > > cat modulesList | while read moduleName; do
> > >
> > >            echo $moduleName
> > >
> > >            A=1
> > >
> > >            while [ "$A" -lt "51" ]; do
> > >
> > >            L=`ls -1 build/test_report.${moduleName}_${A} | grep xml | wc
> > > -l`
> > >
> > >            echo build/test_report.${moduleName}_${A} $L >>
> > > test_report_length
> > >
> > >            A=`expr $A + 1`
> > >
> > >            done
> > > done
> > >
> > > Thanks,
> > > Tanya
> > >
> > >
> >
>
>

Mime
View raw message