harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geir Magnusson Jr." <g...@pobox.com>
Subject Re: [drlvm][classlib unit tests] iterative runs
Date Mon, 27 Nov 2006 15:30:27 GMT


tatyana doubtsova wrote:
> I'm not sure the iterative runs should be incorporated in CC for 2 reasons:
> 1. Those runs are time consuming.

So?  They are important!

Oh, note that I said "possible" - I don't mean that everyone running the 
  CC framework should have to run them - I mean I can setup a second 
machine and just run that target, and get all the goodness of the CC 
framework for free.

> 2. ant is not acceptable to organize loops.

There must be a way to fix this.

geir

> 
> Thanks,
> Tanya
> 
> On 11/24/06, Geir Magnusson Jr. <geir@pobox.com> wrote:
>>
>> This is incredibly useful.  Thanks.  We need to incorporate this as a
>> possible target in CC.
>>
>> geir
>>
>> tatyana doubtsova wrote:
>> > I created lists of intermittent unit tests failures on Win XP in 
>> perTest
>> > mode:
>> > http://issues.apache.org/jira/browse/HARMONY-2282 - swing module
>> > http://issues.apache.org/jira/browse/HARMONY-2284 - security module
>> >
>> > Could anybody take a look at it?
>> >
>> > I did not observe intermittent failures in other modules on Win XP so
>> far.
>> > I'm going to continue iterative classlib tests runs.
>> >
>> > Thanks,
>> > Tanya
>> >
>> > On 11/22/06, Alexei Fedotov <alexei.fedotov@gmail.com> wrote:
>> >
>> >> Tatiana,
>> >> That's great! Feel free to file JIRA issues about new problems.
>> >>
>> >> This is quite interesting that you haven't discovered
>> >> org.apache.harmony.logging.tests.java.util.logging.SocketHandlerTest
>> >> failure using effectively the same revision as I.  This makes me think
>> >> that my problems can be local to my computer.
>> >>
>> >> --
>> >> Thank you,
>> >> Alexei
>> >>
>> >> On 11/21/06, tatyana doubtsova <tatyanadoubtsova@gmail.com> wrote:
>> >> > Hello,
>> >> >
>> >> > I've run classlib unit tests on Win XP in Other VM mode 50 times for
>> >> each
>> >> > module in order to reveal sporadic failures.
>> >> >
>> >> >
>> >> >
>> >> > The script is
>> >> >
>> >> >
>> >> >
>> >> > #!sh
>> >> >
>> >> > cat modulesList | while read moduleName; do
>> >> >
>> >> >            A=0
>> >> >
>> >> >            while [ "$A" -lt "50" ]; do
>> >> >
>> >> >            echo ITERATION $A
>> >> >
>> >> >            ant -
>> >> >
>> >>
>> Dtest.jre.home=c:/users/tdoubtso/harmony/drlvm/trunk/build/win_ia32_msvc_debug/deploy/jre\

>>
>> >>
>> >> >
>> >> >                        -Dbuild.module=$moduleName -
>> >> > Dhy.test.forkmode=perTest \
>> >> >
>> >> >                        test 2>&1 | tee log_${moduleName}_${A}.txt
>> >> >
>> >> >            cp -rf build/test_report
>> >> build/test_report.${moduleName}_${A}
>> >> >
>> >> >            A=`expr $A + 1`
>> >> >
>> >> >            done
>> >> >
>> >> > done
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > $ cat moduleList
>> >> >
>> >> >
>> >> >
>> >> > accessibility
>> >> >
>> >> > archive
>> >> >
>> >> > auth
>> >> >
>> >> > awt
>> >> >
>> >> > beans
>> >> >
>> >> > crypto
>> >> >
>> >> > instrument
>> >> >
>> >> > jndi
>> >> >
>> >> > lang-management
>> >> >
>> >> > logging
>> >> >
>> >> > luni
>> >> >
>> >> > math
>> >> >
>> >> > nio
>> >> >
>> >> > nio_char
>> >> >
>> >> > prefs
>> >> >
>> >> > print
>> >> >
>> >> > regex
>> >> >
>> >> > rmi
>> >> >
>> >> > security
>> >> >
>> >> > sound
>> >> >
>> >> > sql
>> >> >
>> >> > swing
>> >> >
>> >> > text
>> >> >
>> >> > tools
>> >> >
>> >> > x-net
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > I've observed the following failures on svn=r476124
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >  *Module*
>> >> >
>> >> > *Test*
>> >> >
>> >> > *Number of failures*
>> >> >
>> >> > *issue*
>> >> >
>> >> > luni
>> >> >
>> >> >
>> >>
>> org.apache.harmony.tests.internal.net.www.protocol.http.HttpURLConnectionTest 
>>
>> >>
>> >> >
>> >> > 50 out of 50
>> >> >
>> >> > 1879 <http://issues.apache.org/jira/browse/HARMONY-1879>, updated
>> >> >
>> >> > luni
>> >> >
>> >> > tests.api.java.net.InetAddressTest
>> >> >
>> >> > 50 out of 50
>> >> >
>> >> > Test was eliminated from repository after Novenber,17
>> >> >
>> >> > prefs
>> >> >
>> >> > org.apache.harmony.prefs.tests.java.util.prefs.PreferencesTest
>> >> >
>> >> > 50 out of 50
>> >> >
>> >> > 1751 <http://issues.apache.org/jira/browse/HARMONY-1751>
>> >> >
>> >> > swing
>> >> >
>> >> > javax.swing.text.WrappedPlainView_SimpleTest
>> >> >
>> >> > 1 time out of 50
>> >> >
>> >> > 1)
>> >> >
>> >> > swing
>> >> >
>> >> > javax.swing.SpinnerDateModelTest
>> >> >
>> >> > 35 times out of 50
>> >> >
>> >> > 2)
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > 1). Eror
>> >> >
>> >> > Timeout occurred
>> >> >
>> >> > junit.framework.AssertionFailedError: Timeout occurred
>> >> >
>> >> >
>> >> >
>> >> > 2). Failure
>> >> >
>> >> > expected:<0> but was:<7>
>> >> >
>> >> > junit.framework.AssertionFailedError: expected:<0> but was:<7>
at
>> >> > javax.swing.SpinnerDateModelTest.testSpinnerDateModel(
>> >> > SpinnerDateModelTest.java:59) at
>> >> > java.lang.reflect.VMReflection.invokeMethod(Native
>> >> > Method) at javax.swing.BasicSwingTestCase.runBareSuper(
>> >> > BasicSwingTestCase.java) at
>> javax.swing.BasicSwingTestCase.runBareImpl(
>> >> > BasicSwingTestCase.java) at javax.swing.BasicSwingTestCase$1.run(
>> >> > BasicSwingTestCase.java) at java.lang.Thread.run(Unknown Source)
>> >> >
>> >> >
>> >> >
>> >> > 2070 <http://issues.apache.org/jira/browse/HARMONY-2070> was
not
>> >> reproduced
>> >> > during this test run
>> >> >
>> >> >
>> >> >
>> >> > I can't reproduce 1) and 2) today on svn = r477507
>> >> >
>> >> >
>> >> >
>> >> > I've excluded failed tests and restarted multiple test run on Win XP
>> in
>> >> > order to make sure there is no other sporadic failures.
>> >> >
>> >> > Also I'm going to start the multiple classlib tests run on SUSE 9
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > To extract the results I've used commands:
>> >> >
>> >> > cd classlib/trunk/buid
>> >> >
>> >> > # find errors
>> >> >
>> >> > find . -name *Test.xml -exec grep -l "<testsuite errors=\"[1-9]*\"
"
>> {}
>> >> \; >
>> >> > ERRORS
>> >> >
>> >> > # find failures
>> >> >
>> >> > find . -name *Test.xml -exec grep -l "<testsuite errors=\"*\"
>> >> > failures=\"[1-9]*\"" {} \; > FAILURES
>> >> >
>> >> > find . -type f -size 0 | grep *.xml > ZERO_LENGTH
>> >> >
>> >> >
>> >> >
>> >> > Also I've compared the number of files in test_report directories:
>> >> >
>> >> > #!sh
>> >> >
>> >> > cat modulesList | while read moduleName; do
>> >> >
>> >> >            echo $moduleName
>> >> >
>> >> >            A=1
>> >> >
>> >> >            while [ "$A" -lt "51" ]; do
>> >> >
>> >> >            L=`ls -1 build/test_report.${moduleName}_${A} | grep xml
>> >> | wc
>> >> > -l`
>> >> >
>> >> >            echo build/test_report.${moduleName}_${A} $L >>
>> >> > test_report_length
>> >> >
>> >> >            A=`expr $A + 1`
>> >> >
>> >> >            done
>> >> > done
>> >> >
>> >> > Thanks,
>> >> > Tanya
>> >> >
>> >> >
>> >>
>> >
>>
> 

Mime
View raw message