harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Yakushev" <andrey.yakus...@gmail.com>
Subject Re: [testing] M4 candidate (r603534) testing status.
Date Tue, 18 Dec 2007 11:10:52 GMT
On 12/18/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> Hi Andrey,
>
> Thanks for your evaluation. See my comments below.
>
> On 12/17/07, Andrey Yakushev <andrey.yakushev@gmail.com> wrote:
> > My comments on the reliability suite results:
> >
> > > * Reliability
> >
> > There are several tests with known intermittent failures in this suite
> > which were not put to exclude list in order to control new regression
> > appearance. Thus several failures are expected.
> >
> > >
> > > Failed on x86 platforms but they also failed on M3
> > > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)
> >
> > That's because BTI workspace wasn't updated on CC host.
> >
>
> OK. I'll fix it.
>
> > > api.net.SingleConnectTest
> >
> > Known intermittent test issue, the reason was described in HARMONY-5266.
> >
>
> HARMONY-5266 is about HttpConnectionTest.

Yes, But I noted that the same race condition exists in
SingleConnectTest and MultiConnectTest. I will open the corresponding
JIRA ASAP.

>
> > >
> > > - Linux_x86:
> > >
> > > Regresions:
> > > api.kernel.object.ObjectFinalizeTest
> >
> > exception after 1705 successful iterations - probably regression - I
> > couldn't reproduce it yet in my local runs.
> >
> > > api.kernel.throwable.StackTraceExcptsTest
> >
> > hang - could be similar to HARMONY-5288 issue
> >
> > > api.text.DecimalFormat_Locales
> >
> > hang - could be similar to HARMONY-5288 issue
> >
> > > api.zip.ZipEntryIterateThreadingTest
> >
> > Enable to open file - I saw that on M3 - not a regression.
> >
>
> Why not to exclude tests that fails on M3 too?

It's infrequent intermittent failure, presumably covered by
HARMONY-4215. It was not excluded for new regressions monitoring. By
the way, all such rare failures were excluded for Linux x86_64
platform and it's pretty long list - 18 tests currently.

>
> > > api.zip.ZlibTest
> >
> > Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression
> >
> > > vm.finalization.FinalizeThrowRestoreTest
> >
> > VM crash after 142 successful iterations - new issue, could be similar
> > to HARMONY-5304 - regression
> >
> > > vm.stack.StackUnwindTest
> >
> > Hang after 1025 successful iterations - I saw it previously on M3,
> > probably HARMONY-4215 - not a regression
> >
> > >
> > > They also failed on M3:
> > > api.nio.channels.filechannel.FileChannelMapTest
> > > api.nio.channels.filechannel.MappedByteBufferForceTest
> > > api.serialization.SerializableClassesTest
> > >
> >
> > My local run but on the debug release gives only 3 failures:
> >
>
> I think the snapshot build should be tried first to reproduce the failures.

Yes, you are right, but I used default "drlvm" as usual, and it turned
out to be the debug build. It was not M4 release build checking, it
was my usual status checking over weekend.

>
> > api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223
> > successful iterations - I saw that intermittently on M3 - not
> > regression - probably this is HARMONY-4215
> > api.nio.channels.filechannel.MappedByteBufferForceTest - not regression -was on
> > vm.finalization.FinalizeThrowRestoreTest - assertion after 66
> > successful iterations - could be similar to HARMONY-5304
> >
> > There are also several known failures in valid excluded tests (they
> > were not run on CC)
> >
> > api.kernel.thread.LifeSimulationTest.Life
> > vm.classloading.ClassCastTest
> > vm.classloading.ClassReflectionTest
> >
> > > - Windows_x86
> > >
> > > Regresions:
> > > api.kernel.string.InternThreadingTest - regression
> > > api.kernel.thread.Calculation.CalcTest
> > > api.kernel.thread.RecursiveThreadTest
> > > api.kernel.thread.StackTraceTest
> > > api.kernel.thread.Synchronization
> > > api.kernel.thread.ThreadArrayTest
> > > api.kernel.threadgroup.EnumerateTest
> > > api.net.DatagramTest
> > > api.serialization.SerializableClassesTest
> > > vm.stack.StackUnwindTest
> > >
> >
> > My local run results on the release build showed mostly different failures:
> >
>
> Did you mean snapshot build?

Yes, here I used it via "jvm.under.test" option.

> > api.net.NetClient - not regression - known intermittent failure - HARMONY-4215
> > api.net.SingleConnectTest - test issue - HARMONY-5266
> > api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215,
> > but could be a regression
> > api.io.IOTest - i/o failure - require investigation - probably regression
> > api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError
> > after 958 successful iterations
> > api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317
> > successful iterations
> > api.kernel.object.ObjectFinalizeTest - exception after 461 successful
> > iterations
> >
> > There are also several failures in excluded tests (they were not run on CC)
> >
> > vm.classloading.ClassReflectionTest - not regression - HARMONY-3680
> > vm.classloading.ClassAttributesTest - not regression - HARMONY-3680
> > api.kernel.thread.LifeSimulationTest.Life - not regression -
> > OutOfMemoryError after 21507 successful iterations - HARMONY-5266
> > api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172
> >
> > > - Linux_x86_64:
> > >
> > > Regressions:
> > > vm.classloading.ClassCastTest
> >
> > This is HARMONY-5304.I suggest fixing it now if possible.
>
> Is it blocker for releasing M4?

I don't think so, it's rather uncommon failure. I saw Pavel is trying
to fix this. But it would be nice to fix it, because this is a
regression as compared with M3.

>
> >
> > My local run results on the debug build also showed 7 failures only in
> > excluded tests (they were not run on CC):
> >
> > api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest
> > api.kernel.thread.LifeSimulationTest.Life
> > api.nio.channels.filechannel.MappedByteBufferForceTest
> > api.net.DatagramTest
> > vm.classloading.ClassAttributesTest
> > vm.classloading.ClassReflectionTest
> > vm.classloading.DelegationModelTest
> >
> > Details are available at
> > http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM
> >
>
> Wow, the wiki page really informative! (BTW, s/bag/bug/ should be applied to it)

Corrected.

>
> > My resume:
> >
> > My tests run results:
> >
> > Platform | failures | regressions | pass rate
> > Windows x86 | 11 failures | 4 regressions | 87.2%
> > Linux x86 | 6 failures | 1 regressions | 93%
> > Linux x86_64 | 7 failures | 1 regression | 92%
> >
> > This is better results then M3 had. Usually on M3 we had had 11-13
> > failures: http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
> >
> > So I think the reliability in M4 was improved in general although we
> > still have an opportunity for improvements :)
> >
>
> Thanks for the summary!
>
> -Stepan.
>
> > Thanks,
> > Andrey
> >
> >
> > On 12/14/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > > Hi,
> > >
> > > Please see testing status[1] for r603534 below.
> > > Note to my comments:
> > > - I marked tests as regression if they passed on M3 [2]
> > > - I question tests as intermittent if they passed on the previous
> > > snapshot (r603433) [3]
> > > - There are some tests that failed on M3 too. (move them to the exclude lists?)
> > > - Not all failures have JIRA issue assigned so please help me to find
> > > related JIRA number (if there is any).
> > >
> > > All comments and clarifications are greatly appreciated!
> > >
> > > =====================================================
> > > We have the next suites failed on the snapshot.
> > >
> > > * Eclipse Unit Tests 3.3
> > >   - Linux_x86: 4 failures
> > >   - Windows_x86: pass rate 96.92%
> > >   - Linux_x86_64: pass rate 95.29%
> > >
> > > Would be nice to get failures evaluation.
> > > We definitely improved stability of the Eclipse Unit Test suite since
> > > M3 (Thanks to Vladimir Beliaev).
> > > But the are still some issues such as intermittent suite crashes (that
> > > make me very upset).
> > >
> > > * EGAx48:
> > >   - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?)
> > >   - Windows_x86:
> > >      Still no status – the scenario causes CC host down after running
> > > several hours.
> > >      Currently I can not figure out if this is the host's issue or
> > > problem introduced by Harmony.
> > >
> > >
> > > * EHWA(GUI)
> > >   - Linux_x86 & Linux_x86_64 : failed.
> > >   The scenario is quite unstable. Most probably I'll drop the
> > > scenario in M5 from snapshot test and use only EHWA-API.
> > >
> > >
> > > * Functional:
> > >      F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857)
> > >
> > >  - Linux_x86:
> > >      JComponent: intermittent?
> > >  - Linux_x86_64:
> > >      btest6353: fails on M3 too
> > >  - Windows_x86_64:
> > >      DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717,
> > > btest7214 : intermittent?
> > >      btest6353: fails on M3 too
> > >
> > > * GUT:
> > >    ContainerTest, JAASSecurityTest - fail on all platforms
> > >      HARMONY-4895 says that this bug in Geronimo and not a regression.
> > >
> > >  - Linux_x86:
> > >    HARMONY-5302 - says that this a bug in tests (I don't understand
> > > then why they passed on M3)
> > >    security\jaas\ConfigurationEntryTest
> > >    security\jaas\LoginKerberousNonGeronimoTest
> > >    security\jaas\LoginKerberousTest
> > >    security\jaas\LoginSQLTest
> > >    security\network\protocol\SubjectCarryingProtocolTest
> > >    security\remoting\jmx\RemoteLoginTest
> > >
> > >  - Linux_x86_64:   53 failures (11 on M3)
> > >  - Windows_x86_64: 22 failures (28 on M3)
> > >
> > >
> > > * JDKTools
> > >  - Windows_x86_64: 3 failures by TimeoutException.
> > >  Regression? I previously incorrectly reported that they are
> > > intermittent. They stably fail on M4 builds.
> > >
> > > * Reliability
> > >
> > >    Failed on x86 platforms but they also failed on M3
> > >    api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)
> > >    api.net.SingleConnectTest
> > >
> > >  - Linux_x86:
> > >
> > >    Regresions:
> > >    api.kernel.object.ObjectFinalizeTest
> > >    api.kernel.throwable.StackTraceExcptsTest
> > >    api.text.DecimalFormat_Locales
> > >    api.zip.ZipEntryIterateThreadingTest
> > >    api.zip.ZlibTest
> > >    vm.finalization.FinalizeThrowRestoreTest
> > >    vm.stack.StackUnwindTest
> > >
> > >    They also failed on M3:
> > >    api.nio.channels.filechannel.FileChannelMapTest
> > >    api.nio.channels.filechannel.MappedByteBufferForceTest
> > >    api.serialization.SerializableClassesTest
> > >
> > >  - Windows_x86
> > >
> > >    Regresions:
> > >    api.kernel.string.InternThreadingTest - regression
> > >    api.kernel.thread.Calculation.CalcTest
> > >    api.kernel.thread.RecursiveThreadTest
> > >    api.kernel.thread.StackTraceTest
> > >    api.kernel.thread.Synchronization
> > >    api.kernel.thread.ThreadArrayTest
> > >    api.kernel.threadgroup.EnumerateTest
> > >    api.net.DatagramTest
> > >    api.serialization.SerializableClassesTest
> > >    vm.stack.StackUnwindTest
> > >
> > >  - Linux_x86_64:
> > >
> > >    Regressions:
> > >    vm.classloading.ClassCastTest
> > >
> > > * Stress
> > >
> > >  - Linux_x86:
> > >    StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> > > test issue?)
> > >    Fragmentation, FragmentationFinalizer, FragmentationReference:
> > > HARMONY-5159 (wasn't run during M3 so not regression)
> > >
> > >  - Windows_x86:
> > >     StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> > > test issue?)
> > >     FragmentationReference: HARMONY-5159 (wasn't run during M3 so not
> > > regression)
> > >
> > >  - Linux_x86_64:
> > >    jni.arguments,jni.arrays - fails on M3 too so not regression / HARMONY-????
> > >    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
> > >
> > >  - Windows_x86_64:
> > >    NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression
> > >    Fragmentation, FragmentationFinalizer, FragmentationReference:
> > > HARMONY-5159 (wasn't run during M3 so not regression)
> > >    gc.mem.MemoryTest3 - regression / HARMONY-????
> > >    jni.stack - fails on M3 too so not regression / HARMONY-????
> > >    EVENT014- crash (HARMONY-????)/ regression
> > >    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
> > >    StressThreads12Test - crash (HARMONY-????)/ regression
> > >
> > > * VTS VM
> > >  - Windows_x86:
> > >      dstore_20701 - intermittent?
> > >
> > > [1] http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html
> > > [2] http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
> > > [3] http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html
> > >
> > > Thanks,
> > > Stepan Mishura
> > > Intel Enterprise Solutions Software Division
> > >
> >
> >
> > --
> > Thanks,
> > Andrey
> >
>
>
> --
> Stepan Mishura
> Intel Enterprise Solutions Software Division
>


-- 
Thanks,
Andrey

Mime
View raw message