harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Rebriy" <pavel.reb...@gmail.com>
Subject Re: [testing] M4 candidate (r603534) testing status.
Date Mon, 17 Dec 2007 16:17:45 GMT
On 17/12/2007, Andrey Yakushev <andrey.yakushev@gmail.com> wrote:
>
> My comments on the reliability suite results:
>
> > * Reliability
>
> There are several tests with known intermittent failures in this suite
> which were not put to exclude list in order to control new regression
> appearance. Thus several failures are expected.
>
> >
> > Failed on x86 platforms but they also failed on M3
> > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)
>
> That's because BTI workspace wasn't updated on CC host.
>
> > api.net.SingleConnectTest
>
> Known intermittent test issue, the reason was described in HARMONY-5266.
>
> >
> > - Linux_x86:
> >
> > Regresions:
> > api.kernel.object.ObjectFinalizeTest
>
> exception after 1705 successful iterations - probably regression - I
> couldn't reproduce it yet in my local runs.
>
> > api.kernel.throwable.StackTraceExcptsTest
>
> hang - could be similar to HARMONY-5288 issue
>
> > api.text.DecimalFormat_Locales
>
> hang - could be similar to HARMONY-5288 issue
>
> > api.zip.ZipEntryIterateThreadingTest
>
> Enable to open file - I saw that on M3 - not a regression.
>
> > api.zip.ZlibTest
>
> Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression
>
> > vm.finalization.FinalizeThrowRestoreTest
>
> VM crash after 142 successful iterations - new issue, could be similar
> to HARMONY-5304 - regression
>
> > vm.stack.StackUnwindTest
>
> Hang after 1025 successful iterations - I saw it previously on M3,
> probably HARMONY-4215 - not a regression
>
> >
> > They also failed on M3:
> > api.nio.channels.filechannel.FileChannelMapTest
> > api.nio.channels.filechannel.MappedByteBufferForceTest
> > api.serialization.SerializableClassesTest
> >
>
> My local run but on the debug release gives only 3 failures:
>
> api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223
> successful iterations - I saw that intermittently on M3 - not
> regression - probably this is HARMONY-4215
> api.nio.channels.filechannel.MappedByteBufferForceTest - not regression
> -was on
> vm.finalization.FinalizeThrowRestoreTest - assertion after 66
> successful iterations - could be similar to HARMONY-5304
>
> There are also several known failures in valid excluded tests (they
> were not run on CC)
>
> api.kernel.thread.LifeSimulationTest.Life
> vm.classloading.ClassCastTest
> vm.classloading.ClassReflectionTest
>
> > - Windows_x86
> >
> > Regresions:
> > api.kernel.string.InternThreadingTest - regression
> > api.kernel.thread.Calculation.CalcTest
> > api.kernel.thread.RecursiveThreadTest
> > api.kernel.thread.StackTraceTest
> > api.kernel.thread.Synchronization
> > api.kernel.thread.ThreadArrayTest
> > api.kernel.threadgroup.EnumerateTest
> > api.net.DatagramTest
> > api.serialization.SerializableClassesTest
> > vm.stack.StackUnwindTest
> >
>
> My local run results on the release build showed mostly different
> failures:
>
> api.net.NetClient - not regression - known intermittent failure -
> HARMONY-4215
> api.net.SingleConnectTest - test issue - HARMONY-5266
> api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215,
> but could be a regression
> api.io.IOTest - i/o failure - require investigation - probably regression
> api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError
> after 958 successful iterations
> api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317
> successful iterations
> api.kernel.object.ObjectFinalizeTest - exception after 461 successful
> iterations
>
> There are also several failures in excluded tests (they were not run on
> CC)
>
> vm.classloading.ClassReflectionTest - not regression - HARMONY-3680
> vm.classloading.ClassAttributesTest - not regression - HARMONY-3680
> api.kernel.thread.LifeSimulationTest.Life - not regression -
> OutOfMemoryError after 21507 successful iterations - HARMONY-5266
> api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172
>
> > - Linux_x86_64:
> >
> > Regressions:
> > vm.classloading.ClassCastTest
>
> This is HARMONY-5304.I suggest fixing it now if possible.


Ok, I'm looking at it.

My local run results on the debug build also showed 7 failures only in
> excluded tests (they were not run on CC):
>
> api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest
> api.kernel.thread.LifeSimulationTest.Life
> api.nio.channels.filechannel.MappedByteBufferForceTest
> api.net.DatagramTest
> vm.classloading.ClassAttributesTest
> vm.classloading.ClassReflectionTest
> vm.classloading.DelegationModelTest
>
> Details are available at
> http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM
>
> My resume:
>
> My tests run results:
>
> Platform | failures | regressions | pass rate
> Windows x86 | 11 failures | 4 regressions | 87.2%
> Linux x86 | 6 failures | 1 regressions | 93%
> Linux x86_64 | 7 failures | 1 regression | 92%
>
> This is better results then M3 had. Usually on M3 we had had 11-13
> failures:
> http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
>
> So I think the reliability in M4 was improved in general although we
> still have an opportunity for improvements :)
>
> Thanks,
> Andrey
>
>
> On 12/14/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> > Hi,
> >
> > Please see testing status[1] for r603534 below.
> > Note to my comments:
> > - I marked tests as regression if they passed on M3 [2]
> > - I question tests as intermittent if they passed on the previous
> > snapshot (r603433) [3]
> > - There are some tests that failed on M3 too. (move them to the exclude
> lists?)
> > - Not all failures have JIRA issue assigned so please help me to find
> > related JIRA number (if there is any).
> >
> > All comments and clarifications are greatly appreciated!
> >
> > =====================================================
> > We have the next suites failed on the snapshot.
> >
> > * Eclipse Unit Tests 3.3
> >   - Linux_x86: 4 failures
> >   - Windows_x86: pass rate 96.92%
> >   - Linux_x86_64: pass rate 95.29%
> >
> > Would be nice to get failures evaluation.
> > We definitely improved stability of the Eclipse Unit Test suite since
> > M3 (Thanks to Vladimir Beliaev).
> > But the are still some issues such as intermittent suite crashes (that
> > make me very upset).
> >
> > * EGAx48:
> >   - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?)
> >   - Windows_x86:
> >      Still no status – the scenario causes CC host down after running
> > several hours.
> >      Currently I can not figure out if this is the host's issue or
> > problem introduced by Harmony.
> >
> >
> > * EHWA(GUI)
> >   - Linux_x86 & Linux_x86_64 : failed.
> >   The scenario is quite unstable. Most probably I'll drop the
> > scenario in M5 from snapshot test and use only EHWA-API.
> >
> >
> > * Functional:
> >      F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857)
> >
> >  - Linux_x86:
> >      JComponent: intermittent?
> >  - Linux_x86_64:
> >      btest6353: fails on M3 too
> >  - Windows_x86_64:
> >      DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717,
> > btest7214 : intermittent?
> >      btest6353: fails on M3 too
> >
> > * GUT:
> >    ContainerTest, JAASSecurityTest - fail on all platforms
> >      HARMONY-4895 says that this bug in Geronimo and not a regression.
> >
> >  - Linux_x86:
> >    HARMONY-5302 - says that this a bug in tests (I don't understand
> > then why they passed on M3)
> >    security\jaas\ConfigurationEntryTest
> >    security\jaas\LoginKerberousNonGeronimoTest
> >    security\jaas\LoginKerberousTest
> >    security\jaas\LoginSQLTest
> >    security\network\protocol\SubjectCarryingProtocolTest
> >    security\remoting\jmx\RemoteLoginTest
> >
> >  - Linux_x86_64:   53 failures (11 on M3)
> >  - Windows_x86_64: 22 failures (28 on M3)
> >
> >
> > * JDKTools
> >  - Windows_x86_64: 3 failures by TimeoutException.
> >  Regression? I previously incorrectly reported that they are
> > intermittent. They stably fail on M4 builds.
> >
> > * Reliability
> >
> >    Failed on x86 platforms but they also failed on M3
> >    api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed
> it)
> >    api.net.SingleConnectTest
> >
> >  - Linux_x86:
> >
> >    Regresions:
> >    api.kernel.object.ObjectFinalizeTest
> >    api.kernel.throwable.StackTraceExcptsTest
> >    api.text.DecimalFormat_Locales
> >    api.zip.ZipEntryIterateThreadingTest
> >    api.zip.ZlibTest
> >    vm.finalization.FinalizeThrowRestoreTest
> >    vm.stack.StackUnwindTest
> >
> >    They also failed on M3:
> >    api.nio.channels.filechannel.FileChannelMapTest
> >    api.nio.channels.filechannel.MappedByteBufferForceTest
> >    api.serialization.SerializableClassesTest
> >
> >  - Windows_x86
> >
> >    Regresions:
> >    api.kernel.string.InternThreadingTest - regression
> >    api.kernel.thread.Calculation.CalcTest
> >    api.kernel.thread.RecursiveThreadTest
> >    api.kernel.thread.StackTraceTest
> >    api.kernel.thread.Synchronization
> >    api.kernel.thread.ThreadArrayTest
> >    api.kernel.threadgroup.EnumerateTest
> >    api.net.DatagramTest
> >    api.serialization.SerializableClassesTest
> >    vm.stack.StackUnwindTest
> >
> >  - Linux_x86_64:
> >
> >    Regressions:
> >    vm.classloading.ClassCastTest
> >
> > * Stress
> >
> >  - Linux_x86:
> >    StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> > test issue?)
> >    Fragmentation, FragmentationFinalizer, FragmentationReference:
> > HARMONY-5159 (wasn't run during M3 so not regression)
> >
> >  - Windows_x86:
> >     StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> > test issue?)
> >     FragmentationReference: HARMONY-5159 (wasn't run during M3 so not
> > regression)
> >
> >  - Linux_x86_64:
> >    jni.arguments,jni.arrays - fails on M3 too so not regression /
> HARMONY-????
> >    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not
> regression)
> >
> >  - Windows_x86_64:
> >    NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression
> >    Fragmentation, FragmentationFinalizer, FragmentationReference:
> > HARMONY-5159 (wasn't run during M3 so not regression)
> >    gc.mem.MemoryTest3 - regression / HARMONY-????
> >    jni.stack - fails on M3 too so not regression / HARMONY-????
> >    EVENT014- crash (HARMONY-????)/ regression
> >    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not
> regression)
> >    StressThreads12Test - crash (HARMONY-????)/ regression
> >
> > * VTS VM
> >  - Windows_x86:
> >      dstore_20701 - intermittent?
> >
> > [1]
> http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html
> > [2]
> http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
> > [3]
> http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html
> >
> > Thanks,
> > Stepan Mishura
> > Intel Enterprise Solutions Software Division
> >
>
>
> --
> Thanks,
> Andrey
>



-- 
Best regards,
Pavel Rebriy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message