harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Yakushev" <andrey.yakus...@gmail.com>
Subject Re: [testing] M4 candidate (r603534) testing status.
Date Mon, 17 Dec 2007 15:02:54 GMT
My comments on the reliability suite results:

> * Reliability

There are several tests with known intermittent failures in this suite
which were not put to exclude list in order to control new regression
appearance. Thus several failures are expected.

>
> Failed on x86 platforms but they also failed on M3
> api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)

That's because BTI workspace wasn't updated on CC host.

> api.net.SingleConnectTest

Known intermittent test issue, the reason was described in HARMONY-5266.

>
> - Linux_x86:
>
> Regresions:
> api.kernel.object.ObjectFinalizeTest

exception after 1705 successful iterations - probably regression - I
couldn't reproduce it yet in my local runs.

> api.kernel.throwable.StackTraceExcptsTest

hang - could be similar to HARMONY-5288 issue

> api.text.DecimalFormat_Locales

hang - could be similar to HARMONY-5288 issue

> api.zip.ZipEntryIterateThreadingTest

Enable to open file - I saw that on M3 - not a regression.

> api.zip.ZlibTest

Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression

> vm.finalization.FinalizeThrowRestoreTest

VM crash after 142 successful iterations - new issue, could be similar
to HARMONY-5304 - regression

> vm.stack.StackUnwindTest

Hang after 1025 successful iterations - I saw it previously on M3,
probably HARMONY-4215 - not a regression

>
> They also failed on M3:
> api.nio.channels.filechannel.FileChannelMapTest
> api.nio.channels.filechannel.MappedByteBufferForceTest
> api.serialization.SerializableClassesTest
>

My local run but on the debug release gives only 3 failures:

api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223
successful iterations - I saw that intermittently on M3 - not
regression - probably this is HARMONY-4215
api.nio.channels.filechannel.MappedByteBufferForceTest - not regression -was on
vm.finalization.FinalizeThrowRestoreTest - assertion after 66
successful iterations - could be similar to HARMONY-5304

There are also several known failures in valid excluded tests (they
were not run on CC)

api.kernel.thread.LifeSimulationTest.Life
vm.classloading.ClassCastTest
vm.classloading.ClassReflectionTest

> - Windows_x86
>
> Regresions:
> api.kernel.string.InternThreadingTest - regression
> api.kernel.thread.Calculation.CalcTest
> api.kernel.thread.RecursiveThreadTest
> api.kernel.thread.StackTraceTest
> api.kernel.thread.Synchronization
> api.kernel.thread.ThreadArrayTest
> api.kernel.threadgroup.EnumerateTest
> api.net.DatagramTest
> api.serialization.SerializableClassesTest
> vm.stack.StackUnwindTest
>

My local run results on the release build showed mostly different failures:

api.net.NetClient - not regression - known intermittent failure - HARMONY-4215
api.net.SingleConnectTest - test issue - HARMONY-5266
api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215,
but could be a regression
api.io.IOTest - i/o failure - require investigation - probably regression
api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError
after 958 successful iterations
api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317
successful iterations
api.kernel.object.ObjectFinalizeTest - exception after 461 successful
iterations

There are also several failures in excluded tests (they were not run on CC)

vm.classloading.ClassReflectionTest - not regression - HARMONY-3680
vm.classloading.ClassAttributesTest - not regression - HARMONY-3680
api.kernel.thread.LifeSimulationTest.Life - not regression -
OutOfMemoryError after 21507 successful iterations - HARMONY-5266
api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172

> - Linux_x86_64:
>
> Regressions:
> vm.classloading.ClassCastTest

This is HARMONY-5304.I suggest fixing it now if possible.

My local run results on the debug build also showed 7 failures only in
excluded tests (they were not run on CC):

api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest
api.kernel.thread.LifeSimulationTest.Life
api.nio.channels.filechannel.MappedByteBufferForceTest
api.net.DatagramTest
vm.classloading.ClassAttributesTest
vm.classloading.ClassReflectionTest
vm.classloading.DelegationModelTest

Details are available at
http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM

My resume:

My tests run results:

Platform | failures | regressions | pass rate
Windows x86 | 11 failures | 4 regressions | 87.2%
Linux x86 | 6 failures | 1 regressions | 93%
Linux x86_64 | 7 failures | 1 regression | 92%

This is better results then M3 had. Usually on M3 we had had 11-13
failures: http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html

So I think the reliability in M4 was improved in general although we
still have an opportunity for improvements :)

Thanks,
Andrey


On 12/14/07, Stepan Mishura <stepan.mishura@gmail.com> wrote:
> Hi,
>
> Please see testing status[1] for r603534 below.
> Note to my comments:
> - I marked tests as regression if they passed on M3 [2]
> - I question tests as intermittent if they passed on the previous
> snapshot (r603433) [3]
> - There are some tests that failed on M3 too. (move them to the exclude lists?)
> - Not all failures have JIRA issue assigned so please help me to find
> related JIRA number (if there is any).
>
> All comments and clarifications are greatly appreciated!
>
> =====================================================
> We have the next suites failed on the snapshot.
>
> * Eclipse Unit Tests 3.3
>   - Linux_x86: 4 failures
>   - Windows_x86: pass rate 96.92%
>   - Linux_x86_64: pass rate 95.29%
>
> Would be nice to get failures evaluation.
> We definitely improved stability of the Eclipse Unit Test suite since
> M3 (Thanks to Vladimir Beliaev).
> But the are still some issues such as intermittent suite crashes (that
> make me very upset).
>
> * EGAx48:
>   - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?)
>   - Windows_x86:
>      Still no status – the scenario causes CC host down after running
> several hours.
>      Currently I can not figure out if this is the host's issue or
> problem introduced by Harmony.
>
>
> * EHWA(GUI)
>   - Linux_x86 & Linux_x86_64 : failed.
>   The scenario is quite unstable. Most probably I'll drop the
> scenario in M5 from snapshot test and use only EHWA-API.
>
>
> * Functional:
>      F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857)
>
>  - Linux_x86:
>      JComponent: intermittent?
>  - Linux_x86_64:
>      btest6353: fails on M3 too
>  - Windows_x86_64:
>      DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717,
> btest7214 : intermittent?
>      btest6353: fails on M3 too
>
> * GUT:
>    ContainerTest, JAASSecurityTest - fail on all platforms
>      HARMONY-4895 says that this bug in Geronimo and not a regression.
>
>  - Linux_x86:
>    HARMONY-5302 - says that this a bug in tests (I don't understand
> then why they passed on M3)
>    security\jaas\ConfigurationEntryTest
>    security\jaas\LoginKerberousNonGeronimoTest
>    security\jaas\LoginKerberousTest
>    security\jaas\LoginSQLTest
>    security\network\protocol\SubjectCarryingProtocolTest
>    security\remoting\jmx\RemoteLoginTest
>
>  - Linux_x86_64:   53 failures (11 on M3)
>  - Windows_x86_64: 22 failures (28 on M3)
>
>
> * JDKTools
>  - Windows_x86_64: 3 failures by TimeoutException.
>  Regression? I previously incorrectly reported that they are
> intermittent. They stably fail on M4 builds.
>
> * Reliability
>
>    Failed on x86 platforms but they also failed on M3
>    api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)
>    api.net.SingleConnectTest
>
>  - Linux_x86:
>
>    Regresions:
>    api.kernel.object.ObjectFinalizeTest
>    api.kernel.throwable.StackTraceExcptsTest
>    api.text.DecimalFormat_Locales
>    api.zip.ZipEntryIterateThreadingTest
>    api.zip.ZlibTest
>    vm.finalization.FinalizeThrowRestoreTest
>    vm.stack.StackUnwindTest
>
>    They also failed on M3:
>    api.nio.channels.filechannel.FileChannelMapTest
>    api.nio.channels.filechannel.MappedByteBufferForceTest
>    api.serialization.SerializableClassesTest
>
>  - Windows_x86
>
>    Regresions:
>    api.kernel.string.InternThreadingTest - regression
>    api.kernel.thread.Calculation.CalcTest
>    api.kernel.thread.RecursiveThreadTest
>    api.kernel.thread.StackTraceTest
>    api.kernel.thread.Synchronization
>    api.kernel.thread.ThreadArrayTest
>    api.kernel.threadgroup.EnumerateTest
>    api.net.DatagramTest
>    api.serialization.SerializableClassesTest
>    vm.stack.StackUnwindTest
>
>  - Linux_x86_64:
>
>    Regressions:
>    vm.classloading.ClassCastTest
>
> * Stress
>
>  - Linux_x86:
>    StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> test issue?)
>    Fragmentation, FragmentationFinalizer, FragmentationReference:
> HARMONY-5159 (wasn't run during M3 so not regression)
>
>  - Windows_x86:
>     StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> test issue?)
>     FragmentationReference: HARMONY-5159 (wasn't run during M3 so not
> regression)
>
>  - Linux_x86_64:
>    jni.arguments,jni.arrays - fails on M3 too so not regression / HARMONY-????
>    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
>
>  - Windows_x86_64:
>    NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression
>    Fragmentation, FragmentationFinalizer, FragmentationReference:
> HARMONY-5159 (wasn't run during M3 so not regression)
>    gc.mem.MemoryTest3 - regression / HARMONY-????
>    jni.stack - fails on M3 too so not regression / HARMONY-????
>    EVENT014- crash (HARMONY-????)/ regression
>    MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
>    StressThreads12Test - crash (HARMONY-????)/ regression
>
> * VTS VM
>  - Windows_x86:
>      dstore_20701 - intermittent?
>
> [1] http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html
> [2] http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
> [3] http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html
>
> Thanks,
> Stepan Mishura
> Intel Enterprise Solutions Software Division
>


-- 
Thanks,
Andrey

Mime
View raw message