harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ellison <t.p.elli...@gmail.com>
Subject Re: [testing] M4 candidate (r603534) testing status.
Date Fri, 14 Dec 2007 10:27:34 GMT
Stepan Mishura wrote:
> Hi,
> 
> Please see testing status[1] for r603534 below.
> Note to my comments:
> - I marked tests as regression if they passed on M3 [2]
> - I question tests as intermittent if they passed on the previous
> snapshot (r603433) [3]
> - There are some tests that failed on M3 too. (move them to the exclude lists?)
> - Not all failures have JIRA issue assigned so please help me to find
> related JIRA number (if there is any).
> 
> All comments and clarifications are greatly appreciated!
> 
> =====================================================
> We have the next suites failed on the snapshot.
> 
> * Eclipse Unit Tests 3.3
>    - Linux_x86: 4 failures
>    - Windows_x86: pass rate 96.92%
>    - Linux_x86_64: pass rate 95.29%
> 
> Would be nice to get failures evaluation.
> We definitely improved stability of the Eclipse Unit Test suite since
> M3 (Thanks to Vladimir Beliaev).
> But the are still some issues such as intermittent suite crashes (that
> make me very upset).

And yet these numbers are high pass rates, so it implies that the
milestone is quite stable.  Clearly if we were testing a release
candidate we would want to improve these even further.

> * EGAx48:
>    - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?)
>    - Windows_x86:
>       Still no status – the scenario causes CC host down after running
> several hours.
>       Currently I can not figure out if this is the host's issue or
> problem introduced by Harmony.

We should look into these to see if they are regressions.
I'll help.

> * EHWA(GUI)
>    - Linux_x86 & Linux_x86_64 : failed.
>    The scenario is quite unstable. Most probably I'll drop the
> scenario in M5 from snapshot test and use only EHWA-API.

Ok.

> * Functional:
>       F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857)
> 
>   - Linux_x86:
>       JComponent: intermittent?

As Andrey says, these should not stop us declaring M4.

>   - Linux_x86_64:
>       btest6353: fails on M3 too
>   - Windows_x86_64:
>       DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717,
> btest7214 : intermittent?
>       btest6353: fails on M3 too

Intermittent and non-regression failures on the 64-bit platforms I would
suggest we need to log in JIRA and target for the next milestone.

> * GUT:
>     ContainerTest, JAASSecurityTest - fail on all platforms
>       HARMONY-4895 says that this bug in Geronimo and not a regression.

Ok.

>   - Linux_x86:
>     HARMONY-5302 - says that this a bug in tests (I don't understand
> then why they passed on M3)
>     security\jaas\ConfigurationEntryTest
>     security\jaas\LoginKerberousNonGeronimoTest
>     security\jaas\LoginKerberousTest
>     security\jaas\LoginSQLTest
>     security\network\protocol\SubjectCarryingProtocolTest
>     security\remoting\jmx\RemoteLoginTest
> 
>   - Linux_x86_64:   53 failures (11 on M3)
>   - Windows_x86_64: 22 failures (28 on M3)

These need further investigation IMO.

> * JDKTools
>   - Windows_x86_64: 3 failures by TimeoutException.
>   Regression? I previously incorrectly reported that they are
> intermittent. They stably fail on M4 builds.

Assuming these pass on 32-bit platforms?
Is there a JIRA describing this one already?

> * Reliability
> 
>     Failed on x86 platforms but they also failed on M3
>     api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it)
>     api.net.SingleConnectTest
> 
>   - Linux_x86:
> 
>     Regresions:
>     api.kernel.object.ObjectFinalizeTest
>     api.kernel.throwable.StackTraceExcptsTest
>     api.text.DecimalFormat_Locales
>     api.zip.ZipEntryIterateThreadingTest
>     api.zip.ZlibTest
>     vm.finalization.FinalizeThrowRestoreTest
>     vm.stack.StackUnwindTest

Regressions :-(

>     They also failed on M3:
>     api.nio.channels.filechannel.FileChannelMapTest
>     api.nio.channels.filechannel.MappedByteBufferForceTest
>     api.serialization.SerializableClassesTest
> 
>   - Windows_x86
> 
>     Regresions:
>     api.kernel.string.InternThreadingTest - regression
>     api.kernel.thread.Calculation.CalcTest
>     api.kernel.thread.RecursiveThreadTest
>     api.kernel.thread.StackTraceTest
>     api.kernel.thread.Synchronization
>     api.kernel.thread.ThreadArrayTest
>     api.kernel.threadgroup.EnumerateTest
>     api.net.DatagramTest
>     api.serialization.SerializableClassesTest
>     vm.stack.StackUnwindTest

Regressions :-(

>   - Linux_x86_64:
> 
>     Regressions:
>     vm.classloading.ClassCastTest
> 
> * Stress
> 
>   - Linux_x86:
>     StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> test issue?)
>     Fragmentation, FragmentationFinalizer, FragmentationReference:
> HARMONY-5159 (wasn't run during M3 so not regression)
> 
>   - Windows_x86:
>      StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 /
> test issue?)
>      FragmentationReference: HARMONY-5159 (wasn't run during M3 so not
> regression)

Ok.

>   - Linux_x86_64:
>     jni.arguments,jni.arrays - fails on M3 too so not regression / HARMONY-????
>     MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
> 
>   - Windows_x86_64:
>     NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression
>     Fragmentation, FragmentationFinalizer, FragmentationReference:
> HARMONY-5159 (wasn't run during M3 so not regression)
>     gc.mem.MemoryTest3 - regression / HARMONY-????
>     jni.stack - fails on M3 too so not regression / HARMONY-????
>     EVENT014- crash (HARMONY-????)/ regression
>     MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression)
>     StressThreads12Test - crash (HARMONY-????)/ regression

A variety of things to investigate in there it seems.

> * VTS VM
>   - Windows_x86:
>       dstore_20701 - intermittent?
> 
> [1] http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html
> [2] http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html
> [3] http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html

Another great summary Stepan.  Thanks for doing this!

Regards,
Tim

Mime
View raw message