ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Gura (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-2080) JVM crashes on SunOS with SIGBUS (0xa) on frame [libjvm.so+0xc7c438] Unsafe_SetInt+0x14c
Date Thu, 10 Dec 2015 14:52:10 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051058#comment-15051058
] 

Andrey Gura commented on IGNITE-2080:
-------------------------------------

Yes. There is problem with {{GridUnsafeMap}} that uses unaligned pointers. It leads to JVM
crache on CAS operations:

{noformat}
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0xffffffff75fc07e0, pid=6039, tid=377199
#
# JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode solaris-sparc compressed
oops)
# Problematic frame:
# V  [libjvm.so+0x1c07e0]  Unsafe_CompareAndSwapLong+0x320
{noformat}

Stack trace:

{noformat}
Stack: [0xfffffffea9400000,0xfffffffea9500000],  sp=0xfffffffea94fdce0,  free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1c07e0]  Unsafe_CompareAndSwapLong+0x320
J 2924  sun.misc.Unsafe.compareAndSwapLong(Ljava/lang/Object;JJJ)Z (0 bytes) @ 0xffffffff727990d0
[0xffffffff72798fa0+0x130]
j  org.apache.ignite.internal.util.GridUnsafe.compareAndSwapLong(Ljava/lang/Object;JJJ)Z+8
j  org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.casLong(JJJ)Z+5
j  org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Entry.clearQueueAddress(JJLorg/apache/ignite/internal/util/offheap/unsafe/GridUnsafeMemory;)Z+9
j  org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.valuePointer(I[B)Lorg/apache/ignite/lang/IgniteBiTuple;+84
j  org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.valuePointer(I[B)Lorg/apache/ignite/lang/IgniteBiTuple;+7
j  org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.valuePointer(II[B)Lorg/apache/ignite/lang/IgniteBiTuple;+7
j  org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.valuePointer(Ljava/lang/String;ILorg/apache/ignite/internal/processors/cache/KeyCacheObject;[B)Lorg/apache/ignite/lang/IgniteBiTuple;+30
j  org.apache.ignite.internal.processors.cache.GridCacheSwapManager.readOffheapPointer(Lorg/apache/ignite/internal/processors/cache/GridCacheEntryEx;)Lorg/apache/ignite/internal/processors/cache/GridCacheSwapEntry;+51
J 5527 C2 org.apache.ignite.internal.processors.cache.GridCacheMapEntry.unswap(Z)Lorg/apache/ignite/internal/processors/cache/CacheObject;
(418 bytes) @ 0xffffffff72efa400 [0xffffffff72efa280+0x180]
J 8366 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.enlistWriteEntry(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/lang/Object;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;Ljavax/cache/expiry/ExpiryPolicy;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Lorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;JJLorg/apache/ignite/internal/processors/cache/GridCacheReturn;Ljava/util/Collection;ZZZZZZ)Z
(1438 bytes) @ 0xffffffff73950600 [0xffffffff73950280+0x380]
J 10758 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.enlistWrite(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/lang/Object;Ljavax/cache/expiry/ExpiryPolicy;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Lorg/apache/ignite/internal/processors/cache/GridCacheReturn;ZZZ)Lorg/apache/ignite/internal/IgniteInternalFuture;
(180 bytes) @ 0xffffffff7352695c [0xffffffff735267e0+0x17c]
J 9708 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.putAsync0(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Ljava/lang/Object;Ljava/lang/Object;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;Z[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;
(387 bytes) @ 0xffffffff733d79dc [0xffffffff733d76a0+0x33c]
J 8063 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.putAsync(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Ljava/lang/Object;Ljava/lang/Object;Z[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;
(14 bytes) @ 0xffffffff737c86ac [0xffffffff737c8660+0x4c]
j  org.apache.ignite.internal.processors.cache.GridCacheAdapter$34.op(Lorg/apache/ignite/internal/processors/cache/transactions/IgniteTxLocalAdapter;)Ljava/lang/Boolean;+27
j  org.apache.ignite.internal.processors.cache.GridCacheAdapter$34.op(Lorg/apache/ignite/internal/processors/cache/transactions/IgniteTxLocalAdapter;)Ljava/lang/Object;+2
j  org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(Lorg/apache/ignite/internal/processors/cache/GridCacheAdapter$SyncOp;)Ljava/lang/Object;+575
j  org.apache.ignite.internal.processors.cache.GridCacheAdapter.putIfAbsent(Ljava/lang/Object;Ljava/lang/Object;)Z+63
j  org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putIfAbsent(Ljava/lang/Object;Ljava/lang/Object;)Z+57
{noformat}

Problem places in code looks like (comments added by me):

{code:java}
mem.casLong(ptr + 12, qAddr, 0); // need align to 8 byte scale statically
{code}

And worst (comments added by me):

{code:java}
// HEADER hould be aligned by 8 byte scale statically
private static final int HEADER = 4 /*hash*/ + 4 /*key-size*/  + 4 /*value-size*/ + 8 /*queue-address*/
+ 8 /*next-address*/;

new IgniteBiTuple<>(addr + HEADER + keyLen, valLen); // keyLen should be aligned to
8 bytes scale at runtime.
{code}

I think that problem should be fixed as part of a separate issue because proposed fix solves
problems with marshallers and allows use Ignite on Solaris and HP-UX without offheap.

Thoughts?


> JVM crashes on SunOS with SIGBUS (0xa) on frame [libjvm.so+0xc7c438]  Unsafe_SetInt+0x14c
> -----------------------------------------------------------------------------------------
>
>                 Key: IGNITE-2080
>                 URL: https://issues.apache.org/jira/browse/IGNITE-2080
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 1.5, 1.4
>         Environment: SunOS 5.11, JDK 7
>            Reporter: Andrey Gura
>             Fix For: 1.5
>
>         Attachments: 054424_0_localhost.log, benchmark-offheap.properties, hs_err_pid10523.log
>
>
> JVM crashes on SunOS 5.11 with the following message:
> {noformat}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0xa) at pc=0xffffffff76a7c438, pid=10523, tid=652
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode solaris-sparc )
> # Problematic frame:
> # V  [libjvm.so+0xc7c438]  Unsafe_SetInt+0x14c
> {noformat}
> Stack trace:
> {noformat}
> Stack: [0xfffffff0f6f00000,0xfffffff0f7000000],  sp=0xfffffff0f6ffd1f0,  free space=1012k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V  [libjvm.so+0xc7c438]  Unsafe_SetInt+0x14c
> j  sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+-1763873616
> j  sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+0
> j  org.apache.ignite.internal.util.IgniteUtils.writeVersion([BJLorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;)J+135
> j  org.apache.ignite.internal.processors.cache.GridCacheSwapEntryImpl.marshal()[B+128
> j  org.apache.ignite.internal.processors.cache.GridCacheSwapManager.write(Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/nio/ByteBuffer;BLorg/apache/
> ignite/internal/processors/cache/version/GridCacheVersion;JJLorg/apache/ignite/lang/IgniteUuid;Lorg/apache/ignite/lang/IgniteUuid;)V+86
> j  org.apache.ignite.internal.processors.cache.GridCacheMapEntry.swap()V+333
> j  org.apache.ignite.internal.processors.cache.GridCacheMapEntry.evictInternal(ZLorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;[Lorg/apache/ignit
> e/internal/processors/cache/CacheEntryPredicate;)Z+122
> j  org.apache.ignite.internal.processors.cache.GridCacheEvictionManager.evict0(Lorg/apache/ignite/internal/processors/cache/GridCacheAdapter;Lorg/apache/ignite/internal
> /processors/cache/GridCacheEntryEx;Lorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredica
> te;Z)Z+117
> j  org.apache.ignite.internal.processors.cache.GridCacheEvictionManager.touch(Lorg/apache/ignite/internal/processors/cache/GridCacheEntryEx;Lorg/apache/ignite/internal/
> processors/affinity/AffinityTopologyVersion;)V+123
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.unlockEntries(Ljava/util/Collection;Lorg/apache/ignite/internal/processors/affi
> nity/AffinityTopologyVersion;)V+331
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/
> cache/distributed/dht/atomic/GridNearAtomicUpdateRequest;Lorg/apache/ignite/internal/util/typedef/CI2;)V+1023
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/c
> ache/distributed/dht/atomic/GridNearAtomicUpdateRequest;Lorg/apache/ignite/internal/util/typedef/CI2;)V+33
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapSingle(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/cache/
> distributed/dht/atomic/GridNearAtomicUpdateRequest;)V+28
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.access$1200(Lorg/apache/ignite/internal/processors/cache/distributed/dh
> t/atomic/GridNearAtomicUpdateFuture;Ljava/util/UUID;Lorg/apache/ignite/internal/processors/cache/distributed/dht/atomic/GridNearAtomicUpdateRequest;)V+3
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.map(Lorg/apache/ignite/internal/processors/affinity/Affinit
> yTopologyVersion;)V+595
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology()V+253
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map()V+60
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$18.apply()Lorg/apache/ignite/internal/IgniteInternalFuture;+4
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$18.apply()Ljava/lang/Object;+1
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(Lorg/apache/ignite/internal/util/typedef/CO;)Lorg/apache/ignite/internal/IgniteInternalFuture;+86
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsync0(Ljava/util/Map;Ljava/util/Map;[Ljava/lang/Object;Ljava/util/Map;Ljava/util/Map;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Z)Lorg/apache/ignite/internal/IgniteInternalFuture;+331
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;+20
> j  org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;+30
> j  org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Z+4
> j  org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(Ljava/lang/Object;Ljava/lang/Object;)Z+6
> j  org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(Ljava/lang/Object;Ljava/lang/Object;)V+68
> j  org.apache.ignite.yardstick.cache.IgnitePutBenchmark.test(Ljava/util/Map;)Z+28
> j  org.yardstickframework.impl.BenchmarkRunner$2.run()V+242
> j  java.lang.Thread.run()V+11
> v  ~StubRoutines::call_stub
> V  [libjvm.so+0x226c64]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x2fc
> V  [libjvm.so+0x7b1e68]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x1b8
> V  [libjvm.so+0x2d5e34]  void thread_entry(JavaThread*,Thread*)+0x164
> V  [libjvm.so+0xc55e2c]  void JavaThread::thread_main_inner()+0x94
> V  [libjvm.so+0x2d1824]  void JavaThread::run()+0x454
> V  [libjvm.so+0xb09264]  java_start+0x364
> {noformat}
> In order ot reproduce it need to do following:
> # Copy [^benchmark-offheap.properties] file into {{modules/yardstick/config}} directory.
> # Build yardstick {{mvn -B clean package -Prelease,benchmarks -DskipTests -pl modules/yardstick
-am}}
> # Run benchmark {{./bin/benchmark-drivers-start.sh config/benchmark-offheap.properties}}
(do it from {{yardstick}} module directory)
> The problem isn't related with JIT (reproduces with -Xint option).
> Also see attached logs and [core dump|https://drive.google.com/open?id=0Bz6LUqvKtoj6eklXTEp5aEh1MHM].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message