Return-Path: X-Original-To: apmail-ignite-issues-archive@minotaur.apache.org Delivered-To: apmail-ignite-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5A8FE186EE for ; Thu, 10 Dec 2015 14:52:11 +0000 (UTC) Received: (qmail 60952 invoked by uid 500); 10 Dec 2015 14:52:11 -0000 Delivered-To: apmail-ignite-issues-archive@ignite.apache.org Received: (qmail 60923 invoked by uid 500); 10 Dec 2015 14:52:11 -0000 Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list issues@ignite.apache.org Received: (qmail 60900 invoked by uid 99); 10 Dec 2015 14:52:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Dec 2015 14:52:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id F04152C03DA for ; Thu, 10 Dec 2015 14:52:10 +0000 (UTC) Date: Thu, 10 Dec 2015 14:52:10 +0000 (UTC) From: "Andrey Gura (JIRA)" To: issues@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (IGNITE-2080) JVM crashes on SunOS with SIGBUS (0xa) on frame [libjvm.so+0xc7c438] Unsafe_SetInt+0x14c MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/IGNITE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051058#comment-15051058 ] Andrey Gura commented on IGNITE-2080: ------------------------------------- Yes. There is problem with {{GridUnsafeMap}} that uses unaligned pointers. It leads to JVM crache on CAS operations: {noformat} # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0xa) at pc=0xffffffff75fc07e0, pid=6039, tid=377199 # # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode solaris-sparc compressed oops) # Problematic frame: # V [libjvm.so+0x1c07e0] Unsafe_CompareAndSwapLong+0x320 {noformat} Stack trace: {noformat} Stack: [0xfffffffea9400000,0xfffffffea9500000], sp=0xfffffffea94fdce0, free space=1015k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1c07e0] Unsafe_CompareAndSwapLong+0x320 J 2924 sun.misc.Unsafe.compareAndSwapLong(Ljava/lang/Object;JJJ)Z (0 bytes) @ 0xffffffff727990d0 [0xffffffff72798fa0+0x130] j org.apache.ignite.internal.util.GridUnsafe.compareAndSwapLong(Ljava/lang/Object;JJJ)Z+8 j org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.casLong(JJJ)Z+5 j org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Entry.clearQueueAddress(JJLorg/apache/ignite/internal/util/offheap/unsafe/GridUnsafeMemory;)Z+9 j org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.valuePointer(I[B)Lorg/apache/ignite/lang/IgniteBiTuple;+84 j org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.valuePointer(I[B)Lorg/apache/ignite/lang/IgniteBiTuple;+7 j org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.valuePointer(II[B)Lorg/apache/ignite/lang/IgniteBiTuple;+7 j org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.valuePointer(Ljava/lang/String;ILorg/apache/ignite/internal/processors/cache/KeyCacheObject;[B)Lorg/apache/ignite/lang/IgniteBiTuple;+30 j org.apache.ignite.internal.processors.cache.GridCacheSwapManager.readOffheapPointer(Lorg/apache/ignite/internal/processors/cache/GridCacheEntryEx;)Lorg/apache/ignite/internal/processors/cache/GridCacheSwapEntry;+51 J 5527 C2 org.apache.ignite.internal.processors.cache.GridCacheMapEntry.unswap(Z)Lorg/apache/ignite/internal/processors/cache/CacheObject; (418 bytes) @ 0xffffffff72efa400 [0xffffffff72efa280+0x180] J 8366 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.enlistWriteEntry(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/lang/Object;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;Ljavax/cache/expiry/ExpiryPolicy;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Lorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;JJLorg/apache/ignite/internal/processors/cache/GridCacheReturn;Ljava/util/Collection;ZZZZZZ)Z (1438 bytes) @ 0xffffffff73950600 [0xffffffff73950280+0x380] J 10758 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.enlistWrite(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/lang/Object;Ljavax/cache/expiry/ExpiryPolicy;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Lorg/apache/ignite/internal/processors/cache/GridCacheReturn;ZZZ)Lorg/apache/ignite/internal/IgniteInternalFuture; (180 bytes) @ 0xffffffff7352695c [0xffffffff735267e0+0x17c] J 9708 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.putAsync0(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Ljava/lang/Object;Ljava/lang/Object;Ljavax/cache/processor/EntryProcessor;[Ljava/lang/Object;Z[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture; (387 bytes) @ 0xffffffff733d79dc [0xffffffff733d76a0+0x33c] J 8063 C2 org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.putAsync(Lorg/apache/ignite/internal/processors/cache/GridCacheContext;Ljava/lang/Object;Ljava/lang/Object;Z[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture; (14 bytes) @ 0xffffffff737c86ac [0xffffffff737c8660+0x4c] j org.apache.ignite.internal.processors.cache.GridCacheAdapter$34.op(Lorg/apache/ignite/internal/processors/cache/transactions/IgniteTxLocalAdapter;)Ljava/lang/Boolean;+27 j org.apache.ignite.internal.processors.cache.GridCacheAdapter$34.op(Lorg/apache/ignite/internal/processors/cache/transactions/IgniteTxLocalAdapter;)Ljava/lang/Object;+2 j org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(Lorg/apache/ignite/internal/processors/cache/GridCacheAdapter$SyncOp;)Ljava/lang/Object;+575 j org.apache.ignite.internal.processors.cache.GridCacheAdapter.putIfAbsent(Ljava/lang/Object;Ljava/lang/Object;)Z+63 j org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putIfAbsent(Ljava/lang/Object;Ljava/lang/Object;)Z+57 {noformat} Problem places in code looks like (comments added by me): {code:java} mem.casLong(ptr + 12, qAddr, 0); // need align to 8 byte scale statically {code} And worst (comments added by me): {code:java} // HEADER hould be aligned by 8 byte scale statically private static final int HEADER = 4 /*hash*/ + 4 /*key-size*/ + 4 /*value-size*/ + 8 /*queue-address*/ + 8 /*next-address*/; new IgniteBiTuple<>(addr + HEADER + keyLen, valLen); // keyLen should be aligned to 8 bytes scale at runtime. {code} I think that problem should be fixed as part of a separate issue because proposed fix solves problems with marshallers and allows use Ignite on Solaris and HP-UX without offheap. Thoughts? > JVM crashes on SunOS with SIGBUS (0xa) on frame [libjvm.so+0xc7c438] Unsafe_SetInt+0x14c > ----------------------------------------------------------------------------------------- > > Key: IGNITE-2080 > URL: https://issues.apache.org/jira/browse/IGNITE-2080 > Project: Ignite > Issue Type: Bug > Affects Versions: 1.5, 1.4 > Environment: SunOS 5.11, JDK 7 > Reporter: Andrey Gura > Fix For: 1.5 > > Attachments: 054424_0_localhost.log, benchmark-offheap.properties, hs_err_pid10523.log > > > JVM crashes on SunOS 5.11 with the following message: > {noformat} > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGBUS (0xa) at pc=0xffffffff76a7c438, pid=10523, tid=652 > # > # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode solaris-sparc ) > # Problematic frame: > # V [libjvm.so+0xc7c438] Unsafe_SetInt+0x14c > {noformat} > Stack trace: > {noformat} > Stack: [0xfffffff0f6f00000,0xfffffff0f7000000], sp=0xfffffff0f6ffd1f0, free space=1012k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) > V [libjvm.so+0xc7c438] Unsafe_SetInt+0x14c > j sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+-1763873616 > j sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+0 > j org.apache.ignite.internal.util.IgniteUtils.writeVersion([BJLorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;)J+135 > j org.apache.ignite.internal.processors.cache.GridCacheSwapEntryImpl.marshal()[B+128 > j org.apache.ignite.internal.processors.cache.GridCacheSwapManager.write(Lorg/apache/ignite/internal/processors/cache/KeyCacheObject;Ljava/nio/ByteBuffer;BLorg/apache/ > ignite/internal/processors/cache/version/GridCacheVersion;JJLorg/apache/ignite/lang/IgniteUuid;Lorg/apache/ignite/lang/IgniteUuid;)V+86 > j org.apache.ignite.internal.processors.cache.GridCacheMapEntry.swap()V+333 > j org.apache.ignite.internal.processors.cache.GridCacheMapEntry.evictInternal(ZLorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;[Lorg/apache/ignit > e/internal/processors/cache/CacheEntryPredicate;)Z+122 > j org.apache.ignite.internal.processors.cache.GridCacheEvictionManager.evict0(Lorg/apache/ignite/internal/processors/cache/GridCacheAdapter;Lorg/apache/ignite/internal > /processors/cache/GridCacheEntryEx;Lorg/apache/ignite/internal/processors/cache/version/GridCacheVersion;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredica > te;Z)Z+117 > j org.apache.ignite.internal.processors.cache.GridCacheEvictionManager.touch(Lorg/apache/ignite/internal/processors/cache/GridCacheEntryEx;Lorg/apache/ignite/internal/ > processors/affinity/AffinityTopologyVersion;)V+123 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.unlockEntries(Ljava/util/Collection;Lorg/apache/ignite/internal/processors/affi > nity/AffinityTopologyVersion;)V+331 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/ > cache/distributed/dht/atomic/GridNearAtomicUpdateRequest;Lorg/apache/ignite/internal/util/typedef/CI2;)V+1023 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/c > ache/distributed/dht/atomic/GridNearAtomicUpdateRequest;Lorg/apache/ignite/internal/util/typedef/CI2;)V+33 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapSingle(Ljava/util/UUID;Lorg/apache/ignite/internal/processors/cache/ > distributed/dht/atomic/GridNearAtomicUpdateRequest;)V+28 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.access$1200(Lorg/apache/ignite/internal/processors/cache/distributed/dh > t/atomic/GridNearAtomicUpdateFuture;Ljava/util/UUID;Lorg/apache/ignite/internal/processors/cache/distributed/dht/atomic/GridNearAtomicUpdateRequest;)V+3 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.map(Lorg/apache/ignite/internal/processors/affinity/Affinit > yTopologyVersion;)V+595 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology()V+253 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map()V+60 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$18.apply()Lorg/apache/ignite/internal/IgniteInternalFuture;+4 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$18.apply()Ljava/lang/Object;+1 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(Lorg/apache/ignite/internal/util/typedef/CO;)Lorg/apache/ignite/internal/IgniteInternalFuture;+86 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsync0(Ljava/util/Map;Ljava/util/Map;[Ljava/lang/Object;Ljava/util/Map;Ljava/util/Map;ZZ[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;Z)Lorg/apache/ignite/internal/IgniteInternalFuture;+331 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;+20 > j org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Lorg/apache/ignite/internal/IgniteInternalFuture;+30 > j org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(Ljava/lang/Object;Ljava/lang/Object;[Lorg/apache/ignite/internal/processors/cache/CacheEntryPredicate;)Z+4 > j org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(Ljava/lang/Object;Ljava/lang/Object;)Z+6 > j org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(Ljava/lang/Object;Ljava/lang/Object;)V+68 > j org.apache.ignite.yardstick.cache.IgnitePutBenchmark.test(Ljava/util/Map;)Z+28 > j org.yardstickframework.impl.BenchmarkRunner$2.run()V+242 > j java.lang.Thread.run()V+11 > v ~StubRoutines::call_stub > V [libjvm.so+0x226c64] void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x2fc > V [libjvm.so+0x7b1e68] void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x1b8 > V [libjvm.so+0x2d5e34] void thread_entry(JavaThread*,Thread*)+0x164 > V [libjvm.so+0xc55e2c] void JavaThread::thread_main_inner()+0x94 > V [libjvm.so+0x2d1824] void JavaThread::run()+0x454 > V [libjvm.so+0xb09264] java_start+0x364 > {noformat} > In order ot reproduce it need to do following: > # Copy [^benchmark-offheap.properties] file into {{modules/yardstick/config}} directory. > # Build yardstick {{mvn -B clean package -Prelease,benchmarks -DskipTests -pl modules/yardstick -am}} > # Run benchmark {{./bin/benchmark-drivers-start.sh config/benchmark-offheap.properties}} (do it from {{yardstick}} module directory) > The problem isn't related with JIT (reproduces with -Xint option). > Also see attached logs and [core dump|https://drive.google.com/open?id=0Bz6LUqvKtoj6eklXTEp5aEh1MHM]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)