Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1C00E200D36 for ; Mon, 6 Nov 2017 23:48:10 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 1A9C3160BD5; Mon, 6 Nov 2017 22:48:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3AAE1160BEC for ; Mon, 6 Nov 2017 23:48:09 +0100 (CET) Received: (qmail 88802 invoked by uid 500); 6 Nov 2017 22:48:03 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 88791 invoked by uid 99); 6 Nov 2017 22:48:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Nov 2017 22:48:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7AE201A2188 for ; Mon, 6 Nov 2017 22:48:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.971 X-Spam-Level: X-Spam-Status: No, score=0.971 tagged_above=-999 required=6.31 tests=[SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id cvAMkiU9TDEx for ; Mon, 6 Nov 2017 22:47:59 +0000 (UTC) Received: from n6.nabble.com (n6.nabble.com [162.255.23.37]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D75A960F58 for ; Mon, 6 Nov 2017 22:47:58 +0000 (UTC) Received: from n6.nabble.com (localhost [127.0.0.1]) by n6.nabble.com (Postfix) with ESMTP id 1E8E42B62AE0 for ; Mon, 6 Nov 2017 15:47:57 -0700 (MST) Date: Mon, 6 Nov 2017 15:47:57 -0700 (MST) From: smurphy To: user@ignite.apache.org Message-ID: <1510008477122-0.post@n6.nabble.com> In-Reply-To: References: <1508869208730-0.post@n6.nabble.com> <1508989247849-0.post@n6.nabble.com> <1509027528818-0.post@n6.nabble.com> <1509040016144-0.post@n6.nabble.com> <1509040114762-0.post@n6.nabble.com> <1509042438679-0.post@n6.nabble.com> Subject: Re: Grid freezing MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit archived-at: Mon, 06 Nov 2017 22:48:10 -0000 The WARNING message above is not consistently showing up in the logs. Nodes are not going down either. What I have found is that running 1 server and 2 clients does NOT freeze, even under heavy loads. But having 2 ignite servers does freeze up. Two servers are unsuccessful either on separate boxes or on the same box using localhost. Looking at the thread dump of one server, 7 threads are waiting at TransactionProxyImpl.commit (line 259) (Excerpt 1 & 2 below) The thread dump of the other server has 8 thread waiting at IgniteCacheProxy.getAllOutTx (line 1328) (Excerpt 3) The thread number of one of these waiting threads is also waiting in the CacheAffinitySharedManager (Excerpt 4) Even though Excerpt 3 and 4 call getAllOutTx, they are the first place in the the transaction block where an Ignite cache is called. *Excerpt 1:* "pub-#75%dna%" prio=5 tid=96 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park() at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:779) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:103) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461) at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:283) Local Variable: java.lang.NullPointerException#2 at org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:269) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1468) Local Variable: java.lang.NoClassDefFoundError#1 at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:668) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1034) at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3343) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:221) Local Variable: org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler#1 at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepareLocal(GridNearOptimisticSerializableTxPrepareFuture.java:576) at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:493) Local Variable: org.apache.ignite.internal.util.GridLeanMap#24 Local Variable: org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode#1 Local Variable: org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxMapping#4 Local Variable: org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxPrepareRequest#1 at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:407) Local Variable: java.util.AbstractMap$2#1 Local Variable: org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture$MiniFuture#2 Local Variable: java.util.AbstractMap$2#2 Local Variable: java.util.ArrayList$Itr#1 Local Variable: java.util.ArrayList#7898 Local Variable: java.util.HashMap#5544 Local Variable: org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxMapping#1 at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare0(GridNearOptimisticSerializableTxPrepareFuture.java:314) at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:137) Local Variable: org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion#33 Local Variable: org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture#11 at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:74) at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3110) Local Variable: org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture#1 at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3162) Local Variable: org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal#1 at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.commitTxAsync(GridCacheSharedContext.java:929) at org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:259) *Excerpt 2:* "pub-#112%dna%" prio=5 tid=133 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park() at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176) Local Variable: org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture#4 at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:264) *Excerpt 3:* "pub-#35%dna%" prio=5 tid=55 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park() at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:166) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2722) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.getAllOutTx(IgniteCacheProxy.java:1328) Local Variable: java.util.HashSet#226 at com.company.dna.scan.fragment.node.ignite.IgniteFragmentDequeuer._createPriorityWeights(IgniteFragmentDequeuer.java:236) Local Variable: com.company.dna.scan.fragment.node.domain.PriorityWeights#5 Local Variable: double[]#18 Local Variable: org.apache.ignite.internal.processors.cache.IgniteCacheProxy#23 at com.company.dna.scan.fragment.node.ignite.IgniteFragmentDequeuer.*exchange-worker-#30%dna%*(IgniteFragmentDequeuer.java:133) *Excerpt 4:* "*exchange-worker-#30%dna%*" prio=5 tid=48 WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park() at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinity(CacheAffinitySharedManager.java:1402) Local Variable: org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache#4 Local Variable: org.apache.ignite.events.DiscoveryEvent#1 Local Variable: org.apache.ignite.internal.managers.discovery.DiscoCache#1 at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinityOnJoin(CacheAffinitySharedManager.java:1377) Local Variable: java.lang.Integer#3104 Local Variable: java.util.ArrayList#1017 Local Variable: org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtAssignmentFetchFuture#8 at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onServerJoin(CacheAffinitySharedManager.java:1293) Local Variable: org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager#1 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onServerNodeEvent(GridDhtPartitionsExchangeFuture.java:896) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:623) Local Variable: org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion#4 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) Local Variable: org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker#1 at java.lang.Thread.run() -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/