ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From smurphy <smur...@trustwave.com>
Subject Re: Grid freezing
Date Mon, 06 Nov 2017 22:47:57 GMT
The WARNING message above is not consistently showing up in the logs. 
Nodes are not going down either. 
What I have found is that running 1 server and 2 clients does NOT freeze,
even under heavy loads.
But having 2 ignite servers does freeze up. 
Two servers are unsuccessful either on separate boxes or on the same box
using localhost.

Looking at the thread dump of one server, 7 threads are waiting at
TransactionProxyImpl.commit (line 259) (Excerpt 1 & 2 below)

The thread dump of the other server has 8 thread waiting at
IgniteCacheProxy.getAllOutTx (line 1328) (Excerpt 3)
The thread number of one of these waiting threads is also waiting in the
CacheAffinitySharedManager  (Excerpt 4)
Even though Excerpt 3 and 4 call getAllOutTx, they are the first place in
the the transaction block where an Ignite cache is called.

*Excerpt 1:*

"pub-#75%dna%" prio=5 tid=96 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(<unknown string>)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:779)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:103)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
    at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:283)
       Local Variable: java.lang.NullPointerException#2
    at
org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:269)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1468)
       Local Variable: java.lang.NoClassDefFoundError#1
    at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:668)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1034)
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3343)
    at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:221)
       Local Variable:
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler#1
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepareLocal(GridNearOptimisticSerializableTxPrepareFuture.java:576)
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:493)
       Local Variable: org.apache.ignite.internal.util.GridLeanMap#24
       Local Variable:
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode#1
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxMapping#4
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxPrepareRequest#1
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:407)
       Local Variable: java.util.AbstractMap$2#1
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture$MiniFuture#2
       Local Variable: java.util.AbstractMap$2#2
       Local Variable: java.util.ArrayList$Itr#1
       Local Variable: java.util.ArrayList#7898
       Local Variable: java.util.HashMap#5544
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxMapping#1
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare0(GridNearOptimisticSerializableTxPrepareFuture.java:314)
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:137)
       Local Variable:
org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion#33
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture#11
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:74)
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3110)
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture#1
    at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3162)
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal#1
    at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.commitTxAsync(GridCacheSharedContext.java:929)
    at
org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:259)

*Excerpt 2:*

 "pub-#112%dna%" prio=5 tid=133 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(<unknown string>)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture#4
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
    at
org.apache.ignite.internal.processors.cache.transactions.TransactionProxyImpl.commit(TransactionProxyImpl.java:264)


*Excerpt 3:*

"pub-#35%dna%" prio=5 tid=55 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(<unknown string>)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
    at
org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:166)
    at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2722)
    at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.getAllOutTx(IgniteCacheProxy.java:1328)
       Local Variable: java.util.HashSet#226
    at
com.company.dna.scan.fragment.node.ignite.IgniteFragmentDequeuer._createPriorityWeights(IgniteFragmentDequeuer.java:236)
       Local Variable:
com.company.dna.scan.fragment.node.domain.PriorityWeights#5
       Local Variable: double[]#18
       Local Variable:
org.apache.ignite.internal.processors.cache.IgniteCacheProxy#23
    at
com.company.dna.scan.fragment.node.ignite.IgniteFragmentDequeuer.*exchange-worker-#30%dna%*(IgniteFragmentDequeuer.java:133)
    

*Excerpt 4:*

  
"*exchange-worker-#30%dna%*" prio=5 tid=48 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(<unknown string>)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
    at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
    at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinity(CacheAffinitySharedManager.java:1402)
       Local Variable:
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache#4
       Local Variable: org.apache.ignite.events.DiscoveryEvent#1
       Local Variable:
org.apache.ignite.internal.managers.discovery.DiscoCache#1
    at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinityOnJoin(CacheAffinitySharedManager.java:1377)
       Local Variable: java.lang.Integer#3104
       Local Variable: java.util.ArrayList#1017
       Local Variable:
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtAssignmentFetchFuture#8
    at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onServerJoin(CacheAffinitySharedManager.java:1293)
       Local Variable:
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager#1
    at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onServerNodeEvent(GridDhtPartitionsExchangeFuture.java:896)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:623)
       Local Variable:
org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion#4
    at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
    at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
       Local Variable:
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker#1
    at java.lang.Thread.run(<unknown string>)




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message