ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Pereslegin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-8051) Put operation may hang on LOCAL TRANSACTIONAL cache if it was stopped asynchronously.
Date Mon, 26 Mar 2018 16:37:00 GMT
Pavel Pereslegin created IGNITE-8051:
----------------------------------------

             Summary: Put operation may hang on LOCAL TRANSACTIONAL cache if it was stopped asynchronously.
                 Key: IGNITE-8051
                 URL: https://issues.apache.org/jira/browse/IGNITE-8051
             Project: Ignite
          Issue Type: Bug
          Components: cache
    Affects Versions: 2.4
            Reporter: Pavel Pereslegin


Put operation may hang if local cache was destroyed asynchronously. 
It seems that this happens because IgniteTxImplicitSingleStateImpl#topologyReadLock does not check cache status for local cache. 

Simple reproducer: 
{code:java}
public class DestroyLocalCacheTest extends GridCommonAbstractTest {
    /** */
    public void testDestroyAsync() throws Exception {
        try (Ignite node = startGrid()) {
            IgniteCache<Integer, Boolean> locCache = node.createCache(
                new CacheConfiguration<Integer, Boolean>(DEFAULT_CACHE_NAME).setCacheMode(CacheMode.LOCAL)
                    .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL));

            AtomicInteger cntr = new AtomicInteger();

            GridTestUtils.runMultiThreadedAsync(() -> {
                try {
                    int key;

                    while ((key = cntr.getAndIncrement()) < 10_000) {

                        if (key == 1000)
                            locCache.destroy();

                        locCache.put(key, true);
                    }
                }
                catch (Exception ignore) {
                    // No-op
                }

                return null;
            }, 5, "put-thread").get();
        }
    }
}
{code}

Log output:
{noformat}
[2018-03-26 19:10:11,350][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridCacheProcessor] Started cache [name=default, id=1544803905, memoryPolicyName=default, mode=LOCAL, atomicity=TRANSACTIONAL, backups=0]
[2018-03-26 19:10:11,352][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA]
[2018-03-26 19:10:11,353][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[2018-03-26 19:10:11,354][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], err=null]
[2018-03-26 19:10:11,354][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true]
[2018-03-26 19:10:11,355][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, node=fe867235-2fff-465a-b767-206abd291058]
[2018-03-26 19:10:11,590][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=fe867235-2fff-465a-b767-206abd291058, customEvt=DynamicCacheChangeBatch [id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], allowMerge=false]
[2018-03-26 19:10:11,590][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], waitTime=0ms, futInfo=NA]
[2018-03-26 19:10:11,591][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=2]]
[2018-03-26 19:10:11,592][INFO ][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], err=null]
[2018-03-26 19:15:09,191][ERROR][main][root] Test has been timed out and will be interrupted (threads dump will be taken before interruption) [test=testDestroyAsync, timeout=300000]
[2018-03-26 19:15:09,191][WARN ][main][diagnostic] Dumping debug info for node [id=fe867235-2fff-465a-b767-206abd291058, name=local.DestroyLocalCacheTest, order=1, topVer=1, client=false]
[2018-03-26 19:15:09,191][WARN ][main][diagnostic] Ready affinity version: AffinityTopologyVersion [topVer=1, minorTopVer=1]
[2018-03-26 19:15:09,195][WARN ][main][diagnostic] Last exchange future: GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent [customMsg=DynamicCacheChangeBatch [id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], topVer=1, nodeId8=fe867235, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1522080611585]], crd=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], discoEvt=DiscoveryCustomEvent [customMsg=DynamicCacheChangeBatch [id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], topVer=1, nodeId8=fe867235, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1522080611585]], nodeId=fe867235, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=13648335], init=false, lastVer=GridCacheVersion [topVer=133560612, order=1522080613658, nodeOrder=1], partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]], TxReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]]]], exchActions=ExchangeActions [startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, destroy=true], resetParts=null, stateChangeRequest=null], affChangeMsg=null, initTs=1522080611585, centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, done=true, state=DONE, evtLatch=0, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=747464370]]
[2018-03-26 19:15:09,196][WARN ][main][GridCachePartitionExchangeManager] First 10 pending exchange futures [total=0]
[2018-03-26 19:15:09,197][WARN ][main][diagnostic] Last 10 exchange futures (total: 3):
[2018-03-26 19:15:09,197][WARN ][main][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], done=false]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], done=true]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] >>> GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], done=true]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] Pending transactions:
[2018-03-26 19:15:09,200][WARN ][main][diagnostic] >>> [txVer=null, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-1, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613657, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=62, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613657, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297604ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,200][WARN ][main][diagnostic] >>> [txVer=null, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-2, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613653, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=63, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613653, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] >>> [txVer=null, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-5, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613652, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=66, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613652, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] >>> [txVer=null, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-4, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613645, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=65, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613645, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] Pending explicit locks:
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] Pending cache futures:
[2018-03-26 19:15:09,203][WARN ][main][diagnostic] >>> GridNearTxFinishFuture [futId=8c313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-5, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613652, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=66, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613652, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], trackable=true, finishOnePhaseCalled=false, innerFuts=[], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,203][WARN ][main][diagnostic] >>> GridNearTxFinishFuture [futId=fc313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-1, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613657, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=62, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613657, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], trackable=true, finishOnePhaseCalled=false, innerFuts=[], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] >>> GridNearTxFinishFuture [futId=cb313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-4, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613645, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=65, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613645, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], trackable=true, finishOnePhaseCalled=false, innerFuts=[], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] >>> GridNearTxFinishFuture [futId=cc313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false, thread=put-thread-2, mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, order=1522080613653, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=63, startTime=1522080611585, nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion [topVer=133560612, order=1522080613653, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], trackable=true, finishOnePhaseCalled=false, innerFuts=[], super=GridCompoundIdentityFuture [super=GridCompoundFuture [rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] Pending atomic cache futures:
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] Pending data streamer futures:
[2018-03-26 19:15:09,205][WARN ][main][diagnostic] Pending transaction deadlock detection futures:
[19:15:09] (wrn) No deadlocked threads detected.

[19:15:09] (wrn) Thread dump at 2018/03/26 19:15:09 MSK[2018-03-26 19:15:09,226][WARN ][main][root] No deadlocked threads detected.
Thread [name="sys-#69%local.DestroyLocalCacheTest%", id=95, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#68%local.DestroyLocalCacheTest%", id=94, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#67%local.DestroyLocalCacheTest%", id=93, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#66%local.DestroyLocalCacheTest%", id=92, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#65%local.DestroyLocalCacheTest%", id=91, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#64%local.DestroyLocalCacheTest%", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="put-thread-5", id=66, state=WAITING, blockCnt=138, waitCnt=61]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-4", id=65, state=WAITING, blockCnt=109, waitCnt=43]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-3", id=64, state=WAITING, blockCnt=93, waitCnt=56]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.destroy(GatewayProtectedCacheProxy.java:1503)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:30)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-2", id=63, state=WAITING, blockCnt=132, waitCnt=57]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-1", id=62, state=WAITING, blockCnt=135, waitCnt=76]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="async-callable-runner-1", id=61, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=o.a.i.testframework.GridTestSafeThreadFactory$1@1cd072a9, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:871)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:811)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:809)
        at o.a.i.testframework.GridTestUtils$7.run(GridTestUtils.java:979)
        at o.a.i.testframework.GridTestUtils$9.call(GridTestUtils.java:1275)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="ttl-cleanup-worker-#40%local.DestroyLocalCacheTest%", id=60, state=TIMED_WAITING, blockCnt=0, waitCnt=596]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7598)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="srvc-deploy-#39%local.DestroyLocalCacheTest%", id=58, state=WAITING, blockCnt=0, waitCnt=3]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c75222b, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#38%local.DestroyLocalCacheTest%", id=57, state=TIMED_WAITING, blockCnt=0, waitCnt=1489]
    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@48140564, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at o.a.i.i.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:201)
        at o.a.i.i.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:305)
        at o.a.i.i.processors.cache.GridCacheProcessor.stopGateway(GridCacheProcessor.java:2060)
        at o.a.i.i.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2229)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1612)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:2610)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:2408)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1135)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:704)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2344)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4c203ea1
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@27f674d
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@1d251891
Thread [name="disco-event-worker-#37%local.DestroyLocalCacheTest%", id=56, state=WAITING, blockCnt=0, waitCnt=299]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ceff1, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2598)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2580)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-ip-finder-cleaner-#6%local.DestroyLocalCacheTest%", id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1883)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-multicast-addr-sender-#5%local.DestroyLocalCacheTest%", id=54, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked java.net.DatagramPacket@49e4cb85
        - locked java.net.MulticastSocket@2133c8f8
        at o.a.i.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder$AddressSender.body(TcpDiscoveryMulticastIpFinder.java:882)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-srvr-#4%local.DestroyLocalCacheTest%", id=53, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5590)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#3%local.DestroyLocalCacheTest%", id=52, state=TIMED_WAITING, blockCnt=4, waitCnt=29402]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43a25848, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6702)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2602)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-data-loader-flusher-#36%local.DestroyLocalCacheTest%", id=51, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ac3fd8b, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-3-#34%local.DestroyLocalCacheTest%", id=49, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5594a1b5
        - locked java.util.Collections$UnmodifiableSet@6a5fc7f7
        - locked sun.nio.ch.EPollSelectorImpl@3b6eb2ec
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-2-#33%local.DestroyLocalCacheTest%", id=48, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1e643faf
        - locked java.util.Collections$UnmodifiableSet@6e8dacdf
        - locked sun.nio.ch.EPollSelectorImpl@7a79be86
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-1-#32%local.DestroyLocalCacheTest%", id=47, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@34ce8af7
        - locked java.util.Collections$UnmodifiableSet@b684286
        - locked sun.nio.ch.EPollSelectorImpl@880ec60
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-0-#31%local.DestroyLocalCacheTest%", id=46, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3f3afe78
        - locked java.util.Collections$UnmodifiableSet@7f63425a
        - locked sun.nio.ch.EPollSelectorImpl@36d64342
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#30%local.DestroyLocalCacheTest%", id=45, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@39ba5a14
        - locked java.util.Collections$UnmodifiableSet@511baa65
        - locked sun.nio.ch.EPollSelectorImpl@340f438e
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-sharedfs-timeout-worker-#2%local.DestroyLocalCacheTest%", id=44, state=TIMED_WAITING, blockCnt=0, waitCnt=60]
    Lock [object=java.lang.Object@30c7da1e, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-comm-worker-#1%local.DestroyLocalCacheTest%", id=43, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b464ce8, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4093)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-3-#29%local.DestroyLocalCacheTest%", id=42, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@57829d67
        - locked java.util.Collections$UnmodifiableSet@19dfb72a
        - locked sun.nio.ch.EPollSelectorImpl@17c68925
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#28%local.DestroyLocalCacheTest%", id=41, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7e0ea639
        - locked java.util.Collections$UnmodifiableSet@3d24753a
        - locked sun.nio.ch.EPollSelectorImpl@59a6e353
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#27%local.DestroyLocalCacheTest%", id=40, state=RUNNABLE, blockCnt=2, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7a0ac6e3
        - locked java.util.Collections$UnmodifiableSet@71be98f5
        - locked sun.nio.ch.EPollSelectorImpl@6fadae5d
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#26%local.DestroyLocalCacheTest%", id=39, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17f6480
        - locked java.util.Collections$UnmodifiableSet@2d6e8792
        - locked sun.nio.ch.EPollSelectorImpl@2812cbfa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#25%local.DestroyLocalCacheTest%", id=38, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@2acf57e3
        - locked java.util.Collections$UnmodifiableSet@506e6d5e
        - locked sun.nio.ch.EPollSelectorImpl@96532d6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#24%local.DestroyLocalCacheTest%", id=37, state=TIMED_WAITING, blockCnt=1, waitCnt=956]
    Lock [object=java.lang.Object@3796751b, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#17%local.DestroyLocalCacheTest%", id=30, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#16%local.DestroyLocalCacheTest%", id=29, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#15%local.DestroyLocalCacheTest%", id=28, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#14%local.DestroyLocalCacheTest%", id=27, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#13%local.DestroyLocalCacheTest%", id=26, state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#12%local.DestroyLocalCacheTest%", id=25, state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#11%local.DestroyLocalCacheTest%", id=24, state=WAITING, blockCnt=2, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#10%local.DestroyLocalCacheTest%", id=23, state=WAITING, blockCnt=1, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#9%local.DestroyLocalCacheTest%", id=22, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#8%local.DestroyLocalCacheTest%", id=21, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#7%local.DestroyLocalCacheTest%", id=20, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#6%local.DestroyLocalCacheTest%", id=19, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#5%local.DestroyLocalCacheTest%", id=18, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#4%local.DestroyLocalCacheTest%", id=17, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#3%local.DestroyLocalCacheTest%", id=16, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#2%local.DestroyLocalCacheTest%", id=15, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="test-runner-#1%local.DestroyLocalCacheTest%", id=14, state=WAITING, blockCnt=4, waitCnt=5]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.testDestroyAsync(DestroyLocalCacheTest.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at junit.framework.TestCase.runTest(TestCase.java:176)
        at o.a.i.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
        at o.a.i.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
        at o.a.i.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=13, state=TIMED_WAITING, blockCnt=0, waitCnt=5984]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

Thread [name="ignite-clock-for-tests", id=11, state=TIMED_WAITING, blockCnt=0, waitCnt=29548]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.GridTestClockTimer.run(GridTestClockTimer.java:41)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Monitor Ctrl-Break", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name="Finalizer", id=3, state=WAITING, blockCnt=3, waitCnt=4]
    Lock [object=java.lang.ref.ReferenceQueue$Lock@4411d970, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=3, waitCnt=3]
    Lock [object=java.lang.ref.Reference$Lock@6442b0a6, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1324)
        at o.a.i.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1942)
        at junit.framework.TestCase.runBare(TestCase.java:141)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:129)
        at junit.framework.TestSuite.runTest(TestSuite.java:255)
        at junit.framework.TestSuite.run(TestSuite.java:250)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
        at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
        at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
        at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)



[2018-03-26 19:15:09,244][WARN ][main][root] Thread dump at 2018/03/26 19:15:09 MSK
Thread [name="sys-#69%local.DestroyLocalCacheTest%", id=95, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#68%local.DestroyLocalCacheTest%", id=94, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#67%local.DestroyLocalCacheTest%", id=93, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#66%local.DestroyLocalCacheTest%", id=92, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#65%local.DestroyLocalCacheTest%", id=91, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#64%local.DestroyLocalCacheTest%", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="put-thread-5", id=66, state=WAITING, blockCnt=138, waitCnt=61]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-4", id=65, state=WAITING, blockCnt=109, waitCnt=43]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-3", id=64, state=WAITING, blockCnt=93, waitCnt=56]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.destroy(GatewayProtectedCacheProxy.java:1503)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:30)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-2", id=63, state=WAITING, blockCnt=132, waitCnt=57]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-1", id=62, state=WAITING, blockCnt=135, waitCnt=76]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="async-callable-runner-1", id=61, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=o.a.i.testframework.GridTestSafeThreadFactory$1@1cd072a9, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:871)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:811)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:809)
        at o.a.i.testframework.GridTestUtils$7.run(GridTestUtils.java:979)
        at o.a.i.testframework.GridTestUtils$9.call(GridTestUtils.java:1275)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="ttl-cleanup-worker-#40%local.DestroyLocalCacheTest%", id=60, state=TIMED_WAITING, blockCnt=0, waitCnt=596]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7598)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="srvc-deploy-#39%local.DestroyLocalCacheTest%", id=58, state=WAITING, blockCnt=0, waitCnt=3]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c75222b, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#38%local.DestroyLocalCacheTest%", id=57, state=TIMED_WAITING, blockCnt=0, waitCnt=1489]
    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@48140564, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at o.a.i.i.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:201)
        at o.a.i.i.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:305)
        at o.a.i.i.processors.cache.GridCacheProcessor.stopGateway(GridCacheProcessor.java:2060)
        at o.a.i.i.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2229)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1612)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:2610)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:2408)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1135)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:704)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2344)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4c203ea1
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@27f674d
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@1d251891
Thread [name="disco-event-worker-#37%local.DestroyLocalCacheTest%", id=56, state=WAITING, blockCnt=0, waitCnt=299]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ceff1, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2598)
        at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2580)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-ip-finder-cleaner-#6%local.DestroyLocalCacheTest%", id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1883)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-multicast-addr-sender-#5%local.DestroyLocalCacheTest%", id=54, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked java.net.DatagramPacket@49e4cb85
        - locked java.net.MulticastSocket@2133c8f8
        at o.a.i.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder$AddressSender.body(TcpDiscoveryMulticastIpFinder.java:882)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-srvr-#4%local.DestroyLocalCacheTest%", id=53, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5590)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#3%local.DestroyLocalCacheTest%", id=52, state=TIMED_WAITING, blockCnt=4, waitCnt=29404]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43a25848, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6702)
        at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2602)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-data-loader-flusher-#36%local.DestroyLocalCacheTest%", id=51, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ac3fd8b, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-3-#34%local.DestroyLocalCacheTest%", id=49, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5594a1b5
        - locked java.util.Collections$UnmodifiableSet@6a5fc7f7
        - locked sun.nio.ch.EPollSelectorImpl@3b6eb2ec
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-2-#33%local.DestroyLocalCacheTest%", id=48, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1e643faf
        - locked java.util.Collections$UnmodifiableSet@6e8dacdf
        - locked sun.nio.ch.EPollSelectorImpl@7a79be86
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-1-#32%local.DestroyLocalCacheTest%", id=47, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@34ce8af7
        - locked java.util.Collections$UnmodifiableSet@b684286
        - locked sun.nio.ch.EPollSelectorImpl@880ec60
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-client-listener-0-#31%local.DestroyLocalCacheTest%", id=46, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3f3afe78
        - locked java.util.Collections$UnmodifiableSet@7f63425a
        - locked sun.nio.ch.EPollSelectorImpl@36d64342
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#30%local.DestroyLocalCacheTest%", id=45, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@39ba5a14
        - locked java.util.Collections$UnmodifiableSet@511baa65
        - locked sun.nio.ch.EPollSelectorImpl@340f438e
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-sharedfs-timeout-worker-#2%local.DestroyLocalCacheTest%", id=44, state=TIMED_WAITING, blockCnt=0, waitCnt=60]
    Lock [object=java.lang.Object@30c7da1e, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-comm-worker-#1%local.DestroyLocalCacheTest%", id=43, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b464ce8, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4093)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-3-#29%local.DestroyLocalCacheTest%", id=42, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@57829d67
        - locked java.util.Collections$UnmodifiableSet@19dfb72a
        - locked sun.nio.ch.EPollSelectorImpl@17c68925
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#28%local.DestroyLocalCacheTest%", id=41, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7e0ea639
        - locked java.util.Collections$UnmodifiableSet@3d24753a
        - locked sun.nio.ch.EPollSelectorImpl@59a6e353
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#27%local.DestroyLocalCacheTest%", id=40, state=RUNNABLE, blockCnt=2, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7a0ac6e3
        - locked java.util.Collections$UnmodifiableSet@71be98f5
        - locked sun.nio.ch.EPollSelectorImpl@6fadae5d
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#26%local.DestroyLocalCacheTest%", id=39, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17f6480
        - locked java.util.Collections$UnmodifiableSet@2d6e8792
        - locked sun.nio.ch.EPollSelectorImpl@2812cbfa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#25%local.DestroyLocalCacheTest%", id=38, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@2acf57e3
        - locked java.util.Collections$UnmodifiableSet@506e6d5e
        - locked sun.nio.ch.EPollSelectorImpl@96532d6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#24%local.DestroyLocalCacheTest%", id=37, state=TIMED_WAITING, blockCnt=1, waitCnt=956]
    Lock [object=java.lang.Object@3796751b, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#17%local.DestroyLocalCacheTest%", id=30, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#16%local.DestroyLocalCacheTest%", id=29, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#15%local.DestroyLocalCacheTest%", id=28, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#14%local.DestroyLocalCacheTest%", id=27, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#13%local.DestroyLocalCacheTest%", id=26, state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#12%local.DestroyLocalCacheTest%", id=25, state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#11%local.DestroyLocalCacheTest%", id=24, state=WAITING, blockCnt=2, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#10%local.DestroyLocalCacheTest%", id=23, state=WAITING, blockCnt=1, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#9%local.DestroyLocalCacheTest%", id=22, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#8%local.DestroyLocalCacheTest%", id=21, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#7%local.DestroyLocalCacheTest%", id=20, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#6%local.DestroyLocalCacheTest%", id=19, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#5%local.DestroyLocalCacheTest%", id=18, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#4%local.DestroyLocalCacheTest%", id=17, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#3%local.DestroyLocalCacheTest%", id=16, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#2%local.DestroyLocalCacheTest%", id=15, state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="test-runner-#1%local.DestroyLocalCacheTest%", id=14, state=WAITING, blockCnt=4, waitCnt=5]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.processors.cache.local.DestroyLocalCacheTest.testDestroyAsync(DestroyLocalCacheTest.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at junit.framework.TestCase.runTest(TestCase.java:176)
        at o.a.i.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
        at o.a.i.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
        at o.a.i.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=13, state=TIMED_WAITING, blockCnt=0, waitCnt=5984]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

Thread [name="ignite-clock-for-tests", id=11, state=TIMED_WAITING, blockCnt=0, waitCnt=29550]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.GridTestClockTimer.run(GridTestClockTimer.java:41)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Monitor Ctrl-Break", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name="Finalizer", id=3, state=WAITING, blockCnt=3, waitCnt=4]
    Lock [object=java.lang.ref.ReferenceQueue$Lock@4411d970, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=3, waitCnt=3]
    Lock [object=java.lang.ref.Reference$Lock@6442b0a6, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1324)
        at o.a.i.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1944)
        at junit.framework.TestCase.runBare(TestCase.java:141)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:129)
        at junit.framework.TestSuite.runTest(TestSuite.java:255)
        at junit.framework.TestSuite.run(TestSuite.java:250)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
        at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
        at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
        at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message