Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 356C9200B7B for ; Thu, 25 Aug 2016 08:48:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3419B160A94; Thu, 25 Aug 2016 06:48:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6AF55160A93 for ; Thu, 25 Aug 2016 08:48:21 +0200 (CEST) Received: (qmail 64599 invoked by uid 500); 25 Aug 2016 06:48:20 -0000 Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list issues@ignite.apache.org Received: (qmail 64580 invoked by uid 99); 25 Aug 2016 06:48:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Aug 2016 06:48:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 817362C0151 for ; Thu, 25 Aug 2016 06:48:20 +0000 (UTC) Date: Thu, 25 Aug 2016 06:48:20 +0000 (UTC) From: "Semen Boikov (JIRA)" To: issues@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (IGNITE-1905) High contention for CacheLockImpl causes AssertionErrors MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 25 Aug 2016 06:48:23 -0000 [ https://issues.apache.org/jira/browse/IGNITE-1905?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Semen Boikov updated IGNITE-1905: --------------------------------- Priority: Major (was: Critical) > High contention for CacheLockImpl causes AssertionErrors > -------------------------------------------------------- > > Key: IGNITE-1905 > URL: https://issues.apache.org/jira/browse/IGNITE-1905 > Project: Ignite > Issue Type: Bug > Components: cache > Affects Versions: ignite-1.4 > Environment: Windows 7 > Reporter: Denis Magda > Assignee: Semen Boikov > Attachments: ClientTest.java, ServerTest.java > > > When multiple threads, running on the same client node, compete for Cache= LockImpl this leads to AssertionErrors. > Pseudo-code snippet, that is called from multiple threads and causes the = assertions, looks like this: > {noformat} > boolean locked =3D lock.tryLock(100, TimeUnit.MILLISECONDS); > if (locked) > lock.unlock(); > {noformat} > Initially the issue was detected on ignite-1.4. > In server's node logs the following assertion appears > {noformat} > java.lang.AssertionError > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.addOwned(GridDhtLockFuture.java:958) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.map(GridDhtLockFuture.java:918) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.onOwnerChanged(GridDhtLockFuture.java:663) > =09at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$2.= onOwnerChanged(GridCacheMvccManager.java:155) > =09at org.apache.ignite.internal.processors.cache.distributed.GridDistrib= utedCacheEntry.checkOwnerChanged(GridDistributedCacheEntry.java:810) > =09at org.apache.ignite.internal.processors.cache.distributed.GridDistrib= utedCacheEntry.readyLock(GridDistributedCacheEntry.java:516) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.readyLocks(GridDhtLockFuture.java:576) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.map(GridDhtLockFuture.java:764) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.lockAllAsync(GridDhtTransactionalCacheAdapter.jav= a:973) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.processNearLockRequest(GridDhtTransactionalCacheA= dapter.java:557) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.access$000(GridDhtTransactionalCacheAdapter.java:= 88) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:132= ) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:130= ) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.proc= essMessage(GridCacheIoManager.java:580) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMe= ssage0(GridCacheIoManager.java:280) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.hand= leMessage(GridCacheIoManager.java:198) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.acce= ss$000(GridCacheIoManager.java:77) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.on= Message(GridCacheIoManager.java:160) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.pro= cessRegularMessage0(GridIoManager.java:811) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.acc= ess$1500(GridIoManager.java:106) > =09at org.apache.ignite.internal.managers.communication.GridIoManager$5.r= un(GridIoManager.java:774) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > {noformat} > =20 > In addition from time to time a client node also outputs a different kind= of assertion: > {noformat} > Exception in thread "ignite-#4%sys-null%" java.lang.AssertionError > =09at org.apache.ignite.internal.processors.cache.GridCacheExplicitLockSp= an.markOwned(GridCacheExplicitLockSpan.java:196) > =09at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.ma= rkExplicitOwner(GridCacheMvccManager.java:862) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedLockFuture$MiniFuture.onResult(GridDhtColocatedLockFutur= e.java:1412) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedLockFuture.onResult(GridDhtColocatedLockFuture.java:437) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedCache.processLockResponse(GridDhtColocatedCache.java:888= ) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedCache.access$100(GridDhtColocatedCache.java:80) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedCache$3.apply(GridDhtColocatedCache.java:145) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.colocat= ed.GridDhtColocatedCache$3.apply(GridDhtColocatedCache.java:143) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.proc= essMessage(GridCacheIoManager.java:580) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMe= ssage0(GridCacheIoManager.java:280) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.hand= leMessage(GridCacheIoManager.java:198) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.acce= ss$000(GridCacheIoManager.java:77) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.on= Message(GridCacheIoManager.java:160) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.pro= cessRegularMessage0(GridIoManager.java:811) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.acc= ess$1500(GridIoManager.java:106) > =09at org.apache.ignite.internal.managers.communication.GridIoManager$5.r= un(GridIoManager.java:774) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > {noformat} > Ran the test against ignite-1.5 and got a different kind of assertions on= the server node: > {noformat} > Exception in thread "ignite-#246%sys-null%" java.lang.AssertionError: Ent= ry does not own lock for tx [locNodeId=3D8f8c5147-2d8f-44e6-8da8-ef8a5bae1f= d3, entry=3DGridDhtColocatedCacheEntry [super=3DGridDhtCacheEntry [rdrs=3D[= ], locPart=3DGridDhtLocalPartition [id=3D49, mapPubSize=3D5, rmvQueue=3DGri= dCircularBuffer [sizeMask=3D511, idxGen=3D0], state=3DOWNING, reservations= =3D0, empty=3Dfalse, createTime=3D11/13/2015 15:08:42, mapPubSize=3D5], sup= er=3DGridDistributedCacheEntry [super=3DGridCacheMapEntry [key=3DKeyCacheOb= jectImpl [val=3D1, hasValBytes=3Dtrue], val=3DCacheObjectImpl [val=3Dnull, = hasValBytes=3Dtrue], startVer=3D1447412910273, ver=3DGridCacheVersion [topV= er=3D58892922, nodeOrderDrId=3D1, globalTime=3D0, order=3D0], hash=3D-32044= 3512, extras=3DGridCacheMvccEntryExtras [mvcc=3DGridCacheMvcc [locs=3D[Grid= CacheMvccCandidate [nodeId=3D8f8c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGr= idCacheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D14474129= 68535, order=3D1447412964154], timeout=3D100, ts=3D1447412968537, threadId= =3D101, id=3D2245, topVer=3DAffinityTopologyVersion [topVer=3D3, minorTopVe= r=3D0], reentry=3Dnull, otherNodeId=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b,= otherVer=3DGridCacheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalT= ime=3D1447412968535, order=3D1447412964154], mappedDhtNodes=3D[TcpDiscovery= Node [id=3Dce646644-b983-4d81-ba40-c39a057b4cc5, addrs=3D[0:0:0:0:0:0:0:1, = 10.0.0.18, 127.0.0.1], sockAddrs=3D[/10.0.0.18:47501, /0:0:0:0:0:0:0:1:4750= 1, /10.0.0.18:47501, /127.0.0.1:47501], discPort=3D47501, order=3D2, intOrd= er=3D2, lastExchangeTime=3D1447412934615, loc=3Dfalse, ver=3D1.5.0#19700101= -sha1:00000000, isClient=3Dfalse]], mappedNearNodes=3Dnull, ownerVer=3DGrid= CacheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968= 452, order=3D1447412964076], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D1|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968537, order=3D1447412964156], t= imeout=3D100, ts=3D1447412968547, threadId=3D99, id=3D2246, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968537, order=3D14474= 12964156], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296845= 4, order=3D1447412964078], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968539, order=3D1447412964158], t= imeout=3D100, ts=3D1447412968547, threadId=3D75, id=3D2247, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968539, order=3D14474= 12964158], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296845= 6, order=3D1447412964080], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968541, order=3D1447412964160], t= imeout=3D100, ts=3D1447412968547, threadId=3D83, id=3D2248, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968541, order=3D14474= 12964160], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296845= 8, order=3D1447412964082], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968543, order=3D1447412964162], t= imeout=3D100, ts=3D1447412968547, threadId=3D86, id=3D2249, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968543, order=3D14474= 12964162], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296846= 1, order=3D1447412964084], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968545, order=3D1447412964164], t= imeout=3D100, ts=3D1447412968557, threadId=3D69, id=3D2250, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968545, order=3D14474= 12964164], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296846= 3, order=3D1447412964086], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968547, order=3D1447412964166], t= imeout=3D100, ts=3D1447412968557, threadId=3D95, id=3D2251, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968547, order=3D14474= 12964166], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296846= 5, order=3D1447412964088], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968550, order=3D1447412964168], t= imeout=3D100, ts=3D1447412968557, threadId=3D102, id=3D2252, topVer=3DAffin= ityTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNode= Id=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [top= Ver=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968550, order=3D1447= 412964168], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridC= acheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D14474129684= 67, order=3D1447412964090], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968552, order=3D1447412964170], t= imeout=3D100, ts=3D1447412968557, threadId=3D67, id=3D2253, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968552, order=3D14474= 12964170], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296847= 0, order=3D1447412964092], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968554, order=3D1447412964172], t= imeout=3D100, ts=3D1447412968557, threadId=3D78, id=3D2254, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968554, order=3D14474= 12964172], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296847= 2, order=3D1447412964094], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968556, order=3D1447412964174], t= imeout=3D100, ts=3D1447412968567, threadId=3D70, id=3D2255, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968556, order=3D14474= 12964174], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296847= 4, order=3D1447412964096], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968559, order=3D1447412964176], t= imeout=3D100, ts=3D1447412968567, threadId=3D79, id=3D2256, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968559, order=3D14474= 12964176], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296847= 6, order=3D1447412964098], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968561, order=3D1447412964178], t= imeout=3D100, ts=3D1447412968567, threadId=3D103, id=3D2257, topVer=3DAffin= ityTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNode= Id=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [top= Ver=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968561, order=3D1447= 412964178], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridC= acheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D14474129684= 79, order=3D1447412964100], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968563, order=3D1447412964180], t= imeout=3D100, ts=3D1447412968567, threadId=3D90, id=3D2258, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968563, order=3D14474= 12964180], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296848= 1, order=3D1447412964102], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968566, order=3D1447412964182], t= imeout=3D100, ts=3D1447412968577, threadId=3D66, id=3D2259, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968566, order=3D14474= 12964182], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296848= 3, order=3D1447412964104], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968568, order=3D1447412964184], t= imeout=3D100, ts=3D1447412968577, threadId=3D82, id=3D2260, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968568, order=3D14474= 12964184], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296848= 5, order=3D1447412964106], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968570, order=3D1447412964186], t= imeout=3D100, ts=3D1447412968577, threadId=3D91, id=3D2261, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968570, order=3D14474= 12964186], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296848= 7, order=3D1447412964108], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968584, order=3D1447412964188], t= imeout=3D100, ts=3D1447412968597, threadId=3D94, id=3D2262, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968584, order=3D14474= 12964188], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296849= 4, order=3D1447412964114], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968585, order=3D1447412964189], t= imeout=3D100, ts=3D1447412968597, threadId=3D68, id=3D2263, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968585, order=3D14474= 12964189], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296849= 4, order=3D1447412964114], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968586, order=3D1447412964192], t= imeout=3D100, ts=3D1447412968597, threadId=3D84, id=3D2264, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968586, order=3D14474= 12964192], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296849= 4, order=3D1447412964114], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968591, order=3D1447412964194], t= imeout=3D100, ts=3D1447412968597, threadId=3D85, id=3D2265, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968591, order=3D14474= 12964194], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296849= 7, order=3D1447412964116], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968593, order=3D1447412964196], t= imeout=3D100, ts=3D1447412968597, threadId=3D104, id=3D2266, topVer=3DAffin= ityTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNode= Id=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [top= Ver=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968593, order=3D1447= 412964196], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridC= acheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D14474129684= 99, order=3D1447412964118], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968595, order=3D1447412964198], t= imeout=3D100, ts=3D1447412968597, threadId=3D92, id=3D2267, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968595, order=3D14474= 12964198], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296850= 2, order=3D1447412964120], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968596, order=3D1447412964200], t= imeout=3D100, ts=3D1447412968607, threadId=3D88, id=3D2268, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968596, order=3D14474= 12964200], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296850= 4, order=3D1447412964122], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968601, order=3D1447412964202], t= imeout=3D100, ts=3D1447412968607, threadId=3D98, id=3D2269, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968601, order=3D14474= 12964202], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296850= 6, order=3D1447412964124], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968603, order=3D1447412964204], t= imeout=3D100, ts=3D1447412968607, threadId=3D72, id=3D2270, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968603, order=3D14474= 12964204], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296850= 8, order=3D1447412964126], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968605, order=3D1447412964206], t= imeout=3D100, ts=3D1447412968617, threadId=3D96, id=3D2271, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968605, order=3D14474= 12964206], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296851= 0, order=3D1447412964128], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968606, order=3D1447412964208], t= imeout=3D100, ts=3D1447412968617, threadId=3D80, id=3D2272, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968606, order=3D14474= 12964208], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296851= 4, order=3D1447412964132], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968606, order=3D1447412964210], t= imeout=3D100, ts=3D1447412968617, threadId=3D81, id=3D2273, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968606, order=3D14474= 12964210], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296851= 4, order=3D1447412964132], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968610, order=3D1447412964212], t= imeout=3D100, ts=3D1447412968617, threadId=3D97, id=3D2274, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968610, order=3D14474= 12964212], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296851= 6, order=3D1447412964134], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968612, order=3D1447412964214], t= imeout=3D100, ts=3D1447412968617, threadId=3D65, id=3D2275, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968612, order=3D14474= 12964214], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296851= 8, order=3D1447412964136], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968614, order=3D1447412964216], t= imeout=3D100, ts=3D1447412968617, threadId=3D89, id=3D2276, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968614, order=3D14474= 12964216], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 0, order=3D1447412964138], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D1447412964218], t= imeout=3D100, ts=3D1447412968627, threadId=3D73, id=3D2277, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D14474= 12964218], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 5, order=3D1447412964144], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D1447412964220], t= imeout=3D100, ts=3D1447412968627, threadId=3D74, id=3D2278, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D14474= 12964220], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 5, order=3D1447412964144], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D1447412964222], t= imeout=3D100, ts=3D1447412968627, threadId=3D71, id=3D2279, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968616, order=3D14474= 12964222], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 5, order=3D1447412964144], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968621, order=3D1447412964224], t= imeout=3D100, ts=3D1447412968627, threadId=3D87, id=3D2280, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968621, order=3D14474= 12964224], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 7, order=3D1447412964146], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull], GridCacheMvccCandidate [nodeId=3D8f8= c5147-2d8f-44e6-8da8-ef8a5bae1fd3, ver=3DGridCacheVersion [topVer=3D5889292= 4, nodeOrderDrId=3D3, globalTime=3D1447412968624, order=3D1447412964226], t= imeout=3D100, ts=3D1447412968627, threadId=3D76, id=3D2281, topVer=3DAffini= tyTopologyVersion [topVer=3D3, minorTopVer=3D0], reentry=3Dnull, otherNodeI= d=3Df35fe3b2-93e7-4da5-8c59-94325ccd971b, otherVer=3DGridCacheVersion [topV= er=3D58892924, nodeOrderDrId=3D3, globalTime=3D1447412968624, order=3D14474= 12964226], mappedDhtNodes=3Dnull, mappedNearNodes=3Dnull, ownerVer=3DGridCa= cheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime=3D144741296852= 9, order=3D1447412964148], serOrder=3Dnull, key=3DKeyCacheObjectImpl [val= =3D1, hasValBytes=3Dtrue], masks=3Dlocal=3D1|owner=3D0|ready=3D1|reentry=3D= 0|used=3D0|tx=3D0|single_implicit=3D0|dht_local=3D1|near_local=3D0|removed= =3D0, prevVer=3Dnull, nextVer=3Dnull]], rmts=3Dnull]], flags=3D0]]]], mappe= dVer=3DGridCacheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globalTime= =3D1447412968533, order=3D1447412964152], ver=3DGridCacheVersion [topVer=3D= 58892922, nodeOrderDrId=3D1, globalTime=3D0, order=3D0], tx=3Dnull, req=3DG= ridNearLockRequest [topVer=3DAffinityTopologyVersion [topVer=3D3, minorTopV= er=3D0], miniId=3D6b229800151-652e9fe6-66fa-4415-aeb1-1ec44adaf8d2, implici= tTx=3Dfalse, implicitSingleTx=3Dfalse, onePhaseCommit=3Dfalse, dhtVers=3D[n= ull], subjId=3Dnull, taskNameHash=3D0, hasTransforms=3Dfalse, syncCommit=3D= false, accessTtl=3D-1, retVal=3Dtrue, firstClientReq=3Dtrue, filter=3Dnull,= super=3DGridDistributedLockRequest [nodeId=3Df35fe3b2-93e7-4da5-8c59-94325= ccd971b, nearXidVer=3DGridCacheVersion [topVer=3D58892924, nodeOrderDrId=3D= 3, globalTime=3D1447412968533, order=3D1447412964152], threadId=3D77, futId= =3D4b229800151-652e9fe6-66fa-4415-aeb1-1ec44adaf8d2, timeout=3D100, isInTx= =3Dfalse, isInvalidate=3Dfalse, isRead=3Dfalse, isolation=3Dnull, retVals= =3D[true], txSize=3D1, flags=3D0, keysCnt=3D1, super=3DGridDistributedBaseM= essage [ver=3DGridCacheVersion [topVer=3D58892924, nodeOrderDrId=3D3, globa= lTime=3D1447412968533, order=3D1447412964152], candsByIdx=3Dnull, committed= Vers=3Dnull, rolledbackVers=3Dnull, cnt=3D0, super=3DGridCacheMessage [msgI= d=3D4463, depInfo=3Dnull, err=3Dnull, skipPrepare=3Dfalse]]]], err=3Dnull] > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.createLockReply(GridDhtTransactionalCacheAdapter.= java:1132) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.access$400(GridDhtTransactionalCacheAdapter.java:= 88) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$10.apply(GridDhtTransactionalCacheAdapter.java:98= 6) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$10.apply(GridDhtTransactionalCacheAdapter.java:97= 9) > =09at org.apache.ignite.internal.util.future.GridEmbeddedFuture$1.applyx(= GridEmbeddedFuture.java:59) > =09at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncList= ener1.apply(GridEmbeddedFuture.java:297) > =09at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncList= ener1.apply(GridEmbeddedFuture.java:290) > =09at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyList= ener(GridFutureAdapter.java:262) > =09at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyList= eners(GridFutureAdapter.java:250) > =09at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(Gri= dFutureAdapter.java:380) > =09at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(Gri= dFutureAdapter.java:346) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.onComplete(GridDhtLockFuture.java:733) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.onDone(GridDhtLockFuture.java:711) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.onDone(GridDhtLockFuture.java:80) > =09at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(Gri= dFutureAdapter.java:323) > =09at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComp= lete(GridCompoundFuture.java:285) > =09at org.apache.ignite.internal.util.future.GridCompoundFuture.markIniti= alized(GridCompoundFuture.java:276) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.map(GridDhtLockFuture.java:943) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= LockFuture.onOwnerChanged(GridDhtLockFuture.java:666) > =09at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$2.= onOwnerChanged(GridCacheMvccManager.java:156) > =09at org.apache.ignite.internal.processors.cache.distributed.GridDistrib= utedCacheEntry.checkOwnerChanged(GridDistributedCacheEntry.java:814) > =09at org.apache.ignite.internal.processors.cache.distributed.GridDistrib= utedCacheEntry.removeLock(GridDistributedCacheEntry.java:448) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= CacheEntry.removeLock(GridDhtCacheEntry.java:319) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.removeLocks(GridDhtTransactionalCacheAdapter.java= :1486) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.processNearUnlockRequest(GridDhtTransactionalCach= eAdapter.java:1336) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter.access$200(GridDhtTransactionalCacheAdapter.java:= 88) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$5.apply(GridDhtTransactionalCacheAdapter.java:150= ) > =09at org.apache.ignite.internal.processors.cache.distributed.dht.GridDht= TransactionalCacheAdapter$5.apply(GridDhtTransactionalCacheAdapter.java:148= ) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.proc= essMessage(GridCacheIoManager.java:540) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMe= ssage0(GridCacheIoManager.java:273) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.hand= leMessage(GridCacheIoManager.java:197) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager.acce= ss$000(GridCacheIoManager.java:76) > =09at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.on= Message(GridCacheIoManager.java:159) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.pro= cessRegularMessage0(GridIoManager.java:811) > =09at org.apache.ignite.internal.managers.communication.GridIoManager.acc= ess$1500(GridIoManager.java:106) > =09at org.apache.ignite.internal.managers.communication.GridIoManager$5.r= un(GridIoManager.java:774) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:745) > {noformat} > To reproduce run attached sources in the following order: > - Start two server nodes (ServerTest.java); > - Start a client node (ClientTest.java). > If the issue is not reproduced on your machine try to adjust {{lock.tryLo= ck}} timeout value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)