ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roman Shtykh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-7696) Deadlock at GridDhtAtomicCache.lockEntries called through GridDhtAtomicCache.updateAllAsyncInternal
Date Tue, 20 Feb 2018 03:28:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369680#comment-16369680
] 

Roman Shtykh commented on IGNITE-7696:
--------------------------------------

[~frsyuki] I suggest you provide cluster configuration details and raise attention to the
issue in the ml.

> Deadlock at GridDhtAtomicCache.lockEntries called through GridDhtAtomicCache.updateAllAsyncInternal
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-7696
>                 URL: https://issues.apache.org/jira/browse/IGNITE-7696
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>    Affects Versions: 2.3
>         Environment: * Ignite 2.3
>  * OpenJDK version "1.8.0_151"
>  * Linux 4.4.0
>            Reporter: Sadayuki Furuhashi
>            Priority: Major
>
> We observed that all nodes in a cluster completely stalls and put/get/remove operations to
a cache block for ever. When it happens, we can see following log in thread dump:
> {code}
> 2018-02-14_04:21:33.84410 Found one Java-level deadlock:
> 2018-02-14_04:21:33.84410 =============================
> 2018-02-14_04:21:33.84411 "sys-#41%IgniteManager%":
> 2018-02-14_04:21:33.84411   waiting to lock monitor 0x00007f6d5e41a558 (object 0x0000000781083ef0,
a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry),
> 2018-02-14_04:21:33.84411   which is held by "sys-stripe-5-#6%IgniteManager%"
> 2018-02-14_04:21:33.84412 "sys-stripe-5-#6%IgniteManager%":
> 2018-02-14_04:21:33.84412   waiting to lock monitor 0x00007f6d5e41de68 (object 0x0000000781083e70,
a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
> 2018-02-14_04:21:33.84412   in JNI, which is held by "sys-stripe-2-#3%IgniteManager%"
> 2018-02-14_04:21:33.84412 "sys-stripe-2-#3%IgniteManager%":
> 2018-02-14_04:21:33.84413   waiting to lock monitor 0x00007f6d5e41a558 (object 0x0000000781083ef0,
a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
> 2018-02-14_04:21:33.84413   in JNI, which is held by "sys-stripe-5-#6%IgniteManager%"
> 2018-02-14_04:21:33.84413
> 2018-02-14_04:21:33.84414 Java stack information for the threads listed above:
> 2018-02-14_04:21:33.84414 ===================================================
> 2018-02-14_04:21:33.84416 "sys-#41%IgniteManager%":
> 2018-02-14_04:21:33.84416       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.markObsoleteVersion(GridCacheMapEntry.java:2153)
> 2018-02-14_04:21:33.84417       - waiting to lock <0x0000000781083ef0> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry)
> 2018-02-14_04:21:33.84417       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.removeVersionedEntry(GridDhtLocalPartition.java:368)
> 2018-02-14_04:21:33.84418       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.cleanupRemoveQueue(GridDhtLocalPartition.java:392)
> 2018-02-14_04:21:33.84418       at org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1.run(GridCacheProcessor.java:4051)
> 2018-02-14_04:21:33.84418       at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6687)
> 2018-02-14_04:21:33.84419       at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
> 2018-02-14_04:21:33.84419       at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 2018-02-14_04:21:33.84419       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2018-02-14_04:21:33.84420       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2018-02-14_04:21:33.84421       at java.lang.Thread.run(Thread.java:748)
> 2018-02-14_04:21:33.84421 "sys-stripe-5-#6%IgniteManager%":
> 2018-02-14_04:21:33.84421       at sun.misc.Unsafe.monitorEnter(Native Method)
> 2018-02-14_04:21:33.84421       at org.apache.ignite.internal.util.GridUnsafe.monitorEnter(GridUnsafe.java:1207)
> 2018-02-14_04:21:33.84422       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2848)
> 2018-02-14_04:21:33.84422       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1707)
> 2018-02-14_04:21:33.84423       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1629)
> 2018-02-14_04:21:33.84423       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3056)
> 2018-02-14_04:21:33.84424       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:131)
> 2018-02-14_04:21:33.84424       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:267)
> 2018-02-14_04:21:33.84425       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:262)
> 2018-02-14_04:21:33.84425       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
> 2018-02-14_04:21:33.84425       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> 2018-02-14_04:21:33.84426       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> 2018-02-14_04:21:33.84426       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> 2018-02-14_04:21:33.84426       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> 2018-02-14_04:21:33.84427       at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> 2018-02-14_04:21:33.84427       at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
> 2018-02-14_04:21:33.84427       at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> 2018-02-14_04:21:33.84428       at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
> 2018-02-14_04:21:33.84429       at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
> 2018-02-14_04:21:33.84429       at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
> 2018-02-14_04:21:33.84429       at java.lang.Thread.run(Thread.java:748)
> 2018-02-14_04:21:33.84429 "sys-stripe-2-#3%IgniteManager%":
> 2018-02-14_04:21:33.84430       at sun.misc.Unsafe.monitorEnter(Native Method)
> 2018-02-14_04:21:33.84430       at org.apache.ignite.internal.util.GridUnsafe.monitorEnter(GridUnsafe.java:1207)
> 2018-02-14_04:21:33.84430       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:2848)
> 2018-02-14_04:21:33.84431       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1707)
> 2018-02-14_04:21:33.84431       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1629)
> 2018-02-14_04:21:33.84431       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3056)
> 2018-02-14_04:21:33.84433       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:131)
> 2018-02-14_04:21:33.84433       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:267)
> 2018-02-14_04:21:33.84433       at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:262)
> 2018-02-14_04:21:33.84434       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
> 2018-02-14_04:21:33.84434       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> 2018-02-14_04:21:33.84434       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> 2018-02-14_04:21:33.84435       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> 2018-02-14_04:21:33.84435       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> 2018-02-14_04:21:33.84436       at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> 2018-02-14_04:21:33.84437       at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
> 2018-02-14_04:21:33.84437       at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> 2018-02-14_04:21:33.84437       at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
> 2018-02-14_04:21:33.84438       at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
> 2018-02-14_04:21:33.84438       at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
> 2018-02-14_04:21:33.84438       at java.lang.Thread.run(Thread.java:748)
> 2018-02-14_04:21:33.84439
> 2018-02-14_04:21:33.84439 Found 1 deadlock.
> {code}
> This indicates that GridDhtAtomicCache.lockEntries (threads sys-stripe-5-... and sys-stripe-7-...)
is causing deadlock, and GridCacheMapEntry.markObsoleteVersion (sys-#41...) is involved in
it:
>  * Thread "sys-stripe-5-..." locked 0x...3ef0, waits for 0x...3e70
>  * Thread "sys-stripe-7-..." locked 0x...3e70, waits for 0x...3ef0
>  * Thread "sys-#41..." waits for 0x...3ef0
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message