ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vyacheslav Koptilin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-11640) IgnitePdsPageEvictionDuringPartitionClearTest sometimes hangs on node stop
Date Wed, 27 Mar 2019 14:56:00 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vyacheslav Koptilin updated IGNITE-11640:
-----------------------------------------
    Description: 
It seems that this hang can be explained as follows:
 * 'vacuum-cleaner' thread tries to clean up a cache entry and so it needs to acquire the
checkpoint read lock.
 It is important to note that gateway lock for the corresponding cache is already acquired.
 Let's assume the checkpoint read lock cannot be acquired right now because there are many
dirty pages, and, therefore, the 'vacuum' has to wait for a next checkpoint.

{noformat}
"vacuum-cleaner-#484%db.IgnitePdsPageEvictionDuringPartitionClearTest0%" #587 prio=5 os_prio=0
tid=0x00007f5e2c007800 nid=0x1f960e waiting on condition [0x00007f5ee49eb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1565)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2395)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2270)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2188)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748){noformat}
 * On the other hand, the checkpoint thread can be already stopped by Ignition#stop(cancel
= true)
 In that case, scheduledCp.cpFinishFut is not completed (it seems to me, this is definitely
a bug) and leads to a deadlock:

{noformat}
"test-runner-#302%db.IgnitePdsPageEvictionDuringPartitionClearTest%" #388 prio=5 os_prio=0
tid=0x00007f60f461b800 nid=0x1f954b waiting on condition [0x00007f60338fc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000095fe0948> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:219)
at org.apache.ignite.internal.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:315)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.blockGateways(GridCacheProcessor.java:1142)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2399)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2283)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2570)
- locked <0x000000009cd5fcc8> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2533)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:330)
at org.apache.ignite.Ignition.stop(Ignition.java:223){noformat}

  was:
It seems that this hang can be explained as follows:
 # 'vacuum-cleaner' thread tries to clean up a cache entry and so it needs to acquire the
checkpoint read lock.
It is important to note that gateway lock for the corresponding cache is already acquired.
Let's assume the checkpoint read lock cannot be acquired right now because there are many
dirty pages, and, therefore, the 'vacuum' has to wait for a next checkpoint.

{noformat}
"vacuum-cleaner-#484%db.IgnitePdsPageEvictionDuringPartitionClearTest0%" #587 prio=5 os_prio=0
tid=0x00007f5e2c007800 nid=0x1f960e waiting on condition [0x00007f5ee49eb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1565)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2395)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2270)
at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2188)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748){noformat}

 # On the other hand, the checkpoint thread can be already stopped by Ignition#stop(cancel
= true)
In that case, scheduledCp.cpFinishFut is not completed (it seems to me, this is definitely
a bug) and leads to a deadlock:

{noformat}
"test-runner-#302%db.IgnitePdsPageEvictionDuringPartitionClearTest%" #388 prio=5 os_prio=0
tid=0x00007f60f461b800 nid=0x1f954b waiting on condition [0x00007f60338fc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000095fe0948> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:219)
at org.apache.ignite.internal.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:315)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.blockGateways(GridCacheProcessor.java:1142)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2399)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2283)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2570)
- locked <0x000000009cd5fcc8> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2533)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:330)
at org.apache.ignite.Ignition.stop(Ignition.java:223){noformat}


> IgnitePdsPageEvictionDuringPartitionClearTest sometimes hangs on node stop
> --------------------------------------------------------------------------
>
>                 Key: IGNITE-11640
>                 URL: https://issues.apache.org/jira/browse/IGNITE-11640
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 2.7
>            Reporter: Vyacheslav Koptilin
>            Assignee: Vyacheslav Koptilin
>            Priority: Major
>             Fix For: 2.8
>
>
> It seems that this hang can be explained as follows:
>  * 'vacuum-cleaner' thread tries to clean up a cache entry and so it needs to acquire
the checkpoint read lock.
>  It is important to note that gateway lock for the corresponding cache is already acquired.
>  Let's assume the checkpoint read lock cannot be acquired right now because there are
many dirty pages, and, therefore, the 'vacuum' has to wait for a next checkpoint.
> {noformat}
> "vacuum-cleaner-#484%db.IgnitePdsPageEvictionDuringPartitionClearTest0%" #587 prio=5
os_prio=0 tid=0x00007f5e2c007800 nid=0x1f960e waiting on condition [0x00007f5ee49eb000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
> at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1565)
> at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.cleanup(MvccProcessorImpl.java:2395)
> at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.processPartition(MvccProcessorImpl.java:2270)
> at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2188)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748){noformat}
>  * On the other hand, the checkpoint thread can be already stopped by Ignition#stop(cancel
= true)
>  In that case, scheduledCp.cpFinishFut is not completed (it seems to me, this is definitely
a bug) and leads to a deadlock:
> {noformat}
> "test-runner-#302%db.IgnitePdsPageEvictionDuringPartitionClearTest%" #388 prio=5 os_prio=0
tid=0x00007f60f461b800 nid=0x1f954b waiting on condition [0x00007f60338fc000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000095fe0948> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
> at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
> at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:219)
> at org.apache.ignite.internal.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:315)
> at org.apache.ignite.internal.processors.cache.GridCacheProcessor.blockGateways(GridCacheProcessor.java:1142)
> at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2399)
> at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2283)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2570)
> - locked <0x000000009cd5fcc8> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2533)
> at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:330)
> at org.apache.ignite.Ignition.stop(Ignition.java:223){noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message