Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 28 Feb 2017 19:27:45 +0000 (UTC)
From: "Arun Suresh (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13047070.1488309720000.21846.1488310065519@Atlassian.JIRA>
In-Reply-To: <JIRA.13047070.1488309720000@Atlassian.JIRA>
References: <JIRA.13047070.1488309720000@Atlassian.JIRA> <JIRA.13047070.1488309720909@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6251) Fix Scheduler locking issue
 introduced by YARN-6216
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 28 Feb 2017 19:27:50 -0000


    [ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888722#comment-15888722 ] 

Arun Suresh commented on YARN-6251:
-----------------------------------

Posting relevant section of the jstack dump

{noformat}
Found one Java-level deadlock:
=============================
"pool-7-thread-88":
  waiting for ownable synchronizer 0x0000000083635b98, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "SchedulerEventDispatcher:Event Processor"
"SchedulerEventDispatcher:Event Processor":
  waiting for ownable synchronizer 0x00000000f3f6b808, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-7-thread-88"

Java stack information for the threads listed above:
===================================================
"pool-7-thread-88":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000083635b98> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1520)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainerInternal(CapacityScheduler.java:1600)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:602)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyUpdatedContainers(SchedulerApplicationAttempt.java:852)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyIncreasedContainers(SchedulerApplicationAttempt.java:789)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.getAllocation(FiCaSchedulerApp.java:693)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocate(CapacityScheduler.java:925)
        at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.allocate(SLSCapacityScheduler.java:191)
        at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocateInternal(ApplicationMasterService.java:581)
        at org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocateInternal(OpportunisticContainerAllocatorAMService.java:254)
        at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:446)
        - locked <0x00000000f3fe59c0> (a org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService$AllocateResponseLock)
        at org.apache.hadoop.yarn.sls.appmaster.PromotingAMSimulator$1.run(PromotingAMSimulator.java:267)
        at org.apache.hadoop.yarn.sls.appmaster.PromotingAMSimulator$1.run(PromotingAMSimulator.java:264)
        at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.middleStep(AMSimulator.java:179)
        at org.apache.hadoop.yarn.sls.scheduler.TaskRunner$Task.run(TaskRunner.java:96)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
"SchedulerEventDispatcher:Event Processor":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f3f6b808> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.getHeadroom(FiCaSchedulerApp.java:755)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.allocateResource(LeafQueue.java:1578)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.apply(LeafQueue.java:1206)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:560)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2378)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.submitResourceCommitRequest(CapacityScheduler.java:2263)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1248)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1237)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1326)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1124)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:988)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1368)
        at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:254)
        at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:84)
        at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
        at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.
{noformat}

> Fix Scheduler locking issue introduced by YARN-6216
> ---------------------------------------------------
>
>                 Key: YARN-6251
>                 URL: https://issues.apache.org/jira/browse/YARN-6251
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>             Fix For: 3.0.0-alpha3
>
>
> Opening to track a locking issue that was uncovered when running a custom SLS AMSimulator.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org