hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3487) CapacityScheduler.getQueueInfo grabs scheduler lock unnecessarily
Date Tue, 14 Apr 2015 21:36:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494940#comment-14494940
] 

Jason Lowe commented on YARN-3487:
----------------------------------

Sample stacktrace of a blocked thread:

{code}
"IPC Server handler 48 on x" daemon prio=10 tid=0x00007fd4991d5000 nid=0x5d53 waiting for
monitor entry [0x00007fd45cf1a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueInfo(CapacityScheduler.java:909)
        - waiting to lock <0x000000023ae2c938> (a org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:223)
        at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:96)
        at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:501)
        - locked <0x00000002616389e0> (a org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService$AllocateResponseLock)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
        at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
{code}

In 2.6 we started calling getQueueInfo to validate resource requests so we could check the
labels.  Unfortunately getQueueInfo grabs the scheduler lock unnecessarily:

{code}
  public QueueInfo getQueueInfo(String queueName, 
      boolean includeChildQueues, boolean recursive) 
  throws IOException {
    CSQueue queue = null;

    synchronized (this) {
      queue = this.queues.get(queueName); 
    }

    if (queue == null) {
      throw new IOException("Unknown queue: " + queueName);
    }
    return queue.getQueueInfo(includeChildQueues, recursive);
  }
{code}

this.queues is a ConcurrentHashMap, so there's not much utility in grabbing the lock just
to lookup something in that map.  Worse, it adds a lot of contention on an already highly
contentious lock.

> CapacityScheduler.getQueueInfo grabs scheduler lock unnecessarily
> -----------------------------------------------------------------
>
>                 Key: YARN-3487
>                 URL: https://issues.apache.org/jira/browse/YARN-3487
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>
> Recently saw a significant slowdown of applications on a large cluster, and we noticed
there were a large number of blocked threads on the RM.  Most of the blocked threads were
waiting for the CapacityScheduler lock while calling getQueueInfo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message