hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3137) CapacityScheduler.checkAccess unnecessarily grabs the scheduler lock
Date Thu, 11 May 2017 14:07:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006467#comment-16006467

Jason Lowe commented on YARN-3137:

Seeing this lock contention quite a bit in 2.8, and I guess I'm still a little confused why
we need a full CapacityScheduler lock to do this.  Even if we have a lock, we're either checking
just before or just after some kind of hierarchy change.  Queue lookup is not by path but
simply a concurrent hashmap lookup, so that's not an issue.   Either the queue is there or
it's not.  Then we can invoke checkAccess on the queue itself.  The queue knows its path in
the hierarchy, and again either we're checking before the queue is moved or after.  As long
as we lock the specific queue when trying to check the ACL we should be fine, but maybe I'm
missing something.

Can someone elaborate the scenario where moving the checkAccess lock from the highly-contended
CapacityScheduler lock to a queue-specific lock causes problems when the hierarchy changes?

> CapacityScheduler.checkAccess unnecessarily grabs the scheduler lock
> --------------------------------------------------------------------
>                 Key: YARN-3137
>                 URL: https://issues.apache.org/jira/browse/YARN-3137
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.5.0
>            Reporter: Jason Lowe
>            Assignee: Varun Saxena
> The queues are stored in a ConcurrentHashMap and the code already checks for a null queue.
 At best we need to lock individual queues when processing the access check, but I don't see
why we need to grab the highly-contested scheduler lock to lookup which queue to use for the
hasAccess call.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message