hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6025) Few issues in synchronization in CapacityScheduler & AbstractYarnScheduler
Date Sat, 24 Dec 2016 20:32:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15775397#comment-15775397

Wangda Tan commented on YARN-6025:

bq. IMHO FifoScheduler should extend and get it synchronized if required (nodeUpdate is already
done in that way).
Extend in FifoScheduler sounds like a better solution.

bq. May be i dint get it properly but based on my understanding either we need to hold write
lock in CS and invoke super.nodeUpdate..
YARN-5706 changes to use readLock for part of the nodeUpdate and writeLock for rest of the
part, two reasons:
- Use readLock to prevent add/remove of apps / queues / nodes, and to reduce critical section.
Because any changes to above collections need to hold writelock.
- Use writeLock to for container allocation.

> Few issues in synchronization in CapacityScheduler & AbstractYarnScheduler
> --------------------------------------------------------------------------
>                 Key: YARN-6025
>                 URL: https://issues.apache.org/jira/browse/YARN-6025
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler, scheduler
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: YARN-6025.01.patch
> YARN-3139 does optimization on the locks by introducing ReentrantReadWriteLock to remove
synchronized but seems to have some issues.
> # CapacityScheduler
> #* nodeUpdate(RMNode) need not be synchronized, as its the only one to be in the class
> #* setLastNodeUpdateTime in nodeUpdate needs to be updated with readLock ? then getLastNodeUpdateTime
is done without any lock and more over its volatile.
> #* getUserGroupMappingPlacementRule need not be public as its held called within and
not used in test and further is called from initScheduler and reinitialize where both are
holding write locks so i presume getting read locks are of no use.
> # AbstractYarnScheduler
> #* recoverContainersOnNode is synchronized as well as holds write lock on the complete
method so i presume we do not require synchronized here.
> #* nodeUpdate method too is synchronized but if i see the updates done inside i do not
see any place where node update from two different nodes will have any issues (except for
schedulerHealth which is taken care internally with concurrentHashMap), And even if require
we could better use write lock. (also depends on the decision of next point)
> #* readLock is only used in containerLaunchedOnNode which i am not completely sure whether
its required to have a read lock here, suppose we do not require then whether there is any
use of read write locks in AbstractYarnScheduler as in general there is performance overhead
in using readwrite lock over synchronized blocks on frequently accessed code path.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message