hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4002) make ResourceTrackerService.nodeHeartbeat more concurrent
Date Thu, 19 May 2016 20:16:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292058#comment-15292058
] 

Hudson commented on YARN-4002:
------------------------------

SUCCESS: Integrated in Hadoop-trunk-Commit #9825 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9825/])
YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. (jianhe: rev feb90ffcca536e7deac50976b8a8774450fe089f)
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java


> make ResourceTrackerService.nodeHeartbeat more concurrent
> ---------------------------------------------------------
>
>                 Key: YARN-4002
>                 URL: https://issues.apache.org/jira/browse/YARN-4002
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Hong Zhiguo
>            Assignee: Hong Zhiguo
>            Priority: Critical
>             Fix For: 2.8.0
>
>         Attachments: 0001-YARN-4002.patch, YARN-4002-lockless-read.patch, YARN-4002-rwlock-v2.patch,
YARN-4002-rwlock-v2.patch, YARN-4002-rwlock-v3-rebase.patch, YARN-4002-rwlock-v3.patch, YARN-4002-rwlock-v4.patch,
YARN-4002-rwlock-v5.patch, YARN-4002-rwlock-v6.patch, YARN-4002-rwlock.patch, YARN-4002-v0.patch
>
>
> We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By design the method
ResourceTrackerService.nodeHeartbeat should be concurrent enough to scale for large clusters.
> But we have a "BIG" lock in NodesListManager.isValidNode which I think it's unnecessary.
> First, the fields "includes" and "excludes" of HostsFileReader are only updated on "refresh
nodes".  All RPC threads handling node heartbeats are only readers.  So RWLock could be used
to  alow concurrent access by RPC threads.
> Second, since he fields "includes" and "excludes" of HostsFileReader are always updated
by "reference assignment", which is atomic in Java, the reader side lock could just be skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message