hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Zhiguo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4002) make ResourceTrackerService.nodeHeartbeat more concurrent
Date Fri, 31 Jul 2015 02:25:05 GMT
Hong Zhiguo created YARN-4002:

             Summary: make ResourceTrackerService.nodeHeartbeat more concurrent
                 Key: YARN-4002
                 URL: https://issues.apache.org/jira/browse/YARN-4002
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Hong Zhiguo
            Assignee: Hong Zhiguo
            Priority: Critical

We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By design the method
ResourceTrackerService.nodeHeartbeat should be concurrent enough to scale for large clusters.
But we have a "BIG" log in NodesListManager.isValidNode which I think it's unnecessary.
First, the fields "includes" and "excludes" of HostsFileReader are only updated on "refresh
nodes".  All RPC threads handling node heartbeats are only readers.  So RWLock could be used
to have alow concurrently access by RPC threads.
Second, since he fields "includes" and "excludes" of HostsFileReader are always updated by
"reference assignment", which is atomic in Java, the reader side lock could just be skipped.

This message was sent by Atlassian JIRA

View raw message