hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4024) YARN RM should avoid resolving IP when NMs doing heartbeat
Date Thu, 06 Aug 2015 18:17:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660546#comment-14660546

Wangda Tan commented on YARN-4024:

Actually, there're two problems we need to solve:
1. DNS resolve hostname is under synchronized lock of hostsReader:
In NodesListManager:
  public boolean isValidNode(String hostName) {
    synchronized (hostsReader) {
      Set<String> hostsList = hostsReader.getHosts();
      Set<String> excludeList = hostsReader.getExcludedHosts();
      String ip = NetUtils.normalizeHostName(hostName);
      return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
          && !(excludeList.contains(hostName) || excludeList.contains(ip));
We should take {{NetUtils.normalizeHostName(hostName);}} out of the synchronized lock.

2. We may need to cache IP address when NMs doing registering.
If we assume IP address won't change after NM registered, we can cache resolved IP address
for each host. We can add a switch to turn it on when we needed.

> YARN RM should avoid resolving IP when NMs doing heartbeat
> ----------------------------------------------------------
>                 Key: YARN-4024
>                 URL: https://issues.apache.org/jira/browse/YARN-4024
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> Currently, YARN RM NodesListManager will resolve IP address every time when node doing
heartbeat. When DNS server becomes slow, NM heartbeat will be blocked and cannot make progress.

This message was sent by Atlassian JIRA

View raw message