hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
Date Mon, 19 Nov 2018 03:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691219#comment-16691219
] 

Tao Yang commented on YARN-8925:
--------------------------------

Thanks [~cheersyang] for the testing.
I think the node attributes are updated as expected but logs in NM are misunderstanding, RM
will update node attributes when registering nodes and will not update when handling heartbeats
later, because there is no change for node attributes and double check exists in RM side as
follows:
{code:java}
    if (request.getNodeAttributes() != null) {
      try {
        // update node attributes if necessary then update heartbeat response
        boolean updated = updateNodeAttributesIfNecessary(nodeId,
            request.getNodeAttributes());
        nodeHeartBeatResponse.setAreNodeAttributesAcceptedByRM(updated);
      } catch (IOException ex) {
        //ensure the error message is captured and sent across in response
        String errorMsg = ...
        nodeHeartBeatResponse.setDiagnosticsMessage(errorMsg);
        nodeHeartBeatResponse.setAreNodeAttributesAcceptedByRM(false);
      }
    }
{code}
I did't notice the logs in NM side, so just set areNodeAttributesAcceptedByRM=false in heartbeat
response when there is no change for node attributes. I think here should be changed to set
true whether or not it's necessary like this:
{code:java}
    if (request.getNodeAttributes() != null) {
      try {
        // update node attributes if necessary then update heartbeat response  
        // !!!!!!!!!change the following two lines
        updateNodeAttributesIfNecessary(nodeId, request.getNodeAttributes());
        nodeHeartBeatResponse.setAreNodeAttributesAcceptedByRM(true);
      } catch (IOException ex) {
        //ensure the error message is captured and sent across in response
        String errorMsg = ...
        nodeHeartBeatResponse.setDiagnosticsMessage(errorMsg);
        nodeHeartBeatResponse.setAreNodeAttributesAcceptedByRM(false);
      }
    }
{code}
Thoughts?

> Updating distributed node attributes only when necessary
> --------------------------------------------------------
>
>                 Key: YARN-8925
>                 URL: https://issues.apache.org/jira/browse/YARN-8925
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.2.1
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>            Priority: Major
>              Labels: performance
>         Attachments: YARN-8925.001.patch, YARN-8925.002.patch, YARN-8925.003.patch, YARN-8925.004.patch,
YARN-8925.005.patch, YARN-8925.006.patch, YARN-8925.007.patch
>
>
> Currently if distributed node attributes exist, even though there is no change, updating
for distributed node attributes will happen in every heartbeat between NM and RM. Updating
process will hold NodeAttributesManagerImpl#writeLock and may have some influence in a large
cluster. We have found nodes UI of a large cluster is opened slowly and most time it's waiting
for the lock in NodeAttributesManagerImpl. I think this updating should be called only when
necessary to enhance the performance of related process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message