hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
Date Mon, 22 Oct 2018 15:02:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659058#comment-16659058

Weiwei Yang commented on YARN-8925:

Hi [~Tao Yang]

Good catch! I think this was discussed sometime before. Cc [~Naganarasimha], [~sunilg].

I just did a quick look at your patch, following comparing code
NodeAttributesUtils.isNodeAttributesEquals(nodeAttributes, currentNodeAttributes))
in added in each {{nodeHeartbeat}} call.

Instead of doing that, can we move the logic to NM side. See similar code in {{NodeStatusUpdaterImpl#getNodeLabelsForHeartbeat}}.
I think this way it avoids the overhead as the comparing in RM side is still expensive.

Does that make sense to you?

BTW, I am moving this under umbrella YARN-8766.


> Updating distributed node attributes only when necessary
> --------------------------------------------------------
>                 Key: YARN-8925
>                 URL: https://issues.apache.org/jira/browse/YARN-8925
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 3.2.1
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>            Priority: Major
>         Attachments: YARN-8925.001.patch
> Currently if distributed node attributes exist, even though there is no change, updating
for distributed node attributes will happen in every heartbeat between NM and RM. Updating
process will hold NodeAttributesManagerImpl#writeLock and may have some influence in a large
cluster. We have found nodes UI of a large cluster is opened slowly and most time it's waiting
for the lock in NodeAttributesManagerImpl. I think this updating should be called only when
necessary to enhance the performance of related process.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message