hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4454) NM to nodelabel mapping going wrong after RM restart
Date Fri, 18 Dec 2015 19:09:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064513#comment-15064513
] 

Wangda Tan commented on YARN-4454:
----------------------------------

[~bibinchundatt], thanks for reporting and looking at the issue. 

The root cause of this issue is, when the RM restart first time, it will generate a mirror
file which has a complete node->label mappings:
{code}
node1:port=x 
node1=y
{code}

And when we restart the RM again, we will load the mapping, but node1:port loaded first, so
node1=y will overwrite the previous one.

In: {{org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager#checkReplaceLabelsOnNode}}

Instead of directly iterate the map:
{code}
    for (Entry<NodeId, Set<String>> entry : replaceLabelsToNode.entrySet()) {
      NodeId nodeId = entry.getKey();
{code}
We should sort the map so that the node without port should be handled first before node with
port specified to avoid overwriting happens.

Is it make sense to you?

> NM to nodelabel mapping going wrong after RM restart
> ----------------------------------------------------
>
>                 Key: YARN-4454
>                 URL: https://issues.apache.org/jira/browse/YARN-4454
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>            Priority: Critical
>         Attachments: test.patch
>
>
> *Nodelabel mapping with NodeManager  is going wrong if combination of hostname and then
NodeId is used to update nodelabel mapping*
> *Steps to reproduce*
> 1.Create cluster with 2 NM
> 2.Add label X,Y to cluster
> 3.replace  Label of node  1 using <HOSTNAME1:PORT>,x
> 4.replace label for node 1 by <HOSTNAME1>,y
> 5.Again replace label of node 1 by <HOSTNAME1:PORT>,x
> Check cluster label mapping HOSTNAME1 will be mapped with X 
> Now restart RM 2 times NODE LABEL mapping of HOSTNAME1:PORT changes to Y
> {noformat}
> 2015-12-14 17:17:54,901 INFO org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager:
Add labels: [<ResourcePool_1:exclusivity=true>,<ResourcePool_null:exclusivity=true>]
> 2015-12-14 17:17:54,905 INFO org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager:
REPLACE labels on nodes:
> 2015-12-14 17:17:54,906 INFO org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager:
  NM=host-10-19-92-188:64318, labels=[ResourcePool_1]
> 2015-12-14 17:17:54,906 INFO org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager:
  NM=host-10-19-92-188:0, labels=[ResourcePool_null]
> 2015-12-14 17:17:54,906 INFO org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager:
  NM=host-10-19-92-187:64318, labels=[ResourcePool_null]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message