hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1008) MiniYARNCluster with multiple nodemanagers, all nodes have same key for allocations
Date Tue, 13 Aug 2013 13:15:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738175#comment-13738175

Alejandro Abdelnur commented on YARN-1008:

What the patch does:

Introduces a new configuration property in the RM, {{RM_SCHEDULER_USE_PORT_FOR_NODE_NAME}}
(defaulting to {{false}}). 

This is an RM property but it is to be used by the scheduler implementations when matching
a resourcerequest to a node.

If the property is set to {{false}} things work as today, the matching is done using only
the HOSTNAME obtained from the NodeId.

If the property is set to {{true}}, the matching is done using the HOSTNAME:PORT obtained
from the NodeId.

There are not changes in the NM or AM side. 

If the property is set to {{true}}, the AM must be aware of such setting and when creating
 resource requests, it must set the location as HOSTNAME:PORT of the node instead of just

The renaming of {{SchedulerNode#getHostName()}} to {{SchedulerNode#getNodeName()}} is to make
obvious to developers that may not be HOSTNAME. Added javadocs explaing this clearly.

This works with all 3 schedulers.

The main motivation for this change is to be able to use yarn minicluster with multiple NMs
and being able to target a specific NM instance.

We could expose this for production use if there is a need. For that we would need to:

* Expose via ApplicationReport the node matching mode: HOSTNAME or HOSTNAME:PORT
* Provide a mechanism for AMs only aware of HOSTNAME matching mode to work with HOSTNAME:PORT

If there is a usecase for this in a real deployment we should follow up this with another

> MiniYARNCluster with multiple nodemanagers, all nodes have same key for allocations
> -----------------------------------------------------------------------------------
>                 Key: YARN-1008
>                 URL: https://issues.apache.org/jira/browse/YARN-1008
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: YARN-1008.patch, YARN-1008.patch, YARN-1008.patch
> While the NMs are keyed using the NodeId, the allocation is done based on the hostname.

> This makes the different nodes indistinguishable to the scheduler.
> There should be an option to enabled the host:port instead just port for allocations.
The nodes reported to the AM should report the 'key' (host or host:port). 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message