hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Waldyn Benbenek (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6117) Hadoop ignores yarn.nodemanager.hostname for RPC listeners
Date Wed, 15 Oct 2014 18:50:36 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Waldyn Benbenek updated MAPREDUCE-6117:
---------------------------------------
    Release Note: 
This patch has few new tests for the following reasons:
TestTaskAttemptListenerImpl does not test or even perform the service start where the change
is made. This is because that would starting a new process.
TestMRClientService already checks the NM_HOST which change does effect. 
The change pulls the NM_HOST from the environment.  This needs to be passed to a spawned process
which none of the tests do.  
In general , it would be better if NM_HOST were more pervasive, that is, if the property were
passed to the all the parts of the application, in particular the parts that deal with RPC.
 Since that is not the case, I have chosen to pull it from the environment where once can
depend upon its being. 

I have tested it in clusters with multiple networks where the nm host is configured and those
where it is not.  It works as designed.  That is, if the NM host is configured on the node
the TaskAttempt Listner  and the Client Service listen on the give NM host, otherwise they
listen on the node's "hostname".

  was:
This patch has no new tests for the following reasons:
TestTaskAttemptListenerImpl does not test or even perform the service start where the change
is made. This is because that would starting a new process.
TestMRClientService already checks the NM_HOST which change does effect. 
The change pulls the NM_HOST from the environment.  This needs to be passed to a spawned process
which none of the tests do.  
In general , it would be better if NM_HOST were more pervasive, that is, if the property were
passed to the all the parts of the application, in particular the parts that deal with RPC.
 Since that is not the case, I have chosen to pull it from the environment where once can
depend upon its being. 

I have tested it in clusters with multiple networks where the nm host is configured and those
where it is not.  It works as designed.  That is, if the NM host is configured on the node
the TaskAttempt Listner  and the Client Service listen on the give NM host, otherwise they
listen on the node's "hostname".

          Status: Patch Available  (was: Open)

> Hadoop ignores yarn.nodemanager.hostname for RPC listeners
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-6117
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6117
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: client, task
>    Affects Versions: 2.5.1, 2.4.1, 2.2.1
>         Environment: Any mapreduce example with standard cluster.  In our case each node
has four networks.  It is important that all internode communication be done on a specific
network.
>            Reporter: Waldyn Benbenek
>            Assignee: Waldyn Benbenek
>             Fix For: 2.5.1
>
>         Attachments: MapReduce-534.patch
>
>   Original Estimate: 48h
>          Time Spent: 384h
>  Remaining Estimate: 0h
>
> The RPC listeners for an application are using the hostname of the node as the binding
address of the listener,  They ignore yarn.nodemanager.hostname for this.  In our setup we
want all communication between nodes to be done via the network addresses we specify in yarn.nodemanager.hostname
on each node.  
> TaskAttemptListenerImpl.java and MRClientService.java are two places I have found where
the default address is used rather that NM_host.   The node Manager hostname should be used
for all communication between nodes including the RPC listeners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message