hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milan Potocnik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces
Date Tue, 29 Jul 2014 21:35:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078412#comment-14078412
] 

Milan Potocnik commented on YARN-1994:
--------------------------------------

[~cwelch]
I'll try to explain one of the use cases.

Let's say we have following interfaces in our network:
 - 1 ethernet, public network
 - 2 IB, private network. Please note that on Windows, IB does not support teaming

On DNS Server, DNS entry for machine's hostname can resolve to any of the three interfaces
(for each 'hostname' entry - three IP addresses). We also add a special DNS entry for each
machine that resolves only to two IB interfaces, let's say in the form of 'hostname-IB'.

Use case 1: We want internal communication in the cluster to always use IB. We also want to
be fault tolerant if one of the IB fails (remember, no teaming on Windows). In order to bind
to both IB interfaces, we must set bind address to 0.0.0.0. When this is set, clients when
connecting will currently get hostname, which in some cases (DNS server usually returns IPs
by round-robin) will resolve to Ethernet IP address, which could be blocked by firewall, or
it might degrade performance in internal communication.

By setting _BIND_HOST to 0.0.0.0 and _ADDRESS to 'hostname-IB' we avoid the non-determinism
of InetSocketAddress.getHostName()

For outside clients we can also control connectivity by making sure they connect via the public
network, but this is a simpler problem, since they would use different DNS server.



> Expose YARN/MR endpoints on multiple interfaces
> -----------------------------------------------
>
>                 Key: YARN-1994
>                 URL: https://issues.apache.org/jira/browse/YARN-1994
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager, webapp
>    Affects Versions: 2.4.0
>            Reporter: Arpit Agarwal
>            Assignee: Craig Welch
>         Attachments: YARN-1994.0.patch, YARN-1994.1.patch, YARN-1994.11.patch, YARN-1994.11.patch,
YARN-1994.12.patch, YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch, YARN-1994.5.patch,
YARN-1994.6.patch, YARN-1994.7.patch
>
>
> YARN and MapReduce daemons currently do not support specifying a wildcard address for
the server endpoints. This prevents the endpoints from being accessible from all interfaces
on a multihomed machine.
> Note that if we do specify INADDR_ANY for any of the options, it will break clients as
they will attempt to connect to 0.0.0.0. We need a solution that allows specifying a hostname
or IP-address for clients while requesting wildcard bind for the servers.
> (List of endpoints is in a comment below)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message