hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8198) Support multiple network interfaces
Date Wed, 11 Apr 2012 14:23:17 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251617#comment-13251617

Daryn Sharp commented on HADOOP-8198:

As Eli alluded to, I have been trying to formulate a more sustainable model for mapping tokens
to their host.  The current token service model is rapidly breaking with the advent of features
like HA and multi-interface hosts.  The use of ip service (ip:port) provides no portability
of tokens.  Using host-based tokens (host:port) provides portability when a host's ip changes.

HA and multi-interface hosts create a larger challenge that host-based tokens do not address.
 HA currently rewrites the service of a fetched token to an abstract identifier.  The token
is later duplicated and the service is rewritten to each of the HA NN's services.  This approach
only works if all clients are identically configured.  Cross-colo access requires all grids
to share configurations which becomes a difficult maintenance issue.

Multi-interface hosts would require the same concept of HA - an abstract identifier - but
the client cannot chose the identifier w/o further complicated config settings.  What I'd
like to propose is the client no longer manages the service of a token.  The server will set
the service to an abstract id for itself.  The id will be presented in the SASL challenge.
 The client will chose the token based on the id.  This approach would neatly solve the dichotomy
between ips/hosts/HA/multi-interface.

Whether multi-interface support is added, I think the token service model needs to be evolved
because it's becoming fragile and relying too much on the client to manage the service correctly.
 I'll create another jira and write up a design proposal as soon as I get a chance.

> Support multiple network interfaces
> -----------------------------------
>                 Key: HADOOP-8198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8198
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io, performance
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: MultipleNifsv1.pdf, MultipleNifsv2.pdf, MultipleNifsv3.pdf
> Hadoop does not currently utilize multiple network interfaces, which is a common user
request, and important in enterprise environments. This jira covers a proposal for enhancements
to Hadoop so it better utilizes multiple network interfaces. The primary motivation being
improved performance, performance isolation, resource utilization and fault tolerance. The
attached design doc covers the high-level use cases, requirements, a proposal for trunk/0.23,
discussion on related features, and a proposal for Hadoop 1.x that covers a subset of the
functionality of the trunk/0.23 proposal.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message