hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8198) Support multiple network interfaces
Date Wed, 11 Apr 2012 23:57:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252063#comment-13252063

Sanjay Radia commented on HADOOP-8198:

Eli, below is some feedback to improve the document/proposal and some questions. At some point,
the answers should move into the document. 
* Intro
  Not true that "Hadoop does not currently utilize multiple network interfaces". - Hadoop
relies on host-level bonding for multiple nics and this does not allow use case 1.2 but addresses
use case 1.1. Worth clarifying this in the intro.
* Use case 1.1 Layer 7 bonding
** Some folks get thrown off by the "Layer 7 bonding" thinking  that the proposal is to configure
switches, nic, etc to get  layer 7 bonding. Add a sentence to clarify.
** bandwidth and failover is the main motivation for this use case.
*** Q What are the limitations of  host level bonding for workers and masters for this use
case (as opposed to use case 1.2)?
** Add text that worker is DN or TT and Master is NN or JT,
* Use case 1.2 - i understand the text but have hard time buying this use case as a high priority.
 Can you please motivate this better. Can we separate this one for phase 2 (or were you planning
that anyway)?
* Use case 1.3 - you have text and filed jiras for a client config to use a specific nic -
is this a separate use case?
* Insert a section "Scope"
** Your description is general (Masters and Workers) but you limit the detailed sections to
HDFS and MR. How will things work if don't address this problem at all layers? (see my comment
below on issues/risks)
* Insert a section called "Issues/Risks"
** What if other services up the stack have not been modified and depend only on host-level-bonding?
** Minimize changes to DfsCLient so it can remain as thin as possible for future client side
porting to other languages. (There are solutions that do not require changes to the dfsClient
that we should consider in the short term).
* 2 Requirements
** some of the requirements seem to prescribe the solution. Separate out the solutions and
insert in a section between 2 and 3 describing the solution.
** 2.1 - you are not stating a requirement here -- reword - nics can be bonded or multi-homed.
Current wildcard IP config should be used for multi-homing. 
        State that the current multi-homing config does not help with use case 1.1 for worker
** 2.2 Multiple interfaces on master
*** I assume this is motived completely by use case 1.2 and not use case 1.1 since we don't
have bandwidth issues for masters - please add this clarification.
*** What will this do to tokens if tokens are obtained from one of the several interfaces
on a Master.
** 2.5 Enable clients to use multiple local interfaces - what is the motivation for this?
Is this for tasks running inside Hadoop  or for clients outside the hadoop cluster. (or is
this a yet another use case? If so add to use cases section.)
** Add requirements:
*** old configs should run unchanged
*** host level bonding should continue to work
*** security should work for the proposed enhancements
*** solution should work for all protocols - hadoop native, http rpc, etc. (if you are excluding
any please clarify).
*** should work with HA
* 3.1 Example Config ...
** This section could be significantly improved. You switch between "current config"  and
"how the current config's semantics are extended" and "new configuration"; this is sometimes
* 4.3 Security 
** looks like the paragraph is not complete - last sentence is cut off. 
** Also added some details on what change are needed for this to work with security. Part
of the paragraph reads like requirements statements and should move to the requirements section.
* 4.3 HA
** Add to requirements as noted above.
** Are there new issues here? Will have to failover multiple IPs  of the NN etc.
> Support multiple network interfaces
> -----------------------------------
>                 Key: HADOOP-8198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8198
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io, performance
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: MultipleNifsv1.pdf, MultipleNifsv2.pdf, MultipleNifsv3.pdf
> Hadoop does not currently utilize multiple network interfaces, which is a common user
request, and important in enterprise environments. This jira covers a proposal for enhancements
to Hadoop so it better utilizes multiple network interfaces. The primary motivation being
improved performance, performance isolation, resource utilization and fault tolerance. The
attached design doc covers the high-level use cases, requirements, a proposal for trunk/0.23,
discussion on related features, and a proposal for Hadoop 1.x that covers a subset of the
functionality of the trunk/0.23 proposal.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message