Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Wed, 11 Apr 2012 23:57:18 +0000 (UTC)
From: "Sanjay Radia (Commented) (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: 
 <1371226605.15382.1334188638646.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <625973047.1326.1332395902366.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-8198) Support multiple network
 interfaces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252063#comment-13252063 ] 

Sanjay Radia commented on HADOOP-8198:
--------------------------------------

Eli, below is some feedback to improve the document/proposal and some questions. At some point, the answers should move into the document. 
* Intro
  Not true that "Hadoop does not currently utilize multiple network interfaces". - Hadoop relies on host-level bonding for multiple nics and this does not allow use case 1.2 but addresses use case 1.1. Worth clarifying this in the intro.
* Use case 1.1 Layer 7 bonding
** Some folks get thrown off by the "Layer 7 bonding" thinking  that the proposal is to configure switches, nic, etc to get  layer 7 bonding. Add a sentence to clarify.
** bandwidth and failover is the main motivation for this use case.
*** Q What are the limitations of  host level bonding for workers and masters for this use case (as opposed to use case 1.2)?
** Add text that worker is DN or TT and Master is NN or JT,
* Use case 1.2 - i understand the text but have hard time buying this use case as a high priority.  Can you please motivate this better. Can we separate this one for phase 2 (or were you planning that anyway)?
* Use case 1.3 - you have text and filed jiras for a client config to use a specific nic - is this a separate use case?
* Insert a section "Scope"
** Your description is general (Masters and Workers) but you limit the detailed sections to HDFS and MR. How will things work if don't address this problem at all layers? (see my comment below on issues/risks)
* Insert a section called "Issues/Risks"
** What if other services up the stack have not been modified and depend only on host-level-bonding?
** Minimize changes to DfsCLient so it can remain as thin as possible for future client side porting to other languages. (There are solutions that do not require changes to the dfsClient that we should consider in the short term).
* 2 Requirements
** some of the requirements seem to prescribe the solution. Separate out the solutions and insert in a section between 2 and 3 describing the solution.
** 2.1 - you are not stating a requirement here -- reword - nics can be bonded or multi-homed. Current wildcard IP config should be used for multi-homing. 
        State that the current multi-homing config does not help with use case 1.1 for worker nodes.
** 2.2 Multiple interfaces on master
*** I assume this is motived completely by use case 1.2 and not use case 1.1 since we don't have bandwidth issues for masters - please add this clarification.
*** What will this do to tokens if tokens are obtained from one of the several interfaces on a Master.
** 2.5 Enable clients to use multiple local interfaces - what is the motivation for this? Is this for tasks running inside Hadoop  or for clients outside the hadoop cluster. (or is this a yet another use case? If so add to use cases section.)
** Add requirements:
*** old configs should run unchanged
*** host level bonding should continue to work
*** security should work for the proposed enhancements
*** solution should work for all protocols - hadoop native, http rpc, etc. (if you are excluding any please clarify).
*** should work with HA
* 3.1 Example Config ...
** This section could be significantly improved. You switch between "current config"  and "how the current config's semantics are extended" and "new configuration"; this is sometimes confusing. 
* 4.3 Security 
** looks like the paragraph is not complete - last sentence is cut off. 
** Also added some details on what change are needed for this to work with security. Part of the paragraph reads like requirements statements and should move to the requirements section.
* 4.3 HA
** Add to requirements as noted above.
** Are there new issues here? Will have to failover multiple IPs  of the NN etc.
                
> Support multiple network interfaces
> -----------------------------------
>
>                 Key: HADOOP-8198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8198
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io, performance
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: MultipleNifsv1.pdf, MultipleNifsv2.pdf, MultipleNifsv3.pdf
>
>
> Hadoop does not currently utilize multiple network interfaces, which is a common user request, and important in enterprise environments. This jira covers a proposal for enhancements to Hadoop so it better utilizes multiple network interfaces. The primary motivation being improved performance, performance isolation, resource utilization and fault tolerance. The attached design doc covers the high-level use cases, requirements, a proposal for trunk/0.23, discussion on related features, and a proposal for Hadoop 1.x that covers a subset of the functionality of the trunk/0.23 proposal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira