hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by namenode and jobtracker
Date Thu, 31 Jan 2008 17:55:09 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564428#action_12564428
] 

Hairong Kuang commented on HADOOP-1985:
---------------------------------------

A few dfs-related comments:
1. A datanode should not be added the cluster map before its network location is resolved;
2. A datanode's block report should not be processed before its network location is resolved.
Violating (1) cause the datanode be a candiate target of block allocation; Violating (2) may
cause block rereplication or excessive block deletion to wrongly think replicas on the same
rack to be on different racks. Both may result all replicas of a block to end up on the same
rack.

I also think that doing network resolution per datanode is inefficient. It would be nice if
NN could have a list or a range of nodes that are alllowed to join the cluster and then resolve
their network locations by running a script that takes a list or a range of nodes and returns
all racks and the nodes belonged to each rack. Datanode network locations resolutions are
performed before accepting any datanode registration. This would eliminate all the problems
that are caused by lazy network resolution. 

> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.17.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v10.patch, 1985.v11.patch, 1985.v2.patch,
1985.v3.patch, 1985.v4.patch, 1985.v5.patch, 1985.v6.patch, 1985.v9.patch, jobinprogress.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in
both the namenode and job tracker.  Currently the namenode asks the data nodes for this info
and they run a local script to answer this question.  In our environment and others that I
know of there is no reason to push this to each node.  It is easier to maintain a centralized
script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and
invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.
 We can then add this to the namenode to support the current block to switch mapping needs
and simplify the data nodes.  We can also add this same callout to the job tracker and then
implement rack locality logic there without needing to chane the filesystem API or the split
planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also
future compatible to future infrastructures that may derive topology on the fly, etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message