hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by namenode and jobtracker
Date Tue, 02 Oct 2007 18:39:50 GMT
Abstract node to switch mapping into a topology service class used by namenode and jobtracker
---------------------------------------------------------------------------------------------

                 Key: HADOOP-1985
                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
             Project: Hadoop
          Issue Type: New Feature
            Reporter: eric baldeschwieler


In order to implement switch locality in MapReduce, we need to have switch location in both
the namenode and job tracker.  Currently the namenode asks the data nodes for this info and
they run a local script to answer this question.  In our environment and others that I know
of there is no reason to push this to each node.  It is easier to maintain a centralized script
that maps node DNS names to switch strings.

I propose that we build a new class that caches known DNS name to switch mappings and invokes
a loadable class or a configurable system call to resolve unknown DNS to switch mappings.
 We can then add this to the namenode to support the current block to switch mapping needs
and simplify the data nodes.  We can also add this same callout to the job tracker and then
implement rack locality logic there without needing to chane the filesystem API or the split
planning API.

Not only is this the least intrusive path to building racklocal MR I can ID, it is also future
compatible to future infrastructures that may derive topology on the fly, etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message