hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-1985) Abstract node to switch mapping into a topology service class used by namenode and jobtracker
Date Mon, 14 Jan 2008 23:09:36 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558834#action_12558834
] 

owen.omalley edited comment on HADOOP-1985 at 1/14/08 3:09 PM:
----------------------------------------------------------------

I'm worried about the time and memory performance of this. Have you run a sort with dfs cluster
== map/reduce cluster and compared running times and job tracker memory size? We've already
seen cases where the current pollForNewTask causes performance problems...

It bothers me that the max levels is hard coded rather than configurable.

>From a style point of view, I probably would have defined a new class rather than use
nested java.utils containers like List<Map<Node, List<TaskInProgress>>>.
That way if we change the representation later it won't be scattered through the code.  In
particular, I can imagine wanting to have the data structure be something like:
Map<String (rack name), RackInfo> and RackInfo has a Map<String (hostname), List<TaskInProgress>
>. Or even more tree-like...

Did you need to change the definition of findNewTask? I don't see it in the patch.

This needs user documentation in forrest.

The java doc on DNSToSwitchMapping.resolve should probably mention that they must cache if
their operation is expensive. Although there isn't a way to clear or update that cache, which
might be a problem at some point...

You don't really need the Scan example, you could use the GenericMRLoadGenerator with a -keepmap
of 0.

In the longer term I think a configured mapping class would be useful. A class named
org.apache.hadoop.net.ConfiguredNodeMapping that let you set the mapping in your config. Something
like:

{code}
<property>
   <name>hadoop.configured.node.mapping</name>
   <value>host1=/rack1,host2=/rack1,host3=/rack4</value>
</property>
{code}




      was (Author: owen.omalley):
    I'm worried about the time and memory performance of this. Have you run a sort with dfs
cluster == map/reduce cluster and compared running times and job tracker memory size? We've
already seen cases where the current pollForNewTask causes performance problems...

It bothers me that the max levels is hard coded rather than configurable.

>From a style point of view, I probably would have defined a new class rather than use
nested java.utils containers like List<Map<Node, List<TaskInProgress>>>.
That way if we change the representation later it won't be scattered through the code.  In
particular, I can imagine wanting to have the data structure be something like:
Map<String (rack name), RackInfo> and RackInfo has a Map<String (hostname), List<TaskInProgress>
>. Or even more tree-like...

Did you need to change the definition of findNewTask? I don't see it in the patch.

This needs user documentation in forrest.

The java doc on DNSToSwitchMapping.resolve should probably mention that they must cache if
their operation is expensive. Although there isn't a way to clear or update that cache, which
might be a problem at some point...

You don't really need the Scan example, you could use the GenericMRLoadGenerator with a -keepmap
of 0.

In the longer term I think a configured mapping class would be useful. A class named
org.apache.hadoop.net.ConfiguredNodeMapping that let you set the mapping in your config. Something
like:

{code}
<property>
   <name>hadoop.configured.node.mapping</name>
   <value>host1=/rack1,host2=/rack1,host3=/rack4</value>
</property>



  
> Abstract node to switch mapping into a topology service class used by namenode and jobtracker
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1985
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: eric baldeschwieler
>            Assignee: Devaraj Das
>             Fix For: 0.16.0
>
>         Attachments: 1985.new.patch, 1985.v1.patch, 1985.v2.patch, 1985.v3.patch, 1985.v4.patch,
1985.v5.patch, 1985.v6.patch
>
>
> In order to implement switch locality in MapReduce, we need to have switch location in
both the namenode and job tracker.  Currently the namenode asks the data nodes for this info
and they run a local script to answer this question.  In our environment and others that I
know of there is no reason to push this to each node.  It is easier to maintain a centralized
script that maps node DNS names to switch strings.
> I propose that we build a new class that caches known DNS name to switch mappings and
invokes a loadable class or a configurable system call to resolve unknown DNS to switch mappings.
 We can then add this to the namenode to support the current block to switch mapping needs
and simplify the data nodes.  We can also add this same callout to the job tracker and then
implement rack locality logic there without needing to chane the filesystem API or the split
planning API.
> Not only is this the least intrusive path to building racklocal MR I can ID, it is also
future compatible to future infrastructures that may derive topology on the fly, etc, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message