hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2038) Making reduce tasks locality-aware
Date Fri, 27 Aug 2010 21:44:53 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903656#action_12903656

Todd Lipcon commented on MAPREDUCE-2038:

This is a very interesting direction. We have another use case for HBase bulk loads, where
we know that a given reducer partition is going to end up on a particular region server (often
colocated with a TT). Scheduling the reducer to run on the same node or rack will ensure a
local replica of the HFile when it comes time to serve it.

Another interesting use case is for aggregation queries where we can make use of something
like a "rack combiner". We can simply implement a Partitioner that returns the rack index
of the mapper, and then schedule that reduce task on the same rack. Thus we end up with a
result set per rack, and can do a second small job to recombine those. This is not unlike
the multilevel query execution trees used in Dremel - I imagine Hive and Pig's query planners
could make use of plenty of techniques like this.

> Making reduce tasks locality-aware
> ----------------------------------
>                 Key: MAPREDUCE-2038
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2038
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Hong Tang
> Currently Hadoop MapReduce framework does not take into consideration of data locality
when it decides to launch reduce tasks. There are several cases where it could become sub-optimal.
> - The map output data for a particular reduce task are not distributed evenly across
different racks. This could happen when the job does not have many maps, or when there is
heavy skew in map output data.
> - A reduce task may need to access some side file (e.g. Pig fragmented join, or incremental
merge of unsorted smaller dataset with an already sorted large dataset). It'd be useful to
place reduce tasks based on the location of the side files they need to access.
> This jira is created for the purpose of soliciting ideas on how we can make it better.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message