Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <14305318.329841285108472849.JavaMail.jira@thor>
Date: Tue, 21 Sep 2010 18:34:32 -0400 (EDT)
From: "Faraz Ahmad (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Updated: (MAPREDUCE-2083) Run partial reduce instead of
 combiner at reduce node
In-Reply-To: <18705267.329821285108294749.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Faraz Ahmad updated MAPREDUCE-2083:
-----------------------------------

     Original Estimate:     (was: 672h)
    Remaining Estimate:     (was: 672h)

> Run partial reduce instead of combiner at reduce node
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-2083
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2083
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Faraz Ahmad
>             Fix For: 0.20.2
>
>
> Shuffle delays can be large for mapreductions with lots of intermediate data. 
> Some of this shuffle delay can be overlapped with reduce if some of the reduce 
> computation is started on partial intermediate data received by a reduce. 
> Along these lines, the patch ??HADOOP-3226?? runs the combiner
> on the reduce side to prune the data that goes to reduce. However, ??HADOOP-3226?? does not 
> achieve our goal of overlap with the shuffle because: (1) In its original use of reducing intermediate data volume,
> the combiner falls in the critical path at the map side. Therefore, the 
> combiner is usually a simple function which is too  lightweight in its new use 
>  to achieve sufficient overlap with the shuffle.
> (2) Running the combiner  at the reduce side is helpful in overlapping with 
> the shuffle only if  the combiner's functionality is a major portion of the 
> reduce functionality --  otherwise running the combiner at the reduce side 
> achieves only modest overlap with the shuffle.
> In many mapreductions, the  combiner computation is often not part or only a 
> small part of reduce computation.
> Addressing both these points, reduces that are complex often
> have heavier-weight computation than simple combining that can be overlapped 
> with the shuffle.   This heavy-weight computation is specified by a 
> user-supplied "partial reduce" which performs the commutative/associative 
> parts of reduce. The idea is to run partial reduce on subsets of intermediate 
> data as they arrive at a reduce to  overlap with the shuffle, and then run the 
> full-blown final reduce which re-reduces the partially-reduced data. Because
> the shuffle delay is large  for shuffle-heavy mapreductions,  
> partial reduce that are heavier-weight than simple combiner can be hidden 
> under the shuffle delay without extending the critical path of execution.
> Finally, to further ensure that the partial reduce does not extend the 
> critical path, include two easily-tunable thresholds: One to start partial 
> reduce only after enough intermediate data has been received (e.g. mapred.inmem.merge.threshold 
> or a separately defined parameter) so that we do not incur the overhead of invoking partial reduce on small data.
> Another threshold to stop partial reduce after most of the intermediate data 
> has been received so that running partial reduce on the small remainder data 
> does not  delay starting final reduce.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.