hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-591) Reducer sort should even out the pass factors in merging different pass
Date Wed, 25 Oct 2006 22:40:18 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-591?page=comments#action_12444749 ] 
Runping Qi commented on HADOOP-591:

In case there are multiple local disks, the merge process should write merge result files
onto different disks

> Reducer sort should even out the pass factors in merging different pass
> -----------------------------------------------------------------------
>                 Key: HADOOP-591
>                 URL: http://issues.apache.org/jira/browse/HADOOP-591
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Runping Qi
>         Assigned To: Devaraj Das
> When multiple pass merging is needed during sort, the current sort implementation in
SequenceFile class uses a simple "greedy" way to select pass factors, resulting uneven pass
factor in different passes. For example, if the factor pass is 100 (the default), and there
are 101 segments to be merged. The current implementation will first merge the first 100 segments
into one and then merge the big output file with the last segment with pass factor 2. It will
be better off to use pass factors 11 in the first pass and pass factor 10 in the second pass.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message