hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3226) Run combiner when merging spills from map output
Date Mon, 30 Jun 2008 17:43:45 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-3226:
----------------------------------

     Description: 
When merging spills from the map, running the combiner should further diminish the volume
of data we send to the reduce.


  was:When merging spills from the map, running the combiner should further diminish the volume
of data we send to the reduce.

    Release Note: 
Previously, the combiner was run exactly once on each record by the map task. Now, the combiner
may be run multiple times as the map's output is sorted and merged. Additionally, it may be
run on the reduce side as data is merged.

The old semantics are available in Hadoop 0.18 if the user calls: 
job.setCombineOnlyOnce(true);


  was:Previously, the combiner was run exactly once on each record by the map task. Now, the
combiner may be run multiple times as the map's output is sorted and merged. Additionally,
it may be run on the reduce side as data is merged.

    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Reviewed, Incompatible change])

> Run combiner when merging spills from map output
> ------------------------------------------------
>
>                 Key: HADOOP-3226
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3226
>             Project: Hadoop Core
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>             Fix For: 0.18.0
>
>         Attachments: 3226-0.patch, 3226-1.patch, 3226-2.patch, 3226-3.patch
>
>
> When merging spills from the map, running the combiner should further diminish the volume
of data we send to the reduce.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message