hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4502) Node-level aggregation with combining the result of maps
Date Wed, 17 Jul 2013 10:30:52 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tsuyoshi OZAWA updated MAPREDUCE-4502:

    Attachment: MAPREDUCE-4502.9.patch

Sorry, I attached a wrong file. Now I attach a correct one.
> Node-level aggregation with combining the result of maps
> --------------------------------------------------------
>                 Key: MAPREDUCE-4502
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>    Affects Versions: 3.0.0
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: design_v2.pdf, design_v3.pdf, MAPREDUCE-4502.1.patch, MAPREDUCE-4502.2.patch,
MAPREDUCE-4502.3.patch, MAPREDUCE-4502.4.patch, MAPREDUCE-4502.5.patch, MAPREDUCE-4502.6.patch,
MAPREDUCE-4502.7.patch, MAPREDUCE-4502.8.patch, MAPREDUCE-4502.8.patch, MAPREDUCE-4502.9.patch,
MAPREDUCE-4502.9.patch, MAPREDUCE-4525-pof.diff, speculative_draft.pdf
> The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because
the scope of combining is limited within only one MapTask. To solve this problem, it's a good
way to aggregate the result of maps per node/rack by launch combiner.
> This JIRA is to implement the multi-level aggregation infrastructure, including combining
per container(MAPREDUCE-3902 is related), coordinating containers by application master without
breaking fault tolerance of jobs.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message