hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
Date Wed, 19 Sep 2012 02:32:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458359#comment-13458359
] 

Tsuyoshi OZAWA commented on MAPREDUCE-4502:
-------------------------------------------

bq. For the benchmark, we were mostly handling cases without combiners; in our data, each
combiner was too effective to benefit from an intermediate level.

This result is also interesting. It means that the rack-level combiner is useful if its overhead
is minimized. To get more detail, I'll read the tech paper. 

BTW, it takes several days to publish new design note. I'll attach it here as soon as possible.
                
> Multi-level aggregation with combining the result of maps per node/rack
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4502
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, mrv2
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: speculative_draft.pdf
>
>
> The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because
the scope of combining is limited within only one MapTask. To solve this problem, it's a good
way to aggregate the result of maps per node/rack by launch combiner.
> This JIRA is to implement the multi-level aggregation infrastructure, including combining
per container(MAPREDUCE-3902 is related), coordinating containers by application master without
breaking fault tolerance of jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message