hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
Date Wed, 01 Aug 2012 06:46:34 GMT
Tsuyoshi OZAWA created MAPREDUCE-4502:
-----------------------------------------

             Summary: Multi-level aggregation with combining the result of maps per node/rack
                 Key: MAPREDUCE-4502
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: applicationmaster, mrv2
            Reporter: Tsuyoshi OZAWA


The shuffle costs is expensive in Hadoop in spite of the
existence of combiner, because the scope of combining is limited
within only one MapTask. To solve this problem, it's a good way to aggregate the result of
maps per node/rack by launch combiner.

This JIRA is to implement the multi-level aggregation infrastructure, including combining
per container(MAPREDUCE-3902 is related), coordinating containers by application master without
breaking fault tolerance of jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message