hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4863) Adding aggregationWaitMap for node-level combiner.
Date Fri, 04 Jan 2013 08:14:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tsuyoshi OZAWA updated MAPREDUCE-4863:

    Attachment: 0002-Adding-aggregationWaitMap-for-node-level-combiner.patch

First patch.
> Adding aggregationWaitMap for node-level combiner.
> --------------------------------------------------
>                 Key: MAPREDUCE-4863
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4863
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: applicationmaster
>    Affects Versions: 3.0.0
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: 0002-Adding-aggregationWaitMap-for-node-level-combiner.patch
> To manage node/rack-level combining, MRAppMaster needs to have a management information
about outputs of completed MapTasks to be aggregated.  AggregationWaitMap is used so that
MRAppMaster decides whether or not MapTasks start to combine local MapOutputFiles.
> AggregationWaitMap is a abstraction class of ConcurrentHashMap<String, ArrayList<TaskAttemptCompletionEvent>.
These Events are candidate files to be aggregated.
> When MapTasks are completed, MRAppMaster buffer TaskAttemptCompletionEvent into AggregationWaitMap
to delay reducers' fethcing outputs from mappers until node-level aggregation are finished.
 After node-level aggreagtion, MRAppMaster write back mapAttemptCompletionEvents, to restart
reducers' feching outputs from mappers.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message