hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4812) Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
Date Fri, 07 Dec 2012 21:47:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526802#comment-13526802

Mariappan Asokan commented on MAPREDUCE-4812:

Hi Arun,
  Sorry I did not get back sooner.  The intention of {{ReduceInputMerger}} interface is to
have a pluggable {{MergeManager}} implementation.  For a non-local job, {{Shuffle}} and {{MergeManager}}
interact and synchronize with each other using the three methods {{waitForInMemoryMerge(),}}
{{reserve(),}} and {{close()}}.  So in order to use the {{Shuffle}} these methods are captured
in {{ReduceInputMerger}} interface.  I renamed {{waitForInMemoryMerge()}} to a generic name
{{waitForResource()}} since the plugin implementation may not have the concept of in-memory
Since the return value from {{reserve()}} is {{MapOutput}}, I did some refactoring of {{MapOutput}}
so that plugin can return its own implementation of it.  I kept the refactoring done on {{MapOutput}}
in MAPREDUCE-4808.  With just MAPREDUCE-4812, an external plugin is not possible, but it has
the core part of the concepts so that it is easy to review just {{ReduceInputMerger}} design.
 Similarly, for a local job the input is coming from local files.  I enhanced {{ReduceInputMerger}}
with one more method for this.  It is also kept in MAPREDUCE-4808.

Hope I explained well.  Please let me know if you have any more questions.


-- Asokan

> Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
> ---------------------------------------------------------------------------
>                 Key: MAPREDUCE-4812
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>    Affects Versions: 2.0.2-alpha
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>             Fix For: 2.0.3-alpha
>         Attachments: COMBO-mapreduce-4809-4812.patch, COMBO-mapreduce-4809-4812.patch,
mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch
> This is part of MAPREDUCE-2454.  This further breaks down MAPREDUCE-4808

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message