hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable
Date Tue, 15 Jan 2013 21:06:17 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mariappan Asokan updated MAPREDUCE-4808:
----------------------------------------

    Attachment: mapreduce-4808.patch

Hi Chris and Alejandro,
  Thanks to both of you for your comments and suggestions.  I spent some time reflecting on
them.  I totally agree with Chris that the RDMA/network merge will be different from the way
the merge is done by the current {{MergeManager.}} In that case a shuffle plugin is the way
to go.

When the current HTTP based {{Shuffle}} is used, it makes sense to define a {{MergeManagerPlugin}}
to work with that.  So based on Alejandro's suggestion, I updated the patch to make {{MergeManagerPlugin}}
work only with {{Shuffle.}}  I removed the method {{mergeLocalFiles()}} from the interface
as Chris suggested.  The {{Shuffle}} class would instantiate the {{MergeManagerPlugin}} object
in its {{init()}} method.

By the end of the day, Hadoop users will be offered more choices: a shuffle plugin for RDMA/network
merge when the cluster nodes have hardware support and a merge plugin that works with the
current shuffle when the cluster does not have special network hardware.

Please provide any feedback on the updated patch.

Once again, thanks to both of you for the time you are spending on this Jira.

-- Asokan

                
> Allow reduce-side merge to be pluggable
> ---------------------------------------
>
>                 Key: MAPREDUCE-4808
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.0.2-alpha
>            Reporter: Arun C Murthy
>            Assignee: Mariappan Asokan
>             Fix For: 2.0.3-alpha
>
>         Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch,
mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf
>
>
> Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message