hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
Date Sun, 02 Sep 2012 22:08:10 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447040#comment-13447040

Mariappan Asokan commented on MAPREDUCE-4049:

I don’t have conflict of interests with you. 4 months ago, I already welcomed the watchers
of this issue to help you commit your patch.
I hope I didn't offend you by stating that "we seem to have a conflict of interest here."
 I meant the conflict in our designs:)
RDMA is not coupled with any merge and there is no such a thing "RDMA merge".
Perhaps I should have stated "RDMA based merge" instead of "RDMA merge."  From my understanding
of the pdf document attached to this Jira, there is a {{NetMerger}} plugin component which
does the merge on the reduce side.  It takes care of shuffling(small pieces of data using
RDMA) and merge on the reduce side in order to avoid multi-phase merges and associated disk

If you can educate me and others on the relationship between {{NetMerger}} and your {{ShuffleConsumerPlugin}},
perhaps we will understand more.  To me {{NetMerger}} implies it is doing a merge sort on
the reduce side and hence it is qualified to be a sort plugin on the reduce side.
 It is current Hadoop that couples shuffle with merge. You can be relaxed. I don’t “want
to retain that coupling”. My opinion is that your decoupling is correct idea and I encourage
I am glad that you like the idea of decoupling the merge and shuffle.  In order to achieve
this decoupling, {{Shuffle}} class cannot return the key value iterator.  It should only do
the shuffling of data and feed it to the code that merges the map outputs.  It is the responsibility
of the merger to return the key value iterator.  I want to make that code pluggable and hence
the need for {{ReduceSortPlugin.}}  If you want to make shuffle pluggable, as I suggested
before we can make
{{ShuffleRunner}}(which will be implemented by {{Shuffle}} class) pluggable.
Your patch contains more than 7,000 rows, while my patch is only 400 rows. I don’t want
to wait till your patch passes Automatic QA, and code review, and additional rounds. 
Since we are working on overlapping areas of the code, I just want to make sure that we both
get the design concepts right before looking at the number of lines of code touched or which
Jira is to be committed first and so on.

> plugin for generic shuffle service
> ----------------------------------
>                 Key: MAPREDUCE-4049
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task, tasktracker
>    Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
>            Reporter: Avner BenHanoch
>              Labels: merge, plugin, rdma, shuffle
>         Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop
Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapreduce-4049.patch, mapreduce-4049.patch
> Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on shuffle plugin
that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using
the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable
merge approach during the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of
NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn
University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins (currently,
based on 1.0 branch)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message