hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
Date Tue, 27 Nov 2012 14:26:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504633#comment-13504633
] 

Alejandro Abdelnur commented on MAPREDUCE-4049:
-----------------------------------------------

Hi Avner,

I respectfully disagree with your opinion that my behavior is inappropriate. 

First of all, it is not my intention to slow you this JIRA down, but to make sure it is consistent
with the related work in MAPREDUCE-2454 (you can see that in my comments). If that requires
a couple of extra days, is is a small price to pay.

As an Apache Hadoop developer is my responsibility to review and provide feedback on work
posted by other developers, my usual triggers are area of knowledge, related work and area
of interest. 

This JIRA is tightly related to MAPREDUCE-2454, there is not dispute on that. Thus it should
stay as a subtask of it.

MAPREDUCE-2454 is not disputable, as it has been commented in it JIRA, it is almost ready,
it was matter of breaking it up and doing an fast interactive review of its parts. As far
as I can tell, this is already happening there. 

Now going to your comments on my review:

* Yes the *shuffleConsumerPlugin != null*, you are right, I've  noticed that after I've posted
my comments, so you can disregard that done.

* On the marking the ShuffleConsumerPlugin, ShuffleContext as *unstable*, it is not appropriate,
Hadoop wants to keep the right of modifying these APIs in the future, if hte need arises.
You can also see this, no only in MAPREDUCE-2454, but in several places where Hadoop provides
pluggability (ie ResourceManagement, authentication).

* On making the ShuffleConsumerPlugin and interface, that is a good idea, it will align things
with the other sub-tasks.

Looking forward to see the updated patch.

Cheers.


                
> plugin for generic shuffle service
> ----------------------------------
>
>                 Key: MAPREDUCE-4049
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: performance, task, tasktracker
>    Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
>            Reporter: Avner BenHanoch
>              Labels: merge, plugin, rdma, shuffle
>             Fix For: trunk
>
>         Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on shuffle plugin
that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using
the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable
merge approach during the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of
NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn
University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins (currently,
based on 1.0 branch)
> # I am providing link for downloading UDA - Mellanox's open source plugin that implements
generic shuffle service using RDMA and levitated merge.  Note: At this phase, the code is
in C++ through JNI and you should consider it as beta only.  Still, it can serve anyone that
wants to implement or contribute to levitated merge. (Please be advised that levitated merge
is mostly suit in very fast networks) - [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message