hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avner BenHanoch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
Date Tue, 27 Nov 2012 10:25:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504502#comment-13504502

Avner BenHanoch commented on MAPREDUCE-4049:


With all due respect, I think that something in your behavior is inappropriate:
 * You were never involved in this issue; still you gave yourself the liberty to make it a
sub issue of your supported MAPREDUCE-2454 issue, without consulting anyone.
 * This is especially inappropriate since MAPREDUCE-2454 is disputable and has its acceptance
problems regardless of my issue.  Hence, its acceptance problems will affect my issue.
 * Your justification *"As all this JIRAs are small, I think we'll be able to move fast with
all of them."* is inappropriate since you actually created a linkage that will surely postpone
my issue instead of leaving each issue to progress at its own pace!
 * It is not the first time that the persons behind MAPREDUCE-2454 try to disturb this JIRA

Apparently, I don't have the privileges to break this "sub task" linkage; hence, I am asking
that you or someone else will do it.

I am welcoming any comment coming from a professional place with the simple target of making
Hadoop better. Having that said,  I feel that the way you blitzed my patch with any possible
patty comment, sometime with disputable claims, just before the patch is about to be accepted
– is unfair, unprofessional and unfriendly. Especially considering your complete silence
since this JIRA issue has started.

I am not sure that commenting in a blitz way will increase the quality of hadoop.  For example:

"Checking for shuffleConsumerPlugin != null before closing it seems redundant, you would have
never got there if shufflePlugin is NULL."
This is your mistake (I'll reach there in case isLocal == true).  *There is no option to remove
the nullity check!*

"Visibility annotations for the ShuffleConsumerPlugin, ShuffleContext, should be Unstable"
I think it is inappropriate to declare plugin interface as "Unstable", since it must stay
stable for 3rd party vendors.

--- --- --- ---

Personally, I have no problem to implement all the rest of your comments. It should be very
easy for me.  Still, I am raising few points for consideration regarding your following comments:

"The Shuffle class should be renamed to DefaultShuffle."
"The ShuffleConsumerPlugin should be renamed to Shuffle."
I chose the term 'ShuffleConsumerPlugin' and not something like 'Shuffle', because it clarifies
that we are in a *plugin* of *ShuffleConsumer*, rather than a *builtin*  *ShuffleProvider/ShuffleHandler*.
  Also, I didn't take the liberty to rename core classes of Hadoop.  

"ShuffleConsumerPlugin, getShuffleConsumerPlugin() method is not required, instead use the
ReflectionUtils directly in the ReducerTask class."
Here, I only followed existing convention of Hadoop as shown in ResourceCalculatorPlugin.getResourceCalculatorPlugin().
 Personally, I'll be glad to follow your advice, and even to go one step further and make
ShuffleConsumerPlugin an interface instead of AbstractClass.

"use 'mapreduce.job.reduce.shuffle.class' to be consistent with MAPREDUCE-2454."
Here I chose 'mapreduce.shuffle…', since I think it is consistent with the current convention
in hadoop-3 configuration.

--- --- --- ---

I can tell you that Arun & Todd didn't make it easy for me with their requests from this
patch so far. Still, I understand, respect and accept all their comments.  I am sure that
everyone involved only want the best for Hadoop.  
I suggest we hear Arun's consideration and move forward with the patch in the best professional

I think you are very familiar with both Hadoop/MapReduce and this JIRA issue since its inception.
You are also well familiar and involved with MAPREDUCE-2454.  It is also safe to say you know
Alejandro and Asokan better than you know me.  I believe everyone involved will agree that
your sole interest is Hadoop's quality.  *I am asking you and everyone else to help progressing


> plugin for generic shuffle service
> ----------------------------------
>                 Key: MAPREDUCE-4049
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: performance, task, tasktracker
>    Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
>            Reporter: Avner BenHanoch
>              Labels: merge, plugin, rdma, shuffle
>             Fix For: trunk
>         Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch
> Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on shuffle plugin
that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using
the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable
merge approach during the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of
NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn
University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins (currently,
based on 1.0 branch)
> # I am providing link for downloading UDA - Mellanox's open source plugin that implements
generic shuffle service using RDMA and levitated merge.  Note: At this phase, the code is
in C++ through JNI and you should consider it as beta only.  Still, it can serve anyone that
wants to implement or contribute to levitated merge. (Please be advised that levitated merge
is mostly suit in very fast networks) - [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69]

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message