hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariappan Asokan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
Date Wed, 18 Apr 2012 17:26:42 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256720#comment-13256720

Mariappan Asokan commented on MAPREDUCE-4049:

Hi Avner,
  Following are my additional comments and thoughts:

* Does someone need to implement {{ShuffleProviderPlugin}} and {{ShuffleConsumerPlugin}} as
a pair? Or can one of them be implemented and the other be left to the default implementaion?
* For the trunk version, the shuffle provider will probably not require any changes.  The
shuffle consumer on the other hand has to be separated into two parts: one that deals with
tansferring map output data  across the network and the other that consumes the transferred
data.  If you look at the patch I posted in MAPREDUCE-2454, the above are abstracted as {{ShuffleRunner}}
and {{ShuffleCallback}} interfaces respectively.  In the refactored code as part of the patch,
{{MergeManager}} class implements {{ShuffleCallback}} and {{Shuffle}} class implements {{ShuffleRunner}}.
 I can probably enhance the {{ShuffleRunner}} interface as below with an added {{initialize()}}
method which basically gets all the arguments that current {{Shuffle}} constructor gets:
public interface ShuffleRunner extends ExceptionReporter {
  public void initialize(TaskAttemptID reduceId, JobConf jobConf,
                         TaskUmbilicalProtocol umbilical,
                         Reporter reporter,
                         Counters.Counter shuffledMapsCounter,
                         Counters.Counter reduceShuffleBytes,
                         Counters.Counter failedShuffleCounter,
                         TaskStatus status,
                         Progress copyPhase,
                         Task reduceTask);
  public void run(ShuffleCallback shuffleCallback)
    throws IOException, InterruptedException;
* If you want to implement your own merge(using RDMA to get shuffled data as described in
the technical paper attached to this Jira) you can implement the interface {{ReduceSortPlugin}}
in addition to {{ShuffleRunner}}.  In the {{ReduceSortPlugin}} your merge class can implement
{{ShuffleCallback}}.  Currently, since the merge and shuffle are running in separate threads,
the synchronization is done by {{waitForResource()}} method in   {{ShuffleCallback}}.  If
you are using RDMA to fetch map outputs, your merge will be in full control.  It has to coordinate
with your implementation of {{ShuffleRunner}} to fetch specific mapper output you want in
the merge.

If you have any questions, please let me know.
> plugin for generic shuffle service
> ----------------------------------
>                 Key: MAPREDUCE-4049
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task, tasktracker
>    Affects Versions: 1.0.3
>            Reporter: Avner BenHanoch
>              Labels: merge, plugin, rdma, shuffle
>         Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, HADOOP-1.0.x.patch, Hadoop
Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch,
mapred-site.xml, mapred.diff, src.tgz, test.diff
> Support generic shuffle service as set of two plugins: ShuffleProvider & ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on shuffle plugin
that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using
the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable
merge approach during the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of
NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn
University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins (currently,
based on 1.0 branch)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message