hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avner BenHanoch (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3060) Generic shuffle service
Date Sun, 25 Mar 2012 15:59:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237899#comment-13237899

Avner BenHanoch commented on MAPREDUCE-3060:

Please see MAPREDUCE-4049.
> Generic shuffle service
> -----------------------
>                 Key: MAPREDUCE-3060
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3060
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>              Labels: shuffle
>             Fix For: 0.24.0
> When I was talking to Owen about MAPREDUCE-2600, we came across (again, talked about
it with Chris before) the shuffle dependency issue. NodeManager currently has an implicit
(hidden by the service plugin mechanism) dependency of a specific version of mapreduce shuffle.
While this works in many cases, as long as we don't change shuffle headers and the usage of
mapred security tokens, it's a hack to make things work none the less. It's generally agreed
upon that nodemanager should only load generic services that are mapreduce framework neutral.
> In this particular case, the right solution seems to be a generic shuffle handler that
can serve data for a particular partition securely. The ShuffleHandler currently only depends
on mapreduce for task tokens and shuffle header, which is only used for writing data, i.e.,
the shuffle handler has no semantic dependency on mapreduce.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message