hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2480) MR-279: mr app should not depend on hard-coded version of shuffle
Date Tue, 10 May 2011 18:15:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031281#comment-13031281

Luke Lu commented on MAPREDUCE-2480:

bq. To further clarify, the dependency, ideally, should not be on the implementation but rather
on some kind of interface exposed by the shuffle service.

That's exactly my point. It should not depend on a specific implementation class but rather
an SPI that both the app and the shuffle implementation depend on. In this case, the SPI is
the service id at the moment. If you rename the ShuffleHandler to ShuffleHandlerImpl the code
would look wrong too.

Now you have a runtime classpath dependency on a specific shuffle implementation for mr app.
If you replace the jar with a different class and update the config, the NM will happily load
the service but the mr app won't even run due to class not found exception. 

> MR-279: mr app should not depend on hard-coded version of shuffle
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-2480
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2480
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>         Attachments: mr-2480-shuffle-version-v1.patch
> The following commit introduced a dependency of shuffle with hard-coded version for mr
> {noformat}
> commit 6f69742140516be7493c9a9177b81d0516cc9539
> Author: Vinod Kumar Vavilapalli <vinodkv@apache.org>
> Date:   Wed May 4 06:53:52 2011 +0000
>     Adding user log handling for YARN. Making NM put the user-logs on DFS and providing
log-dump tools. Contributed by Vinod Kumar Vavilapalli.
> {noformat}

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message