hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-967) TaskTracker does not need to fully unjar job jars
Date Wed, 23 Sep 2009 18:09:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758807#action_12758807

Todd Lipcon commented on MAPREDUCE-967:

bq. Please also make sure the new class in mapreduce is in org.apache.hadoop.mapreduce.util
and create a deprecate class in org.apache.hadoop.util which uses functionality from org.apache.hadoop.mapreduce.util.RunJar.

Can we do a real dependency in this direction or do I have to use reflection to cross the
project boundary? I was under the impression that cyclic dependencies were bad.

I can certainly mark the existing implementation as Deprecated, but keep the implementation
there. Then I can add a non-deprecated shim class in mapreduce.util to wrap the old version
with deprecation warnings supressed. We can then do the move for reals next version.

Sound good?

> TaskTracker does not need to fully unjar job jars
> -------------------------------------------------
>                 Key: MAPREDUCE-967
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-967
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-967-branch-0.20.txt
> In practice we have seen some users submitting job jars that consist of 10,000+ classes.
Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant
cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message