hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-967) TaskTracker does not need to fully unjar job jars
Date Fri, 11 Sep 2009 04:26:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753987#action_12753987

Vinod K V commented on MAPREDUCE-967:

Now that there are changes to RunJar, for trunk, can you move RunJar to mapreduce from common
as is generally desired (https://issues.apache.org/jira/browse/MAPREDUCE-727?focusedCommentId=12728372&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12728372)

Also, I think, it will be good to make the filter to specify the directories/files in job.jar
to be un-jarred as _configurable_. This way we can also maintain backward compatibility to
the current scenario where in we un-jar everything. The configuration can be a comma separated
list of files/dires for example. You may also need changes to the JarEntryFilter to accept
wild-card entries. Thoughts?

> TaskTracker does not need to fully unjar job jars
> -------------------------------------------------
>                 Key: MAPREDUCE-967
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-967
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-967-branch-0.20.txt
> In practice we have seen some users submitting job jars that consist of 10,000+ classes.
Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant
cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message