hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-967) TaskTracker does not need to fully unjar job jars
Date Tue, 08 Dec 2009 14:09:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787470#action_12787470

Vinod K V commented on MAPREDUCE-967:

Your argument w.r.t the classpath (points 1 and 3) sounds reasonable. The newly added documentation
also mitigates some of my previous issues. Still, I think that with the default value for
the configuration property as {{(lib|classes)}} and not (.*), we need to mark this as incompatible
and the release note should clearly reflect the breakage of the streaming like use-cases(your
2nd point above) and the available work-around(the configuration property) as to how to get
everything unjarred.

Given that, the patch looks good to me. +1, pending commit of HADOOP-6346.

> TaskTracker does not need to fully unjar job jars
> -------------------------------------------------
>                 Key: MAPREDUCE-967
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-967
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-967-branch-0.20.txt, mapreduce-967.txt, mapreduce-967.txt,
mapreduce-967.txt, mapreduce-967.txt
> In practice we have seen some users submitting job jars that consist of 10,000+ classes.
Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant
cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message