hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2206) The task-cleanup tasks should be optional
Date Thu, 03 Feb 2011 21:46:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990328#comment-12990328

Joydeep Sen Sarma commented on MAPREDUCE-2206:

+1 from my side.

one thing is that very very few people will be aware that this can be turned off. in particular
- i think the default outputformats don't need a task cleanup. i am wondering how this can
be turned on automatically for more use cases.
- we can make the setting a default one in hive-default.xml - i will file a jira for that.
- how about hadoop streaming? can we turn task cleanup off if hadoop streaming is used with
the (default) fileoutputformat?

> The task-cleanup tasks should be optional
> -----------------------------------------
>                 Key: MAPREDUCE-2206
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2206
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.23.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.23.0
>         Attachments: MAPREDUCE-2206.txt
> For job does not use OutputCommitter.abort(), this should be able to turn off.
> This improves the latency of the job because failed tasks are often the bottleneck of
the jobs.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message