hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-947) OutputCommitter should have an abortJob method
Date Fri, 09 Oct 2009 07:19:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763884#action_12763884
] 

Amar Kamat commented on MAPREDUCE-947:
--------------------------------------

Adding more details on the patch
# OutputCommitter.abortJob() will be called for failed and killed jobs. Whether the job has
failed or was killed is passed as a boolean paramater named _failed_ in abortJob(). 
# By default abortJob() in OutputCommitter calls cleanupJob() so that we are backward compatible.

# FileOutputCommitter overrides abortJob() to do just the temp dir cleanup
# FileOutputCommitter's cleanupJob apart from cleaning up temp dir also creates a _success
in the output dir. This is optional and is the default behavior. Creation of _success can
be controlled by turning off the config param "mapred.fileoutputcommitter.marksuccessfuljobs".


> OutputCommitter should have an abortJob method
> ----------------------------------------------
>
>                 Key: MAPREDUCE-947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-947
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.21.0
>
>         Attachments: mapred-948-v1.12-branch-0.20-internal.patch, mapred-948-v1.12.patch,
mapred-948-v1.2.patch, mapred-948-v1.3.patch, mapred-948-v1.4.patch, mapred-948-v1.7.patch,
mapred-948-v2.1-branch-0.20.patch
>
>
> The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently
there is no way to distinguish between failed or succeeded jobs, making it impossible to write
output promotion code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message