hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-947) OutputCommitter should have an abortJob method
Date Thu, 15 Oct 2009 08:20:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765963#action_12765963
] 

Arun C Murthy commented on MAPREDUCE-947:
-----------------------------------------

After thinking some more, I think we should take Devaraj's suggestion and deprecate OutputCommitter.cleanupJob
and add new methods: OutputCommitter.{commitJob|abortJob}, this is as good a time as any to
fix it. For BC both the new apis should call the cleanupJob, thus they need to be concrete
(not abstract) methods.

Some more comments:
# Please remove FileOutputCommitter.setOutputDirMarking since we are making it a hidden feature
for 0.22, also make FileOutputCommitter.SUCCESSFUL_JOB_OUTPUT_DIR_MARKER package-private for
similar reasons.
# Ditto for o.a.h.mapred.FileOutputCommitter
# o.a.h.mapred.FileOutputCommitter.SUCCEEDED_FILE_NAME is public while o.a.h.mapreduce.FileOutputCommiter.SUCCEEDED_FILE_NAME
is protected. Shouldn't both be package-private or public?
# I think I've changed my mind about o.a.h.mapred.OutputCommitter.abortJob - it should take
an int rather than o.a.h.mapreduce.JobStatus.State. It is odd that o.a.h.mapred.* depends
on o.a.h.mapreduce.*
# o.a.h.{mapred|mapreduce}.OutputFormat.{commitJob|abortJob} should assert that the job's
state is SUCCESS or FAILED/KILLED appropriately in the javadoc (I see that Task.java checks
it appropriately).
# I don't think we should specify mapreduce.fileoutputcommitter.marksuccessfuljobs to be 'false'
in src/test/mapred-site.xml. Rather, we should hunt down and fix test-cases which fail appropriately!
(via PathFilter to 'ls' etc.)

> OutputCommitter should have an abortJob method
> ----------------------------------------------
>
>                 Key: MAPREDUCE-947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-947
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.21.0
>
>         Attachments: mapred-948-v1.12-branch-0.20-internal.patch, mapred-948-v1.12.patch,
mapred-948-v1.13-branch-0.20-internal.patch, mapred-948-v1.2.patch, mapred-948-v1.3.patch,
mapred-948-v1.4.patch, mapred-948-v1.7.patch, mapred-948-v2.1-branch-0.20.patch, mapred-948-v2.3-branch-0.20.patch,
mapred-948-v2.3.patch, mapred-948-v3.1.patch, mapred-948-v3.2.patch
>
>
> The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently
there is no way to distinguish between failed or succeeded jobs, making it impossible to write
output promotion code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message