hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-947) OutputCommitter should have an abortJob method
Date Mon, 05 Oct 2009 10:41:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762146#action_12762146
] 

Amar Kamat commented on MAPREDUCE-947:
--------------------------------------

Food for thought :
# For distinction between running job and completed (succeeded/failed/killed) jobs we can
extend the task promotion technique to job i.e while the job is running write the output to
a temporary/staging area and then upon job completion promote the output to output.dir. So
the simple check is 
{code}
if (output.dir exists) {
  job is complete
} else {
  job is incomplete
}
{code}
One small addition to this is to serialize the job status object to the final output dir so
that the clients can simply use the output.dir to get the job status offline.

# Having the framework do if-else for successful/unsuccessful job is probably not the right
direction. I think the framework should simply leave it to the uses to implement it via OutputCommitter.
Hence I think the framework should simply invoke _committer.cleanupJob()_. FileOutputCommitter
should create
 ## _success for successful jobs
 ## _killed for killed jobs
 ## _failed for failed jobs

The only catch is that the OutputCommitter should know the job state (successful/killed/failed
etc). 

Thoughts?

> OutputCommitter should have an abortJob method
> ----------------------------------------------
>
>                 Key: MAPREDUCE-947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-947
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.21.0
>
>         Attachments: mapred-948-v1.2.patch, mapred-948-v1.3.patch, mapred-948-v1.4.patch
>
>
> The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently
there is no way to distinguish between failed or succeeded jobs, making it impossible to write
output promotion code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message