hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Iyappan Srinivasan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-947) OutputCommitter should have an abortJob method
Date Wed, 21 Oct 2009 13:45:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768236#action_12768236
] 

Iyappan Srinivasan commented on MAPREDUCE-947:
----------------------------------------------

+1 from QA for mr-947-y20-new.patch.

1) When mapreduce.fileoutputcomitter.marksuccessfuljobs parm is set to false

a) Successful jobs passes normally without creating a _SUCCESS file in the output directory.
b) Failed jobs fails normally without creating any file in the output directory.

2) When mapreduce.fileoutputcomitter.marksuccessfuljobs is set to true, then

a) Successful jobs passes and creates _SUCCESS under output directory.
b) Failed jobs, which are running midway, does not create _ SUCCESS when killed
c) Failed jobs, when doing their cleanup does not create _SUCCESS file when killed. 
d) Failed jobs, when doing setup, does not create any output directory and so does not create
any file under it.

3) After JT restart, and When mapreduce.fileoutputcomitter.marksuccessfuljobs is set to true,
then

a) Successful jobs passes and creates _ SUCCESS under output directory.
b) Failed jobs, which are running midway, does not create _ SUCCESS when killed
c) Failed jobs, when doing setup, does not create any output directory and so does not create
any file under it.

4) Run 100 jobs and kill/fail some jobs in the middle. The killed/failed ones should fail
properly and no string displayed. The successful jobs should display success.

5) Run randomwriter and sort. Both should work properly.

> OutputCommitter should have an abortJob method
> ----------------------------------------------
>
>                 Key: MAPREDUCE-947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-947
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.22.0
>
>         Attachments: mapred-948-v1.12-branch-0.20-internal.patch, mapred-948-v1.12.patch,
mapred-948-v1.13-branch-0.20-internal.patch, mapred-948-v1.2.patch, mapred-948-v1.3.patch,
mapred-948-v1.4.patch, mapred-948-v1.7.patch, mapred-948-v2.1-branch-0.20.patch, mapred-948-v2.3-branch-0.20.patch,
mapred-948-v2.3.patch, mapred-948-v3.1.patch, mapred-948-v3.2.patch, mapred-948-v3.4.patch,
mr-947-trunk-new.patch, mr-947-trunk-new.patch, mr-947-trunk.patch, mr-947-trunk.patch, mr-947-trunk.patch,
mr-947-y20-new.patch, mr-947-y20.patch
>
>
> The OutputCommitter needs an abortJob method to clean up from failed jobs. Currently
there is no way to distinguish between failed or succeeded jobs, making it impossible to write
output promotion code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message