hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-4831) Task commit can occur more than once due to AM retries
Date Wed, 27 Feb 2013 23:39:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe resolved MAPREDUCE-4831.
-----------------------------------

    Resolution: Not A Problem

After discussing this at length with Bobby, we came to the conclusion that this isn't a problem.
 Task commit can already occur multiple times due to things like fetch failure, etc.  The
OutputCommitter is responsible for guarding against task double-commit if that indeed is a
problem.
                
> Task commit can occur more than once due to AM retries
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-4831
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4831
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 0.23.0, 2.0.0-alpha
>            Reporter: Jason Lowe
>            Priority: Critical
>
> If a task attempt begins committing but the AM crashes before the task attempt completes
then we could end up having the task commit again when the AM is relaunched.  The subsequent
AM attempt will not see the task having completed, so it will re-run the task and it will
commit again.  The output committer is user code, and the task commit may not be something
repeatable.  Therefore we should treat an AM crash during a task attempt commit the same as
we do for a commit failure by the task attempt, i.e.: the task should fail since we do not
know how to recover from a commit failure.
> This is similar to MAPREDUCE-4819, as this involves commit at the task level and that
involves commit at the job-level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message