hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4831) Task commit can occur more than once due to AM retries
Date Thu, 29 Nov 2012 20:14:58 GMT
Jason Lowe created MAPREDUCE-4831:

             Summary: Task commit can occur more than once due to AM retries
                 Key: MAPREDUCE-4831
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4831
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mr-am
    Affects Versions: 2.0.0-alpha, 0.23.0
            Reporter: Jason Lowe
            Priority: Critical

If a task attempt begins committing but the AM crashes before the task attempt completes then
we could end up having the task commit again when the AM is relaunched.  The subsequent AM
attempt will not see the task having completed, so it will re-run the task and it will commit
again.  The output committer is user code, and the task commit may not be something repeatable.
 Therefore we should treat an AM crash during a task attempt commit the same as we do for
a commit failure by the task attempt, i.e.: the task should fail since we do not know how
to recover from a commit failure.

This is similar to MAPREDUCE-4813, as this involves commit at the task level and that involves
commit at the job-level.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message