hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Hao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
Date Wed, 04 Feb 2015 12:52:35 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305036#comment-14305036

Yang Hao commented on MAPREDUCE-5718:

Hi, I have read the comment twice. It may not be safe to recover under this situation. 

I'm wondering whether you are worring about missing data. But during the recovery, task output
data will be moved to the new output dir, just as the normal AM failover does, then it's safe
to recover

Our cluster will add this feature. Can you give more specific information on the unsafe situation?
Thanks a lot

> MR job will fail after commit fail
> ----------------------------------
>                 Key: MAPREDUCE-5718
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.3.0, 2.6.0
>            Reporter: Karthik Kambatla
>            Assignee: Yang Hao
>             Fix For: 2.6.0
>         Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch
> when any of this happens:
> * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit,

> * When testing preempting, if the MR AM fails over during the middle of a commit
> the subsequent AM gets spawned but dies with a diagnostic message - "We crashed durring
a commit". 

This message was sent by Atlassian JIRA

View raw message