hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered
Date Thu, 27 Feb 2014 18:29:22 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914835#comment-13914835
] 

Zhijie Shen commented on MAPREDUCE-5547:
----------------------------------------

This issue has been left for a while. According to the previous discussion, as to the problem
that the history file is available on JSH while AM fails unregistration, the straightforward
solution is to move the history file to the intermediate done dir after unregistration. If
we don't do that, the job has a second retry, and the second retry fails, we will still see
a successful job on JHS, because according to my investigation, JHS will neglect the duplicate
history file.

Jason has the concern that if we do that, the client will be redirected to JHS before the
history file is copied. I've done more investigation. The client service seems to be the last
one to be stopped by MRAppMaster. On the other side, realProxy will cached until the connect
get failed. Therefore, we just need to make sure copying the history file to the intermediate
done dir after unregistration and before stopping the client service, and we should still
be safe. Please correct me if I'm wrong.

Any thoughts?

> Job history should not be flushed to JHS until AM gets unregistered
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5547
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message