hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5547) Job history should not be flushed to JHS until AM gets unregistered
Date Thu, 03 Oct 2013 06:30:44 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784858#comment-13784858

Zhijie Shen commented on MAPREDUCE-5547:

I thought about the problem again. This jira described the case that when the AM has moved
the job history file to JHS, and then somehow failed at unregister. Then the 2nd AM attempt
will start, but from the aspect of JHS, user will see the job is already finished. Actually,
it should be a common case.

If we changed to unregistered AM before moving the history file to JHS, it would happen that
RM tells the client that the job is already finished, and the client will resort to JHS to
get the information. However, the history file hasn't arrived JHS (copying big files may take
time). Therefore, the job information will be unavailable for some time. Unfortunately, if
we did this change, the unavailability of the job information would be a common case around
every jobs' completion, no matter they succeed, fail, reboot.

Therefore, IMHO, it's not good to fix a bug in a rare case at the cost of troubling the common
case. Probably we need to find some approach else.

> Job history should not be flushed to JHS until AM gets unregistered
> -------------------------------------------------------------------
>                 Key: MAPREDUCE-5547
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen

This message was sent by Atlassian JIRA

View raw message