hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5929) Cleanup JobHistory file naming to do with job recovery
Date Thu, 28 May 2009 03:56:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713850#action_12713850

Devaraj Das commented on HADOOP-5929:

Also, we should remove the <date> field from the job history filename format. Anyway
the date is already there in the jobID and that is a part of the filename.

> Cleanup JobHistory file naming to do with job recovery
> ------------------------------------------------------
>                 Key: HADOOP-5929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5929
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Devaraj Das
>             Fix For: 0.21.0
> The JobTracker uses the job history files for doing job recovery upon startup. To handle
cases where JobTracker goes down again while the recovered job is running, there is some logic
that plays with files and it ends up having two history files for some window of time during
the life of the job - actual history file, .recover file. The idea being that upon the next
restart we should be able to the maximal number of events for the job. It led to performance
problems in the job submission / recovery (part of which got addressed in HADOOP-4372). It
also looks pretty unlikely that a running job will traverse across multiple JT restarts. Even
if it did, without the .recover file, it'd only mean that we lose some tasks that got completed
in a subsequent restart. I propose that we remove the .recover file logic and base the recovery
on only the original job history file. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message