hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3245) Provide ability to persist running jobs (extend HADOOP-1876)
Date Mon, 21 Jul 2008 06:57:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615158#action_12615158
] 

dhruba borthakur commented on HADOOP-3245:
------------------------------------------

+1 to Owen's comment.

A pressing need of our cluster is to not interrupt running jobs if the jobtracker has to be
restarted. This means  that job states have to be persisted in the form of a transaction log.
This requirement is all the more beneficial to sites that have long-running job trackers (instead
of HOD).

However, isn't it better to be able to store state in HDFS? It is true that HDFS stores its
transaction log in local files, but with the current focus on improving HDFS read/write latencies,
HDFS itself is considering whether to store one copy of the transaction log in HDFS blocks
(instead of NFS). In fact, if the JobTracker stores information in a org.apache.hadoop.fs.FileSystem,
then a typical customer install could plug in various forms of storage to support the JobTracker
transaction log.



> Provide ability to persist running jobs (extend HADOOP-1876)
> ------------------------------------------------------------
>
>                 Key: HADOOP-3245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3245
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Amar Kamat
>         Attachments: HADOOP-3245-v2.5.patch, HADOOP-3245-v2.6.5.patch, HADOOP-3245-v2.6.9.patch,
HADOOP-3245-v4.1.patch
>
>
> This could probably extend the work done in HADOOP-1876. This feature can be applied
for things like jobs being able to survive jobtracker restarts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message