hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3245) Provide ability to persist running jobs (extend HADOOP-1876)
Date Tue, 05 Aug 2008 22:54:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620077#action_12620077
] 

dhruba borthakur commented on HADOOP-3245:
------------------------------------------

Regarding the file being zero size, you can work around like this (on trunk):

1. open the file with the FileSysterm.append() API. in a loop unteil it stops geenrating AlreadyBeingCreatedException.
2. Then close the file and reopen using FileSystem.open(). Yoiu should see the new contents
of the file that was present when you did the sync().

Opening a file for "append" tells the namenode to recover a lease, but only if the soft-limit
time has expired. The soft-limit timer is 1 minute. The namenode then starts lease recovery
and finds out the size of the last block of the file. Nowm if the client re-opens the file,
it gets a new block list with updated block size.

> Provide ability to persist running jobs (extend HADOOP-1876)
> ------------------------------------------------------------
>
>                 Key: HADOOP-3245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3245
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Amar Kamat
>         Attachments: HADOOP-3245-v2.5.patch, HADOOP-3245-v2.6.5.patch, HADOOP-3245-v2.6.9.patch,
HADOOP-3245-v4.1.patch, HADOOP-3245-v5.13.patch
>
>
> This could probably extend the work done in HADOOP-1876. This feature can be applied
for things like jobs being able to survive jobtracker restarts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message