hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3245) Provide ability to persist running jobs (extend HADOOP-1876)
Date Fri, 12 Sep 2008 20:21:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amar Kamat updated HADOOP-3245:
-------------------------------

    Attachment: HADOOP-3245-v5.36.1-no-log.patch

Attaching a patch that is updated to trunk. Following are the changes
1) There seems to be a bug in {{RawLocalFileSystem}}, see [here|HADOOP-4167]. A simple fix
is to synchronize all such accesses to history directory in {{JobHistory}}. Clearly this solution
wont work if some external process changes/deletes files from jobhistory directory. The attached
patch synchronizes all the apis that might get invoked simultaneously.
2) {{JobHistory.parseHistoryFromFS()}} allows incomplete log lines to be passed to the listener.
This is an issue for {{RecoveryManager}} as partial updates might lead to inconsistent state.
One way to overcome this is to have a _LINE_DELIMITER_ just to indicate that the line is complete.
The only drawback with this approach is that  its a backward incompatible change where the
new parser (with line-delimiters) wont be able to parse old jobhistory files. Will open a
jira to address this. For now I have implemented the _LINE_DELIMITER_ approach.


> Provide ability to persist running jobs (extend HADOOP-1876)
> ------------------------------------------------------------
>
>                 Key: HADOOP-3245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3245
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Amar Kamat
>         Attachments: HADOOP-3245-v2.5.patch, HADOOP-3245-v2.6.5.patch, HADOOP-3245-v2.6.9.patch,
HADOOP-3245-v4.1.patch, HADOOP-3245-v5.13.patch, HADOOP-3245-v5.14.patch, HADOOP-3245-v5.26.patch,
HADOOP-3245-v5.30-nolog.patch, HADOOP-3245-v5.31.3-nolog.patch, HADOOP-3245-v5.33.1.patch,
HADOOP-3245-v5.35.3-no-log.patch, HADOOP-3245-v5.36-no-log.patch, HADOOP-3245-v5.36.1-no-log.patch
>
>
> This could probably extend the work done in HADOOP-1876. This feature can be applied
for things like jobs being able to survive jobtracker restarts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message