hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server
Date Thu, 20 Feb 2014 21:53:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907574#comment-13907574

Zhijie Shen commented on MAPREDUCE-5641:

bq. could you point us to how the AHS gets this information for AMs that crash. We might be
able to re-use some of that if the RM side of things for doing this is stable.

No matter an application is finished, removed or killed, it is supposed to be recorded by
AHS. However, it depends on what you need. If you're looking for the generic information,
AHS should meet your requirement. Otherwise, you still need to walk around before per framework
information of MR can be recorded.

> History for failed Application Masters should be made available to the Job History Server
> -----------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5641
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, jobhistoryserver
>    Affects Versions: 2.2.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-5641.patch, MAPREDUCE-5641.patch
> Currently, the JHS has no information about jobs whose AMs have failed.  This is because
the History is written by the AM to the intermediate folder just before finishing, so when
it fails for any reason, this information isn't copied there.  However, it is not lost as
its in the AM's staging directory.  To make the History available in the JHS, all we need
to do is have another mechanism to move the History from the staging directory to the intermediate
directory.  The AM also writes a "Summary" file before exiting normally, which is also unavailable
when the AM fails.  

This message was sent by Atlassian JIRA

View raw message