hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server
Date Fri, 14 Feb 2014 19:58:23 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901870#comment-13901870

Vinod Kumar Vavilapalli commented on MAPREDUCE-5641:

Haven't yet read the discussion, but overall, we don't need yet another solution for this.
YARN-321 already is enabling generic history and so has record of killed/failed applications.
If at all we need a fix,
 - For the short term, we should make JHS invoke web-services on RM and/or AHS to obtain this
 - Medium/longer term, the generic data and timeline data (YARN-1530) will merge to expose
all information about apps via web-services. And JHS (if it still exists by that time) should
just use them.

> History for failed Application Masters should be made available to the Job History Server
> -----------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-5641
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, jobhistoryserver
>    Affects Versions: 2.2.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-5641.patch
> Currently, the JHS has no information about jobs whose AMs have failed.  This is because
the History is written by the AM to the intermediate folder just before finishing, so when
it fails for any reason, this information isn't copied there.  However, it is not lost as
its in the AM's staging directory.  To make the History available in the JHS, all we need
to do is have another mechanism to move the History from the staging directory to the intermediate
directory.  The AM also writes a "Summary" file before exiting normally, which is also unavailable
when the AM fails.  

This message was sent by Atlassian JIRA

View raw message