hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-321) Generic application history service
Date Wed, 09 Oct 2013 16:54:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790604#comment-13790604

Zhijie Shen commented on YARN-321:

bq. I like the diagrams, but I want to understand if the generic application history service
is intended to replace the job history server, or to just augment it?

Yes, there's some previous discussion on recording the per application type history data,
but we plan to exclude it from the initial version of AHS. Eventually, we'd like to integrate
JHS into AHS in some way, the details of which we could discuss in the follow jiras. To me,
it would be better if we can design a common per application type plugin framework, such that
we can easily integrate the HS of other applications on YARN.

> Generic application history service
> -----------------------------------
>                 Key: YARN-321
>                 URL: https://issues.apache.org/jira/browse/YARN-321
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: AHS Diagram.pdf, ApplicationHistoryServiceHighLevel.pdf, HistoryStorageDemo.java
> The mapreduce job history server currently needs to be deployed as a trusted server in
sync with the mapreduce runtime. Every new application would need a similar application history
server. Having to deploy O(T*V) (where T is number of type of application, V is number of
version of application) trusted servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and history data
into a particular directory for later serving. Job history data is already stored as json
(or binary avro). I propose that we create only one trusted application history server, which
can have a generic UI (display json as a tree of strings) as well. Specific application/version
can deploy untrusted webapps (a la AMs) to query the application history server and interpret
the json for its specific UI and/or analytics.

This message was sent by Atlassian JIRA

View raw message