hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-321) [Umbrella] Generic application history service
Date Fri, 01 May 2015 22:48:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524113#comment-14524113
] 

Zhijie Shen edited comment on YARN-321 at 5/1/15 10:47 PM:
-----------------------------------------------------------

Close this umbrella jira with few sub tasks open. Generic history service has been implemented
and rides on timeline server, but is not production ready. The subtasks have come into multiple
versions, but mostly before 2.6. YARN-2271 is left open to track one possible performance
issue to fetch all the applications stored in the timeline store.


was (Author: zjshen):
Close this umbrella jira with few sub tasks open. Generic history service has been implemented
and rides on timeline server. YARN-2271 is left open to track one possible performance issue
to fetch all the applications stored in the timeline store.

> [Umbrella] Generic application history service
> ----------------------------------------------
>
>                 Key: YARN-321
>                 URL: https://issues.apache.org/jira/browse/YARN-321
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Luke Lu
>         Attachments: AHS Diagram.pdf, ApplicationHistoryServiceHighLevel.pdf, Generic
Application History - Design-20131219.pdf, HistoryStorageDemo.java
>
>
> The mapreduce job history server currently needs to be deployed as a trusted server in
sync with the mapreduce runtime. Every new application would need a similar application history
server. Having to deploy O(T*V) (where T is number of type of application, V is number of
version of application) trusted servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and history data
into a particular directory for later serving. Job history data is already stored as json
(or binary avro). I propose that we create only one trusted application history server, which
can have a generic UI (display json as a tree of strings) as well. Specific application/version
can deploy untrusted webapps (a la AMs) to query the application history server and interpret
the json for its specific UI and/or analytics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message