hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-321) Generic application history service
Date Fri, 12 Jul 2013 07:09:52 GMT

    [ https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706730#comment-13706730
] 

Zhijie Shen commented on YARN-321:
----------------------------------

bq. How about the Job History Server which is currently handling this for MR Jobs? Are we
moving this responsibility from JHS to AHS to handle for all types of applications?

IMHO, AHS is used to record the history of some generic data of RM, such as RMApp, RMAppAttempt
and RMContainer, regardless what kind of application it is. Currently, JHS of MR can coexist
with AHS, recording the MR specific data, such as Job, Task and TaskAttempt. Perhaps in the
future, AHS can be enhanced to record the specific data of a certain type of application by
loading the per-framework "plugin", and JHS may be turned into such a plugin. Agree with [~vinodkv]
on keeping hosting/serving per-framework data out of the scope now.
                
> Generic application history service
> -----------------------------------
>
>                 Key: YARN-321
>                 URL: https://issues.apache.org/jira/browse/YARN-321
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Vinod Kumar Vavilapalli
>
> The mapreduce job history server currently needs to be deployed as a trusted server in
sync with the mapreduce runtime. Every new application would need a similar application history
server. Having to deploy O(T*V) (where T is number of type of application, V is number of
version of application) trusted servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and history data
into a particular directory for later serving. Job history data is already stored as json
(or binary avro). I propose that we create only one trusted application history server, which
can have a generic UI (display json as a tree of strings) as well. Specific application/version
can deploy untrusted webapps (a la AMs) to query the application history server and interpret
the json for its specific UI and/or analytics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message