hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-321) Generic application history service
Date Fri, 12 Jul 2013 06:21:53 GMT

    [ https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706686#comment-13706686

Zhijie Shen commented on YARN-321:

bq. ResoureManager will write per-application data to a (hopefully very) thin HistoryStorage

Here's more ellaboration about the per-application data. There should be three objects to
record: RMApp, RMAppAttempt and RMContainer. Bellow are properties of each object:

Completed Application:
* Application ID
* Application Name
* Application Type
* User
* Queue
* Submit Time
* Start Time
* Finish Time
* Diagnostics Info
* Final Application Status
* Num of Application Attempts

Completed Application Attempt:
* Application Attempt ID
* Application ID
* Host
* RPC Port
* Tracking URL
* Original Tracking URL (not sure it is necessary)
* Diagnostics Info
* Final Application Status
* Master Container ID
* Num of Containers

Completed Container:
* Container ID
* Application Attempt ID
* Final Container Status
* Resource
* Priority
* Node ID
* Log URL

Application has one-to-many relationship with Application Attempt, while Application Attempt
has one-to-one relationship with Container.

WRT the concrete information to record, here's more idea about the interface of HistoryStorage.
The follow APIs should be useful for RM to persist application history and for AHS to query
* Iterable<CompletedApplication> getApplications([conditions...])
* CompletedApplication getApplication(ApplicationId)
* Iterable<CompletedApplicationAttempt> getApplicationAttempts(ApplicationId)
* CompletedApplicationAttempt getApplicationAttempt(ApplicationAttemptId)
* CompletedContainer getContainer(ApplicationAttemptId)
* CompletedContainer getContainer(ContainerId)
* void addApplication(CompletedApplication)
* void addApplicationAttempt(CompletedApplicationAttempt)
* void addContainer(CompletedContainer)

In addition, HistoryStorage APIs may involve a lot of I/O operations such that the response
of an API will be long. Therefore, it is likely to be good to make the API non-blocking.
> Generic application history service
> -----------------------------------
>                 Key: YARN-321
>                 URL: https://issues.apache.org/jira/browse/YARN-321
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Vinod Kumar Vavilapalli
> The mapreduce job history server currently needs to be deployed as a trusted server in
sync with the mapreduce runtime. Every new application would need a similar application history
server. Having to deploy O(T*V) (where T is number of type of application, V is number of
version of application) trusted servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and history data
into a particular directory for later serving. Job history data is already stored as json
(or binary avro). I propose that we create only one trusted application history server, which
can have a generic UI (display json as a tree of strings) as well. Specific application/version
can deploy untrusted webapps (a la AMs) to query the application history server and interpret
the json for its specific UI and/or analytics.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message