Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Fri, 12 Jul 2013 07:59:49 +0000 (UTC)
From: "Zhijie Shen (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12523887.1316625031717.38829.1373615989477@arcas>
In-Reply-To: <JIRA.12523887.1316625031717@arcas>
References: <JIRA.12523887.1316625031717@arcas>
Subject: [jira] [Commented] (YARN-321) Generic application history service
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706773#comment-13706773 ] 

Zhijie Shen commented on YARN-321:
----------------------------------

bq. Are we moving aggregated log management(i.e deletion after expiry) responsibility to AHS?

Sorry for misunderstanding your previous question. IMHO, in the recent future, we're not moving the aggregated log management, but duplicate it, which Both AHS and JHS can serve the same aggregated logs. However, AHS and JHS see the same logs from different point of views. AHS simply considers them as container logs, no matter what application it is, while JHS know they are the MR job logs. [~vinodkv], would you please confirm it?
                
> Generic application history service
> -----------------------------------
>
>                 Key: YARN-321
>                 URL: https://issues.apache.org/jira/browse/YARN-321
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Vinod Kumar Vavilapalli
>
> The mapreduce job history server currently needs to be deployed as a trusted server in sync with the mapreduce runtime. Every new application would need a similar application history server. Having to deploy O(T*V) (where T is number of type of application, V is number of version of application) trusted servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and history data into a particular directory for later serving. Job history data is already stored as json (or binary avro). I propose that we create only one trusted application history server, which can have a generic UI (display json as a tree of strings) as well. Specific application/version can deploy untrusted webapps (a la AMs) to query the application history server and interpret the json for its specific UI and/or analytics.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira