Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Thu, 25 Feb 2016 02:12:18 +0000 (UTC)
From: "Ming Ma (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12941258.1456190961000.140666.1456366338216@Atlassian.JIRA>
In-Reply-To: <JIRA.12941258.1456190961000@Atlassian.JIRA>
References: <JIRA.12941258.1456190961000@Atlassian.JIRA>
 <JIRA.12941258.1456190961344@arcas>
Subject: [jira] [Commented] (YARN-4720) Skip unnecessary NN operations in
 log aggregation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15166575#comment-15166575 ] 

Ming Ma commented on YARN-4720:
-------------------------------

It seems that {{LogAggregationStatus.RUNNING}} implies the log aggregation service is running, it doesn't necessarily mean NM actually aggregate any logs. So if the long running service is running and hasn't generate any logs since it starts, it is better to return {{LogAggregationStatus.RUNNING}}.

Yes, NM can send several {{LogAggregationReport}}s in the list which is ordered; that is the API between NM and RM. Then on RM side, it will retrieve all elements from the list.

> Skip unnecessary NN operations in log aggregation
> -------------------------------------------------
>
>                 Key: YARN-4720
>                 URL: https://issues.apache.org/jira/browse/YARN-4720
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Jun Gong
>         Attachments: YARN-4720.01.patch, YARN-4720.02.patch
>
>
> Log aggregation service could have unnecessary NN operations in the following scenarios:
> * No new local log has been created since the last upload for the long running service scenario.
> * NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for certain containers.
> In the following code snippet, even though {{pendingContainerInThisCycle}} is empty, it still creates the writer and then removes the file later. Thus it introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't aggregate logs for an app.
>   
> {noformat}
> AppLogAggregatorImpl.java
> ......
>         writer =
>             new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
>                 this.userUgi);
> ......
>       for (ContainerId container : pendingContainerInThisCycle) {
> ......
>       }
> ......
>             if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
>               if (rename) {
>                 remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
>               } else {
>                 remoteFS.delete(remoteNodeTmpLogFileForApp, false);
>               }
>             }
> ......
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)