Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Fri, 14 Aug 2015 16:20:47 +0000 (UTC)
From: "Ming Ma (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12616458.1353091549000.69555.1439569247689@Atlassian.JIRA>
In-Reply-To: <JIRA.12616458.1353091549000@Atlassian.JIRA>
References: <JIRA.12616458.1353091549000@Atlassian.JIRA>
 <JIRA.12616458.1353091549470@arcas>
Subject: [jira] [Commented] (YARN-221) NM should provide a way for AM to
 tell it not to aggregate logs.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697314#comment-14697314 ] 

Ming Ma commented on YARN-221:
------------------------------

The unit test failures aren't related. The tests pass on the local machine.

Another thing Xuan and I discussed is how other frameworks on YARN such as MR, Tez can use this feature; for example if they need to make config and/or code change to allow framework applications specify the policy at per application basis. There are several approaches.

* Have MR define its own configurations to config these policies. Make code change at YarnRunner to retrieve these configurations and set the values at ASC. That means Tez needs to do the same thing.
* Define some common YARN configurations such as yarn.logaggregation.policy.class. YarnRunner still needs to retrieve these configurations and set the values at ASC. But at least MR and Tez can share the same configuration names.
* Define some common YARN configurations such as yarn.logaggregation.policy.class. YarnClientImpl take care of fixing up ASC based on the configurations. In that way, no code change is required at the MR or Tez layer.

Eventually, we prefer to go with the first approach, which is used by other existing MR properties. If we want to define some common YARN properties used by different YARN applications, we can have a separate jira for it.

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Ming Ma
>         Attachments: YARN-221-6.patch, YARN-221-7.patch, YARN-221-8.patch, YARN-221-9.patch, YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch, YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch, YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the logs should not be aggregated, that they should be aggregated with a high priority, or that they should be aggregated but with a lower priority.  The AM should be able to do this in the ContainerLaunch context to provide a default value, but should also be able to update the value when the container is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid connection to the NN at all.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)