hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.
Date Fri, 14 Aug 2015 16:20:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697314#comment-14697314
] 

Ming Ma commented on YARN-221:
------------------------------

The unit test failures aren't related. The tests pass on the local machine.

Another thing Xuan and I discussed is how other frameworks on YARN such as MR, Tez can use
this feature; for example if they need to make config and/or code change to allow framework
applications specify the policy at per application basis. There are several approaches.

* Have MR define its own configurations to config these policies. Make code change at YarnRunner
to retrieve these configurations and set the values at ASC. That means Tez needs to do the
same thing.
* Define some common YARN configurations such as yarn.logaggregation.policy.class. YarnRunner
still needs to retrieve these configurations and set the values at ASC. But at least MR and
Tez can share the same configuration names.
* Define some common YARN configurations such as yarn.logaggregation.policy.class. YarnClientImpl
take care of fixing up ASC based on the configurations. In that way, no code change is required
at the MR or Tez layer.

Eventually, we prefer to go with the first approach, which is used by other existing MR properties.
If we want to define some common YARN properties used by different YARN applications, we can
have a separate jira for it.

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Ming Ma
>         Attachments: YARN-221-6.patch, YARN-221-7.patch, YARN-221-8.patch, YARN-221-9.patch,
YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch, YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch,
YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the logs should
not be aggregated, that they should be aggregated with a high priority, or that they should
be aggregated but with a lower priority.  The AM should be able to do this in the ContainerLaunch
context to provide a default value, but should also be able to update the value when the container
is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid connection
to the NN at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message