hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.
Date Sat, 01 Mar 2014 06:39:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916958#comment-13916958
] 

Ming Ma commented on YARN-221:
------------------------------

Thanks, Jason.

To fix the racing between container's exit by itself and MRAppMaster's stopContainer, I will
upload the patch to https://issues.apache.org/jira/browse/MAPREDUCE-5465.

To support the feature of being able to have X% of container logs aggregated, we can do it
at the yarn layer instead at MR layer. In that way, other applications can get it from free.

If AM doesn't specify any log aggregation policy as part of ContainerLaunchContext, yarn's
default log aggregation policy will be applied. The default polices could be like:

For worker containers,
    1. Always aggregate logs of failed or killed containers.
    2. Aggregate subset of container logs. The sample rate is configurable and it is specific
to the application.

For AM containers,
    1. Always aggregate logs of failed or killed containers.
    2. By default, AM log will be aggregated regardless of the status. It can be disabled
via configuration and that will only impact succeeded container.

Comments?

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Chris Trezzo
>         Attachments: YARN-221-trunk-v1.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the logs should
not be aggregated, that they should be aggregated with a high priority, or that they should
be aggregated but with a lower priority.  The AM should be able to do this in the ContainerLaunch
context to provide a default value, but should also be able to update the value when the container
is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid connection
to the NN at all.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message