hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode
Date Wed, 26 Apr 2017 18:12:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985289#comment-15985289

Junping Du commented on YARN-6521:

[~piaoyu zhang], I think what [~rkanter] mentioned above should at least partially work for
you. If that is still not enough, I am concerning disabling log aggregation and allow some
app's aggregated log on demand will bring many other issues, like: how to manage local container
logs and how to make sure app's log still exists when log CLI runs, etc. 
An alternative way is to set TTL to aggregated logs to keep latest finished/running apps only,
it can be flexible in policies, like: time based, e.g. latest 15 days; app number based, e.g.
latest 1000 apps or even size based (considering long running apps).
CC [~xgong], [~jlowe].

> Yarn Log-aggregation Transform Enable not to Spam the NameNode
> --------------------------------------------------------------
>                 Key: YARN-6521
>                 URL: https://issues.apache.org/jira/browse/YARN-6521
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: zhangyubiao
> Nowdays We have large cluster,with the apps incr and container incr ,We split an  namespace
to store applogs. 
> But the  log-aggregation /tmp/app-logs incr also make the namespace responce slow.  
> We want to chang yarn.log-aggregation-enable true -> false,But Transform the the yarn
log cli service also can get the app-logs.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message