hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode
Date Wed, 26 Apr 2017 17:40:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985230#comment-15985230

Robert Kanter commented on YARN-6521:

I think what [~piaoyu zhang] is asking is that when you disable log aggregation (because it
was making too many small files and causing a problem for the NN), he still wants to be able
to get logs from the YARN CLI logs command, which requires log aggregation.  

[~piaoyu zhang], instead of disabling log aggregation, you could periodically run the Archive
Logs tool (https://hadoop.apache.org/docs/r2.8.0/hadoop-archive-logs/HadoopArchiveLogs.html),
which combines all of the log files for an Application into one file.  The YARN CLI logs command
and JHS can still read these files.  It does require Hadoop 2.8.0+ though.

> Yarn Log-aggregation Transform Enable not to Spam the NameNode
> --------------------------------------------------------------
>                 Key: YARN-6521
>                 URL: https://issues.apache.org/jira/browse/YARN-6521
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: zhangyubiao
> Nowdays We have large cluster,with the apps incr and container incr ,We split an  namespace
to store applogs. 
> But the  log-aggregation /tmp/app-logs incr also make the namespace responce slow.  
> We want to chang yarn.log-aggregation-enable true -> false,But Transform the the yarn
log cli service also can get the app-logs.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message