hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4953) Delete completed container log folder when rolling log aggregation is enabled
Date Fri, 03 Jun 2016 07:17:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313775#comment-15313775

Rohith Sharma K S commented on YARN-4953:

bq. The main issue with aggregating as containers complete is the additional load on the namenode
Right, this is major issue in large cluster.

Since log rolling is supported I think it is worth to delete aggregated completed container
log folders when log rolling is enabled.  Any potential issues with it. Thoughts?

> Delete completed container log folder when rolling log aggregation is enabled
> -----------------------------------------------------------------------------
>                 Key: YARN-4953
>                 URL: https://issues.apache.org/jira/browse/YARN-4953
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
> There would be potential bottle neck when cluster is running with very large number of
containers on the same NodeManager for single application. The linux limits the subfolders
count to 32K. If number of containers is greater than 32K for an application, there would
be container launch failure. At this point of time, there are no more containers can be launched
in this node.
> Currently log folders are deleted after app is finished. Rolling log aggregation aggregates
logs to hdfs periodically. 
> I think if aggregation is completed for finished containers, then clean up can be done
i.e deleting log folder for finished containers. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message