hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4953) Delete completed container log folder when rolling log aggregation is enabled
Date Sat, 13 Aug 2016 03:23:22 GMT

    [ https://issues.apache.org/jira/browse/YARN-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419790#comment-15419790
] 

Rohith Sharma K S commented on YARN-4953:
-----------------------------------------

Thanks Jason for your inputs. Let me create version-0 patch initially.

> Delete completed container log folder when rolling log aggregation is enabled
> -----------------------------------------------------------------------------
>
>                 Key: YARN-4953
>                 URL: https://issues.apache.org/jira/browse/YARN-4953
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>
> There would be potential bottle neck when cluster is running with very large number of
containers on the same NodeManager for single application. The linux limits the subfolders
count to 32K. If number of containers is greater than 32K for an application, there would
be container launch failure. At this point of time, there are no more containers can be launched
in this node.
> Currently log folders are deleted after app is finished. Rolling log aggregation aggregates
logs to hdfs periodically. 
> I think if aggregation is completed for finished containers, then clean up can be done
i.e deleting log folder for finished containers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message