hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4953) Delete completed container log folder when rolling log aggregation is enabled
Date Wed, 13 Apr 2016 04:30:25 GMT
Rohith Sharma K S created YARN-4953:
---------------------------------------

             Summary: Delete completed container log folder when rolling log aggregation is
enabled
                 Key: YARN-4953
                 URL: https://issues.apache.org/jira/browse/YARN-4953
             Project: Hadoop YARN
          Issue Type: Bug
          Components: nodemanager
            Reporter: Rohith Sharma K S
            Assignee: Rohith Sharma K S


There would be potential bottle neck when cluster is running with very large number of containers
on the same NodeManager for single application. The linux limits the subfolders count to 32K.
If number of containers is greater than 32K for an application, there would be container launch
failure. At this point of time, there are no more containers can be launched in this node.

Currently log folders are deleted after app is finished. Rolling log aggregation aggregates
logs to hdfs periodically. 

I think if aggregation is completed for finished containers, then clean up can be done i.e
deleting log folder for finished containers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message