hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4953) Delete completed container log folder when rolling log aggregation is enabled
Date Thu, 14 Apr 2016 04:23:25 GMT

     [ https://issues.apache.org/jira/browse/YARN-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rohith Sharma K S updated YARN-4953:
------------------------------------
    Target Version/s: 2.9.0

> Delete completed container log folder when rolling log aggregation is enabled
> -----------------------------------------------------------------------------
>
>                 Key: YARN-4953
>                 URL: https://issues.apache.org/jira/browse/YARN-4953
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>
> There would be potential bottle neck when cluster is running with very large number of
containers on the same NodeManager for single application. The linux limits the subfolders
count to 32K. If number of containers is greater than 32K for an application, there would
be container launch failure. At this point of time, there are no more containers can be launched
in this node.
> Currently log folders are deleted after app is finished. Rolling log aggregation aggregates
logs to hdfs periodically. 
> I think if aggregation is completed for finished containers, then clean up can be done
i.e deleting log folder for finished containers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message