hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karan Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-882) Specify per user quota for private/application cache and user log files
Date Mon, 31 Jul 2017 11:54:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107187#comment-16107187

Karan Singh commented on YARN-882:

Currently  yarn.nodemanager.localizer.cache.target-size-mb and yarn.nodemanager.localizer.cache.cleanup.interval-ms
triggers deletion serivce for non-running conatiners. 

For containers that are running and spilling data to {'yarn.nodemanager.local-dirs'}/usercache/<user>/appcache/<app_id>
deletion service does not come into action, as a result filesystem gets full, nodes are marked
unhealthy and application gets stuck.

> Specify per user quota for private/application cache and user log files
> -----------------------------------------------------------------------
>                 Key: YARN-882
>                 URL: https://issues.apache.org/jira/browse/YARN-882
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Omkar Vinit Joshi
>            Assignee: Omkar Vinit Joshi
> At present there is no limit on the number of files / size of the files localized by
single user. Similarly there is no limit on the size of the log files created by user via
running containers.
> We need to restrict the user for this.
> For LocalizedResources; this has serious concerns in case of secured environment where
malicious user can start one container and localize resources whose total size >= DEFAULT_NM_LOCALIZER_CACHE_TARGET_SIZE_MB.
Thereafter it will either fail (if no extra space is present on disk) or deletion service
will keep removing localized files for other containers/applications. 
> The limit for logs/localized resources should be decided by RM and sent to NM via secured
containerToken. All these configurations should per container instead of per user or per nm.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message