hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3496) Add a configuration to disable/enable storing localization state in NMLeveldbStateStore
Date Thu, 16 Apr 2015 14:37:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498088#comment-14498088
] 

Jason Lowe commented on YARN-3496:
----------------------------------

How are we going to support recovery of containers without tracking which resources we have
localized?  We cannot re-localize resources for already running containers -- they're already
running on the existing localized resources.  If we fail to track those resources then we
leak them on the local filesystem, potentially colliding with them later.  So from my perspective
there already is a way to enable/disable this behavior, and that's enabling/disabling NM recovery.
 I don't see how we can support recovery without storing localization state.

Do we have hard evidence storing to leveldb is a huge NM performance impact, or is this theoretical?
 In our testing we have not seen enabling NM recovery as having a significant impact on NM
performance.

> Add a configuration to disable/enable storing localization state in NMLeveldbStateStore
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-3496
>                 URL: https://issues.apache.org/jira/browse/YARN-3496
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>
> Add a configuration to disable/enable storing localization state in NMLeveldbStateStore.
> Store Localization state in the levelDB may have some overhead, which may affect NM performance.
> It would better to have a configuration to disable/enable it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message