Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 31 Oct 2016 17:59:59 +0000 (UTC)
From: "Jason Lowe (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12545669.1331221158000.134911.1477936799703@Atlassian.JIRA>
In-Reply-To: <JIRA.12545669.1331221158000@Atlassian.JIRA>
References: <JIRA.12545669.1331221158000@Atlassian.JIRA> <JIRA.12545669.1331221158972@arcas>
Subject: [jira] [Commented] (YARN-65) Reduce RM app memory footprint once
 app has completed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 31 Oct 2016 18:00:02 -0000


    [ https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15622866#comment-15622866 ] 

Jason Lowe commented on YARN-65:
--------------------------------

I believe the request is still valid.  This isn't so much about adjusting the configurable number as it is reducing the memory waste per application.  Even with fewer applications retained there are opportunities to reduce the resourcemanager's memory requirements.

RMAppManager may only be tracking ApplicationId objects directly in completedApps, but the RMContext is tracking the full RMAppImpl objects that are looked up by those IDs (see RMContext#getRMApps and RMAppManager#checkAppNumCompletedLimit).  It doesn't make sense for the RM to track only the ApplicationID for completed applications since that alone is not enough to build an application report should someone come along and query it.  So the RM needs to track enough information to fill out a report for completed applications, but the submissionContext is a significant chunk of memory (it can be many megabytes in some cases) that can be freed once the application has finished.  There are a couple of items from the submission context that are referenced when filling out an app report (e.g.: priority, whether the AM was unmanaged), but those can be copied out, allowing the submission context to be released once the app finishes.

At one extreme we could take a similar approach to the MR job history server, using a cheaper representation for tracking completed applications, i.e.: when an application completes, replace the RMAppImpl object with something like a RMCompletedAppImpl that in turn has references to RMCompletedAppAttemptImpl rather than RMAppAttemptImpl.  Then we can dump the unnecessary concurrent hash maps necessary when the job is active and all the one-off tracking fields that only make sense when the app is alive.  Of course there are diminishing returns there and an increased maintenance burden, so it may not be worth it.  However releasing the submissionContext probably gets us most of the memory savings with what should be a relatively straightforward change.

> Reduce RM app memory footprint once app has completed
> -----------------------------------------------------
>
>                 Key: YARN-65
>                 URL: https://issues.apache.org/jira/browse/YARN-65
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 0.23.3
>            Reporter: Jason Lowe
>
> The ResourceManager holds onto a configurable number of completed applications (yarn.resource.max-completed-applications, defaults to 10000), and the memory footprint of these completed applications can be significant.  For example, the {{submissionContext}} in RMAppImpl contains references to protocolbuffer objects and other items that probably aren't necessary to keep around once the application has completed.  We could significantly reduce the memory footprint of the RM by releasing objects that are no longer necessary once an application completes.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org