hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikandan R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-65) Reduce RM app memory footprint once app has completed
Date Fri, 28 Jul 2017 05:35:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104478#comment-16104478
] 

Manikandan R commented on YARN-65:
----------------------------------

Based on earlier suggestions, made changes to free memory by clearing unused variables explicitly
with either null or empty values, particularly submissioncontext object. As of now, we are
clearing up 

app.submissionContext.getAMContainerSpec().setTokensConf(null);

inside AddApplicationToSchedulerTransition, RMAppRecoveredTransition & FinalTransition
class in RMAppImpl.java. Created a new method to clear all unused fields including above mentioned
specific field so that it can be called from above mentioned classes.

Attached initial patch for review.

Couple of junits test cases like 

TestContainerAllocation.testLogAggregationContextPassedIntoContainerToken
TestRMWebServicesAppsModification.testGetNewApplicationAndSubmit

is failing because actual value would have been cleared and hence not matching with the expected
value. To confirm whether those fields has been used in application report, checked the AppBlock.java
and don't see any usage over there? Please correct me if this understanding is wrong. Based
on feedback, we may need to fix the Junit test cases accordingly.

In addition, to understand the memory savings, submitted an simple yarn app to RM (local setup)
in series and tried to visualize memory usage using jconsole during GC before and after this
code changes. I am not seeing any significant difference as may be because of app is very
light and size of each field inside submissioncontext etc.

> Reduce RM app memory footprint once app has completed
> -----------------------------------------------------
>
>                 Key: YARN-65
>                 URL: https://issues.apache.org/jira/browse/YARN-65
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 0.23.3
>            Reporter: Jason Lowe
>            Assignee: Naganarasimha G R
>         Attachments: YARN-65.001.patch
>
>
> The ResourceManager holds onto a configurable number of completed applications (yarn.resource.max-completed-applications,
defaults to 10000), and the memory footprint of these completed applications can be significant.
 For example, the {{submissionContext}} in RMAppImpl contains references to protocolbuffer
objects and other items that probably aren't necessary to keep around once the application
has completed.  We could significantly reduce the memory footprint of the RM by releasing
objects that are no longer necessary once an application completes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message