hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-65) Reduce RM app memory footprint once app has completed
Date Thu, 10 Aug 2017 12:20:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121530#comment-16121530

Rohith Sharma K S commented on YARN-65:

thanks [~manirajv06@gmail.com] for the patch. 
Clearing AMContainer launch context should not be done unless application is completed. It
is always buggy!! And also I see existing code also a buggy which clears {{app.submissionContext.getAMContainerSpec().setTokensConf(null);}}.
After application is submitted, at any point of time application can be updated. During application
update, we also stores application submission context back to RMStateStore which overrides.
And if RM restart after app update then it is a issue.

As a solution, unused fields must be cleared only after app is finished may be in final transition.

> Reduce RM app memory footprint once app has completed
> -----------------------------------------------------
>                 Key: YARN-65
>                 URL: https://issues.apache.org/jira/browse/YARN-65
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 0.23.3
>            Reporter: Jason Lowe
>            Assignee: Manikandan R
>         Attachments: YARN-65.001.patch, YARN-65.002.patch, YARN-65.003.patch, YARN-65.004.patch,
YARN-65.005.patch, YARN-65.006.patch, YARN-65.007.patch
> The ResourceManager holds onto a configurable number of completed applications (yarn.resource.max-completed-applications,
defaults to 10000), and the memory footprint of these completed applications can be significant.
 For example, the {{submissionContext}} in RMAppImpl contains references to protocolbuffer
objects and other items that probably aren't necessary to keep around once the application
has completed.  We could significantly reduce the memory footprint of the RM by releasing
objects that are no longer necessary once an application completes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message