hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-5341) DefaultMetricsSystem leaks the source name when a source unregisters
Date Fri, 08 Jul 2016 21:57:11 GMT

     [ https://issues.apache.org/jira/browse/YARN-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated YARN-5341:
-----------------------------
    Summary: DefaultMetricsSystem leaks the source name when a source unregisters  (was: Nodemanager
leaks ContainerResource names in UniqueNames for each container)

This looks like a problem in DefaultMetricsSystem.  There are two UniqueNames instances in
that class, mBeanNames and sourceNames.  When a ContainerMetrics source unregisters the names
are getting cleaned out properly from mBeanNames but not from sourceNames.  Each source will
register a new name in sourceNames, but the only way to clear them out is by shutting down
the DefaultMetricsSystem instance.  There needs to be a way to remove elements from sourceNames
when a source unregisters.

I updated the summary based on this analysis, and I'll move the JIRA to the HADOOP project
since that's where the fix is needed.

> DefaultMetricsSystem leaks the source name when a source unregisters
> --------------------------------------------------------------------
>
>                 Key: YARN-5341
>                 URL: https://issues.apache.org/jira/browse/YARN-5341
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.2
>            Reporter: Jason Lowe
>            Priority: Critical
>
> Ran across a nodemanager that was spending most of its time in GC.  Upon examination
of the heap most of the memory was going to the map of names in org.apache.hadoop.metrics2.lib.UniqueNames.
 In this case the map had almost 2 million entries.  Looking at a few of the map showed entries
like "ContainerResource_container_e01_1459548490386_8560138_01_002020", "ContainerResource_container_e01_1459548490386_2378745_01_000410",
etc.
> Looks like the ContainerMetrics for each container will cause a unique name to be registered
with UniqueNames and the name will never be unregistered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message