hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14938) Configuration.updatingResource map should be initialized lazily
Date Thu, 12 Oct 2017 05:03:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201444#comment-16201444
] 

Daniel Templeton commented on HADOOP-14938:
-------------------------------------------

>From what I've read, you're almost correct on DCL.  The one difference is that the JVM
may do operator reordering, which can cause the value to be returned before the cache flushing
is complete (or something like that) unless the value is volatile.

LGTM, but someone else should take a closer look.  I keep doing these reviews while on the
run.

> Configuration.updatingResource map should be initialized lazily
> ---------------------------------------------------------------
>
>                 Key: HADOOP-14938
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14938
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Misha Dmitriev
>            Assignee: Misha Dmitriev
>         Attachments: HADOOP-14938.01.patch, HADOOP-14938.02.patch
>
>
> Using jxray (www.jxray.com), I've analyzed a heap dump of YARN RM running in a big cluster.
The tool uncovered several inefficiencies in the RM memory. It turns out that one of the biggest
sources of memory waste, responsible for almost 1/4 of used memory, is empty ConcurrentHashMap
instances in org.apache.hadoop.conf.Configuration.updatingResource:
> {code}
> 905,551K (24.0%): java.util.concurrent.ConcurrentHashMap: 22118 / 100% of empty 905,551K
(24.0%)
> ↖org.apache.hadoop.conf.Configuration.updatingResource
> ↖{j.u.WeakHashMap}.keys
> ↖Java Static org.apache.hadoop.conf.Configuration.REGISTRY
> {code}
> That is, there are 22118 empty ConcurrentHashMaps here, and they collectively waste ~905MB
of memory. This is caused by eager initialization of these maps. To address this problem,
we should initialize them lazily.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message