hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4852) Resource Manager Ran Out of Memory
Date Thu, 03 Nov 2016 15:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15633063#comment-15633063
] 

Rohith Sharma K S commented on YARN-4852:
-----------------------------------------

[~sharadag] [~slukog] As per analysis in earlier discussions, YARN-4862 was identified as
reason for OOM. 
Should this JIRA can be resolved since YARN-4862 patch got committed? Does YARN-4862 need
to backported to Hadoop-2.6?

> Resource Manager Ran Out of Memory
> ----------------------------------
>
>                 Key: YARN-4852
>                 URL: https://issues.apache.org/jira/browse/YARN-4852
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Gokul
>         Attachments: threadDump.log
>
>
> Resource Manager went out of memory (max heap size: 8 GB, CMS GC) and shut down itself.

> Heap dump analysis reveals that 1200 instances of RMNodeImpl class hold 86% of memory.
When digging  deeper, there are around 0.5 million objects of UpdatedContainerInfo (nodeUpdateQueue
inside RMNodeImpl). This in turn contains around 1.7 million objects of YarnProtos$ContainerIdProto,
ContainerStatusProto, ApplicationAttemptIdProto, ApplicationIdProto each of which retain around
1 GB heap.
> Back to Back Full GC kept on happening. GC wasn't able to recover any heap and went OOM.
JVM dumped the heap before quitting. We analyzed the heap. 
> RM's usual heap usage is around 4 GB but it suddenly spiked to 8 GB in 20 mins time and
went OOM.
> There are no spike in job submissions, container numbers at the time of issue occurrence.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message