hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chenjianxun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16220) Memory leak when creating a table using location and NameNode in HA
Date Tue, 12 Mar 2019 06:15:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790264#comment-16790264
] 

chenjianxun commented on HIVE-16220:
------------------------------------

is there any progess for this issue?

i have got the same problem, the instances of "org.apache.hadoop.hive.conf.HiveConf" keep
growing.

and the dump file of hiveserver2 shows that there is a memory leak problem.

 

 

Leak Suspects
1,041 instances of *"org.apache.hadoop.hive.conf.HiveConf"*, loaded by *"sun.misc.Launcher$AppClassLoader
@ 0xeaaa0058"* occupy *270,268,056 (52.70%)* bytes. These instances are referenced from one
instance of *"java.util.HashMap$Node[]"*, loaded by *"<system class loader>"*

> Memory leak when creating a table using location and NameNode in HA
> -------------------------------------------------------------------
>
>                 Key: HIVE-16220
>                 URL: https://issues.apache.org/jira/browse/HIVE-16220
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 1.2.1
>         Environment: HDP-2.4.0.0
>            Reporter: Angel Alvarez Pascua
>            Priority: Major
>
> The following simple DDL
> CREATE TABLE `test`(`field` varchar(1)) LOCATION 'hdfs://benderHA/apps/hive/warehouse/test'
> ends up generating a huge memory leak in the HiveServer2 service.
> After two weeks without a restart, the service stops suddenly because of OutOfMemory
errors.
> This only happens when we're in an environment in which the NameNode is in HA,  otherwise,
nothing (so weird) happens. If the location clause is not present, everything is also fine.
> It seems, multiples instances of Hadoop configuration are created when we're in an HA
environment:
> <AFTER ONE EXECUTIONS OF CREATE TABLE WITH LOCATION>
> 2.618 instances of "org.apache.hadoop.conf.Configuration", loaded by "sun.misc.Launcher$AppClassLoader
@ 0x4d260de88" 
> occupy 350.263.816 (81,66%) bytes. These instances are referenced from one instance of
"java.util.HashMap$Node[]", 
> loaded by "<system class loader>"
> <AFTER TWO EXECUTIONS OF CREATE TABLE WITH LOCATION>
> 5.216 instances of "org.apache.hadoop.conf.Configuration", loaded by "sun.misc.Launcher$AppClassLoader
@ 0x4d260de88" 
> occupy 699.901.416 (87,32%) bytes. These instances are referenced from one instance of
"java.util.HashMap$Node[]", 
> loaded by "<system class loader>"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message