hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Jeffrey <bryan.jeff...@gmail.com>
Subject Hive 0.13.0 Memory Leak
Date Thu, 15 May 2014 13:29:43 GMT
All,

We are running Hadoop 2.2.0 and Hive 0.13.0.  One typical application is to
load data (as text), and then convert that data to ORC to decrease query
time.  When running these processes we are seeing significant memory leaks
(leaking 4 GB in about 5 days).

We're running HiveServer2 with the following options:

root     28136     1 51 May14 ?        09:51:09 /usr/java/latest/bin/java
-Xmx2048m -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40
-XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit -XX:MaxPermSize=1024m
-XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15
-XX:-UseGCOverheadLimit -XX:MaxPermSize=1024m
-Dhadoop.log.dir=/opt/hadoop/latest-hadoop/logs
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/latest-hadoop
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Djava.library.path=/opt/hadoop/latest-hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
/opt/hadoop/latest-hive/lib/hive-service-0.13.0.jar
org.apache.hive.service.server.HiveServer2

Typical ORC conversion query looks like the following:

HIVE -u jdbc:hive2://hive_server:10002/db -n root --hiveconf
hive.exec.dynamic.partition.mode=nonstrict --hiveconf
hive.enforce.sorting=true --hiveconf $SET_QUEUE -e "insert into table
orc_table partition (range) select * from loading_text_table; "

I saw a couple of tickets for memory leaks, but they seemed to deal with
failed queries.  The memory usage increases in a linear fashion.  Jobs all
succeed until memory limit exceeded.

Is there an open bug for memory leaks associated with successful jobs in
HS2?  Is there a fix for this issue?

Regards,

Bryan Jeffrey

Mime
View raw message