hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Sprague <sprag...@gmail.com>
Subject Re: Hive Server 2 memory leak
Date Wed, 11 Jun 2014 19:46:12 GMT
searching this list will in fact show you're not alone.   what is being
done about it is another matter.


On Wed, Jun 11, 2014 at 10:42 AM, Benjamin Bowman <bbowman410@gmail.com>
wrote:

> All,
>
> I am running Hadoop 2.4 and Hive 0.13.  I consistently run out of Hive
> heap space when running for a long period of time.  If I bump up the heap
> memory - it will run longer, but still eventually throws an out of memory
> error and becomes unresponsive.  The memory usage has a clearly linear
> trend which leads me to believe that there is a significant memory leak.
>  Has any one else had this problem or have any insight as to why I am
> seeing this?  Below I have included (1) the hive calls I am making, (2) the
> resulting java processes spawned off by these commands,  and (3) the top 50
> memory users according to a heap dump performed on Hive after it had run
> out of memory.  Note that each of these calls is being run several times a
> minute.
>
> Thanks,
> Benjamin Bowman
>
>
> (1) Hive/Hadoop calls:
> //Copying data to HDFS
> $HADOOP fs -copyFromLocal ${fullFile} $LOADING_DIR/${file}
> //Loading data
> $HIVE -u jdbc:hive2://$HIVE_MASTER:10002/database-n root -e "load data
> inpath '$LOADING_DIR/${file}' into table ${LOADING_TABLE} partition
> (range=$partitionDate);"
>  //ORC data
> $HIVE -u jdbc:hive2://$HIVE_MASTER:10002/database -n root --hiveconf
> hive.exec.dynamic.partition.mode=nonstrict --hiveconf
> hive.enforce.sorting=true --hiveconf $SET_QUEUE -e "insert into table
> ${TABLE_NAME} partition (range) select * from ${LOADING_TABLE};"
>
> (2) Java processes:
> // Copying data to HDFS
> /usr/java/latest/bin/java -Xmx1000m
> -Dhadoop.log.dir=/opt/hadoop/latest-hadoop/logs
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/latest-hadoop
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/opt/hadoop/latest-hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.fs.FsShell
> -copyFromLocal /data/file1.txt /database/loading/file1.txt
>
> // Loading data
> /usr/java/latest/bin/java -Xmx4096m -XX:NewRatio=12 -Xms10m
> -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit
> -XX:MaxPermSize=1024m -Dhadoop.log.dir=/opt/hadoop/latest-hadoop/logs
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/latest-hadoop
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/opt/hadoop/latest-hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
> /opt/hadoop/latest-hive/lib/hive-cli-0.13.0.jar
> org.apache.hive.beeline.BeeLine --hiveconf
> hive.aux.jars.path=file:///opt/hadoop/latest-hive/lib/hive-exec-0.13.0.jar,file:///opt/hadoop/latest-hive/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.jar
> -u jdbc:hive2://HIVE_MASTER:10002/database -n root -e load data inpath
> '/database/loading/file1.txt' into table loading_table partition
> (range=1402506000);
>
> //ORC data
> /usr/java/latest/bin/java -Xmx4096m -XX:NewRatio=12 -Xms10m
> -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit
> -XX:MaxPermSize=1024m -Dhadoop.log.dir=/opt/hadoop/latest-hadoop/logs
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/hadoop/latest-hadoop
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/opt/hadoop/latest-hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar
> /opt/hadoop/latest-hive/lib/hive-cli-0.13.0.jar
> org.apache.hive.beeline.BeeLine --hiveconf
> hive.aux.jars.path=file:///opt/hadoop/latest-hive/lib/hive-exec-0.13.0.jar,file:///opt/hadoop/latest-hive/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.jar
> -u jdbc:hive2://HIVE_MASTER:10002/database -n root --hiveconf
> hive.exec.dynamic.partition.mode=nonstrict --hiveconf
> hive.enforce.sorting=true --hiveconf mapred.job.queue.name=orc_queue -e
> insert into table TABLE_NAME partition (range) select * from loading_table;
>
> (3) Top 50 memory users of Hive heap space after 4 GB was consumed by Hive
> (Size is in Bytes)
>
>   Class Count Size  class [Ljava.util.HashMap$Entry; 6073306 1372508848  class
> [C 13007633 1034931150  class java.util.HashMap$Entry 19706437 551780236  class
> java.util.HashMap 6072908 291499584  class java.lang.String 13008539
> 260170780  class [Ljava.lang.Object; 2201432 180233440  class
> org.datanucleus.ExecutionContextThreadedImpl 439411 119959203  class
> com.mysql.jdbc.JDBC4ResultSet 352849 91740740  class [I 2433025 83659960  class
> org.datanucleus.FetchPlanForClass 2421672 79915176  class
> com.mysql.jdbc.StatementImpl 352825 75151725  class
> org.datanucleus.util.SoftValueMap$SoftValueReference 1483828 71223744  class
> org.datanucleus.FetchPlan 439412 33395312  class
> org.datanucleus.TransactionImpl 439411 33395236  class
> java.lang.ref.ReferenceQueue 1318447 31642728  class java.util.ArrayList
> 1759798 28156768  class org.datanucleus.api.jdo.JDOPersistenceManager
> 439411 25046427  class java.util.HashSet 2989733 23917864  class
> java.util.concurrent.locks.ReentrantLock$NonfairSync 745445 20872460  class
> java.util.IdentityHashMap 439419 19334436  class
> java.util.HashMap$KeyIterator 352852 14114080  class
> org.datanucleus.util.SoftValueMap 878903 14062448  class
> org.datanucleus.api.jdo.JDOCallbackHandler 439410 14061120  class
> java.util.HashMap$KeySet 1277338 10218704  class
> java.util.concurrent.ConcurrentHashMap$Segment 305936 9789952  class
> java.util.LinkedHashMap$Entry 219662 9665128  class
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsData 226336 8148096  class
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; 305936 7376992  class
> java.lang.ref.WeakReference 228210 7302720  class
> org.datanucleus.util.WeakValueMap 439441 7031056  class
> java.util.HashMap$Values 878871 7030968  class
> org.datanucleus.api.jdo.JDOTransaction 439411 7030576  class
> [Lcom.mysql.jdbc.Field; 352849 5645872  class java.util.LinkedList$Entry
> 226598 5438352  class org.datanucleus.ExecutionContextImpl$1 439411
> 5272932  class [B 37621 3863013  class java.util.HashMap$EntrySet 439686
> 3517488  class java.util.concurrent.locks.ReentrantLock 439465 3515720  class
> org.datanucleus.api.jdo.JDOFetchPlan 439411 3515288  class
> org.datanucleus.cache.SoftRefCache 439411 3515288  class
> org.datanucleus.properties.BasePropertyStore 439411 3515288  class
> java.util.IdentityHashMap$Values 439410 3515280  class
> [Ljava.util.concurrent.ConcurrentHashMap$Segment; 19126 2753504  class [S
> 21523 2370068  class java.util.Hashtable$Entry 71292 1996176  class
> org.datanucleus.state.LockManagerImpl 221721 1773768  class
> java.util.concurrent.atomic.AtomicBoolean 353119 1412476  class
> java.lang.Class 9397 1353168  class java.util.concurrent.ConcurrentHashMap
> 19126 1071056  class [Ljava.util.Hashtable$Entry; 544 1018368
>
>
>

Mime
View raw message