hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Vonkip <mattvon...@yahoo.com>
Subject drop table -> java.lang.OutOfMemoryError: Java heap space
Date Wed, 04 Jan 2012 23:07:08 GMT
Hi folks,

I am using elastic-mapreduce in the Amazon EC2 eco-system and would like to upgrade from Hive
0.5 on Hadoop 0.20 to Hive 0.7.1 on Hadoop 0.20.205.  I created a new metastore (on S3) to
support testing the latter and have run into some problems.  I have about 15000 partitions
in S3 and in the old version of Hive/Hadoop, I have no problem creating a table, recovering
the partitions, and then dropping the table.  In the new version of Hive/Hadoop, the first
two steps are successful, but I run into a "java.lang.OutOfMemoryError: Java heap space" error
when I try to drop the table.

When I look at the output of "set;" from the hive prompt, I see several environment variables
related to heap size.  I was able to augment HADOOP_DATANODE_HEAPSIZE and HADOOP_NAMENODE_HEAPSIZE
each to 4096 (2048 is sufficient in 0.5/0.20), but I see other parameters including HADOOP_HEAPSIZE
that I cannot seem to change.  To be fair, I'm just shooting in the dark here and unable
to decipher from the error message *which* heap is too small.

If this is already documented somewhere (neither basic tutorials nor google searches helped),
I would be grateful for a reference and happy to summarize what I learn here.  Or, if you
simply have an answer ... well, any help would be most appreciated!

Sincerely,
Matt Vonkip

Mime
View raw message