hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ameet chaubal <ameetchau...@yahoo.com>
Subject large sql file creating large num of columns
Date Mon, 16 Jan 2012 17:46:58 GMT
Hi All,

I have a SQL file of size 30mb which is a single create table statement with about 800,000
columns, hence the size. 

I am trying to execute it using hive -f 
<file>. Initially, hive ran the command with 256mb heap size and 
gave me an OOM error. I increased the heap size using export 
HADOOP_HEAPSIZE to 1 gb and eventually 2gb which made the OOM error go 
away. However, the hive command ran for 5 hours without actually 
creating the table. The JVM was running.
However,
1. running a strace on the process showed that it was stuck on a futex call.
2. I am using mysql for metastore and there were no rows added to either TBLS or COLUMNS table.

Question.
1. can hive do this create table of 800k columns from a sql file of 30mb?
2. if theoretically possible, what could be happening that's taking it over 5 hours and still
not succeeding?
Running it with debug, it spews the following,
---snip ----

stored as textfile location '<myfile>'
12/01/16 11:28:54 INFO parse.ParseDriver: Parse Completed
12/01/16 11:28:54 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
12/01/16 11:28:54 INFO parse.SemanticAnalyzer: Creating table my_table position=22
----

and it's stuck there at SemanticAnalyzer...

thanks for any insight.

sincerely,

ameet
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message