hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hadoopman <hadoop...@gmail.com>
Subject Hive dynamic partition error?
Date Mon, 11 Jul 2011 03:01:43 GMT
So we're seeing the following error during some of our hive loads:

2011-07-05 12:26:52,927 Stage-2 map = 100%,  reduce = 100%
Ended Job = job_201106302113_3864
Loading data to table default.merged_weblogs partition (day=null)
Failed with exception Number of dynamic partitions created is 1013, 
which is more than 1000. To solve this try to set 
hive.exec.max.dynamic.partitions to at least 1013.
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask

Here is a sample script we're running:

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions.pernode=10000;
SET hive.exec.max.dynamic.partitions=10000;
SET hive.exec.max.created.files=150000;

SET hive.exec.compress.intermediate=true;
SET hive.intermediate.compression.codec=com.hadoop.compression.lzo.LzoCodec;
SET hive.intermediate.compression.type=BLOCK;
SET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;

SET hive.exec.compress.output=true;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
SET mapred.output.compression.type=BLOCK;

FROM (
   SELECT hostname, name, ip, day
   FROM logsStaging
   UNION ALL
   SELECT hostname, name, ip, day
   FROM logs
) a

INSERT OVERWRITE TABLE logs PARTITION(day)
SELECT DISTINCT hostname, name, ip, day
DISTRIBUTE BY day;

QUIT;

Has anyone run into this problem before?  And I've noticed that 
increasing the number of partitions hasn't been working.  Been looking 
for the config.xml setting already configured with 'final' in the 
properties but no go so far.  I believe the default is 100 partitions 
and the job (when running) does show 10000 partitions (from the above 
script)

thoughts on what else to look at?
Thanks!

Mime
View raw message