Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (athena.apache.org: domain of hadoopman@gmail.com
 designates 209.85.214.176 as permitted sender)
Message-ID: <4E1A6797.90305@gmail.com>
Date: Sun, 10 Jul 2011 21:01:43 -0600
From: hadoopman <hadoopman@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: user@hive.apache.org
Subject: Hive dynamic partition error?
References: <4F6B25AFFFCAFE44B6259A412D5F9B10331837F5@ExchMBX104.netflix.com>
 <CAENxBwxZ2xUMArF4Ci+R1-djypRUqiyJZD57V-7SSJa8As6oGw@mail.gmail.com>
In-Reply-To: 
 <CAENxBwxZ2xUMArF4Ci+R1-djypRUqiyJZD57V-7SSJa8As6oGw@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------040600080303030601010706"

This is a multi-part message in MIME format.
--------------040600080303030601010706
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

So we're seeing the following error during some of our hive loads:

2011-07-05 12:26:52,927 Stage-2 map = 100%,  reduce = 100%
Ended Job = job_201106302113_3864
Loading data to table default.merged_weblogs partition (day=null)
Failed with exception Number of dynamic partitions created is 1013, 
which is more than 1000. To solve this try to set 
hive.exec.max.dynamic.partitions to at least 1013.
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask

Here is a sample script we're running:

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions.pernode=10000;
SET hive.exec.max.dynamic.partitions=10000;
SET hive.exec.max.created.files=150000;

SET hive.exec.compress.intermediate=true;
SET hive.intermediate.compression.codec=com.hadoop.compression.lzo.LzoCodec;
SET hive.intermediate.compression.type=BLOCK;
SET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;

SET hive.exec.compress.output=true;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
SET mapred.output.compression.type=BLOCK;

FROM (
   SELECT hostname, name, ip, day
   FROM logsStaging
   UNION ALL
   SELECT hostname, name, ip, day
   FROM logs
) a

INSERT OVERWRITE TABLE logs PARTITION(day)
SELECT DISTINCT hostname, name, ip, day
DISTRIBUTE BY day;

QUIT;

Has anyone run into this problem before?  And I've noticed that 
increasing the number of partitions hasn't been working.  Been looking 
for the config.xml setting already configured with 'final' in the 
properties but no go so far.  I believe the default is 100 partitions 
and the job (when running) does show 10000 partitions (from the above 
script)

thoughts on what else to look at?
Thanks!

--------------040600080303030601010706
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="Content-Type">
  <title></title>
</head>
<body text="#000000" bgcolor="#ffffff">
So we're seeing the following error during some of our hive loads:
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">2011-07-05 12:26:52,927 Stage-2 map = 100%,&nbsp;
reduce = 100%<o:p></o:p><br>
Ended Job = job_201106302113_3864<br>
Loading data to table default.merged_weblogs partition
(day=null)<o:p></o:p><br>
Failed with exception Number of dynamic partitions created
is 1013, which is more than 1000. To solve this try to set
hive.exec.max.dynamic.partitions to at least 1013.<o:p></o:p><br>
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Here is a sample script we&#8217;re running:<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">SET hive.exec.dynamic.partition=true;<o:p></o:p><br>
SET hive.exec.dynamic.partition.mode=nonstrict;<o:p></o:p><br>
SET hive.exec.max.dynamic.partitions.pernode=10000;<o:p></o:p><br>
SET hive.exec.max.dynamic.partitions=10000;<o:p></o:p><br>
SET hive.exec.max.created.files=150000;<o:p> <br>
</o:p></p>
<p class="MsoNormal">SET hive.exec.compress.intermediate=true;<o:p></o:p><br>
SET
hive.intermediate.compression.codec=com.hadoop.compression.lzo.LzoCodec;<o:p></o:p><br>
SET hive.intermediate.compression.type=BLOCK;<o:p></o:p><br>
SET
mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;<o:p></o:p></p>
<p class="MsoNormal">SET hive.exec.compress.output=true;<o:p></o:p><br>
SET mapred.output.compress=true;<o:p></o:p><br>
SET
mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;<o:p></o:p><br>
SET mapred.output.compression.type=BLOCK;<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">FROM (<o:p></o:p><br>
&nbsp; SELECT hostname, name, ip, day<o:p></o:p><br>
&nbsp; FROM logsStaging<o:p></o:p><br>
&nbsp; UNION ALL<o:p></o:p><br>
&nbsp; SELECT hostname, name, ip, day<o:p></o:p><br>
&nbsp; FROM logs<o:p></o:p><br>
) a<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">INSERT OVERWRITE TABLE logs PARTITION(day)<o:p></o:p><br>
SELECT DISTINCT hostname, name, ip, day<o:p></o:p><br>
DISTRIBUTE BY day;<o:p></o:p></p>
QUIT;<br>
<br>
Has anyone run into this problem before?&nbsp; And I've noticed that
increasing the number of partitions hasn't been working.&nbsp; Been looking
for the config.xml setting already configured with 'final' in the
properties but no go so far.&nbsp; I believe the default is 100 partitions
and the job (when running) does show 10000 partitions (from the above
script)<br>
<br>
thoughts on what else to look at?<br>
Thanks!<br>
</body>
</html>

--------------040600080303030601010706--