hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristof Vanbecelaere <>
Subject Re: select list for dynamic partition insert
Date Tue, 22 Jul 2014 20:24:12 GMT
I see. The last column in u_data is unixtime while I wanted to partition on
rating. I just assumed Hive would use the same-named column as the one
mentioned in the partition spec. Thanks for clarifying this, I missed that
bit in the documentation.

On Tue, Jul 22, 2014 at 10:13 PM, Prasanth Jayachandran <> wrote:

> From the error msg it looks like there are too many distinct values in
> partition column. Try increasing the count
> hive.exec.max.dynamic.partitions.pernode to a number >100. If you already
> know the number of distinct values in partition column, try a value greater
> than or equal to that number.
> Hive uses the last column of select * query as the partition column.
> Projecting the list of columns in select query is not always mandatory.
> Thanks
> Prasanth Jayachandran
> On Jul 22, 2014, at 1:07 PM, Kristof Vanbecelaere <
>> wrote:
> While playing with the movielens data set to learn about dynamic
> partitions I ran
> from u_data insert overwrite table u_data_p partition (rating) select *
> This failed with
> [Error 20004]: Fatal error occurred when node tried to create too many
> dynamic partitions. The maximum number of dynamic partitions is controlled
> by hive.exec.max.dynamic.partitions and
> hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
> Apparently select * cannot be used and one should explicitly list the
> fields in the select list. I guess a beter error message is required here.
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

View raw message