hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumanth V <vsumant...@gmail.com>
Subject Re: Query using dynamic partitions
Date Sun, 18 Sep 2011 02:56:05 GMT
Hi Adriaan,

To use dynamic partition, follow the following steps inside hive shell -

#Set the following values -

set hive.exec.dynamic.partition.mode=nonstrict;

set hive.exec.dynamic.partition=true

#Create another table -

create table raw_2
(
 data string
)
partitioned by (partition1 string, partition2 string);

#Now insert the values stored in table raw into table raw_2 using the
following query -

from raw
insert overwrite table raw_2 partition (partition1, partition2)
select data, partition1, partition2;

This will dynamically create the 2 partitions based on the values of
partition1 and partition2 and insert the values of 'data' in the appropriate
partition.

Regards,
Sumanth



On Sat, Sep 17, 2011 at 2:18 PM, Adriaan Tijsseling
<adriaan@tijsseling.com>wrote:

> Hi,
>
> I have a table created with
>
> CREATE TABLE raw(partition1 string, partition2 string, data string) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
>
> I want to further process "data" and put it in a partition (partition1,
> partition2) defined by the values in the relevant row.
>
> I'm however stuck at trying to use dynamic partitions in a query. With
> predefined partition values it's straightforward:
>
> FROM (
>  FROM raw
>  SELECT TRANSFORM(raw.data)
>         USING 'python parser.py' AS (foo STRING, date STRING, bar
> MAP<STRING,STRING>)
>  CLUSTER BY date
> ) tmap
> INSERT OVERWRITE TABLE polished PARTITION (partition1='p1',
> partition2='p2') SELECT foo, date, bar;
>
> What would be the best way to define the partition using raw.partition1 and
> raw.partition2 as values?
>
> Thanks much,
>
> Adriaan
>
>

Mime
View raw message