hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jasper Knulst <jasper.knu...@vlc.nl>
Subject Re: can I use hive dynamic partition while loading data into tables?
Date Fri, 15 Apr 2011 07:51:01 GMT
Hi,

I think you are looking for this:

ALTER TABLE page_view ADD PARTITION (date='2008-06-08', country='US', type)
LOCATION '/<hdfs path relative to location of base hdfs/hive table dir>';

So you can go on adding new hdfs subdir's (based on the partition naming) to
you base hive hdfs table dir, moving extra files to these subdirs and making
them available to your hive users by adding new partitions afterwards.

This works only in case of EXTERNAL tables I guess.


Jasper


2011/4/15 Ning Zhang <nzhang@fb.com>

>  The INSERT OVERWRITE command will not overwrite the whole table. If you
> specify a partition in that table, it will only overwrite that partition. If
> you specify dynamic partitions, it will only create/overwrite partitions
> that will be seen from the input query (pvs.country in the example).
>
>
>  On Apr 15, 2011, at 12:31 AM, Erix Yao wrote:
>
> Does this mean if I want the type field as the partition key , I will have
> to split the raw data by myself and load the files into the target table?
>
>  I see there's an example in tutorial:
>
> FROM page_view_stg pvs
>     INSERT OVERWRITE TABLE page_view PARTITION(dt='2008-06-08', country)
>            SELECT pvs.viewTime, pvs.userid, pvs.page_url, pvs.referrer_url, null, null,
pvs.ip, pvs.country
>
>  but insert must overwrite the table whole table partition.
> Can I insert without the overwrite key word?
>
>
> 2011/4/15 Ning Zhang <nzhang@fb.com>
>
>> The LOAD DATA command only copy the files to the destination directory. It
>> doesn't read the records of the input file, so it cannot do partitioning
>> based on record values.
>>
>>  On Apr 14, 2011, at 10:52 PM, Erix Yao wrote:
>>
>> hi,all
>>     The dynamic partition function is amazing ,but only works in insert
>> clause. Can I use it while loading data into table?
>>
>>      For example: load data  LOAD DATA LOCAL INPATH
>> `/tmp/pv_2008-06-08_us.txt` INTO TABLE page_view
>> PARTITION(date='2008-06-08', country='US', type);
>> type is the dynamic partition key in the raw data?
>>
>>  This will be very cool! If is supported, I will not have to category the
>> raw data according to the type column.
>>
>>
>>
>> --
>> haitao.yao@Beijing
>>
>>
>>
>>
>>
>>
>
>
> --
> haitao.yao@Beijing
>
>
>
>
>
>


-- 
Kind Regards \ Met Vriendelijke Groet,





Jasper Knulst

BI Consultant





VLC Den Haag
Gildeweg 5B
2632 BD  Nootdorp


M: +31 (0)6 19 66 75 11

T: +31 (0)15 764 07 50
------------------------------------------------------------

Skype: jasper_knulst_vlc

Mime
View raw message