hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirav Patel <npa...@xactlycorp.com>
Subject Re: Wildcard and Automatic partitioning
Date Thu, 03 Aug 2017 18:41:39 GMT
What is the point if I have to rename hdfs directories to hive
format  (key=value/key=value etc.).  Why it can't just be normal directory
like everyone has. That entire directory can be considered as a key
("period" in my example) and hive add all values as partitions.

On Thu, Aug 3, 2017 at 11:25 AM, Sergey Shelukhin <sergey@hortonworks.com>
wrote:

> The typical, although not technically intended for the purpose, approach
> is to use msck to “repair” the table and create the partitions. The
> partitions have to be in the standard Hive format (key=value/key=value
> etc.) and the table must be created with the corresponding partition keys.
> It may actually be good to have a feature to do it in a standard manner
> for external tables only, however it would probably be restricted to the
> same format. So, the example below probably won’t work because of the star
> in the middle.
>
>
> From: Nirav Patel <npatel@xactlycorp.com>
> Reply-To: "user@hive.apache.org" <user@hive.apache.org>
> Date: Thursday, August 3, 2017 at 11:00
> To: "user@hive.apache.org" <user@hive.apache.org>
> Subject: Wildcard and Automatic partitioning
>
> Hi, is there a way in hive when I create an external table I can specify
> wild card in LOCATION and have hive automatically identify partitions.
>
> I have opened HIVE-17236
> <https://issues.apache.org/jira/browse/HIVE-17236> for wildcard support.
> same time I also have issue of specifying partitions.  I can use tedious
> ALTER TABLE to add partition directory. But since data already exist in
> separate partition why can't hive identify it ?
>
> Here's such example of a directory:
> /user/mycomp/customers/*/departments/partition/*
>
> I can have n number of customer and for each m number of partition for
> departments object.
>
> I think if I use following sql to create external table then it should be
> able to identify all the partitions.
>
> CREATE EXTERNAL TABLE testTable (val map<string, string>)
> PARTITIONED BY (period string)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> LOCATION '/user/mycomp/customers/*/departments/partition/*';
>
> If my partition directory (lets say p-12345) have multiple files insides
> it that doesn't start with "part-" prefix then I should be able to specify
> that prefix so hive can find the right filesets.
>
> Thanks
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
> <https://twitter.com/Xactly>  [image: Facebook]
> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> <http://www.youtube.com/xactlycorporation>
>

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

<https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>

Mime
View raw message