hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Oskarsson (JIRA)" <>
Subject [jira] Commented: (HIVE-91) Allow external tables with different partition directory structure
Date Thu, 11 Dec 2008 14:11:44 GMT


Johan Oskarsson commented on HIVE-91:

My approach would be to have a command to add partitions manually, I have created a jira ticket
for it: HIVE-115. There's also already a method for this in the metastore thrift interface
if I'm not mistaken. For us it would be fairly simple to add another command after loading
our data into hdfs.
It would also be a bit tricky to automatically find partitions from HDFS if they have a custom
format. I can't think of a way off the top of my head if you have directories like so: /dataset/2008/12/10/spain
where 2008/12/11 is one partition and spain is another. Then we'd have to save more information
on the exact directory structure for each partition and it seems to get more complex then
it has to at this stage.

> Allow external tables with different partition directory structure
> ------------------------------------------------------------------
>                 Key: HIVE-91
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: Metastore
>            Reporter: Johan Oskarsson
>            Assignee: Johan Oskarsson
>            Priority: Minor
> A lot of users have datasets in a directory structures similar to this in hdfs: /dataset/yyyy/MM/dd/<one
or more files>
> Instead of loading these into Hive the normal way it would be useful to create an external
table with the /dataset location and then one partition per yyyy/mm/dd. This would require
the partition "naming to directory"-function to be made more flexible.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message