hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl Steinbach (JIRA)" <>
Subject [jira] Created: (HIVE-951) Selectively include EXTERNAL TABLE source files via REGEX
Date Tue, 24 Nov 2009 03:10:40 GMT
Selectively include EXTERNAL TABLE source files via REGEX

                 Key: HIVE-951
             Project: Hadoop Hive
          Issue Type: Improvement
          Components: Query Processor
            Reporter: Carl Steinbach

CREATE EXTERNAL TABLE should allow users to cherry-pick files via regular expression. 
CREATE EXTERNAL TABLE was designed to allow users to access data that exists outside of Hive,
currently makes the assumption that all of the files located under the supplied path should
be included
in the new table. Users frequently encounter directories containing multiple
datasets, or directories that contain data in heterogeneous schemas, and it's often
impractical or impossible to adjust the layout of the directory to meet the requirements of

CREATE EXTERNAL TABLE. A good example of this problem is creating an external table based
on the contents of an S3 bucket. 

One way to solve this problem is to extend the syntax of CREATE EXTERNAL TABLE
as follows:

LOCATION path [file_regex]

For example:

CREATE EXTERNAL TABLE mytable1 ( a string, b string, c string )
LOCATION 's3://my.bucket/' 'folder/2009.*\.bz2$';

Creates mytable1 which includes all files in s3:/my.bucket with a filename matching 'folder/2009*.bz2'

CREATE EXTERNAL TABLE mytable2 ( d string, e int, f int, g int )
LOCATION 'hdfs://data/' 'xyz.*2009????.bz2$';

Creates mytable2 including all files matching 'xyz*2009????.bz2' located under hdfs://data/

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message