hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Subramanian <Sanjay.Subraman...@wizecommerce.com>
Subject Re: Noob question on creating tables
Date Fri, 29 Mar 2013 17:40:41 GMT
Hi

CREATE EXTERNAL TABLE IF NOT EXISTS log_data(col1 datatype1, col2
datatype2, . . . colN datatypeN) PARTITIONED BY (YEAR INT, MONTH INT, DAY
INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';


ALTER table log_data ADD PARTITION (YEAR=2013 , MONTH=2, DAY=27) LOCATION
'/path/to/YEAR/MONTH/DAY/directory/ON/HDFS';"

Hive will read gzip and bz2 files out of the box.(so suppose you had
hourly log files in gzip format in your /YEAR/MONTH/DAY directory then it
will be read)
Snappy and LZO will need some jar installs and configs
https://github.com/toddlipcon/hadoop-lzo

https://code.google.com/p/snappy/


Note that for example - gzip format is not splittable..so huge gzip files
without splits are not recommended as input to maps

Hope this helps

sanjay


On 3/29/13 10:19 AM, "Mark" <static.void.dev@gmail.com> wrote:

>We have existing log data in directories in the format of YEAR/MONTH/DAY.
>
>- How can we create a table over this table without hive modifying and/or
>moving it?
>- How can we tell Hive to partition this data so it knows about each day
>of logs?
>- Does hive out of the box work with reading compressed files?
>
>Thanks


CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply email and destroy all copies of the original message along with any attachments,
from your computer system. If you are the intended recipient, please be advised that the content
of this message is subject to access, review and disclosure by the sender's Email System Administrator.


Mime
View raw message