hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Harris <>
Subject RE: Loading multiple file format in hive
Date Tue, 25 Aug 2015 18:27:26 GMT
A few things..
1) If you are using spark streaming, I don't see any reason why the output of your spark streaming
can't match the necessary destination shouldn't need a second job to read the
output from Spark Streaming and convert to parquet.  Do a search for spark streaming and lambda

2) a simple solution to your problem (even without using spark streaming) would be to simply
have an external "staging" table, a hive managed "destination" table and then use a view to
UNION the two together.
CREATE EXTERNAL TABLE raw_staging (inputline STRING) LOCATION '/staging/';
CREATE TABLE parsed as SELECT split(inputline,',') as field_array from raw_staging;
CREATE VIEW combined as SELECT field_array FROM(SELECT split(inputline,',') as field_array
from raw_staging UNION ALL select field_array from pared) subq_u;

Even if possible to mix and match the schema on a per-partition, I wouldn't recommend doing

From: Jeetendra G []
Sent: Tuesday, August 25, 2015 12:37 AM
Subject: Re: Loading multiple file format in hive

If I write to staging area and then run job to convert this data to parquet , there wont be
delay of this much time? mean to say this data wont be available to hive until it converts
to parquet and write to hive location?

On Tue, Aug 25, 2015 at 11:53 AM, Nitin Pawar <<>>
Is it possible for you to write the data into staging area and run a job on that and then
convert ito paraquet table ?
so you are looking to have two table .. one temp for holding data till 15mins and then your
job loads this temp data to to your parquet backed table
sorry for my misunderstanding .. you can though set fileformat at each partition level but
then you need to entirely redesign your table to have staging partition and real data partition

On Tue, Aug 25, 2015 at 11:46 AM, Jeetendra G <<>>
Thanks Nitin for reply.

I have data coming from RabbitMQ and i have spark streaming API which take this events and
dump into HDFS.
I cant really convert data events to some format like parquet/orc because I dont have schema
Once I dump to HDFS i am writing one job which read this data  and convert into Parquet.
By this time I will have some raw events right?

On Tue, Aug 25, 2015 at 11:35 AM, Nitin Pawar <<>>
file formats in a hive is a table level property.
I am not sure why would you have data at 15mins interval to your actual table instead of a
staging table and do the conversion or have the raw file in the format you want and load it
directly into table

On Tue, Aug 25, 2015 at 11:27 AM, Jeetendra G <<>>
I tried searching how to set multiple format with multiple partitions , could not find much
Can please share some good material around this if you have any.

On Mon, Aug 24, 2015 at 10:49 PM, Daniel Haviv <<>>
You can set a different file format per partition.
You can't mix files in the same directory (You could theoretically write some kind of custom


On Mon, Aug 24, 2015 at 6:15 PM, Jeetendra G <<>>
Can anyone put some light on this please?

On Mon, Aug 24, 2015 at 12:32 PM, Jeetendra G <<>>
HI All,

I have a directory where I have json formatted and parquet files in same folder. can hive
load these?

I am getting Json data and storing in HDFS. later I am running job to convert JSon to Parquet(every
15 mins). so we will habe 15 mins Json data.

Can i provide multiple serde in hive?


Nitin Pawar

Nitin Pawar

information that is privileged and exempt from disclosure under applicable law. If you are
neither the intended recipient nor responsible for delivering the message to the intended
recipient, please note that any dissemination, distribution, copying or the taking of any
action in reliance upon the message is strictly prohibited. If you have received this communication
in error, please notify the sender immediately.  Thank you.
View raw message