hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <>
Subject Dynamic Partitioning not working in HCatalog 0.11?
Date Thu, 10 Oct 2013 20:43:19 GMT
Here's some simple Pig that reads from one Hive table and writes to another
(same data, same schema):

sigs_in = load 'signals' using org.apache.hcatalog.pig.HCatLoader();
sigs = filter sigs_in by datetime_partition == '2013-10-07_0000';
STORE sigs INTO 'signals_orc' USING org.apache.hcatalog.pig.HCatStorer();

the signals_orc table is defined to have the datetime_partition partition,
as in:

create external table signals_orc (
 signal_id string,
) partitioned by (datetime_partition string)
stored as ORC
location '/user/hive/external/signals_orc'
tblproperties ("orc.compress"="snappy");

After running the job, I end up with the following directory in HDFS:


When clearly the filter in the Pig proves the datetime_partition field in
my data is valid. If I change the store clause in my Pig script to:

STORE sigs INTO 'signals_orc' USING

then I get the correct output:


So it appears to me that there's something broken in the dynamic
partitioning code in HCatalog 0.11.0.


View raw message