incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Potter <thelabd...@gmail.com>
Subject Pig partition filter using operator other than ==
Date Mon, 19 Nov 2012 19:53:29 GMT
Hi,

I'm using HCatalog 0.4.0 with Pig 0.10 and am not having success using an
operator other than (==) with my partition field.

For example, the following works (day is my partition field):

signals = load 'signals' using org.apache.hcatalog.pig.HCatLoader();

signals_for_day = filter signals by (day == '2012-10-30_1200' AND service IS
NOT NULL);

samp1 = sample signals_for_day 0.01;

dump samp1;


but, if I change my filter to: signals_for_day = filter signals by (day >=
'2012-10-30_1200' AND service IS NOT NULL);

Then I get the following error:

Caused by: java.io.IOException:
org.apache.thrift.transport.TTransportException
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:42)
at org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:90)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:380)
... 19 more
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_by_filter(ThriftHiveMetastore.java:1511)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_by_filter(ThriftHiveMetastore.java:1495)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByFilter(HiveMetaStoreClient.java:691)
at
org.apache.hcatalog.mapreduce.InitializeInput.getSerializedHcatKeyJobInfo(InitializeInput.java:98)
at
org.apache.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:73)
at
org.apache.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:40)
... 21 more

I can start debugging but would like to know if HCatalog is supposed to
support this type of filtering by partition fields?

Thanks.
Tim

Mime
View raw message