incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohini Palaniswamy <rohini.adi...@gmail.com>
Subject Re: Pig creates files on HDFS but partitions not getting added
Date Fri, 06 Apr 2012 21:44:08 GMT
David,
I looked at the mapred log for your job.

2012-04-06 17:32:48,452 INFO hive.metastore: Trying to connect to
metastore with URI thrift://gsbl90831.blue.ygrid.yahoo.com:9080
2012-04-06 17:32:48,496 INFO hive.metastore: Connected to metastore.
2012-04-06 17:32:48,522 ERROR hive.metastore: Unable to shutdown local
metastore client
org.apache.thrift.transport.TTransportException: SASL authentication
not complete
	at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:443)
	at org.apache.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)

.........

        at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)
	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.cleanupJob(FileOutputCommitterContainer.java:430)
	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:255)


You are hitting HCATALOG-236. hcat branch-0.4 uses hive-0.8 and you should
not be hitting this with that. Trunk uses hive 0.9 snapshot and that throws
this error. As part of the patch to MultiOutputFormat (HCAT-36) I at least
made the code to ignore exceptions when connection to metastore fails and
just log a warning. So when that gets checked in you can probably use trunk
builds. Till then can use hcat branch-0.4 builds?

Regards,
Rohini

On Fri, Apr 6, 2012 at 1:21 PM, David Capwell <dcapwell@gmail.com> wrote:

> I am using trunk and it seems that pig is not updating partitions for me.
>  Is anyone else seeing pig scripts not updated the partitions?  Below is
> the following commands run.
>
> CREATE TABLE tmp_table (
>   data string
> )
> partitioned by (
>   datestamp string
>  ,srcid string
>  ,action string
>  ,testid string
> )
> stored as rcfile
> location '/tmp/hcat_tmp_tables/tmp_table';
>
> The end of my pigscript looks like this:
>
> store b into 'default.tmp_table' using
>
> org.apache.hcatalog.pig.HCatStorer('datestamp=20091103,srcid=19174,action=click,testid=NOTESTID');
>
>
> On HDFS I see the following:
>
> -rw-rw-r--   3 dcapwell hdfs   13990632 2012-04-06
>
> 17:32/tmp/hcat_tmp_tables/tmp_table/action=click/datestamp=20091103/srcid=19174/testid=NOTESTID/part-m-00000
>
>
> When I check the partitions in Hive I see the following:
> ./bin/hive -e "show partitions
> tmp_table"
> OK
> Time taken: 1.461 seconds
>
> So it looks like the pig job finishes just fine, writes the files to HDFS
> but
> the partitions are not getting updated?
>
>
> thanks for your time reading this email.
>

Mime
View raw message