incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Capwell <dcapw...@gmail.com>
Subject Re: Pig creates files on HDFS but partitions not getting added
Date Mon, 09 Apr 2012 22:41:35 GMT
This is an issue in 0.4 as well.  The only difference between 0.4 and trunk
is the error text itself is this:

2012-04-09 19:28:52,898 INFO hive.metastore: Trying to connect to metastore
with URI thrift://hcatserver.example.com:9080
2012-04-09 19:28:52,937 INFO hive.metastore: Connected to metastore.
2012-04-09 19:28:52,963 ERROR hive.metastore: Unable to shutdown local
metastore client
2012-04-09 19:28:52,963 ERROR hive.metastore:
[Ljava.lang.StackTraceElement;@1127b52

Looks like https://issues.apache.org/jira/browse/HCATALOG-236 is the jira
for this.

On Fri, Apr 6, 2012 at 2:44 PM, Rohini Palaniswamy
<rohini.aditya@gmail.com>wrote:

> David,
> I looked at the mapred log for your job.
>
> 2012-04-06 17:32:48,452 INFO hive.metastore: Trying to connect to metastore with URI
thrift://gsbl90831.blue.ygrid.yahoo.com:9080
> 2012-04-06 17:32:48,496 INFO hive.metastore: Connected to metastore.
> 2012-04-06 17:32:48,522 ERROR hive.metastore: Unable to shutdown local metastore client
> org.apache.thrift.transport.TTransportException: SASL authentication not complete
> 	at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:443)
> 	at org.apache.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)
>
> .........
>
>         at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)
> 	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.cleanupJob(FileOutputCommitterContainer.java:430)
> 	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:255)
>
>
> You are hitting HCATALOG-236. hcat branch-0.4 uses hive-0.8 and you should
> not be hitting this with that. Trunk uses hive 0.9 snapshot and that throws
> this error. As part of the patch to MultiOutputFormat (HCAT-36) I at least
> made the code to ignore exceptions when connection to metastore fails and
> just log a warning. So when that gets checked in you can probably use trunk
> builds. Till then can use hcat branch-0.4 builds?
>
> Regards,
> Rohini
>
> On Fri, Apr 6, 2012 at 1:21 PM, David Capwell <dcapwell@gmail.com> wrote:
>
>> I am using trunk and it seems that pig is not updating partitions for me.
>>  Is anyone else seeing pig scripts not updated the partitions?  Below is
>> the following commands run.
>>
>> CREATE TABLE tmp_table (
>>   data string
>> )
>> partitioned by (
>>   datestamp string
>>  ,srcid string
>>  ,action string
>>  ,testid string
>> )
>> stored as rcfile
>> location '/tmp/hcat_tmp_tables/tmp_table';
>>
>> The end of my pigscript looks like this:
>>
>> store b into 'default.tmp_table' using
>>
>> org.apache.hcatalog.pig.HCatStorer('datestamp=20091103,srcid=19174,action=click,testid=NOTESTID');
>>
>>
>> On HDFS I see the following:
>>
>> -rw-rw-r--   3 dcapwell hdfs   13990632 2012-04-06
>>
>> 17:32/tmp/hcat_tmp_tables/tmp_table/action=click/datestamp=20091103/srcid=19174/testid=NOTESTID/part-m-00000
>>
>>
>> When I check the partitions in Hive I see the following:
>> ./bin/hive -e "show partitions
>> tmp_table"
>> OK
>> Time taken: 1.461 seconds
>>
>> So it looks like the pig job finishes just fine, writes the files to HDFS
>> but
>> the partitions are not getting updated?
>>
>>
>> thanks for your time reading this email.
>>
>
>

Mime
View raw message