incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From agateaaa <agate...@gmail.com>
Subject Intermittent failure inserting into hcatalog table get AlreadyExistsException(message:Partition already exists error
Date Thu, 29 Nov 2012 20:07:24 GMT
Hi All:


I am see that sometimes my jobs inserting data into a partition fail
with this error AlreadyExistsException(message:Partition already
exists:Partition(values:[CLICKS, US, CA, 2012-11-01.log.txt, pending,
2012-11-29T19:16:47], dbName:default,


As far as I can tell the partition that the job is trying to write to
doesnt exist before the job fires u, and partition  should have been
created by the job itself.


The workaround that I have had to do is to restart the hcatalog server
(hive metastore server). After the restart (and deleting the
partitions from hive cli) the error goes away and the insert that
previously choked works fine.


Has any one seen this error before?



java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:184)
	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1055)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:357)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:182)
	... 7 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error
adding partition to metastore. Cause :
org.apache.thrift.transport.TTransportException
	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:618)
	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:174)
	... 12 more
Caused by: org.apache.thrift.transport.TTransportException
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions(ThriftHiveMetastore.java:947)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions(ThriftHiveMetastore.java:934)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:345)
	at org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:594)
	... 13 more

java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:184)
	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1055)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:357)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:182)
	... 7 more
Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error
adding partition to metastore. Cause :
AlreadyExistsException(message:Partition already
exists:Partition(values:[CLICKS, US, CA, 2012-11-01.log.txt, pending,
2012-11-29T19:16:47], dbName:default,




Thanks

Agatea

Mime
View raw message