Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ACD9117A26 for ; Thu, 12 Mar 2015 09:39:36 +0000 (UTC) Received: (qmail 48202 invoked by uid 500); 12 Mar 2015 09:38:38 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 48177 invoked by uid 500); 12 Mar 2015 09:38:38 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 48119 invoked by uid 99); 12 Mar 2015 09:38:38 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 09:38:38 +0000 Date: Thu, 12 Mar 2015 09:38:38 +0000 (UTC) From: "Chaoyu Tang (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-9720) Metastore does not properly migrate column stats when renaming a table across databases. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-9720: ------------------------------ Attachment: HIVE-9720.patch Remove invalid column stats entries from TAB_COL_STATS and PART_COL_STATS after any change of db, table, partition, or column. > Metastore does not properly migrate column stats when renaming a table across databases. > ---------------------------------------------------------------------------------------- > > Key: HIVE-9720 > URL: https://issues.apache.org/jira/browse/HIVE-9720 > Project: Hive > Issue Type: Bug > Components: Metastore > Affects Versions: 0.13.1 > Reporter: Alexander Behm > Assignee: Chaoyu Tang > Attachments: HIVE-9720.patch > > > It appears that the Hive Metastore does not properly migrate column statistics when renaming a table across databases. While renaming across databases is not supported in HiveQL, it can be done via the Metastore Thrift API. > The problem is that such a newly renamed table cannot be dropped (unless renamed back to its original database/name). > Here are steps for reproducing the issue. > 1. From the Hive shell/beeline: > {code} > create database db1; > create database db2; > create table db1.mv (i int); > use db1; > analyze table mv compute statistics for columns i; > {code} > 2. From a Java program: > {code} > public static void main(String[] args) throws Exception { > HiveConf conf = new HiveConf(MetaStoreClientPool.class); > HiveMetaStoreClient hiveClient = new HiveMetaStoreClient(conf); > Table t = hiveClient.getTable("db1", "mv"); > t.setDbName("db2"); > t.setTableName("mv2"); > hiveClient.alter_table("db1", "mv", t); > } > {code} > 3. From the Hive shell/beeline: > {code} > drop table db2.mv2; > {code} > Stack shown when running 3: > {code} > FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore > at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106) > at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown Source) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > NestedThrowablesStackTrace: > java.sql.BatchUpdateException: Batch entry 0 DELETE FROM "TBLS" WHERE "TBL_ID"='1621' was aborted. Call getNextException to see the cause. > at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598) > at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) > at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407) > at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2737) > at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424) > at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) > at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628) > at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596) > at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683) > at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86) > at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454) > at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:203) > at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:267) > at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) > at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106) > at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown Source) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > ) > hive> drop table db2.mv2; > FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore > at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) > at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106) > at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown Source) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > NestedThrowablesStackTrace: > java.sql.BatchUpdateException: Batch entry 0 DELETE FROM "TBLS" WHERE "TBL_ID"='1621' was aborted. Call getNextException to see the cause. > at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2598) > at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) > at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407) > at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2737) > at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424) > at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) > at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628) > at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596) > at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683) > at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86) > at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454) > at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:203) > at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:267) > at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) > at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:411) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) > at com.sun.proxy.$Proxy0.commitTransaction(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1389) > at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1525) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106) > at com.sun.proxy.$Proxy1.drop_table_with_environment_context(Unknown Source) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8072) > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:8056) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > ) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)