hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Acherkan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6113) Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
Date Tue, 08 Apr 2014 14:55:15 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963055#comment-13963055
] 

Eli Acherkan commented on HIVE-6113:
------------------------------------

The exact same issue reproduces here. Hive 0.12 on MapR 3.1.0 with MySQL metastore. The exception
appears when there are several processes working with Hive concurrently.

>From our analysis the problem seems related to the one described here: http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E

h5. Analysis:
At certain times, Hive's DataNucleus decides to create and then drop tables called "DELETEME"+timestamp
in the metastore schema on MySQL (see [ProbleTable|http://sourceforge.net/p/datanucleus/code/HEAD/tree/platform/store.rdbms/tags/datanucleus-rdbms-3.2.2/src/java/org/datanucleus/store/rdbms/table/ProbeTable.java]).

During other flows, DataNucleus queries MySQL for the list of all the columns of all the tables
(see [RDBMSSchemaHandler.refreshTableData|http://sourceforge.net/p/datanucleus/code/HEAD/tree/platform/store.rdbms/tags/datanucleus-rdbms-3.2.2/src/java/org/datanucleus/store/rdbms/schema/RDBMSSchemaHandler.java#l872]).
MySQL's JDBC driver implements the DatabaseMetaData.getColumns method by querying the DB for
a list of all the tables, and then iterating over that list and querying for each table's
columns (see [com.mysql.jdbc.DatabaseMetaData|http://bazaar.launchpad.net/~mysql/connectorj/5.1/view/head:/src/com/mysql/jdbc/DatabaseMetaData.java#L2581]).
If a table is deleted from the DB during this operation, DatabaseMetaData.getColumns will
throw an exception.

This exception is interpreted by Hive to mean that the "default" Hive database doesn't exist.
Hive tries to create it, inserting a row into the metastore.DBS table in MySQL, which triggers
the "Duplicate entry 'default' for key 'UNIQUE_DATABASE'" exception.

I'm not completely clear about the conditions for a) DataNucleus creating and dropping a "DELETEME"
table, and b) DataNucleus calling DatabaseMetaData.getColumns, so unfortunately I can't yet
provide a clear test case. But in our lab environment under load we were able to reproduce
the exception once every few minutes.

h5. Workaround:
As suggested by the link above, setting the *datanucleus.fixedDatastore* property to *true*
(e.g. in hive-site.xml or elsewhere) seems to solve the problem. However, it means that the
metastore schema is no longer automatically created on-demand, and requires using Hive's schematool
command to manually create the metastore schema.

> Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> --------------------------------------------------------------------------
>
>                 Key: HIVE-6113
>                 URL: https://issues.apache.org/jira/browse/HIVE-6113
>             Project: Hive
>          Issue Type: Bug
>          Components: Database/Schema
>    Affects Versions: 0.12.0
>         Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0
>            Reporter: William Stone
>            Priority: Critical
>              Labels: HiveMetaStoreClient, metastore, unable_instantiate
>
> When I exccute SQL "use fdm; desc formatted fdm.tableName;"  in python, throw Error as
followed.
> but when I tryit again , It will success.
> 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> 	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139)
> 	... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
> 	... 25 more
> Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> 	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> 	at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> 	at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:358)
> 	at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:404)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
> 	at $Proxy9.createDatabase(Unknown Source)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:422)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
> 	... 30 more
> Caused by: java.sql.BatchUpdateException: Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> 	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2028)
> 	at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1451)
> 	at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)
> 	at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
> 	at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628)
> 	at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596)
> 	at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683)
> 	at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86)
> 	at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454)
> 	at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:199)
> 	at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:263)
> 	at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
> 	... 46 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry 'default' for key 'UNIQUE_DATABASE'
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> 	at com.mysql.jdbc.Util.getInstance(Util.java:386)
> 	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1039)
> 	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
> 	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
> 	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
> 	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
> 	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
> 	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
> 	at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2427)
> 	at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1980)
> 	... 57 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message