hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Zhang <jiezh2...@gmail.com>
Subject embedded hive metastore fail when upgrading from 0.14.0 to 2.1.0
Date Tue, 14 Feb 2017 22:41:02 GMT
We are working on upgrading hive from apache hive 0.14.0 to 2.1.0, however,
hit the problem in a testcase which uses embedded metatore.

here is the code snippet and stack trace. Does anyone encounter the similar
problem and any suggestion to resolve it? Thanks for the help!

Connection connection =
DriverManager.getConnection("jdbc:hive2:///default", "hive", "");


 INFO 2017-02-14 14:23:21,749 [main] [HiveMetaStore] [line 564] 0:
Opening raw store with implementation
class:org.apache.hadoop.hive.metastore.ObjectStore
 INFO 2017-02-14 14:23:21,793 [main] [ObjectStore] [line 325]
ObjectStore, initialize called
 INFO 2017-02-14 14:23:23,014 [main] [ObjectStore] [line 429] Setting
MetaStore object pin classes with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/02/14 14:23:24 WARN DataNucleus.Query: Query for candidates of
org.apache.hadoop.hive.metastore.model.MDatabase and subclasses
resulted in no possible candidates
Required table missing : "DBS" in Catalog "" Schema "". DataNucleus
requires this table to perform its persistence operations. Either your
MetaData is incorrect, or you need to enable
"datanucleus.schema.autoCreateTables"
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required
table missing : "DBS" in Catalog "" Schema "". DataNucleus requires
this table to perform its persistence operations. Either your MetaData
is incorrect, or you need to enable
"datanucleus.schema.autoCreateTables"
	at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:606)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3365)
	at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2877)
	at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1608)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:671)
	at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:425)
	at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:864)
	at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:346)
	at org.datanucleus.store.query.Query.executeQuery(Query.java:1805)
	at org.datanucleus.store.query.Query.executeWithArray(Query.java:1733)
	at org.datanucleus.store.query.Query.execute(Query.java:1715)
	at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:371)
	at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:213)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.ensureDbInit(MetaStoreDirectSql.java:197)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:143)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:332)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:294)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:581)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:546)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1626)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3317)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3356)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
	at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
	at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
	at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:513)
	at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
	at org.apache.hive.service.cli.CLIService.init(CLIService.java:110)
	at org.apache.hive.service.cli.thrift.EmbeddedThriftBinaryCLIService.init(EmbeddedThriftBinaryCLIService.java:45)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:153)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at com.apple.athena.feeder.HiveJDBCTest.connect(HiveJDBCTest.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)


java.lang.RuntimeException: Error applying authorization policy on hive
configuration: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:Hive metastore database is not initialized. Please
use schematool (e.g. ./schematool -initSchema -dbType ...) to create the
schema. If needed, don't forget to include the option to auto-create the
underlying database in your JDBC connection string (e.g.
?createDatabaseIfNotExist=true for mysql))
at org.apache.hive.service.cli.CLIService.init(CLIService.java:112)
at
org.apache.hive.service.cli.thrift.EmbeddedThriftBinaryCLIService.init(EmbeddedThriftBinaryCLIService.java:45)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:153)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.apple.athena.feeder.HiveJDBCTest.connect(HiveJDBCTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:Hive metastore database is not initialized. Please
use schematool (e.g. ./schematool -initSchema -dbType ...) to create the
schema. If needed, don't forget to include the option to auto-create the
underlying database in your JDBC connection string (e.g.
?createDatabaseIfNotExist=true for mysql))
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:513)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:110)
... 32 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:Hive metastore database is not initialized. Please
use schematool (e.g. ./schematool -initSchema -dbType ...) to create the
schema. If needed, don't forget to include the option to auto-create the
underlying database in your JDBC connection string (e.g.
?createDatabaseIfNotExist=true for mysql))
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
... 35 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
MetaException(message:Hive metastore database is not initialized. Please
use schematool (e.g. ./schematool -initSchema -dbType ...) to create the
schema. If needed, don't forget to include the option to auto-create the
underlying database in your JDBC connection string (e.g.
?createDatabaseIfNotExist=true for mysql))
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3593)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
... 40 more
Caused by: MetaException(message:Hive metastore database is not
initialized. Please use schematool (e.g. ./schematool -initSchema -dbType
...) to create the schema. If needed, don't forget to include the option to
auto-create the underlying database in your JDBC connection string (e.g.
?createDatabaseIfNotExist=true for mysql))
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3364)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)
... 42 more


Jessica

Mime
View raw message