hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Steinbach <c...@cloudera.com>
Subject Re: how to load hive table schema programatically?
Date Mon, 14 Mar 2011 17:56:53 GMT
Hi Jae,

Sounds like your problem is related to HIVE-1435 (
https://issues.apache.org/jira/browse/HIVE-1435). You need to make sure that
the Datanucleus ORM layer is getting initialized with the configuration
property datanucleus.identifierFactory=datanucleus. Probably the easiest way
to fix this problem is to make sure that the 0.7.0 version of
hive-default.xml is available on the CLASSPATH and is getting loaded into
HiveConf. Try dumping the contents of your HiveConf object and make sure
that the values match those that appear in the 0.7.0 version of
hive-default.xml

Hope this helps.

Carl

On Mon, Mar 14, 2011 at 10:41 AM, Jae Lee <jlee119@gmail.com> wrote:

> just a bit more information from my debugging so far
>
> my mysql hive metastore have columns like
> "integer_idx" at "columns" table
> "integer_idx" at "sort_cols" table
>
> those columns looks pretty suspicious in that it is similar to "idx"
> columns
> that HiveMetaSotreClient complains missing.
>
> It looks like expectation of having "idx" column is auto-generated (not
> from
> package.jdo document)
> Can anybody tell me whether "integer_idx" column should have been "idx"
> column at "columns" table?
> or am I suppose to have custom package.jdo file that specify the index
> column name to "integer_idx" instead of "idx" column?
>
> J
>
> On Mon, Mar 14, 2011 at 2:27 PM, Jae Lee <jlee119@gmail.com> wrote:
>
> > Hi,
> >
> > I've had this code below working with Hive 0.5
> >
> > String databaseName = "default";
> > String tableName = "foobar";
> > List<org.apache.hadoop.hive.metastore.api.FieldSchema> hiveTable = new
> > HiveMetaStoreClient(new HiveConf(new Configuration(),
> > SessionState.class)).getSchema(databaseName, tableName);
> >
> >
> > to produce list of FieldSchema for a table foobar in default database
> >
> > I've recently upgraded hive to 0.7, and the same code now generates an
> > error messages such as
> >
> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was thrown
> > while adding/validating class(es) : Required columns missing from table
> > "`COLUMNS`" : `IDX`. Perhaps your MetaData is incorrect, or you havent
> > enabled "datanucleus.autoCreateColumns".
> > Required columns missing from table "`COLUMNS`" : `IDX`. Perhaps your
> > MetaData is incorrect, or you havent enabled
> > "datanucleus.autoCreateColumns".
> > org.datanucleus.store.rdbms.exceptions.MissingColumnException: Required
> > columns missing from table "`COLUMNS`" : `IDX`. Perhaps your MetaData is
> > incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >  at
> >
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> > at
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> > at
> >
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >  at
> >
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> > at
> >
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >  at
> > org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> > at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >  at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> > at
> >
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >  at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >  at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >  at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >  at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >  at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >
> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was thrown
> > while adding/validating class(es) : Required columns missing from table
> > "`SORT_COLS`" : `IDX`. Perhaps your MetaData is incorrect, or you havent
> > enabled "datanucleus.autoCreateColumns".
> > Required columns missing from table "`SORT_COLS`" : `IDX`. Perhaps your
> > MetaData is incorrect, or you havent enabled
> > "datanucleus.autoCreateColumns".
> > org.datanucleus.store.rdbms.exceptions.MissingColumnException: Required
> > columns missing from table "`SORT_COLS`" : `IDX`. Perhaps your MetaData
> is
> > incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >  at
> >
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> > at
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> > at
> >
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >  at
> >
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> > at
> >
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >  at
> > org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> > at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >  at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> > at
> >
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >  at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >  at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >  at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >  at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >  at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >
> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was thrown
> > while adding/validating class(es) : Expected primary key for table
> > `SORT_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not found in existing keys
> PRIMARY
> > KEY (`SD_ID`)
> > Expected primary key for table `SORT_COLS` PRIMARY KEY (`SD_ID`,`IDX`)
> not
> > found in existing keys PRIMARY KEY (`SD_ID`)
> > org.datanucleus.store.rdbms.exceptions.WrongPrimaryKeyException: Expected
> > primary key for table `SORT_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not found
> in
> > existing keys PRIMARY KEY (`SD_ID`)
> >  at
> >
> org.datanucleus.store.rdbms.table.TableImpl.validatePrimaryKey(TableImpl.java:368)
> > at
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:180)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> > at
> >
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >  at
> >
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> > at
> >
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >  at
> > org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> > at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >  at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> > at
> >
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >  at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >  at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >  at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >  at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >  at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >
> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was thrown
> > while adding/validating class(es) : Required columns missing from table
> > "`BUCKETING_COLS`" : `IDX`. Perhaps your MetaData is incorrect, or you
> > havent enabled "datanucleus.autoCreateColumns".
> > Required columns missing from table "`BUCKETING_COLS`" : `IDX`. Perhaps
> > your MetaData is incorrect, or you havent enabled
> > "datanucleus.autoCreateColumns".
> > org.datanucleus.store.rdbms.exceptions.MissingColumnException: Required
> > columns missing from table "`BUCKETING_COLS`" : `IDX`. Perhaps your
> MetaData
> > is incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >  at
> >
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> > at
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> > at
> >
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >  at
> >
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> > at
> >
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >  at
> > org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> > at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >  at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> > at
> >
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >  at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >  at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >  at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >  at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >  at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >
> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was thrown
> > while adding/validating class(es) : Expected primary key for table
> > `BUCKETING_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not found in existing keys
> > PRIMARY KEY (`SD_ID`)
> > Expected primary key for table `BUCKETING_COLS` PRIMARY KEY
> (`SD_ID`,`IDX`)
> > not found in existing keys PRIMARY KEY (`SD_ID`)
> > org.datanucleus.store.rdbms.exceptions.WrongPrimaryKeyException: Expected
> > primary key for table `BUCKETING_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not
> found
> > in existing keys PRIMARY KEY (`SD_ID`)
> >  at
> >
> org.datanucleus.store.rdbms.table.TableImpl.validatePrimaryKey(TableImpl.java:368)
> > at
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:180)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> > at
> >
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >  at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >  at
> >
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> > at
> >
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> > at
> >
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >  at
> > org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> > at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >  at
> >
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> > at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> > at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >  at
> >
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> > at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >  at
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> > at
> >
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >  at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >  at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >  at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >  at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >  at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >
> > I've debugged through the code and found that those missing columns were
> > added by datanucleus JDO code. I'm not sure what this means, rest of the
> > hive applications are working perfectly fine.
> >
> > Is there anything that I should try to figure out what's going on? or
> just
> > generally is it a right way to get a schema from hive table?
> >
> > J
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message