hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Steinbach <c...@cloudera.com>
Subject Re: how to load hive table schema programatically?
Date Mon, 14 Mar 2011 19:01:05 GMT
Hi Ed,

I'm pretty sure HiveMetaStoreClient is intended to be a public API.


On Mon, Mar 14, 2011 at 11:49 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

> On Mon, Mar 14, 2011 at 2:44 PM, Jae Lee <jlee119@gmail.com> wrote:
> > Ah... thanks alot... that worked :)
> >
> > Is there any other recommended way to load hive table meta data? I
> suppose
> > accessing meta-store via HiveMetaStoreClient make it possible to change
> > underlying data storage implementation choice.
> >
> > J
> >
> > On Mon, Mar 14, 2011 at 5:56 PM, Carl Steinbach <carl@cloudera.com>
> wrote:
> >
> >> Hi Jae,
> >>
> >> Sounds like your problem is related to HIVE-1435 (
> >> https://issues.apache.org/jira/browse/HIVE-1435). You need to make sure
> >> that the Datanucleus ORM layer is getting initialized with the
> configuration
> >> property datanucleus.identifierFactory=datanucleus. Probably the easiest
> way
> >> to fix this problem is to make sure that the 0.7.0 version of
> >> hive-default.xml is available on the CLASSPATH and is getting loaded
> into
> >> HiveConf. Try dumping the contents of your HiveConf object and make sure
> >> that the values match those that appear in the 0.7.0 version of
> >> hive-default.xml
> >>
> >> Hope this helps.
> >>
> >> Carl
> >>
> >>
> >> On Mon, Mar 14, 2011 at 10:41 AM, Jae Lee <jlee119@gmail.com> wrote:
> >>
> >>> just a bit more information from my debugging so far
> >>>
> >>> my mysql hive metastore have columns like
> >>> "integer_idx" at "columns" table
> >>> "integer_idx" at "sort_cols" table
> >>>
> >>> those columns looks pretty suspicious in that it is similar to "idx"
> >>> columns
> >>> that HiveMetaSotreClient complains missing.
> >>>
> >>> It looks like expectation of having "idx" column is auto-generated (not
> >>> from
> >>> package.jdo document)
> >>> Can anybody tell me whether "integer_idx" column should have been "idx"
> >>> column at "columns" table?
> >>> or am I suppose to have custom package.jdo file that specify the index
> >>> column name to "integer_idx" instead of "idx" column?
> >>>
> >>> J
> >>>
> >>> On Mon, Mar 14, 2011 at 2:27 PM, Jae Lee <jlee119@gmail.com> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I've had this code below working with Hive 0.5
> >>> >
> >>> > String databaseName = "default";
> >>> > String tableName = "foobar";
> >>> > List<org.apache.hadoop.hive.metastore.api.FieldSchema> hiveTable
=
> new
> >>> > HiveMetaStoreClient(new HiveConf(new Configuration(),
> >>> > SessionState.class)).getSchema(databaseName, tableName);
> >>> >
> >>> >
> >>> > to produce list of FieldSchema for a table foobar in default database
> >>> >
> >>> > I've recently upgraded hive to 0.7, and the same code now generates
> an
> >>> > error messages such as
> >>> >
> >>> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was
> thrown
> >>> > while adding/validating class(es) : Required columns missing from
> table
> >>> > "`COLUMNS`" : `IDX`. Perhaps your MetaData is incorrect, or you
> havent
> >>> > enabled "datanucleus.autoCreateColumns".
> >>> > Required columns missing from table "`COLUMNS`" : `IDX`. Perhaps your
> >>> > MetaData is incorrect, or you havent enabled
> >>> > "datanucleus.autoCreateColumns".
> >>> > org.datanucleus.store.rdbms.exceptions.MissingColumnException:
> Required
> >>> > columns missing from table "`COLUMNS`" : `IDX`. Perhaps your MetaData
> is
> >>> > incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> >>> > at
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >>> >  at
> >>> >
> org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> >>> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> >>> > at
> org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >>> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> >>> > at
> >>> >
> >>>
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> >  at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> >>> > at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >>> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> >>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >>> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >>> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >>> > at
> >>> >
> >>>
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >>> >  at
> >>> >
> >>>
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> >>> > at
> >>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >>> >
> >>> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was
> thrown
> >>> > while adding/validating class(es) : Required columns missing from
> table
> >>> > "`SORT_COLS`" : `IDX`. Perhaps your MetaData is incorrect, or you
> havent
> >>> > enabled "datanucleus.autoCreateColumns".
> >>> > Required columns missing from table "`SORT_COLS`" : `IDX`. Perhaps
> your
> >>> > MetaData is incorrect, or you havent enabled
> >>> > "datanucleus.autoCreateColumns".
> >>> > org.datanucleus.store.rdbms.exceptions.MissingColumnException:
> Required
> >>> > columns missing from table "`SORT_COLS`" : `IDX`. Perhaps your
> MetaData
> >>> is
> >>> > incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> >>> > at
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >>> >  at
> >>> >
> org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> >>> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> >>> > at
> org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >>> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> >>> > at
> >>> >
> >>>
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> >  at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> >>> > at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >>> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> >>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >>> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >>> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >>> > at
> >>> >
> >>>
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >>> >  at
> >>> >
> >>>
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> >>> > at
> >>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >>> >
> >>> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was
> thrown
> >>> > while adding/validating class(es) : Expected primary key for table
> >>> > `SORT_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not found in existing keys
> >>> PRIMARY
> >>> > KEY (`SD_ID`)
> >>> > Expected primary key for table `SORT_COLS` PRIMARY KEY
> (`SD_ID`,`IDX`)
> >>> not
> >>> > found in existing keys PRIMARY KEY (`SD_ID`)
> >>> > org.datanucleus.store.rdbms.exceptions.WrongPrimaryKeyException:
> >>> Expected
> >>> > primary key for table `SORT_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not
> found
> >>> in
> >>> > existing keys PRIMARY KEY (`SD_ID`)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validatePrimaryKey(TableImpl.java:368)
> >>> > at
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:180)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >>> >  at
> >>> >
> org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> >>> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> >>> > at
> org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >>> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> >>> > at
> >>> >
> >>>
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> >  at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> >>> > at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >>> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> >>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >>> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >>> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >>> > at
> >>> >
> >>>
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >>> >  at
> >>> >
> >>>
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> >>> > at
> >>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >>> >
> >>> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was
> thrown
> >>> > while adding/validating class(es) : Required columns missing from
> table
> >>> > "`BUCKETING_COLS`" : `IDX`. Perhaps your MetaData is incorrect, or
> you
> >>> > havent enabled "datanucleus.autoCreateColumns".
> >>> > Required columns missing from table "`BUCKETING_COLS`" : `IDX`.
> Perhaps
> >>> > your MetaData is incorrect, or you havent enabled
> >>> > "datanucleus.autoCreateColumns".
> >>> > org.datanucleus.store.rdbms.exceptions.MissingColumnException:
> Required
> >>> > columns missing from table "`BUCKETING_COLS`" : `IDX`. Perhaps your
> >>> MetaData
> >>> > is incorrect, or you havent enabled "datanucleus.autoCreateColumns".
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validateColumns(TableImpl.java:282)
> >>> > at
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:175)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >>> >  at
> >>> >
> org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> >>> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> >>> > at
> org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >>> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> >>> > at
> >>> >
> >>>
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> >  at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> >>> > at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >>> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> >>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >>> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >>> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >>> > at
> >>> >
> >>>
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >>> >  at
> >>> >
> >>>
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> >>> > at
> >>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >>> >
> >>> > 11/03/14 14:22:02 ERROR DataNucleus.Datastore: An exception was
> thrown
> >>> > while adding/validating class(es) : Expected primary key for table
> >>> > `BUCKETING_COLS` PRIMARY KEY (`SD_ID`,`IDX`) not found in existing
> keys
> >>> > PRIMARY KEY (`SD_ID`)
> >>> > Expected primary key for table `BUCKETING_COLS` PRIMARY KEY
> >>> (`SD_ID`,`IDX`)
> >>> > not found in existing keys PRIMARY KEY (`SD_ID`)
> >>> > org.datanucleus.store.rdbms.exceptions.WrongPrimaryKeyException:
> >>> Expected
> >>> > primary key for table `BUCKETING_COLS` PRIMARY KEY (`SD_ID`,`IDX`)
> not
> >>> found
> >>> > in existing keys PRIMARY KEY (`SD_ID`)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validatePrimaryKey(TableImpl.java:368)
> >>> > at
> >>>
> org.datanucleus.store.rdbms.table.TableImpl.validate(TableImpl.java:180)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:2711)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:2503)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2148)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:113)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:986)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:952)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:919)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:356)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.ExtentHelper.getExtent(ExtentHelper.java:48)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.RDBMSStoreManager.getExtent(RDBMSStoreManager.java:1332)
> >>> >  at
> >>> >
> org.datanucleus.ObjectManagerImpl.getExtent(ObjectManagerImpl.java:4149)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compileCandidates(JDOQLQueryCompiler.java:411)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:312)
> >>> > at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
> >>> > at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
> >>> >  at
> >>> >
> >>>
> org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
> >>> > at
> org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
> >>> >  at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:266)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:775)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:709)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1076)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$17.run(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:307)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1073)
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_schema(HiveMetaStore.java:1785)
> >>> >  at
> >>> >
> >>>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getSchema(HiveMetaStoreClient.java:857)
> >>> > at
> >>> >
> >>>
> HiveMetaStoreClientTest.shouldGetSchemaFromMetaStore(HiveMetaStoreClientTest.java:10)
> >>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> >  at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >  at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
> >>> > at
> >>> >
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
> >>> >  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> >>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> >>> >  at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> >>> > at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >  at
> >>> >
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> >>> >  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> >>> > at
> >>> >
> >>>
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:94)
> >>> >  at
> >>> >
> >>>
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196)
> >>> > at
> >>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
> >>> >
> >>> > I've debugged through the code and found that those missing columns
> were
> >>> > added by datanucleus JDO code. I'm not sure what this means, rest of
> the
> >>> > hive applications are working perfectly fine.
> >>> >
> >>> > Is there anything that I should try to figure out what's going on?
or
> >>> just
> >>> > generally is it a right way to get a schema from hive table?
> >>> >
> >>> > J
> >>> >
> >>>
> >>
> >>
> >
>
> Correct you should not interface with the Metastore this way because
> it is not stable API you are working with.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message