hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
Date Tue, 17 Mar 2015 10:04:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364857#comment-14364857
] 

Pavan Srinivas commented on HIVE-9543:
--------------------------------------

Hey [~brocknoland], as [~lijy83] pointed out, when ever there is a MysqlException, a duplicate
entry will be created. The problem here is with https://db.apache.org/jdo/api20/apidocs/javax/jdo/Query.html#execute%28%29
execute() function of Query.java.  "execute()" does not throw any exceptions. 

Some reproducible steps: 
1) Make sure metastore db is reachable&running. 
2) Enable debugger and stop at a breakpoint inside checkSchema().
3) Stop the metastore db or make it unreachable. 
4) Continue on the debugger, you would see execute call of getMSchemaVersion() does not throw
exceptions of DB reachability. It just returns an empty array as if there are no entries in
the table. Thus making the getMSchemaVersion() throw NoSuchObjectException. 


This issue happens when we have a flacky network connection to DB. 



> MetaException(message:Metastore contains multiple versions)
> -----------------------------------------------------------
>
>                 Key: HIVE-9543
>                 URL: https://issues.apache.org/jira/browse/HIVE-9543
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.13.1
>            Reporter: Junyong Li
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable
to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
>     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>     at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
>     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
>     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>     at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
>     at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
>     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
>     ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>     at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
>     ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
>     at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
>     at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
>     at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
>     at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
>     at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
>     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
>     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
>     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
>     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
>     at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
>     at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
>     at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
>     at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
>     ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source code, i
found following code maybe cause the problem: 
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
>     String schemaVer = getMetaStoreSchemaVersion();
>     if (schemaVer == null) {
>       // metastore has no schema version information
>       if (strictValidation) {
>             throw new MetaException("Version information not found in metastore. ");
>           } else {
>             LOG.warn("Version information not found in metastore. "
>                 + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
>                 " is not enabled so recording the schema version " +
>                 MetaStoreSchemaInfo.getHiveSchemaVersion());
>             setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
>                 "Set by MetaStore");
>         }
>     }
> {noformat}
> If there is exception in the {{getMetaStoreSchemaVersion()}}, Hive will insert a duplication
record
> And i also found my another exception maybe the same cause as above.the exception is
:
> {noformat}
>  Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable
to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
>         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
>         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>         at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
>         ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
>         ... 12 more
> Caused by: javax.jdo.JDODataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MRoleMap@2fd02622"
using statement "INSERT INTO `ROLE_MAP` (`ROLE_GRANT_ID`,`GRANTOR`,`A
> DD_TIME`,`ROLE_ID`,`GRANTOR_TYPE`,`GRANT_OPTION`,`PRINCIPAL_NAME`,`PRINCIPAL_TYPE`) VALUES
(?,?,?,?,?,?,?,?)" failed : Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
> NestedThrowables:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate
entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
>         at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>         at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
>         at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
>         at org.apache.hadoop.hive.metastore.ObjectStore.grantRole(ObjectStore.java:3058)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
>         at com.sun.proxy.$Proxy9.grantRole(Unknown Source)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addAdminUsers(HiveMetaStore.java:611)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
>         at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
>         ... 17 more
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
>         at com.mysql.jdbc.Util.getInstance(Util.java:381)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
>         at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
>         at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
>         at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
>         at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:410)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
>         at org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784)
>         at org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760)
>         at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219)
>         at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
>         at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
>         at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
>         at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
>         ... 32 more
> {noformat}
> The following code maybe cause the exception, in In the org.apache.hadoop.hive.metastore.ObjectStore.java:3018:
> {noformat}
>       MRole nameCheck = this.getMRole(roleName);
>       if (nameCheck != null) {
>         throw new InvalidObjectException("Role " + roleName + " already exists.");
>       }
>       int now = (int)(System.currentTimeMillis()/1000);
>       MRole mRole = new MRole(roleName, now,
>           ownerName);
>       pm.makePersistent(mRole);
>       commited = commitTransaction();
>       success = true;
> {noformat} 
> If there is exception in the {{getMRole(roleName)}} function, Hive will try to insert
a duplicate record into table ROLE_MAP, this will cause exception above.
> So can we just throw the exception instead of ignoring the exception in the {{getMRole(ObjectStore.java:3292)}}
and in the {{getMetaStoreSchemaVersion(ObjectStore.java:6326)}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message