hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8519) Hive metastore lock wait timeout
Date Tue, 06 Oct 2015 00:11:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944297#comment-14944297
] 

Sushanth Sowmyan commented on HIVE-8519:
----------------------------------------

I notice a similar issue when I try to drop a table with about 50000 partitions.

Essentially, what seems to be happening with that flow is the following:

a) Deleting a table requires deleting all partition objects for that table, Table->Partition
is a 1:many mapping
b) Deleting the partition objects requires deleting a all SD objects associated with the partitions,
Partition->SD is a 1:1 mapping
c) Deleting SD objects requires looking for all CDs pointed to by the SDs, and wherever a
CD has no more SDs pointing to it, we need to drop the CD in question, SD->CD is a many:1
mapping.
d) If a CD is to be deleted, we need to drop all List<MFieldSchema> associated with
it (COLUMNS_V2 where CD_ID in list of CDs to delete.)

The big inefficiency here is that SD->CD is a many:1 mapping with a goal of reusing CDs
for efficiency, but in practice, we don't. But the fact that it is many:1, not 1:1, means
we need to do that additional check before dropping rather than simply dropping. This combination
hits us in the worst way possible for both of those.

We need to rethink the way we use our objects and either drop the many:1 intent or actually
make sure that we create a unique CD for every SD, or this is not going to be scalable. Other
solutions that bypass this wonky model may also exist that we have to work out.

> Hive metastore lock wait timeout
> --------------------------------
>
>                 Key: HIVE-8519
>                 URL: https://issues.apache.org/jira/browse/HIVE-8519
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.10.0
>            Reporter: Liao, Xiaoge
>
> We got a lot of exception as below when doing a drop table partition, which made hive
query every every slow. For example, it will cost 250s while executing use db_test;
> Log:
> 2014-10-17 04:04:46,873 ERROR Datastore.Persist (Log4JLogger.java:error(115)) - Update
of object "org.apache.hadoop.hive.metastore.model.MStorageDescriptor@13c9c4b3" using statement
"UPDATE `SDS` SET `CD_ID`=? WHERE `SD_ID`=?" failed : java.sql.SQLException: Lock wait timeout
exceeded; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4096)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4028)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2734)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2458)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2375)
>         at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2359)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
>         at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
>         at org.datanucleus.store.rdbms.request.UpdateRequest.execute(UpdateRequest.java:374)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateTable(RDBMSPersistenceHandler.java:417)
>         at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.updateObject(RDBMSPersistenceHandler.java:390)
>         at org.datanucleus.state.JDOStateManager.flush(JDOStateManager.java:5012)
>         at org.datanucleus.FlushOrdered.execute(FlushOrdered.java:106)
>         at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4019)
>         at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450)
>         at org.datanucleus.store.query.Query.prepareDatastore(Query.java:1575)
>         at org.datanucleus.store.query.Query.executeQuery(Query.java:1760)
>         at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
>         at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243)
>         at org.apache.hadoop.hive.metastore.ObjectStore.listStorageDescriptorsWithCD(ObjectStore.java:2185)
>         at org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:2131)
>         at org.apache.hadoop.hive.metastore.ObjectStore.preDropStorageDescriptor(ObjectStore.java:2162)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionCommon(ObjectStore.java:1361)
>         at org.apache.hadoop.hive.metastore.ObjectStore.dropPartition(ObjectStore.java:1301)
>         at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
>         at $Proxy4.dropPartition(Unknown Source)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition_common(HiveMetaStore.java:1865)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partition(HiveMetaStore.java:1911)
>         at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>         at $Proxy5.drop_partition(Unknown Source)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message