hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Kuzmenko <f1she...@gmail.com>
Subject Re: Hive TxnHandler::lock method run into dead lock.
Date Mon, 20 Mar 2017 21:17:05 GMT
Sorry miss clicked.

2) TxnHandler::lock method request new connection when executing this line
of code:
ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);

3) After that folowing this stacktrace:
- TxnHandler::lock
- TxnHandler::checkLockWithRetry
- TxnHandler::checkLock

In checkLock method we reach this line:
handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());

4) acquireLock method requests another connection to DB:
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);

So all in all if i call TxnHandler::lock method in 10 threads same time at
first I get all connections to db, that stored in pool, and at acquireLock
I will stuck because there's no free connection.

Does anyone run into this problem? How can I avoid this problem?

Code was taken from here:
https://github.com/hortonworks/hive-release/blob/HDP-2.5.0.0-tag/metastore/
src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java

I guess the closest branch in apach repo is:
https://github.com/apache/hive/blob/branch-2.1/metastore/src/java/org/apache
/hadoop/hive/metastore/txn/TxnHandler.java

On Tue, Mar 21, 2017 at 12:07 AM, Igor Kuzmenko <f1sherox@gmail.com> wrote:

> Hello I'm running Hortonworks data platform 2.5.0.0 with included hive.
> I'm using storm hive bolt to load data into my hive. But launching many
> hive bolt always leads me to TimeoutException on calling hive metastore.
> Metastore logs full of Exception like this:
>
> 2017-03-15 18:46:12,436 ERROR [pool-5-thread-11]: txn.TxnHandler
> (TxnHandler.java:getDbConn(1834)) - There is a problem with a connection
> from the pool, retrying(rc=7): Timed out waiting for a free available
> connection. (SQLState=08001, ErrorCode=0)
> java.sql.SQLException: Timed out waiting for a free available connection.
> at com.jolbox.bonecp.DefaultConnectionStrategy.getConnectionInternal(
> DefaultConnectionStrategy.java:88)
> at com.jolbox.bonecp.AbstractConnectionStrategy.getConnection(
> AbstractConnectionStrategy.java:90)
> at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:553)
> at com.jolbox.bonecp.BoneCPDataSource.getConnection(
> BoneCPDataSource.java:131)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.
> getDbConn(TxnHandler.java:1827)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(
> TxnHandler.java:873)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(
> TxnHandler.java:814)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$
> HMSHandler.lock(HiveMetaStore.java:5751)
>
> After looking through code I found out
>
> 1) TxnHandler  class uses connection pool to get db connections and it's
> size is 10.
> 2) TxnHandler::lock method requset new connection whe executing this line
> of code:
>
>

Mime
View raw message