db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Stuckman" <stuck...@umd.edu>
Subject Statement cache deadlock
Date Thu, 05 Mar 2009 08:17:17 GMT
Hello,

I have spent the past three days trying to find the cause of a seemingly
"impossible" deadlock in my Derby application. This was hard to debug
because the deadlock only occurs under heavy load. I think I have found the
cause -- due to a design flaw in the statement cache, a deadlock can occur
if a prepared statement becomes out-of-date.

I will illustrate this with the following example:

The application is using the embedded Derby driver. The application has two
threads, and each thread uses its own connection.

There is a table named MYTABLE with column MYCOLUMN.

1. A thread prepares and executes the query SELECT MYCOLUMN FROM MYTABLE.
The prepared statement is stored in the statement cache (see
org.apache.derby.impl.sql.GenericStatement for this logic)
2. After some time, the prepared statement becomes invalid or out-of-date
for some reason (see org.apache.derby.impl.sql.GenericPreparedStatement)
3. Thread 1 begins a transaction and executes LOCK TABLE MYTABLE IN
EXCLUSIVE MODE
4. Thread 2 begins a transaction and executes SELECT MYCOLUMN FROM MYTABLE.
The statement is in the statement cache but it is out-of-date. The thread
begins to recompile the statement. To compile the statement, the thread
needs a shared lock on MYTABLE. Thread 1 already has an exclusive lock on
MYTABLE. Thread 2 waits.
5. Thread 1 executes SELECT MYCOLUMN FROM MYTABLE. The statement is in the
statement cache but it is being compiled. Thread 1 waits on the statement's
monitor.
6. We have a deadlock. Derby eventually detects a lock timeout, but the
error message is not descriptive. The stacks at the time of the deadlock
are:

Thread 1:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
org.apache.derby.impl.sql.GenericPreparedStatement.rePrepare(Unknown Source)
org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown
Source)

Thread 2 (some frames omitted):
org.apache.derby.impl.services.locks.ActiveLock.waitForGrant(Unknown Source)
org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown
Source)
org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source)
.....
org.apache.derby.impl.sql.compile.ResultColumnList.generateHolderMethod(Unkn
own Source)
org.apache.derby.impl.sql.compile.FromBaseTable.getScanArguments(Unknown
Source)
.....
org.apache.derby.impl.sql.compile.IndexToBaseRowNode.generate(Unknown
Source)
org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(Unknown
Source)
....
org.apache.derby.impl.sql.compile.ScrollInsensitiveResultSetNode.generate(Un
known Source)
.....
org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
org.apache.derby.impl.sql.GenericPreparedStatement.rePrepare(Unknown Source)
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown
Source)

I discovered an undocumented property that allows me to disable the
statement cache. Setting derby.language.statementCacheSize=0 will disable
the statement cache and prevent the deadlock, because the
GenericPreparedStatement objects are no longer shared between sessions/JDBC
connections.

In my situation, this deadlock is impossible to prevent through careful
database design. You are only safe if all of your transactions are very
simple and cannot be interleaved in a sequence that causes the deadlock.
(For the sake of simplicity, I used LOCK TABLE in my example, but any UPDATE
statement would fit.)

I'm asking any knowledgeable parties to help me with the following:
1. Read my deadlock scenario and see if my analysis of the situation is
correct.
2. Help me write a test case, so I can post this as an issue. (I'm hoping
that we can fix this, so nobody else spends 3 days debugging this like I
just did.) I'm not sure how to invalidate the cache entry so the issue can
be reproduced.
3. If someone is familiar with the statement caching code, maybe we could
find possible solutions. Maybe we could wait until all the locks for
compilation have been acquired before "grabbing" the
GenericPreparedStatement object (compilingStatement=true). At the very
least, the statement complier could recognize the significance of the lock
timeout and suggest a solution in the exception text.
4. Document the property derby.language.statementCacheSize

Thanks,
Jeff



Mime
View raw message