db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject Help with a lock dump relating to: RE: Have an issue in a production system and need some help with a (logical) deadlock
Date Thu, 14 Mar 2013 20:10:45 GMT
Still working on this problem and it occurred a few more times since yesterday.   In trying to debug this a bit to find a trigger I noticed that there is a lock timeout.   Part of the lock table dump looks like:

XID       |TYPE         |MODE|LOCKCOUNT|LOCKNAME                                                                        |STATE|TABLETYPE / LOCKOBJ                   |INDEXNAME / CONTAINER_ID / (MODE for LATCH only)  |TABLENAME / CONGLOM_ID                |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*** The following row is the victim ***
1949098515|TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |ALARMEVENT_ALARMEVENTPROPERTY         |
*** The above row is the victim ***
null      |ROW          |X   |0        |(9211297,6)                                                                     |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
1948989697|ROW          |S   |0        |(327381,12)                                                                     |WAIT |T                                     |NULL                                              |CONFIGURATION_BUNDLE_9145E            |
1949103169|ShExLockable |S   |0        |org.apache.derby.iapi.services.locks.ShExLockable@1c20eb7                       |WAIT |S                                     |NULL                                              |*** INTERNAL TABLE -1                 |
1948989697|ROW          |S   |0        |(113966,44)                                                                     |WAIT |T                                     |NULL                                              |CONFIGURATION_BUNDLE                  |
1948989697|TABLE        |IS  |2        |Tablelock                                                                       |GRANT|T                                     |NULL                                              |COSEDDROPPROFILEDSCPTABLEBUNDLE_COSEDD|
1949103169|TABLE        |IS  |0        |Tablelock                                                                       |WAIT |S                                     |NULL                                              |SYSSCHEMAS                            |
1949103197|TABLE        |IS  |0        |Tablelock                                                                       |WAIT |S                                     |NULL                                              |SYSSCHEMAS                            |
null      |ROW          |S   |0        |(7,90)                                                                          |WAIT |T                                     |NULL                                              |NPARESULTSCIRCUIT                     |
1949103169|TABLE        |IS  |0        |Tablelock                                                                       |WAIT |S                                     |NULL                                              |SYSALIASES                            |
null      |TABLE        |IS  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTSPROFILE                     |
1948989697|TABLE        |IX  |4        |Tablelock                                                                       |GRANT|T                                     |NULL                                              |ALARM_EVENT                           |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IX  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTS_WEEK_10                    |
null      |TABLE        |IS  |0        |Tablelock                                                                       |WAIT |T                                     |NULL                                              |NPARESULTSCIRCUIT                     |
1948989697|TABLE        |IX  |6        |Tablelock                                                                       |GRANT|T                                     |NULL                                              |ALARMEVENT_ALARMACTIONDETAIL          |
1948989697|TABLE        |X   |1        |Tablelock                                                                       |GRANT|T                                     |NULL                                              |ALARMEVENT_ALARMACTIONDETAIL          |
1948989697|ROW          |S   |0        |(31106,6)                                                                       |WAIT |T                                     |NULL                                              |COS_ED_DROP_PROFILE_DSCP_CONFIG_BUNDLE|
1948989697|TABLE        |IS  |1        |Tablelock                                                                       |GRANT|T                                     |NULL                                              |CONFIGURATION_BUNDLE                  |
null      |ROW          |S   |0        |(5920,9)                                                                        |WAIT |T                                     |NULL                                              |NPARESULTSADDRESS                     |
null      |ROW          |S   |0        |(3,69)                                                                          |WAIT |T                                     |NULL                                              |NPARESULTSCIRCUIT                     |

Any idea why some of the rows in this dump have "null" for the XID?

Also, it appears that there is some sort of transaction that is exceeding the configured "derby.jdbc.xaTransactionTimeout" which is set to "1800".    The last time this problem occurred, I see that Derby determined that this had been exceeded and tried to cleanup from this and that is when all hell broke loose.

Below is some of the messages in the "derby.log".   It almost seems that somehow these transactions really don't exist or something similar as Derby is getting a NullPointerException trying to cancel some.   Note that there is a possibility that the application server has determined that a transaction is taking too long and canceled it itself.   This time is set to "600" seconds which is definitely less than what the "derby.jdbc.xaTransactionTimeout".    I wonder if this were to happen that the XA transaction is being canceled but somehow Derby does not know about it?

The XA transaction timed out and is going to be rolled back. The transaction Xid is (4871251,e37c140026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,fdad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,7e99160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f4ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,fbad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f3ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f6ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f8ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f7ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,fcad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f2ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f9ad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,ffad160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,01ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,00ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,02ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,04ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,03ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,06ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,08ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,07ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0aae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,09ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0bae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0dae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0cae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0fae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,0eae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,14ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,13ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,12ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,11ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,10ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,15ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,17ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,18ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,19ae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,1aae160026049c68636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
Exception in thread "DRDAConnThread_15" java.lang.NullPointerException
      at org.apache.derby.impl.jdbc.EmbedConnection.cancelRunningStatement(Unknown Source)
      at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
      at org.apache.derby.jdbc.ResourceAdapterImpl.cancelXATransaction(Unknown Source)
      at org.apache.derby.impl.drda.DRDAXAProtocol.rollbackCurrentTransaction(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)


From: Bergquist, Brett [mailto:BBergquist@canoga.com]
Sent: Wednesday, March 13, 2013 3:41 PM
To: derby-dev@db.apache.org
Subject: Have an issue in a production system and need some help with a (logical) deadlock

We have a product system that has an issue.   The database is used as part of a network management system.   An event in the network triggers an update on a bunch of records in the database and then a cascade of errors occurs.

Some background:

-          Application Server is Glassfish V2.1.1

-          Derby 10.8.2.1 running in the network server mode

-          XA Transactions being used

-          The system is pretty busy processing  about 70 transactions/second almost continuous 24x7

Here is a snippet of the derby.log when the problem occurs:

---vvv  part of derby.log vvv---
Wed Mar 13 17:13:59 GMT 2013 Thread[DRDAConnThread_19,5,main] (XID = 19447377525), (SESSIONID = 6271961), (DATABASE = csemdb), (DRDAID = ????????.??-44129633
21674897252{16075}), Cleanup action starting
Wed Mar 13 17:13:59 GMT 2013 Thread[DRDAConnThread_19,5,main] (XID = 19447377525), (SESSIONID = 6271961), (DATABASE = csemdb), (DRDAID = ????????.??-44129633
21674897252{16075}), Failed Statement is: null
java.lang.NullPointerException
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.getMetaData(Unknown Source)
        at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.getMetaData(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.writeSQLDARD(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
Cleanup action completed
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,88aa2f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
Exception in thread "DRDAConnThread_19" java.lang.NullPointerException
        at org.apache.derby.impl.jdbc.EmbedConnection.cancelRunningStatement(Unknown Source)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
        at org.apache.derby.jdbc.ResourceAdapterImpl.cancelXATransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAXAProtocol.rollbackCurrentTransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,52a12f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f6a12f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,f5ad2f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,4db02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,4bb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,4cb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,46b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,49b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,4fb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,51b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,4eb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
Exception in thread "DRDAConnThread_38" java.lang.NullPointerException
        at org.apache.derby.impl.jdbc.EmbedConnection.cancelRunningStatement(Unknown Source)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
        at org.apache.derby.jdbc.ResourceAdapterImpl.cancelXATransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAXAProtocol.rollbackCurrentTransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,6ab02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,6cb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
Exception in thread "DRDAConnThread_75" java.lang.NullPointerException
        at org.apache.derby.impl.jdbc.EmbedConnection.cancelRunningStatement(Unknown Source)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)There was an XA transaction associated with the connection being closed. The transa
ction is going to be rolled back. The transaction Xid is (4871251,6db02f0069100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c73
65727665722c50333730302c00).

        at org.apache.derby.jdbc.ResourceAdapterImpl.cancelXATransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAXAProtocol.rollbackCurrentTransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
Exception in thread "DRDAConnThread_72" java.lang.NullPointerException
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,6bb02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
        at org.apache.derby.impl.jdbc.EmbedConnection.cancelRunningStatement(Unknown Source)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
        at org.apache.derby.jdbc.ResourceAdapterImpl.cancelXATransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAXAProtocol.rollbackCurrentTransaction(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.closeSession(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,71b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,72b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
There was an XA transaction associated with the connection being closed. The transaction is going to be rolled back. The transaction Xid is (4871251,73b02f00
69100763636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
---^^^  part of derby.log ^^^---

As you can see, something pretty drastic happened but with the NullPointerExceptions, it is hard to tell what.  Also there appears to be problems in cancelling the transactions as well also with NullPointerExceptions.

Here is the stack trace of derby after this event:

---- vvv derby stack trace vvv ---
Derby stacktrace at Wed Mar 13 17:52:56 GMT 2013
2013-03-13 17:52:57
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

"DRDAConnThread_81" prio=3 tid=0x0000000101d7d800 nid=0x77 runnable [0xfffffffd2e0fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"Attach Listener" daemon prio=3 tid=0x000000010594a800 nid=0x74 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DRDAConnThread_80" prio=3 tid=0x0000000101d0e000 nid=0x73 in Object.wait() [0xfffffffd266ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_79" prio=3 tid=0x000000010270d800 nid=0x72 runnable [0xfffffffd268ff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_78" prio=3 tid=0x0000000101dac000 nid=0x71 in Object.wait() [0xfffffffd26aff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_77" prio=3 tid=0x000000010502e800 nid=0x70 runnable [0xfffffffd26cff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_76" prio=3 tid=0x00000001022ed800 nid=0x6f in Object.wait() [0xfffffffd26eff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_74" prio=3 tid=0x00000001020c0000 nid=0x6d in Object.wait() [0xfffffffd272ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_73" prio=3 tid=0x0000000103668000 nid=0x6c runnable [0xfffffffd274fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_71" prio=3 tid=0x000000010404f000 nid=0x6a in Object.wait() [0xfffffffd278ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_70" prio=3 tid=0x00000001029b5800 nid=0x69 in Object.wait() [0xfffffffd27aff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_69" prio=3 tid=0x0000000104317800 nid=0x68 in Object.wait() [0xfffffffd27cff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_68" prio=3 tid=0x00000001024dd000 nid=0x67 in Object.wait() [0xfffffffd27eff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_67" prio=3 tid=0x00000001034ad000 nid=0x66 in Object.wait() [0xfffffffd280ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_66" prio=3 tid=0x0000000103733000 nid=0x65 runnable [0xfffffffd282ff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_65" prio=3 tid=0x0000000102af4000 nid=0x64 runnable [0xfffffffd284fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_64" prio=3 tid=0x00000001024db000 nid=0x63 in Object.wait() [0xfffffffd286ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_63" prio=3 tid=0x0000000102ad9000 nid=0x62 in Object.wait() [0xfffffffd288ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_62" prio=3 tid=0x0000000104c70000 nid=0x61 in Object.wait() [0xfffffffd28aff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_61" prio=3 tid=0x0000000103996800 nid=0x60 runnable [0xfffffffd28cfe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.getExtData(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readBytes(Unknown Source)
      at org.apache.derby.impl.drda.DRDAXAProtocol.parseXID(Unknown Source)
      at org.apache.derby.impl.drda.DRDAXAProtocol.parseSYNCCTL(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_60" prio=3 tid=0x0000000104b89000 nid=0x5f in Object.wait() [0xfffffffd28eff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_58" prio=3 tid=0x000000010269f000 nid=0x5d in Object.wait() [0xfffffffd292ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_57" prio=3 tid=0x0000000102258000 nid=0x5c in Object.wait() [0xfffffffd294ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_56" prio=3 tid=0x000000010259d000 nid=0x5b in Object.wait() [0xfffffffd296ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_55" prio=3 tid=0x0000000103f8c000 nid=0x5a runnable [0xfffffffd298ff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_54" prio=3 tid=0x0000000104c73000 nid=0x59 in Object.wait() [0xfffffffd29aff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_53" prio=3 tid=0x0000000103f8b000 nid=0x58 runnable [0xfffffffd29cfe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_52" prio=3 tid=0x0000000102030800 nid=0x57 in Object.wait() [0xfffffffd29eff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_51" prio=3 tid=0x0000000102f88000 nid=0x56 in Object.wait() [0xfffffffd2a0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_50" prio=3 tid=0x0000000102f87000 nid=0x55 in Object.wait() [0xfffffffd2a2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_49" prio=3 tid=0x0000000102e3e000 nid=0x54 in Object.wait() [0xfffffffd2a4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_48" prio=3 tid=0x0000000102372000 nid=0x53 in Object.wait() [0xfffffffd2a6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_47" prio=3 tid=0x00000001020cb000 nid=0x52 in Object.wait() [0xfffffffd2a8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_46" prio=3 tid=0x0000000101ac9800 nid=0x51 in Object.wait() [0xfffffffd2aaff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_45" prio=3 tid=0x0000000102141800 nid=0x50 in Object.wait() [0xfffffffd2acff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_44" prio=3 tid=0x00000001025e4000 nid=0x4f in Object.wait() [0xfffffffd2aeff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_43" prio=3 tid=0x0000000101aa9800 nid=0x4e in Object.wait() [0xfffffffd2b0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_42" prio=3 tid=0x000000010256b800 nid=0x4d in Object.wait() [0xfffffffd2b2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_41" prio=3 tid=0x0000000102105000 nid=0x4c runnable [0xfffffffd2b4fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_40" prio=3 tid=0x0000000101c7a000 nid=0x4b runnable [0xfffffffd2b6fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_39" prio=3 tid=0x000000010268d800 nid=0x4a in Object.wait() [0xfffffffd2b8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_37" prio=3 tid=0x0000000102097800 nid=0x48 in Object.wait() [0xfffffffd2bcff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_36" prio=3 tid=0x0000000101c8f000 nid=0x47 runnable [0xfffffffd2beff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_35" prio=3 tid=0x0000000102499800 nid=0x46 in Object.wait() [0xfffffffd2c0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_34" prio=3 tid=0x00000001022e9000 nid=0x45 in Object.wait() [0xfffffffd2c2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_33" prio=3 tid=0x0000000101f56000 nid=0x44 in Object.wait() [0xfffffffd2c4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_32" prio=3 tid=0x00000001027bd000 nid=0x43 in Object.wait() [0xfffffffd2c6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_31" prio=3 tid=0x000000010212a800 nid=0x42 in Object.wait() [0xfffffffd2c8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_30" prio=3 tid=0x00000001023b0000 nid=0x41 runnable [0xfffffffd2caff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_29" prio=3 tid=0x0000000102259800 nid=0x40 in Object.wait() [0xfffffffd2ccff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_28" prio=3 tid=0x00000001020f3800 nid=0x3f in Object.wait() [0xfffffffd2ceff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_27" prio=3 tid=0x00000001023c3800 nid=0x3e in Object.wait() [0xfffffffd2d0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_26" prio=3 tid=0x0000000102500000 nid=0x3d in Object.wait() [0xfffffffd2d2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_25" prio=3 tid=0x00000001024fe000 nid=0x3c in Object.wait() [0xfffffffd2d4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_24" prio=3 tid=0x00000001024fc000 nid=0x3b runnable [0xfffffffd2d6fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_23" prio=3 tid=0x0000000101c41800 nid=0x3a in Object.wait() [0xfffffffd2d8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_22" prio=3 tid=0x0000000101c40800 nid=0x39 in Object.wait() [0xfffffffd2daff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_21" prio=3 tid=0x0000000101c44800 nid=0x38 runnable [0xfffffffd2dcff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_20" prio=3 tid=0x00000001021cd800 nid=0x37 in Object.wait() [0xfffffffd2deff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_18" prio=3 tid=0x0000000101ab4000 nid=0x35 runnable [0xfffffffd2e2fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_17" prio=3 tid=0x0000000101ab3000 nid=0x34 runnable [0xfffffffd2e4fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_16" prio=3 tid=0x0000000101bdb800 nid=0x33 runnable [0xfffffffd2e6fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_15" prio=3 tid=0x0000000101c08000 nid=0x32 runnable [0xfffffffd2e8ff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_14" prio=3 tid=0x0000000101e44000 nid=0x31 runnable [0xfffffffd2eaff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_13" prio=3 tid=0x00000001022c1000 nid=0x30 runnable [0xfffffffd2ecff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_12" prio=3 tid=0x00000001021e9800 nid=0x2f runnable [0xfffffffd2eeff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"derby.rawStoreDaemon" daemon prio=3 tid=0x0000000101e75800 nid=0x2e in Object.wait() [0xfffffffd2f0ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
      - locked <0xfffffffd75a57618> (a org.apache.derby.impl.services.daemon.BasicDaemon)
      at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:662)

"DRDAConnThread_11" prio=3 tid=0x0000000101ee1800 nid=0x2d runnable [0xfffffffd2f2ff000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_10" prio=3 tid=0x0000000102588800 nid=0x2c in Object.wait() [0xfffffffd2f4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_9" prio=3 tid=0x0000000101cf4000 nid=0x2b in Object.wait() [0xfffffffd2f6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_8" prio=3 tid=0x0000000101eac000 nid=0x2a in Object.wait() [0xfffffffd2f8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_7" prio=3 tid=0x0000000101eee800 nid=0x29 in Object.wait() [0xfffffffd2faff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_6" prio=3 tid=0x0000000101abd000 nid=0x28 in Object.wait() [0xfffffffd2fcff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_5" prio=3 tid=0x000000010011c800 nid=0x27 in Object.wait() [0xfffffffd2feff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_4" prio=3 tid=0x000000010230c000 nid=0x26 runnable [0xffffffff706fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
      at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"derby.rawStoreDaemon" daemon prio=3 tid=0x00000001025ac800 nid=0x25 in Object.wait() [0xffffffff70eff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
      - locked <0xfffffffd756aea68> (a org.apache.derby.impl.services.daemon.BasicDaemon)
      at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:662)

"DRDAConnThread_3" prio=3 tid=0x0000000101c2d800 nid=0x24 in Object.wait() [0xffffffff711ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
      - locked <0xfffffffd7704cb40> (a java.util.Vector)
      at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"NetworkServerThread_2" prio=3 tid=0x000000010258c000 nid=0x23 runnable [0xffffffff713fe000]
   java.lang.Thread.State: RUNNABLE
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
      - locked <0xfffffffd760158e8> (a java.net.SocksSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:453)
      at java.net.ServerSocket.accept(ServerSocket.java:421)
      at org.apache.derby.impl.drda.ClientThread$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.apache.derby.impl.drda.ClientThread.run(Unknown Source)

"Timer-0" daemon prio=3 tid=0x0000000101ff5000 nid=0x22 in Object.wait() [0xffffffff717ff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at java.util.TimerThread.mainLoop(Timer.java:483)
      - locked <0xfffffffd7704d378> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=3 tid=0x0000000101a26000 nid=0x20 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=3 tid=0x0000000101a20000 nid=0x1f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=3 tid=0x0000000101a1d800 nid=0x1e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=3 tid=0x0000000101a1b800 nid=0x1d runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=3 tid=0x00000001019f4800 nid=0x1c in Object.wait() [0xffffffff72dff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      - locked <0xfffffffd76a8f888> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=3 tid=0x00000001019f2800 nid=0x1b in Object.wait() [0xffffffff72fff000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:485)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      - locked <0xfffffffd76c75d90> (a java.lang.ref.Reference$Lock)

"main" prio=3 tid=0x0000000100117000 nid=0x2 in Object.wait() [0xffffffff7c1fd000]
   java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0xfffffffd760158a0> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:485)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.blockingStart(Unknown Source)
      - locked <0xfffffffd760158a0> (a java.lang.Object)
      at org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(Unknown Source)
      at org.apache.derby.drda.NetworkServerControl.main(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.sun.enterprise.cli.commands.DerbyControl.invokeNetworkServerControl(DerbyControl.java:132)
      at com.sun.enterprise.cli.commands.DerbyControl.main(DerbyControl.java:215)

"VM Thread" prio=3 tid=0x00000001019ee800 nid=0x1a runnable

"GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100127000 nid=0x3 runnable

"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100128800 nid=0x4 runnable

"GC task thread#2 (ParallelGC)" prio=3 tid=0x000000010012a800 nid=0x5 runnable

"GC task thread#3 (ParallelGC)" prio=3 tid=0x000000010012c000 nid=0x6 runnable

"GC task thread#4 (ParallelGC)" prio=3 tid=0x000000010012d800 nid=0x7 runnable

"GC task thread#5 (ParallelGC)" prio=3 tid=0x000000010012f000 nid=0x8 runnable

"GC task thread#6 (ParallelGC)" prio=3 tid=0x0000000100131000 nid=0x9 runnable

"GC task thread#7 (ParallelGC)" prio=3 tid=0x0000000100132800 nid=0xa runnable

"GC task thread#8 (ParallelGC)" prio=3 tid=0x0000000100134000 nid=0xb runnable

"GC task thread#9 (ParallelGC)" prio=3 tid=0x0000000100135800 nid=0xc runnable

"GC task thread#10 (ParallelGC)" prio=3 tid=0x0000000100137800 nid=0xd runnable

"GC task thread#11 (ParallelGC)" prio=3 tid=0x000000010013b000 nid=0xe runnable

"GC task thread#12 (ParallelGC)" prio=3 tid=0x000000010013c800 nid=0xf runnable

"GC task thread#13 (ParallelGC)" prio=3 tid=0x000000010013e000 nid=0x10 runnable

"GC task thread#14 (ParallelGC)" prio=3 tid=0x0000000100140000 nid=0x11 runnable

"GC task thread#15 (ParallelGC)" prio=3 tid=0x0000000100141800 nid=0x12 runnable

"GC task thread#16 (ParallelGC)" prio=3 tid=0x0000000100143000 nid=0x13 runnable

"GC task thread#17 (ParallelGC)" prio=3 tid=0x0000000100145000 nid=0x14 runnable

"GC task thread#18 (ParallelGC)" prio=3 tid=0x0000000100146800 nid=0x15 runnable

"GC task thread#19 (ParallelGC)" prio=3 tid=0x0000000100148000 nid=0x16 runnable

"GC task thread#20 (ParallelGC)" prio=3 tid=0x0000000100149800 nid=0x17 runnable

"GC task thread#21 (ParallelGC)" prio=3 tid=0x000000010014b800 nid=0x18 runnable

"GC task thread#22 (ParallelGC)" prio=3 tid=0x000000010014d000 nid=0x19 runnable

"VM Periodic Task Thread" prio=3 tid=0x0000000101a38800 nid=0x21 waiting on condition

JNI global references: 1902
---^^^ derby stack trace ^^^ ---

The Glassfish application server reported this at the same time:

---vvv Glassfish stack trace vvv ---
[#|2013-03-13T17:13:59.257+0000|SEVERE|sun-appserver2.1|javax.enterprise.system.core.transaction|_ThreadID=18;_ThreadName=PMCollectionActivationManager.samReset
Detected;org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe;rollback;_RequestID=f75d8235-7e90-4578-b116-9c5bf0838529;|JTS5031: Exception [org.
omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.|#]

[#|2013-03-13T17:13:59.271+0000|SEVERE|sun-appserver2.1|javax.enterprise.system.container.ejb.mdb|_ThreadID=18;_ThreadName=PMCollectionActivationManager.samRese
tDetected;csem:PMCollectionServiceReceiver;javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0
minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No;_RequestID=f75d8235-7e90-4578-b116-9c5bf0838529;|MDB
00049: Message-driven bean [csem:PMCollectionServiceReceiver]: Exception in postinvoke : [javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Ex
ception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource [rollback] operation.  vmcid: 0x0  minor code: 0  completed: No]|#]

[#|2013-03-13T17:13:59.271+0000|SEVERE|sun-appserver2.1|javax.enterprise.system.container.ejb.mdb|_ThreadID=18;_ThreadName=PMCollectionActivationManager.samRese
tDetected;_RequestID=f75d8235-7e90-4578-b116-9c5bf0838529;|javax.transaction.SystemException
javax.transaction.SystemException: org.omg.CORBA.INTERNAL: JTS5031: Exception [org.omg.CORBA.INTERNAL:   vmcid: 0x0  minor code: 0 completed: Maybe] on Resource
[rollback] operation.  vmcid: 0x0  minor code: 0  completed: No
        at com.sun.jts.jta.TransactionManagerImpl.rollback(TransactionManagerImpl.java:350)
        at com.sun.enterprise.distributedtx.J2EETransactionManagerImpl.rollback(J2EETransactionManagerImpl.java:1143)
        at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.rollback(J2EETransactionManagerOpt.java:433)
        at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:3801)
        at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:3605)
        at com.sun.ejb.containers.MessageBeanContainer.afterMessageDeliveryInternal(MessageBeanContainer.java:1226)
        at com.sun.ejb.containers.MessageBeanContainer.afterMessageDelivery(MessageBeanContainer.java:1197)
        at com.sun.ejb.containers.MessageBeanListenerImpl.afterMessageDelivery(MessageBeanListenerImpl.java:79)
        at com.sun.enterprise.connectors.inflow.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:205)
        at $Proxy126.samResetDetected(Unknown Source)
        at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivation.samResetDetected(PMCollectionActivation.java:96)
        at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivationManager$3.run(PMCollectionActivationManager.java:236)
        at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:77)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
|#]

[#|2013-03-13T17:13:59.365+0000|SEVERE|sun-appserver2.1|com.canoga.nms.PMCollectionActivationManager|_ThreadID=18;_ThreadName=PMCollectionActivationManager.samR
esetDetected;_RequestID=f75d8235-7e90-4578-b116-9c5bf0838529;|PMCollectionActivation.pmResetDetected
javax.ejb.EJBException: message-driven bean method public abstract void com.canoga.j2ee.connector.pmra.api.PMCollectionNotificationListener.samResetDetected(jav
a.net.InetAddress,long) system exception
        at com.sun.ejb.containers.MessageBeanContainer.deliverMessage(MessageBeanContainer.java:1146)
        at com.sun.ejb.containers.MessageBeanListenerImpl.deliverMessage(MessageBeanListenerImpl.java:74)
        at com.sun.enterprise.connectors.inflow.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:179)
        at $Proxy126.samResetDetected(Unknown Source)
        at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivation.samResetDetected(PMCollectionActivation.java:96)
        at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivationManager$3.run(PMCollectionActivationManager.java:236)
        at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:77)
        at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.1.3.v20091002-r5404): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: DERBY SQL error: SQLCODE: -1, SQLSTATE: XJ001, SQLERRMC: java.lang.NullPointerException^T^TXJ001.U
Error Code: -1
Call: UPDATE PCS_V1.NPASAMRESULTS_WEEK_40 SET ENDREASON = ? WHERE (((MASTERIP = ?) AND (BOOTCOUNT < CAST (? AS INTEGER ))) AND (ENDREASON = ?))
        bind => [7, [B@565bd26b, 202, 0]
Query: UpdateAllQuery(name="NpaSamResultWeek40.endActiveSamResultsForAddress" referenceClass=NpaSamResultWeek40 sql="UPDATE PCS_V1.NPASAMRESULTS_WEEK_40 SET END
REASON = ? WHERE (((MASTERIP = ?) AND (BOOTCOUNT < CAST (? AS INTEGER ))) AND (ENDREASON = ?))")
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:332)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1386)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:659)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:512)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeCall(AbstractSession.java:872)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:205)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:191)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeNoSelectCall(DatasourceCallQueryMechanism.java:234)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.updateAll(DatasourceCallQueryMechanism.java:715)
        at org.eclipse.persistence.queries.UpdateAllQuery.executeDatabaseQuery(UpdateAllQuery.java:153)
        at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:664)
        at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:583)
        at org.eclipse.persistence.queries.ModifyAllQuery.executeInUnitOfWork(ModifyAllQuery.java:143)
---vvv Glassfish stack trace vvv ---

Glassfish, detecting a connection problem tries to close all connections in the connection pool and then that fails with a logical deadlock.

Here is a snippet of a small portion of the stacktrace of the Glassfish threads relating to this:

---vvv Glassfish stacktraces vvv---

        ====Many thread  stuck in the following state====

"httpSSLWorkerThread-8080-11" daemon prio=3 tid=0x00000001050fa800 nid=0x4474 waiting for monitor entry [0xfffffff8cb7fa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at com.sun.enterprise.resource.AbstractResourcePool.getResourceFromPool(AbstractResourcePool.java:773)
        - waiting to lock <0xfffffff97333c948> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:667)
        at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:609)
        at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:458)
        at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
        at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
        at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:130)
        at com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm.getConnection(JDBCRealm.java:527)
        at com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm.isUserValid(JDBCRealm.java:383)
        at com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm.authenticate(JDBCRealm.java:301)
        at com.sun.enterprise.security.auth.login.JDBCLoginModule.authenticate(JDBCLoginModule.java:72)
        at com.sun.enterprise.security.auth.login.PasswordLoginModule.authenticateUser(PasswordLoginModule.java:90)
        at com.sun.appserv.security.AppservPasswordLoginModule.login(AppservPasswordLoginModule.java:184)
        at sun.reflect.GeneratedMethodAccessor1004.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at com.sun.enterprise.security.auth.LoginContextDriver.doPasswordLogin(LoginContextDriver.java:319)
        at com.sun.enterprise.security.auth.LoginContextDriver.login(LoginContextDriver.java:177)
        at com.sun.enterprise.security.auth.LoginContextDriver.login(LoginContextDriver.java:130)
        at com.sun.web.security.RealmAdapter.authenticate(RealmAdapter.java:522)
        at com.sun.web.security.RealmAdapter.authenticate(RealmAdapter.java:462)
        at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:269)
        at org.apache.catalina.authenticator.AuthenticatorBase.processSecurityCheck(AuthenticatorBase.java:1040)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:638)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:625)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
        at com.sun.enterprise.web.connector.grizzly.comet.CometEngine.executeServlet(CometEngine.java:616)
        at com.sun.enterprise.web.connector.grizzly.comet.CometEngine.handle(CometEngine.java:362)
        at com.sun.enterprise.web.connector.grizzly.comet.CometAsyncFilter.doFilter(CometAsyncFilter.java:84)
        at com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.invokeFilters(DefaultAsyncExecutor.java:189)
        at com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncExecutor.interrupt(DefaultAsyncExecutor.java:164)
        at com.sun.enterprise.web.connector.grizzly.async.AsyncProcessorTask.doTask(AsyncProcessorTask.java:92)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

     ==== and  the other important one ====

"PMCollectionActivationManager.newPmTestResult" daemon prio=3 tid=0x0000000104765000 nid=0x366 runnable [0xfffffff8d31fb000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.derby.client.net.Reply.fill(Unknown Source)
        at org.apache.derby.client.net.Reply.ensureALayerDataInBuffer(Unknown Source)
        at org.apache.derby.client.net.Reply.readDssHeader(Unknown Source)
        at org.apache.derby.client.net.Reply.startSameIdChainParse(Unknown Source)
        at org.apache.derby.client.net.NetXAConnectionReply.readLocalXACommit(Unknown Source)
        at org.apache.derby.client.net.NetXAConnection.readLocalXACommit_(Unknown Source)
        at org.apache.derby.client.net.NetXAConnection.readCommit(Unknown Source)
        at org.apache.derby.client.net.NetConnection.readXACommit_(Unknown Source)
       at org.apache.derby.client.am.Connection.readCommit(Unknown Source)
        at org.apache.derby.client.am.Connection.readAutoCommit(Unknown Source)
        at org.apache.derby.client.am.Connection.flowClose(Unknown Source)
        at org.apache.derby.client.am.Connection.closeResourcesX(Unknown Source)
        at org.apache.derby.client.am.Connection.closeResources(Unknown Source)
        - locked <0xfffffff9c9513c50> (a org.apache.derby.client.net.NetConnection40)
        at org.apache.derby.client.net.NetConnection.closeResources(Unknown Source)
        - locked <0xfffffff9c9513c50> (a org.apache.derby.client.net.NetConnection40)
        at org.apache.derby.client.ClientPooledConnection.close(Unknown Source)
        - locked <0xfffffff9c9513b30> (a org.apache.derby.client.ClientXAConnection40)
        at org.apache.derby.client.ClientXAConnection.close(Unknown Source)
        - locked <0xfffffff9c9513b30> (a org.apache.derby.client.ClientXAConnection40)
        at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:276)
        at com.sun.enterprise.resource.ConnectorAllocator.destroyResource(ConnectorAllocator.java:175)
        at com.sun.enterprise.resource.AbstractResourcePool.destroyResource(AbstractResourcePool.java:986)
        at com.sun.enterprise.resource.AbstractResourcePool.emptyPool(AbstractResourcePool.java:1635)
        - locked <0xfffffff97333c948> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.AbstractResourcePool.doFailAllConnectionsProcessing(AbstractResourcePool.java:1213)
        at com.sun.enterprise.resource.AbstractResourcePool.resourceErrorOccurred(AbstractResourcePool.java:1163)
        - locked <0xfffffff97333c948> (a com.sun.enterprise.resource.SJSASResourcePool)
        at com.sun.enterprise.resource.PoolManagerImpl.putbackResourceToPool(PoolManagerImpl.java:445)
        at com.sun.enterprise.resource.PoolManagerImpl.resourceErrorOccurred(PoolManagerImpl.java:411)
        at com.sun.enterprise.resource.ConnectorAllocator$ConnectionListenerImpl.connectionErrorOccurred(ConnectorAllocator.java:98)
        at com.sun.gjc.spi.ManagedConnection.connectionErrorOccurred(ManagedConnection.java:662)
        at com.sun.gjc.spi.ManagedConnectionFactory.getInvalidConnections(ManagedConnectionFactory.java:249)
        at com.sun.enterprise.resource.AbstractConnectorAllocator.getInvalidConnections(AbstractConnectorAllocator.java:99)
        at com.sun.enterprise.resource.AbstractConnectorAllocator.isConnectionValid(AbstractConnectorAllocator.java:111)
        at com.sun.enterprise.resource.AbstractResourcePool.isConnectionValid(AbstractResourcePool.java:722)
        at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:668)
        at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:609)
        at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:458)
        at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
        at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
---^^^ Glassfish stacktraces ^^^---

In the second thread it appears that Glassfish gets a connection from the pool, realizes that the connection is invalid and attempts to close the connection (an XA connection).  Along the way it locks the resource pool.  Derby never responds to the close request and the system is then logically deadlocked.

Note while in this state, derby is up and will accept new connections.   I know this because when in this state, the script that gathers all of this information for me uses IJ to connect to the database and query the current locks and transactions.   Here is that output:

---vvv IJ output vvv---
ij version 10.8
ij> ij> XID            |GLOBAL_XID                                                                                                                      |USERNAM
E                                                                                                                        |TYPE                          |STATUS
|FIRST_INSTANT       |SQL_TEXT
----------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------

0 rows selected
ij> XID            |TYPE |MODE|TABLENAME                                                                                                                       |
LOCKNAME            |STATE|TABLETYPE|LOCK&|INDEXNAME

----------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------
-------

0 rows selected
ij>
---^^^ IJ output ^^^---

At this point, I don't know how to proceed.   I could upgrade to the latest version of Derby but I don't have a good feeling that this will not happen there either.   I have been unable to replicate this in a lab environment.

Any help will be much appreciated.

Brett


Mime
View raw message