db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject Problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1.1
Date Wed, 21 Dec 2011 14:14:44 GMT
I have a problem with a deadlock with Derby 10.8.1.2 and Glassfish V2.1.1 but at this point I cannot tell if it is a Derby problem or a Glassfish problem so I am looking for some guidance.

A database pool is created using the "org.apache.derby.jdbc.ClientXADataSource" XADataSource.   Connection validation is turned on and uses "auto-commit" to validate the connection and attempts to close all connections on any validation failure.

The deadlock is triggered when I have a couple of transactions that cause a Derby locks deadlock.  Deadlock detection and tracing is turned on in Derby and it correctly detects the locking deadlock and terminates one of the transactions.   Soon after this, however,  Glassfish becomes stuck.  Here is the only reference to Derby in the "jstack" output of Glassfish:

"PMCollectionActivationManager.newPmTestResult" daemon prio=3 tid=0x0000000103926800 nid=0xaf runnable [0xfffffffc218fa000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.client.net.Reply.fill(Unknown Source)
                at org.apache.derby.client.net.Reply.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.client.net.Reply.readDssHeader(Unknown Source)
                at org.apache.derby.client.net.Reply.startSameIdChainParse(Unknown Source)
                at org.apache.derby.client.net.NetXAConnectionReply.readLocalXACommit(Unknown Source)
                at org.apache.derby.client.net.NetXAConnection.readLocalXACommit_(Unknown Source)
                at org.apache.derby.client.net.NetXAConnection.readCommit(Unknown Source)
                at org.apache.derby.client.net.NetConnection.readXACommit_(Unknown Source)
                at org.apache.derby.client.am.Connection.readCommit(Unknown Source)
                at org.apache.derby.client.am.Connection.readAutoCommit(Unknown Source)
                at org.apache.derby.client.am.Connection.flowClose(Unknown Source)
                at org.apache.derby.client.am.Connection.closeResourcesX(Unknown Source)
                at org.apache.derby.client.am.Connection.closeResources(Unknown Source)
                - locked <0xfffffffc91831e18> (a org.apache.derby.client.net.NetConnection40)
                at org.apache.derby.client.net.NetConnection.closeResources(Unknown Source)
                - locked <0xfffffffc91831e18> (a org.apache.derby.client.net.NetConnection40)
                at org.apache.derby.client.ClientPooledConnection.close(Unknown Source)
                - locked <0xfffffffc91831cf8> (a org.apache.derby.client.ClientXAConnection40)
                at org.apache.derby.client.ClientXAConnection.close(Unknown Source)
                - locked <0xfffffffc91831cf8> (a org.apache.derby.client.ClientXAConnection40)
                at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:276)
                at com.sun.enterprise.resource.ConnectorAllocator.destroyResource(ConnectorAllocator.java:175)
                at com.sun.enterprise.resource.AbstractResourcePool.destroyResource(AbstractResourcePool.java:986)
                at com.sun.enterprise.resource.AbstractResourcePool.emptyPool(AbstractResourcePool.java:1635)
                - locked <0xfffffffc714c8da0> (a com.sun.enterprise.resource.SJSASResourcePool)
                at com.sun.enterprise.resource.AbstractResourcePool.doFailAllConnectionsProcessing(AbstractResourcePool.java:1213)
                at com.sun.enterprise.resource.AbstractResourcePool.resourceErrorOccurred(AbstractResourcePool.java:1163)
                - locked <0xfffffffc714c8da0> (a com.sun.enterprise.resource.SJSASResourcePool)
                at com.sun.enterprise.resource.PoolManagerImpl.putbackResourceToPool(PoolManagerImpl.java:445)
                at com.sun.enterprise.resource.PoolManagerImpl.resourceErrorOccurred(PoolManagerImpl.java:411)
                at com.sun.enterprise.resource.ConnectorAllocator$ConnectionListenerImpl.connectionErrorOccurred(ConnectorAllocator.java:98)
                at com.sun.gjc.spi.ManagedConnection.connectionErrorOccurred(ManagedConnection.java:662)
                at com.sun.gjc.spi.ManagedConnectionFactory.getInvalidConnections(ManagedConnectionFactory.java:249)
                at com.sun.enterprise.resource.AbstractConnectorAllocator.getInvalidConnections(AbstractConnectorAllocator.java:99)
                at com.sun.enterprise.resource.AbstractConnectorAllocator.isConnectionValid(AbstractConnectorAllocator.java:111)
                at com.sun.enterprise.resource.AbstractResourcePool.isConnectionValid(AbstractResourcePool.java:722)
                at com.sun.enterprise.resource.AbstractResourcePool.getUnenlistedResource(AbstractResourcePool.java:668)
                at com.sun.enterprise.resource.AbstractResourcePool.internalGetResource(AbstractResourcePool.java:609)
                at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:458)
                at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
                at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
                at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
                at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
                at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
                at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
                at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:108)
                at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:133)
                at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:94)
                at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
                at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:324)
                at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:276)
                at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:546)
                at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1400)
                at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:300)
                - locked <0xfffffffc9d592f48> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
                at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:556)
                at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:512)
                at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:536)
                at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:205)
                at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:191)
                at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:262)
                at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:599)
                at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2518)
                at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2476)
                at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:481)
                at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:928)
                at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:664)
                at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:889)
                at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:458)
                at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:952)
                at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2750)
                at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1181)
                at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1165)
                at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1139)
                at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:389)
                at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:594)
                at com.canoga.nms.pkg.pcs.v1.entity.NpaResultsCircuitFacade.findByCircuit(NpaResultsCircuitFacade.java:57)
                at sun.reflect.GeneratedMethodAccessor599.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1011)
                at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:175)
                at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2929)
                at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4020)
                at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:197)
                at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:83)
                at $Proxy76.findByCircuit(Unknown Source)
                at com.canoga.nms.pkg.pcs.v1.collection.service.PMCollectionServiceReceiver.getOrCreateCircuit(PMCollectionServiceReceiver.java:671)
                at com.canoga.nms.pkg.pcs.v1.collection.service.PMCollectionServiceReceiver.insertPmTestResult(PMCollectionServiceReceiver.java:381)
                at com.canoga.nms.pkg.pcs.v1.collection.service.PMCollectionServiceReceiver.newPmTestResult(PMCollectionServiceReceiver.java:208)
                at sun.reflect.GeneratedMethodAccessor611.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at com.sun.enterprise.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1011)
                at com.sun.enterprise.security.SecurityUtil.invoke(SecurityUtil.java:175)
                at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:2929)
                at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4020)
                at com.sun.ejb.containers.MessageBeanContainer.deliverMessage(MessageBeanContainer.java:1111)
                at com.sun.ejb.containers.MessageBeanListenerImpl.deliverMessage(MessageBeanListenerImpl.java:74)
                at com.sun.enterprise.connectors.inflow.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:179)
                at $Proxy154.newPmTestResult(Unknown Source)
                at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivation.newPmTestResult(PMCollectionActivation.java:76)
                at com.canoga.j2ee.connector.pmra.spi.PMCollectionActivationManager$2.run(PMCollectionActivationManager.java:191)
                at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:77)
                at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)

All of the other threads that need to access the database are stuck waiting for "0xfffffffc714c8da0" which is locked in the thread above.   It appears that this thread never makes any progress as I run "jstack" over and over and this thread is always at this point.

The Derby "jstack" trace looks like:

2011-12-21 05:43:35
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

"Attach Listener" daemon prio=3 tid=0x000000010481a800 nid=0x122 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DRDAConnThread_99" prio=3 tid=0x00000001044e3800 nid=0xe4 in Object.wait() [0xfffffffd262ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_98" prio=3 tid=0x000000010466c000 nid=0xe3 in Object.wait() [0xfffffffd264ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_97" prio=3 tid=0x00000001045c7000 nid=0xe2 in Object.wait() [0xfffffffd266ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_96" prio=3 tid=0x0000000101628800 nid=0xe1 runnable [0xfffffffd268fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_95" prio=3 tid=0x000000010449c000 nid=0xe0 in Object.wait() [0xfffffffd26aff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_94" prio=3 tid=0x0000000104111000 nid=0xdf runnable [0xfffffffd26cff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_93" prio=3 tid=0x00000001015e3800 nid=0xde in Object.wait() [0xfffffffd26efe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_92" prio=3 tid=0x0000000101ed4800 nid=0xdd in Object.wait() [0xfffffffd270fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_91" prio=3 tid=0x0000000102443000 nid=0xdc in Object.wait() [0xfffffffd272ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_90" prio=3 tid=0x0000000100e9b800 nid=0xdb in Object.wait() [0xfffffffd274ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_89" prio=3 tid=0x00000001034f9000 nid=0xda runnable [0xfffffffd276fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_88" prio=3 tid=0x0000000101092800 nid=0xd9 in Object.wait() [0xfffffffd278ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_87" prio=3 tid=0x00000001011ee800 nid=0xd8 in Object.wait() [0xfffffffd27aff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_86" prio=3 tid=0x0000000103a90800 nid=0xd7 in Object.wait() [0xfffffffd27cff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_85" prio=3 tid=0x000000010445b800 nid=0xd6 in Object.wait() [0xfffffffd27efe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_84" prio=3 tid=0x000000010445b000 nid=0xd5 in Object.wait() [0xfffffffd280fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_83" prio=3 tid=0x0000000104566000 nid=0xd4 in Object.wait() [0xfffffffd282ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_82" prio=3 tid=0x00000001016c1000 nid=0xd3 in Object.wait() [0xfffffffd284ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_81" prio=3 tid=0x0000000101f20000 nid=0xd2 in Object.wait() [0xfffffffd286ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_80" prio=3 tid=0x0000000101153800 nid=0xd1 in Object.wait() [0xfffffffd288ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_79" prio=3 tid=0x0000000103bc5000 nid=0xd0 in Object.wait() [0xfffffffd28aff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_78" prio=3 tid=0x0000000102968800 nid=0xcf in Object.wait() [0xfffffffd28cff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_77" prio=3 tid=0x0000000103c69800 nid=0xce in Object.wait() [0xfffffffd28efe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_76" prio=3 tid=0x0000000102b2b000 nid=0xcd in Object.wait() [0xfffffffd290fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_75" prio=3 tid=0x000000010359e000 nid=0xcc in Object.wait() [0xfffffffd292ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_74" prio=3 tid=0x0000000102d41800 nid=0xcb in Object.wait() [0xfffffffd294ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_73" prio=3 tid=0x00000001022e8800 nid=0xca runnable [0xfffffffd296ff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_72" prio=3 tid=0x0000000103674800 nid=0xc9 in Object.wait() [0xfffffffd298ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_71" prio=3 tid=0x0000000102a5d000 nid=0xc8 in Object.wait() [0xfffffffd29aff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_70" prio=3 tid=0x000000010143b800 nid=0xc7 in Object.wait() [0xfffffffd29cff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_69" prio=3 tid=0x0000000102331800 nid=0xc6 in Object.wait() [0xfffffffd29efe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_68" prio=3 tid=0x00000001035be000 nid=0xc5 in Object.wait() [0xfffffffd2a0fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_67" prio=3 tid=0x00000001019fc000 nid=0xc4 in Object.wait() [0xfffffffd2a2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_66" prio=3 tid=0x0000000102289800 nid=0xc3 in Object.wait() [0xfffffffd2a4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_65" prio=3 tid=0x00000001013fa000 nid=0xc2 in Object.wait() [0xfffffffd2a6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_64" prio=3 tid=0x0000000103028000 nid=0xc1 in Object.wait() [0xfffffffd2a8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_63" prio=3 tid=0x0000000101a43800 nid=0xc0 in Object.wait() [0xfffffffd2aaff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_62" prio=3 tid=0x0000000101c79800 nid=0xbf in Object.wait() [0xfffffffd2acff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_61" prio=3 tid=0x0000000103ab9800 nid=0xbe in Object.wait() [0xfffffffd2aefe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_60" prio=3 tid=0x0000000101a82000 nid=0xbd in Object.wait() [0xfffffffd2b0fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_59" prio=3 tid=0x00000001014d4800 nid=0xbc in Object.wait() [0xfffffffd2b2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_58" prio=3 tid=0x0000000101e93800 nid=0xbb runnable [0xfffffffd2b4fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_57" prio=3 tid=0x000000010224c000 nid=0xba in Object.wait() [0xfffffffd2b6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_56" prio=3 tid=0x00000001034d9000 nid=0xb9 in Object.wait() [0xfffffffd2b8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_55" prio=3 tid=0x0000000102c96800 nid=0xb6 in Object.wait() [0xfffffffd2bafe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_54" prio=3 tid=0x000000010222f000 nid=0xb5 in Object.wait() [0xfffffffd2bcfe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_53" prio=3 tid=0x00000001016e6000 nid=0xb4 in Object.wait() [0xfffffffd2beff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_52" prio=3 tid=0x00000001010d0000 nid=0xb3 in Object.wait() [0xfffffffd2c0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_51" prio=3 tid=0x0000000101288800 nid=0xb2 in Object.wait() [0xfffffffd2c2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_50" prio=3 tid=0x0000000100ddd000 nid=0xb1 in Object.wait() [0xfffffffd2c4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_49" prio=3 tid=0x00000001039d9000 nid=0xb0 in Object.wait() [0xfffffffd2c6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_48" prio=3 tid=0x00000001014e8000 nid=0xaf in Object.wait() [0xfffffffd2c8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_47" prio=3 tid=0x0000000102cb9800 nid=0xae runnable [0xfffffffd2cafe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_46" prio=3 tid=0x0000000102346800 nid=0xad in Object.wait() [0xfffffffd2ccfe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_45" prio=3 tid=0x0000000102372800 nid=0xac in Object.wait() [0xfffffffd2ceff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_44" prio=3 tid=0x0000000102a28800 nid=0xab in Object.wait() [0xfffffffd2d0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_43" prio=3 tid=0x0000000103112800 nid=0xaa in Object.wait() [0xfffffffd2d2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_42" prio=3 tid=0x0000000100e10000 nid=0xa9 in Object.wait() [0xfffffffd2d4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_41" prio=3 tid=0x000000010247f800 nid=0xa8 in Object.wait() [0xfffffffd2d6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_40" prio=3 tid=0x0000000102945000 nid=0xa7 in Object.wait() [0xfffffffd2d8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_39" prio=3 tid=0x0000000102347800 nid=0xa6 in Object.wait() [0xfffffffd2dafe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_38" prio=3 tid=0x0000000102e03000 nid=0xa5 runnable [0xfffffffd2dcfe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_37" prio=3 tid=0x0000000101910800 nid=0xa4 in Object.wait() [0xfffffffd2deff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_36" prio=3 tid=0x00000001022bd800 nid=0xa3 in Object.wait() [0xfffffffd2e0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_35" prio=3 tid=0x00000001029a1800 nid=0xa2 in Object.wait() [0xfffffffd2e2ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_34" prio=3 tid=0x0000000102108800 nid=0xa1 in Object.wait() [0xfffffffd2e4ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_33" prio=3 tid=0x0000000102226000 nid=0xa0 in Object.wait() [0xfffffffd2e6ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_32" prio=3 tid=0x00000001029a5000 nid=0x9f in Object.wait() [0xfffffffd2e8ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_31" prio=3 tid=0x00000001010ff000 nid=0x9e in Object.wait() [0xfffffffd2eafe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_30" prio=3 tid=0x0000000101bad000 nid=0x9d in Object.wait() [0xfffffffd2ecfe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_29" prio=3 tid=0x0000000101dbe000 nid=0x32 runnable [0xfffffffd2eefe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_28" prio=3 tid=0x00000001011c4000 nid=0x31 in Object.wait() [0xfffffffd2f0ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_27" prio=3 tid=0x00000001021fd000 nid=0x2f runnable [0xfffffffd2f4ff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_26" prio=3 tid=0x0000000100120000 nid=0x2e in Object.wait() [0xfffffffd2f6fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_25" prio=3 tid=0x000000010011e000 nid=0x2d runnable [0xfffffffd2f8fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_24" prio=3 tid=0x000000010011d000 nid=0x2c in Object.wait() [0xfffffffd2faff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_23" prio=3 tid=0x000000010011c000 nid=0x2b in Object.wait() [0xfffffffd2fcff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_22" prio=3 tid=0x0000000100fd9800 nid=0x2a runnable [0xfffffffd2feff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_21" prio=3 tid=0x0000000100fd9000 nid=0x29 in Object.wait() [0xffffffff706ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_20" prio=3 tid=0x0000000101dfa000 nid=0x28 in Object.wait() [0xffffffff708ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_19" prio=3 tid=0x00000001029d7000 nid=0x27 runnable [0xffffffff70aff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_18" prio=3 tid=0x00000001029d8000 nid=0x26 runnable [0xffffffff70cfe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_17" prio=3 tid=0x00000001022ba800 nid=0x25 runnable [0xffffffff70efe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_16" prio=3 tid=0x0000000102374000 nid=0x24 runnable [0xffffffff710fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_15" prio=3 tid=0x00000001021fc000 nid=0x23 runnable [0xffffffff712ff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_14" prio=3 tid=0x0000000100d07000 nid=0x22 in Object.wait() [0xffffffff714ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_13" prio=3 tid=0x0000000101b4a000 nid=0x21 runnable [0xffffffff716ff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_12" prio=3 tid=0x000000010108b000 nid=0x20 runnable [0xffffffff718ff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_11" prio=3 tid=0x0000000101960800 nid=0x1f runnable [0xffffffff71aff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_10" prio=3 tid=0x0000000102362800 nid=0x1e in Object.wait() [0xffffffff71cfe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_9" prio=3 tid=0x0000000100df9800 nid=0x1d runnable [0xffffffff71efe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_8" prio=3 tid=0x0000000100d47000 nid=0x1c in Object.wait() [0xffffffff720ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_7" prio=3 tid=0x0000000101134000 nid=0x1b in Object.wait() [0xffffffff722ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_6" prio=3 tid=0x0000000100fe8000 nid=0x1a runnable [0xffffffff724fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"DRDAConnThread_5" prio=3 tid=0x0000000100d2e000 nid=0x19 in Object.wait() [0xffffffff726ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"derby.rawStoreDaemon" daemon prio=3 tid=0x0000000100fe7000 nid=0x18 in Object.wait() [0xffffffff728ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
                - locked <0xfffffffd77095100> (a org.apache.derby.impl.services.daemon.BasicDaemon)
                at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
                at java.lang.Thread.run(Thread.java:662)

"DRDAConnThread_4" prio=3 tid=0x0000000101365000 nid=0x17 runnable [0xffffffff72aff000]
   java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.derby.impl.drda.DDMReader.fill(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.ensureALayerDataInBuffer(Unknown Source)
                at org.apache.derby.impl.drda.DDMReader.readDssHeader(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"derby.rawStoreDaemon" daemon prio=3 tid=0x0000000100d3b000 nid=0x16 in Object.wait() [0xffffffff732ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
                - locked <0xfffffffd76fd1e90> (a org.apache.derby.impl.services.daemon.BasicDaemon)
                at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
                at java.lang.Thread.run(Thread.java:662)

"DRDAConnThread_3" prio=3 tid=0x000000010130d000 nid=0x15 in Object.wait() [0xffffffff734fe000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.getNextSession(Unknown Source)
                - locked <0xfffffffd73baa288> (a java.util.Vector)
                at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)

"NetworkServerThread_2" prio=3 tid=0x0000000100fdd000 nid=0x14 runnable [0xffffffff736fe000]
   java.lang.Thread.State: RUNNABLE
                at java.net.PlainSocketImpl.socketAccept(Native Method)
                at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
                - locked <0xfffffffd77039ae8> (a java.net.SocksSocketImpl)
                at java.net.ServerSocket.implAccept(ServerSocket.java:453)
                at java.net.ServerSocket.accept(ServerSocket.java:421)
                at org.apache.derby.impl.drda.ClientThread$1.run(Unknown Source)
                at java.security.AccessController.doPrivileged(Native Method)
                at org.apache.derby.impl.drda.ClientThread.run(Unknown Source)

"Timer-0" daemon prio=3 tid=0x000000010104b800 nid=0x13 in Object.wait() [0xffffffff73aff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                - waiting on <0xfffffffd77054aa0> (a java.util.TaskQueue)
                at java.lang.Object.wait(Object.java:485)
                at java.util.TimerThread.mainLoop(Timer.java:483)
                - locked <0xfffffffd77054aa0> (a java.util.TaskQueue)
                at java.util.TimerThread.run(Timer.java:462)

"Low Memory Detector" daemon prio=3 tid=0x0000000100b09000 nid=0x11 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=3 tid=0x0000000100b06000 nid=0x10 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=3 tid=0x0000000100b03000 nid=0xf waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=3 tid=0x0000000100b01000 nid=0xe runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=3 tid=0x0000000100adf000 nid=0xd in Object.wait() [0xffffffff7b7ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
                - locked <0xfffffffd77054f00> (a java.lang.ref.ReferenceQueue$Lock)
                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
                at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=3 tid=0x0000000100ad8000 nid=0xc in Object.wait() [0xffffffff7b9ff000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:485)
                at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
                - locked <0xfffffffd73df7618> (a java.lang.ref.Reference$Lock)

"main" prio=3 tid=0x0000000100117000 nid=0x2 in Object.wait() [0xffffffff7c6fd000]
   java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                - waiting on <0xfffffffd77039aa0> (a java.lang.Object)
                at java.lang.Object.wait(Object.java:485)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.blockingStart(Unknown Source)
                - locked <0xfffffffd77039aa0> (a java.lang.Object)
                at org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(Unknown Source)
                at org.apache.derby.drda.NetworkServerControl.main(Unknown Source)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at com.sun.enterprise.cli.commands.DerbyControl.invokeNetworkServerControl(DerbyControl.java:132)
                at com.sun.enterprise.cli.commands.DerbyControl.main(DerbyControl.java:215)

"VM Thread" prio=3 tid=0x0000000100ad4000 nid=0xb runnable

"GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100127000 nid=0x3 runnable

"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100128800 nid=0x4 runnable

"GC task thread#2 (ParallelGC)" prio=3 tid=0x000000010012a000 nid=0x5 runnable

"GC task thread#3 (ParallelGC)" prio=3 tid=0x000000010012b800 nid=0x6 runnable

"GC task thread#4 (ParallelGC)" prio=3 tid=0x000000010012d800 nid=0x7 runnable

"GC task thread#5 (ParallelGC)" prio=3 tid=0x000000010012f000 nid=0x8 runnable

"GC task thread#6 (ParallelGC)" prio=3 tid=0x0000000100130800 nid=0x9 runnable

"GC task thread#7 (ParallelGC)" prio=3 tid=0x0000000100132000 nid=0xa runnable

"VM Periodic Task Thread" prio=3 tid=0x0000000100b1e000 nid=0x12 waiting on condition

JNI global references: 955

Nothing really looks suspicious here except the lack of any observable work being done.    The only other thing that is interesting is that there 374 transactions still active in the database as reported by:

1 row selected
ij>  select count(*) from syscs_diag.transaction_table where status != 'IDLE';
1
-----------
374

Unfortunately I was not able to look at the transactions before the server got reset, but I am in the process of trying to replicate this again.

It is a if the ClientXADataSource is trying to read from a connection that does not exist anymore.

I have no problem with trying to debug this as I have both Glassfish built from source and Derby built from source.  What I am hoping to get is some direction on where to look.  From looking at the source of the client driver where it appears to be stuck I see:

        do {
            try {
                // oops, we shouldn't expose the agent's input stream here, collapse this into a read method on the agent
                actualBytesRead = netAgent_.getInputStream().read(buffer_, count_, buffer_.length - count_);
            } catch (java.io.IOException ioe) {
                netAgent_.throwCommunicationsFailure(ioe);
            } finally {
                if (agent_.loggingEnabled()) {
                    ((NetLogWriter) netAgent_.logWriter_).traceProtocolFlow(buffer_,
                            count_,
                            actualBytesRead,
                            NetLogWriter.TYPE_TRACE_RECEIVE,
                            "Reply",
                            "fill",
                            2); // tracepoint
                }
            }
            // DERBY-2747: only count if we actually read something
            if (actualBytesRead > 0) {
                count_ += actualBytesRead;
                totalBytesRead += actualBytesRead;
            }

        } while ((totalBytesRead < minimumBytesNeeded) && (actualBytesRead != -1));

So it seems that there are too possible ways this could get stuck.  Either "actualBytesRead" is returning "-1" but no IOException being thrown or  "actualBytesRead" is returning "0".   Any thoughts on this?

Any and all help, pointers where to look, etc. will be greatly appreciated.

Mime
View raw message