I am using Derby as a database as part of a Glassfish installation.  I am processing JMS messages with each one causing access to the Derby database via Eclipselink.  After about 80K to 100K messages processed, I have a hung system. 

 

I run jstack on the Derby network server and there is no deadlock.  Also I run jstack on the Glassfish and there is no deadlock.  In the thread stack for Glassfish I see a thread with the following stack trace:

 

"p: thread-pool-1; w: 3" daemon prio=6 tid=0x5de34800 nid=0xb94 runnable [0x60abe000..0x60abfd94]

   java.lang.Thread.State: RUNNABLE

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.read(SocketInputStream.java:129)

        at org.apache.derby.client.net.Reply.fill(Unknown Source)

        at org.apache.derby.client.net.Reply.ensureALayerDataInBuffer(Unknown Source)

        at org.apache.derby.client.net.Reply.readDssHeader(Unknown Source)

        at org.apache.derby.client.net.Reply.startSameIdChainParse(Unknown Source)

        at org.apache.derby.client.net.NetConnectionReply.readExchangeServerAttributes(Unknown Source)

        at org.apache.derby.client.net.NetConnection.readServerAttributesAndKeyExchange(Unknown Source)

        at org.apache.derby.client.net.NetConnection.readAllConnectCommandsChained(Unknown Source)

        at org.apache.derby.client.net.NetConnection.readDeferredReset(Unknown Source)

        at org.apache.derby.client.net.NetAgent.readDeferredResetConnection(Unknown Source)

        at org.apache.derby.client.net.NetAgent.beginReadChain(Unknown Source)

        at org.apache.derby.client.am.Agent.flow(Unknown Source)

        at org.apache.derby.client.am.Statement.flowExecute(Unknown Source)

        at org.apache.derby.client.am.Statement.executeQueryX(Unknown Source)

        at org.apache.derby.client.am.Statement.executeQuery(Unknown Source)

        - locked <0x1d90dba0> (a org.apache.derby.client.net.NetConnection40)

        at org.apache.derby.client.am.Connection.getTransactionIsolation(Unknown Source)

        at org.apache.derby.client.am.LogicalConnection.getTransactionIsolation(Unknown Source)

        at com.sun.gjc.spi.ManagedConnectionFactory.resetIsolation(ManagedConnectionFactory.java:463)

        at com.sun.gjc.spi.ManagedConnection.resetConnectionProperties(ManagedConnection.java:354)

        at com.sun.gjc.spi.ManagedConnection.getConnection(ManagedConnection.java:328)

        at com.sun.enterprise.resource.ConnectorAllocator.fillInResourceObjects(ConnectorAllocator.java:155)

        at com.sun.enterprise.resource.AbstractResourcePool.getResource(AbstractResourcePool.java:514)

        at com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)

        at com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)

        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)

        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)

        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)

        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)

 

This thread is making no progress at all.  This is the one hanging my system.  The derby log shows:

 

2010-11-15 23:57:46.593 GMT : Apache Derby Network Server - 10.5.3.0 - (802917) started and ready to accept connections on port 1527

----------------------------------------------------------------

2010-11-16 00:00:20.078 GMT:

 Booting Derby version The Apache Software Foundation - Apache Derby - 10.5.3.0 - (802917): instance a816c00e-012c-51f9-12b2-0000376d74d3

on database directory C:\Documents and Settings\Brett Bergquist\.netbeans-derby\csemdb

 

Database Class Loader started - derby.database.classpath='CSEM.csemderby'

(END)

 

The rest of Glassfish is up and running as well as other parts of my application.  This one thread, however, is causing my JMS processing to be hung.

 

This is the second time in two days of testing that this has occurred.  The previous lockup has exactly the same thread stack traceback in a few other threads, all from different database queries, all stuck at the same point.  Restarting the derby network server, caused the condition to clear up as far as the application server is concerned, so I am pretty confident it is related to derby.

 

Any help on where to look to find this problem will be greatly appreciated.

 

Brett