This has happened twice now in about two months apart.   See my initial report of a problem here:

 

http://markmail.org/search/?q=Had+a+problem+with+SYSCS_FREEZE_DATABASE+and+am+wondering+is+something+can+be+done+make+this+more+robust#query:Had%20a%20problem%20with%20SYSCS_FREEZE_DATABASE%20and%20am%20wondering%20is%20something%20can%20be%20done%20make%20this%20more%20robust+page:1+mid:rvpzp7q4aanslswf+state:results

 

An earlier problem that I had here:

 

http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:e54e5pc43qoymdja+state:results

 

My utility code is in the first link and when run it output this:

 

Failed to unfreeze the database: Insufficient data while reading from the network - expected a minimum of 6 bytes and received only 0 bytes.  The connection has been terminated.

 

From looking at the time that the backup tar file was created (the utility does  freeze/zfs snapshot/unfreeze/create tar file) I can see that the tar file did not get created  until this morning when the Derby network server was terminated, indicating that the utility was hung in the unfreeze and the above message is because of the Network server being terminated.

 

At this point the database was frozen and all connections to the database were blocked and the Derby network server had to be forcefully killed with a “kill –9” as a connection could not be acquired to issue the shutdown command. 

 

The derby.log showed this:

 

The XA transaction timed out and is going to be rolled back. The transaction Xid is (4871251,1a193600a379ea74636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).

----------------------------------------------------------------

Fri Oct 19 07:37:22 EDT 2012: Shutting down Derby engine

----------------------------------------------------------------

Fri Oct 19 07:38:20 EDT 2012 : Security manager installed using the Basic server security policy.

Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and ready to accept connections on port 1527

Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and ready to accept connections on port 1527

----------------------------------------------------------------

Fri Oct 19 07:38:49 EDT 2012:

Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance a816c00e-013a-78d1-1c49-000065089f97

on database directory /opt/canoga/canogaview/glassfish/databases/csemdb  with class loader sun.misc.Launcher$AppClassLoader@61e63e3d

Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar

java.vendor=Sun Microsystems Inc.

java.runtime.version=1.6.0_22-b04

user.dir=/

derby.system.home=/opt/canoga/canogaview/glassfish/databases

Database Class Loader started - derby.database.classpath='CSEM.csemderby:PCS_V1.pcsderby'

----------------------------------------------------------------

Fri Oct 19 07:39:34 EDT 2012:

Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance 601a400f-013a-78d1-1c49-000065089f97

on database directory /opt/canoga/canogaview/glassfish/databases/csemrptsvrdb  with class loader sun.misc.Launcher$AppClassLoader@61e63e3d

Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar

java.vendor=Sun Microsystems Inc.

java.runtime.version=1.6.0_22-b04

user.dir=/

derby.system.home=/opt/canoga/canogaview/glassfish/databases

Database Class Loader started - derby.database.classpath=''

Loaded com.canoga.derby.fcn.NpaResultsQuery from database jar "PCS_V1"."PCSDERBY"

Loaded com.canoga.derby.fcn.NpaResultsTableFunction from database jar "PCS_V1"."PCSDERBY"

Loaded com.canoga.derby.fcn.NpaSamResultsQuery from database jar "PCS_V1"."PCSDERBY"

Loaded com.canoga.derby.fcn.NpaSamResultsTableFunction from database jar "PCS_V1"."PCSDERBY"

 

There is a background job that we have running that performs an UPDATE_STATISTICS that just so happens to be scheduled to be run at the same instance as the backup utility.   In the second email thread above, I saw a lockup related to this when trying to perform a backup.    I think this happened again however I could not get a stack trace this time as the customer killed the Derby network server process before this could be done.

 

My analysis in that second email thread points out the pattern of the lockup.   I am going to try to fix this but if anyone could look at the stack trace of the email thread:

 

http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:3dy474zwlvi637bn+state:results

 

and see if they see anything obvious, it would be greatly appreciated.