db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject An issue with FREEZE/UNFREEZE at a customer happened
Date Fri, 19 Oct 2012 16:52:44 GMT
This has happened twice now in about two months apart.   See my initial report of a problem
here:

http://markmail.org/search/?q=Had+a+problem+with+SYSCS_FREEZE_DATABASE+and+am+wondering+is+something+can+be+done+make+this+more+robust#query:Had%20a%20problem%20with%20SYSCS_FREEZE_DATABASE%20and%20am%20wondering%20is%20something%20can%20be%20done%20make%20this%20more%20robust+page:1+mid:rvpzp7q4aanslswf+state:results

An earlier problem that I had here:

http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:e54e5pc43qoymdja+state:results

My utility code is in the first link and when run it output this:

Failed to unfreeze the database: Insufficient data while reading from the network - expected
a minimum of 6 bytes and received only 0 bytes.  The connection has been terminated.

>From looking at the time that the backup tar file was created (the utility does  freeze/zfs
snapshot/unfreeze/create tar file) I can see that the tar file did not get created  until
this morning when the Derby network server was terminated, indicating that the utility was
hung in the unfreeze and the above message is because of the Network server being terminated.

At this point the database was frozen and all connections to the database were blocked and
the Derby network server had to be forcefully killed with a "kill -9" as a connection could
not be acquired to issue the shutdown command.

The derby.log showed this:

The XA transaction timed out and is going to be rolled back. The transaction Xid is (4871251,1a193600a379ea74636776776e6a73767230312c7365727665722c5033373030,636776776e6a73767230312c7365727665722c50333730302c00).
----------------------------------------------------------------
Fri Oct 19 07:37:22 EDT 2012: Shutting down Derby engine
----------------------------------------------------------------
Fri Oct 19 07:38:20 EDT 2012 : Security manager installed using the Basic server security
policy.
Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and
ready to accept connections on port 1527
Fri Oct 19 07:38:21 EDT 2012 : Apache Derby Network Server - 10.8.2.2 - (cp1) started and
ready to accept connections on port 1527
----------------------------------------------------------------
Fri Oct 19 07:38:49 EDT 2012:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance
a816c00e-013a-78d1-1c49-000065089f97
on database directory /opt/canoga/canogaview/glassfish/databases/csemdb  with class loader
sun.misc.Launcher$AppClassLoader@61e63e3d
Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar
java.vendor=Sun Microsystems Inc.
java.runtime.version=1.6.0_22-b04
user.dir=/
derby.system.home=/opt/canoga/canogaview/glassfish/databases
Database Class Loader started - derby.database.classpath='CSEM.csemderby:PCS_V1.pcsderby'
----------------------------------------------------------------
Fri Oct 19 07:39:34 EDT 2012:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.8.2.2 - (cp1): instance
601a400f-013a-78d1-1c49-000065089f97
on database directory /opt/canoga/canogaview/glassfish/databases/csemrptsvrdb  with class
loader sun.misc.Launcher$AppClassLoader@61e63e3d
Loaded from file:/opt/canoga/canogaview/glassfish/javadb/lib/derby.jar
java.vendor=Sun Microsystems Inc.
java.runtime.version=1.6.0_22-b04
user.dir=/
derby.system.home=/opt/canoga/canogaview/glassfish/databases
Database Class Loader started - derby.database.classpath=''
Loaded com.canoga.derby.fcn.NpaResultsQuery from database jar "PCS_V1"."PCSDERBY"
Loaded com.canoga.derby.fcn.NpaResultsTableFunction from database jar "PCS_V1"."PCSDERBY"
Loaded com.canoga.derby.fcn.NpaSamResultsQuery from database jar "PCS_V1"."PCSDERBY"
Loaded com.canoga.derby.fcn.NpaSamResultsTableFunction from database jar "PCS_V1"."PCSDERBY"

There is a background job that we have running that performs an UPDATE_STATISTICS that just
so happens to be scheduled to be run at the same instance as the backup utility.   In the
second email thread above, I saw a lockup related to this when trying to perform a backup.
   I think this happened again however I could not get a stack trace this time as the customer
killed the Derby network server process before this could be done.

My analysis in that second email thread points out the pattern of the lockup.   I am going
to try to fix this but if anyone could look at the stack trace of the email thread:

http://markmail.org/search/?q=Had+Derby+10.8.2.2+lockup+%28was+Another+backup+question%29#query:Had%20Derby%2010.8.2.2%20lockup%20(was%20Another%20backup%20question)+page:1+mid:3dy474zwlvi637bn+state:results

and see if they see anything obvious, it would be greatly appreciated.


Mime
View raw message