db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject Re: Had a problem with SYSCS_FREEZE_DATABASE and am wondering is something can be done make this more robust
Date Tue, 11 Sep 2012 20:22:31 GMT
The utility that performs the freeze/zfs snapshot/unfreeze is run at a customer sit via cron
and unfortunately the output was overwritten by the next nights run which blocked trying to
freeze the database.  I verified that by looking at the stack traces of derby using jstack.
Other than the database being frozen and all connections in the pool to derby in use, derby
engine looked ok.

The reason for the utility failure was lost but neither the try/finally nor the JVM shutdown
hook worked to unfreeze the database.  Maybe the JVM running the utility crashed but at this
point I don't know.

I did extensive testing of the utility with normal possibilities like the zfs snapshot failing,
interrupting the utility with SIGINT etc and these were handled and the database unfreeze
called.

So at this point I don't know what failed but something did and left the database frozen so
an automatic mechanism in the derby engine would be most welcome.

I will try to implement this maybe as a param to the system procedure and supply a patch for
possible inclusion into derby

On Sep 11, 2012, at 2:21 PM, "Dag Wanvik" <dag.wanvik@oracle.com<mailto:dag.wanvik@oracle.com>>
wrote:



On 10.09.2012 20:07, Bergquist, Brett wrote:
From the stack traces of the Derby engine, it appears that something causes the utility to
fail after the database was frozen and neither the shutdown hook nor the try/finally unfroze
the database.   So after that point, the database was effectively locked up.   The system
was still operating and connections were being made trying to access the  database exhausting
all of the connections.

So if none of your unfreeze attempt worked here, what happened to that process/VM? You say
"failed", did it hang, did it complete normally, albeit with no effect, or was it killed off?
Curious, since If we were to implement an automatic unfreeze at connection close, if the method
you used didn't work, an automatic unfreeze might fail too (if there is a bug in the code
that prevents unfreeze from doing its thing). If it hangs without unfreezing it might be interesting
to see the VM state at that point, e.g. via jstack.


So I was thinking that maybe the database engine should have some sort of protection if this
were to happen.   Maybe the database engine should automatically unfreeze the database if
the connection that freezes the database terminates/closes.   Or maybe a timer to be added
to the freeze command to automatically unfreeze the database after the fact.

I am thinking this because I was told on a previous emailing when trying to build this utility
totally from a script point of view using IJ to freeze the database, SH to perform the ZFS
snapshot and IJ to unfreeze the database that it was not expected that the freeze/unfreeze
would be done from separate connections.  I fact I ran into a problem with the utility at
that point where the IJ connection to unfreeze could not be created because the database was
frozen.

So I guess is there ever a use case that would require a database to be frozen and not unfrozen
before the connection is closed/lost?


Mime
View raw message