db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject Re: Is there any debug logging available when a database is being recovered
Date Wed, 20 Apr 2016 04:43:33 GMT
We are doing a “truss -p <pid> 2>&1 | grep open” but what I am seeing right
now is that one complete pass was made through the logs and I thought that it was about done.
 Now it is going into a pattern where it processes something like

        logxxxx359.dat
        logxxxx358.dat
        …
        logxxxx242.dat

It is updating some “seg0” files (either re-doing or undoing) and then sometimes creates
“logxxxx360.dat” (ie. the next number) and then starts the process over again.   Been
doing recovery now for over 5 hours.  Ouch indeed.

Restoring from a backup is about a 5 hour process as well as the database is about 550G. 
 Just un-taring the backup file takes a substantial amount of time.

The problem occurred when an external system all of a sudden issued 128 concurrent requests
to our server where each request removes 10’s of thousands of records in a transaction.
 These all caused a gridlock on locking, lock timeouts, the cleanup messages being output
to derby-?.log (rolling log support), etc.   The system administrator forcefully shutdown
Derby and restarted.

I know the deletion of 10’s of thousands of records in a transaction is bad design.  It
did not start out at this scale (a few hundred records to start) but grew into this problem.
  The item being deleted represents a piece of network equipment and we can’t have a partial
piece of equipment laying around in the database.   Shortly this will be re-designed to mark
as deleted and then perform the deletion in the background at little chunks at a time, but
of course the issue arose before the solution is complete.

So what would be useful would be something like:

Performing database recovery
Starting analysis pass
215 transactions detected to be processed
Starting redo pass
…. anything that could give some feedback
Starting undo pass
….anything that could give some feedback





> On Apr 20, 2016, at 12:10 AM, Bryan Pendleton <bpendleton.derby@gmail.com> wrote:
>
>> Another issue with about 1100 log files needing to be
>> processed after a restart of the database network server.
>
> Ouch. :(
>
>> is any logging that can tell when pass the database recovery is on
>
> Perhaps use an operating system level monitoring tool (Process Monitor
> on Windows, strace on Linux, etc.) to see if you can watch the store
> opening and accessing each log file.
>
> I'm supposing that the recovery processing essentially reads those
> logs sequentially, so if you watch it for a while you can see how
> long it takes to finish one log and move on to the next?
>
> bryan
>


Canoga Perkins
20600 Prairie Street
Chatsworth, CA 91311
(818) 718-6300

This e-mail and any attached document(s) is confidential and is intended only for the review
of the party to whom it is addressed. If you have received this transmission in error, please
notify the sender immediately and discard the original message and any attachment(s).

Mime
View raw message