db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oystein.Grov...@Sun.COM (Øystein Grøvlen)
Subject Re: [jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Date Mon, 25 Jul 2005 14:32:50 GMT


I have some questions/comments.  Some of them probably reflects that I
have not yet a detailed understanding of all Store mechanisms.
However, I have earlier participated in the implementation of
non-blocking backup in another RDBMS so I have some knowledge of the


>>>>> "ST(" == Suresh Thalamati (JIRA) <derby-dev@db.apache.org> writes:

    ST(> Currently backup contains mainly data files (seg0/*) and the
    ST(> transaction log files(log/*) that are there when the backup
    ST(> started. On restore from the backup, transactions are
    ST(> replayed, similar to crash-recovery to bring the database to
    ST(> a consistent state. New online backup also will work same
    ST(> way, except that all the transaction log must be copied to
    ST(> the backup, only after all the data files are backed up.

I guess the idea is to do a log switch when the data files have been
copied and copy all log from before this log switch.  One will have to
prevent any log files that is needed from being garbage collected
before they are copied.  I guess that can be achieved by running
backup in a transaction and log the start of backup.  That way, the
open backup transaction should prevent garbage-collection of relevant
log files.

One question I have is whether compensation log records (CLR) in Derby
are self-contained.  If they depend on the original non-CLR, log
records for transactions that were rolled back and terminated long
before the backup is finished, will be needed to be able to redo the

    ST(> To avoid backup process reading partial written pages, some
    ST(> kind of synchronization mechanism that does not allow reading
    ST(> a page to write to the back backup when the same page is
    ST(> being written to the disk.  This can be implemented by one of
    ST(> the following approaches:

    ST(> a) By latching on a page key (container id, page number)
    ST(>    while doing the write of the page from cache to disk and
    ST(>    while reading the page from the disk/cache to write to the
    ST(>    backup. This approach has small overhead of acquiring an
    ST(>    extra latch during the page cache writes when the backup
    ST(>    is in progress.

If I understand correctly, you are here talking about not reading
pages into the the page cache but copying them to the backup some
other ways.  I do not think that sounds like a good idea since you
will have to care about the pages that are already in the cache

However, I do not understand the part about an extra latch.  Don't you
have to latch a page while it is written to disk anyhow to prevent
concurrent updates during the write?

    ST(> or 

    ST(> b) read each pages in to the page cache first and then latch
    ST(>    the page in the cache until a temporary copy of it is
    ST(>    made. This approach does not have extra overhead of extra
    ST(>    latches on the page keys during writes , but will pollute
    ST(>    the page cache with the pages that are only required by
    ST(>    the backup; this might have impact on user operations
    ST(>    because active user pages may have been replaced by the
    ST(>    backup pages in the page cache.

    ST(> or 

    ST(> c) read pages into buffer pool and latch them while making a
    ST(>    copy similar to the above approach, but some how make sure
    ST(>    that user pages are not kicked out of the buffer pool.

I think that page cache contention is a separate issue.  This is not
just relevant to backup, but to any sequential table scans (e.g.,
select *).  

    ST(> 2) Committed Non logged operation:
    ST(>   Basic requirement to have consistent database backup is
    ST(>   after the checkpoint for the backup all changes to the
    ST(>   database will be available in the transaction log.  But
    ST(>   Derby provides some non logged operations for performance
    ST(>   reasons , for example CREATE INDEX , IMPORT to a empty
    ST(>   table ..etc.
    ST(>   This was not a issue in the old backup mechanism because no
    ST(>   operations will be committed once the backup starts. So any
    ST(>   non logged operations will be rolled back similar to the
    ST(>   regular crash recovery.

Generally, we cannot give a guarantee that operations that are
performed during backup are reflected in the backup.  If I have
understand correctly, transactions that commits after the data copying
is finished, will not be reflected.  Since a user will not be able to
distiguish between operations committed during data copying and
operations committed during log copying, he cannot be sure concurrent
operations is reflected in the backup.

This is not more of an issue for a new backup mechanism than it is
currently for roll-forward recovery.  Roll-forward recovery will not be
able recover non-logged operations either.  If users needs that, we
should provide logged version of these operations.

    ST(> 6) checkpoints when the backup is in progress.

    ST(>    I think it not necessary to allow checkpoints when the backup is in
    ST(>    progress. But if some one thinks otherwise , following should
    ST(>    be addressed: 

Is checkpoints the only way to write data pages to disk?  If yes,
backup cannot block checkpoints since then the backup will no longer
be non-blocking when the entire page cache becomes dirty.


View raw message