db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Thalamati <suresh.thalam...@gmail.com>
Subject Re: [jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.
Date Mon, 25 Jul 2005 23:41:45 GMT
Thanks for the input , Øystein.   My comments are in-line. 
Øystein Grøvlen wrote:

>>>>>>"ST(" == Suresh Thalamati (JIRA) <derby-dev@db.apache.org> writes:
>    ST(> Currently backup contains mainly data files (seg0/*) and the
>    ST(> transaction log files(log/*) that are there when the backup
>    ST(> started. On restore from the backup, transactions are
>    ST(> replayed, similar to crash-recovery to bring the database to
>    ST(> a consistent state. New online backup also will work same
>    ST(> way, except that all the transaction log must be copied to
>    ST(> the backup, only after all the data files are backed up.
>I guess the idea is to do a log switch when the data files have been
>copied and copy all log from before this log switch.  

That's  what I plan to do, except the  the log switch will not  happen  
until  it is time to determine the last log
file that needs to go into the backup after copying the data files and 
the rest of the log files.  From the user perspective ,
Derby  online backup will include all the transactions that were 
committed the backup.  i.e  if the backup starts at
9PM and ends at 10PM,  backup will  have data approximately  until 10PM 
almost , not 9PM.

My  understanding of what goes into the backup is that ,  data in the 
backup  need not match to the
exact real-time. please  correct if this is wrong  assumption.

>One will have to
>prevent any log files that is needed from being garbage collected
>before they are copied.  I guess that can be achieved by running
>backup in a transaction and log the start of backup.  That way, the
>open backup transaction should prevent garbage-collection of relevant
>log files.

I agree it by writing log record for start of backup, we can prevent 
garbage-collection of log files.
My initial  thought is to simply disable garbage-collection  of log 
files for the duration of the backup. 
unless there are some specific advantages in writing backup-start log 

>One question I have is whether compensation log records (CLR) in Derby
>are self-contained.  If they depend on the original non-CLR, log
>records for transactions that were rolled back and terminated long
>before the backup is finished, will be needed to be able to redo the

I think CLR's are not self contained in Derby. CLR log records contain, 
log instant of  the original
operation log record that was rolled back.

>    ST(> To avoid backup process reading partial written pages, some
>    ST(> kind of synchronization mechanism that does not allow reading
>    ST(> a page to write to the back backup when the same page is
>    ST(> being written to the disk.  This can be implemented by one of
>    ST(> the following approaches:
>    ST(> a) By latching on a page key (container id, page number)
>    ST(>    while doing the write of the page from cache to disk and
>    ST(>    while reading the page from the disk/cache to write to the
>    ST(>    backup. This approach has small overhead of acquiring an
>    ST(>    extra latch during the page cache writes when the backup
>    ST(>    is in progress.
>If I understand correctly, you are here talking about not reading
>pages into the the page cache but copying them to the backup some
>other ways.  I do not think that sounds like a good idea since you
>will have to care about the pages that are already in the cache
   I agree with you.   I was thinking  along  these lines to use 
jdk1.4.2  file to file transfer  mechanisms
to have a  super-fast backup :-)

>However, I do not understand the part about an extra latch.  Don't you
>have to latch a page while it is written to disk anyhow to prevent
>concurrent updates during the write?
   Yes. page is  in latched state  when it written to the disk. But this 
latching mechanism is based
   on the  page objects in the page cache; Unless backup also goes 
through the page cache,  it can not
   latch the same  page object.

>    ST(> 2) Committed Non logged operation:
>    ST(>   Basic requirement to have consistent database backup is
>    ST(>   after the checkpoint for the backup all changes to the
>    ST(>   database will be available in the transaction log.  But
>    ST(>   Derby provides some non logged operations for performance
>    ST(>   reasons , for example CREATE INDEX , IMPORT to a empty
>    ST(>   table ..etc.
>    ST(>   This was not a issue in the old backup mechanism because no
>    ST(>   operations will be committed once the backup starts. So any
>    ST(>   non logged operations will be rolled back similar to the
>    ST(>   regular crash recovery.
>Generally, we cannot give a guarantee that operations that are
>performed during backup are reflected in the backup.  If I have
>understand correctly, transactions that commits after the data copying
>is finished, will not be reflected.  Since a user will not be able to
>distiguish between operations committed during data copying and
>operations committed during log copying, he cannot be sure concurrent
>operations is reflected in the backup.

I agree with you  that ,  one can not  absolutely  guarantee  that 
backup will include operations committed  till
a particular time are included in the backup.  But the  backup design 
depends on the
transactions log to bring the database to consistent state , because 
when data files are being
copied , it is possible that  some of the page are written to the disk.  
So we need  the
transaction log until  the data files are copied for sure.  If a user 
commits a non-logged
operation when data files are being copied , he/she would expect it to 
be there in the  backup,
similar to a logged operation.  Please note that   non-logging operation 
in Derby are not
explicit to the users,  most of  non-logging stuff is done  by the 
system without the user knowledge.

>This is not more of an issue for a new backup mechanism than it is
>currently for roll-forward recovery.  Roll-forward recovery will not be
>able recover non-logged operations either. 
Yes. Roll-forward recovery has same issues,  once the log archive mode 
that is required for roll-forward recovery
 is enabled all the operations are logged including the operations that 
are not logged normally like create index.
 But I think the currently derby does not handle correctly .  it does 
not force logging for
non-logged operations that were started before log archive mode is enabled.

> If users needs that, we
>should provide logged version of these operations.
I think,  during backup  non-logged operations should be   logged by the 
system or block them. 
If user is really concerned of performance they will not  execute them 
in parallel. 

>    ST(> 6) checkpoints when the backup is in progress.
>    ST(>    I think it not necessary to allow checkpoints when the backup is in
>    ST(>    progress. But if some one thinks otherwise , following should
>    ST(>    be addressed: 
>Is checkpoints the only way to write data pages to disk?  If yes,
>backup cannot block checkpoints since then the backup will no longer
>be non-blocking when the entire page cache becomes dirty.
     No. pages are written to the disk  by the cache manager to prevent 
cache full state. 
 I believe  checkpoint can  run in parallel to backup , if I make  a 
copy of  log control file at the 
start of  the backup and disable  log-files garbage collection.    On  
second thought, I don't
see any reason for  not allowing checkpoints during backup.



View raw message