db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: Can anyone give me some suggestions?
Date Thu, 18 Aug 2005 21:24:34 GMT
Could you define "fuzzy" checkpoint or site what update interference
you are trying to avoid.

Derby does the following (suresh let me know if I am off here):

    o loops once around the page cache and queues async writes of
      of every dirty page.  Note that on all the JVM's/OS's I have
      looked at this write just copies the data to OS buffer so is
      fast - thus the latch on the page is not held very long.
    o At end syncs all writes to each table affected using single jvm
      call per file.  These syncs may take awhile, but since they
      happen at the end, many if not all of the writes have already
      hit disk so usually there is nothing to wait for.
    o logs the checkpoint record

As Oystein says I think the main reason to tune checkpoint interval is
some sort of balance between recovery time and performance overhead of
the checkpoint.  The last change in this area bumped the log file size
from 100k to 1 meg, and the checkpoint size from 1 log file to 10 log
files.  Even this change makes us do many more checkpoints than
competing databases, especially in benchmark situations when other dbs
don't do any checkpoints.  Some time based approach may be interesting,
with inputs of how long does a checkpoint take and how long since the
last checkpoint and total log volume rate. There is also the log file
garbage collection issue.


I don't think checkpointing is the way to control # dirty pages in the
cache.  The current system tries to do this in the clock algorithm, by
having a background thread "clean" dirty pages when it thinks there
are too many.  This buffer cache, clock algorithm is pretty simple so
there may be improvements possible there - I believe some have already
been discussed on the list with respect to weighting various pages.  It
is also interesting in that it should be possible to create a separate
implementation of it as a separate module and one could pick an choose
and easily compare performance differences.  Again the interesting
problem is to maintain as many dirty pages as possible while while
maintaining enough clean pages such that a user transaction never has
to wait for a write to get a buffer.

Note that log switching and checkpoints are 2 orthogonal issues.
Historically the code has done checkpoints at the same time as
log switches, but there is not a requirement.  The reason was mostly
that log garbage collection is done at the log file level in the
current log implementation.  Be careful making log files too big, as
there are some JVM's/OS's where the sync performance is linearly
dependent on the size of the file.  I don't believe this is true
of the default syncing used on the log file, but is true in the
pre-1.4.1 log file sync implementation.

Raymond Raymond wrote:

> Oystein.Grovlen,Thanks your for your suggestions. It is exactly what I
> am thinking
> about.I am considering two aspects of the checkpointing issue.
> 1. How to make the engine tune the interval of checkpointing by itsself.
> I think
> It depends on the database buffer size, log buffer size and how many
> dirty pages in the
> database buffer. And you give me a good suggestion about the machine
> performance
> factor. I will take that into account.
> 2. Although the derby implemeted ARIES algorithsm in its recovery
> function, it did not
> adopt fuzzy checkpointing. The current checkpointing approach is not
> very efficient, just
> as what you said, it will interfere with updates requires from other
> transactions. I am
> trying to find a better way to do that.
> 
> Anyone else has any good ideas about that?^_^.
> 
> 
> 
> Raymond
> 
> 
>> From: Oystein.Grovlen@Sun.COM (Øystein Grøvlen)
>>
>> Currently, you can configure the checkpoint interval and log file size
>> of Derby by setting the properties:
>>
>> derby.storage.logSwitchInterval  (default 1 MB)
>> derby.storage.checkpointInterval (default 10 MB)
>>
>> (None of these seems to be documented in the manuals, and the JavaDoc
>> for LogFactory.recover() gives wrong (out-dated?) defaults).
>>
>> This means that by default all log files will be 1 MB, and a checkpoint
>> is made for every tenth log file.
>>
>> In order to know when it is useful to change the defaults, one has to
>> consider the purpose of a checkpoint:
>>
>>   1) Reduced recovery times.  Only log create after the penultimate
>>      checkpoint needs to be redone at recovery.  This also means that
>>      older log files may be garbage-collected (as long as they do not
>>      contain log records for transactions that are still not
>>      terminated.)
>>
>>      To get short recovery times, one should keep the checkpoint
>>      interval low.  The trade-off is that frequent checkpoints will
>>      increase I/O since you will have less updates to the same page
>>      between two checkpoints.  Hence, you will get more I/O per
>>      database operation.
>>
>>   2) Flush dirty pages to disk.  A checkpoint is a much more efficient
>>      way to clean dirty pages in the db cache than to do it on demand
>>      on a single page when one need to replace it with another.
>>      Hence, one should make sure to do checkpoints often enough to
>>      avoid that the whole cache is dirty.
>>
>> Based on 2), one could initiate a new checkpoint when many pages in
>> the cache are dirty (e.g., 50% of the pages) and postpone a checkpoint
>> if few pages are dirty.  The difficult part would be to determine how
>> long checkpoint intervals is acceptable with respect to impact on
>> recovery times.
>>
>> I guess one could argue that for recovery times, it is the clock time
>> that matters.  Hence, one could automatically increase the value of
>> derby.storage.checkpointInterval on more performant computers since it
>> will be able to process more log per time unit.
>>
>> When would want to change the log switch interval?  I think few would
>> care, but since the log files per default are preallocated, space will
>> be wasted if operations that perform a log switch (e.g., backup) is
>> performed when the current log file is nearly empty.  On the other
>> hand, a small log file size will result many concurrent log files if
>> the checkpoint interval is very large.
>>
>> Hope this helps a little,
>>
>> -- 
>> Øystein
>>
> 
> _________________________________________________________________
> Scan and help eliminate destructive viruses from your inbound and
> outbound e-mail and attachments.
> http://join.msn.com/?pgmarket=en-ca&page=byoa/prem&xAPID=1994&DI=1034&SU=http://hotmail.com/enca&HL=Market_MSNIS_Taglines
>  Start enjoying all the benefits of MSN® Premium right now and get the
> first two months FREE*.
> 
> 

Mime
View raw message