db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: [jira] Created: (DERBY-218) Add Relaxed Durability option
Date Wed, 13 Apr 2005 00:15:05 GMT
I agree that checkpoints should happen, I was suggesting just not sync
them, but I am not even that strong about that.  Checkpoints happen
in background so do not affect performance as much as inline syncs like
page allocation and log commit.  One can already control how often
checkpoints happen with existing properties.  Note that currently this
I/O first async writes all the data for each of the files in the cache
and then at the very end calls sync before completing the operation.

Do you think if an option is provided to stop #1 and #3, does that mean
system will gurantee a consistent bootable database - but just that some
transactions thought previously committed may not be there after failure
recovery?  If so that seems like a good first level of relaxed
durability.  I have a feeling there is a very small allocation hole that
could be fixed - need to simulate a crash with an allocation and an
insert into a table where redo finds the pages unallocated and runs out
of disk space attempting to do the allocate (easy to simulate in
debugger - hard to write reproducible test case).  The system already
pretty flexibly handles encountering file sizes that don't match the
bit map entries dynamically so it may already handle it, or it may not
be hard to increase the dynamic handling.

Also some code will have to be added as I don't think the system tracks
the difference between #1 and #2.

Suresh Thalamati wrote:
> I believe providing  mode which can cause non-bootable database to gain
> performace  is a bad  idea.
> Most of the users like me does not read manuals in detail  and will find
> hard way that the database is hosed :-)
> Say for example  I want to use Derby to store system monitoring
> information with relaxed durabilty mode.
> It   would be ok to lose some data , but it will not be acceptable if
> the database can not be booted.
> I think durabilty option#2 should not be available,  the  transaction
> log should be synced when a data page is forced to disk to
> avoid non-recoverable datbases for sure. These kind of  log syncs can 
> be reduced by just making the page cache bigger,
> and also there should be checkpoint atleast once in a while may be for a
> 100MB(just a randon number)
> or on a  boot otherwise the  disk space used by the transaction log will
> never be released.
> Thanks
> -suresh
> Mike Matrigali wrote:
>> I would also like to see the in memory implementation contributed, but
>> I think that is a different discussion.
>> From responses to Dan's original post on building a system with the
>> sync options disabled it seemed like there was enough response that
>> those options should be made available.  I admit I am worried because
>> this system can no longer guarantee recoverability.  It would be
>> interesting to know how people would use such a configuration.
>> Obviously for a database that need not last longer than a connection
>> this option would work, and in memory would probably work better.
>> For databases that last longer than a connection, what risk of
>> database corruption is an acceptable trade off to better performance?
>> This is a new idea for me, as all databases I have worked up for now
>> have not provided a less than durable option.
>> I believe the current durability options being discussed are:
>> 1) sync of the log file at each commit
>> 2) sync of the log file before data page is forced to disk
>> 3) sync of page allocation when file is grown
>> 4) sync of data writes during checkpoint
>> The simple change is just to allow each sync to be disabled, maybe
>> allowing control over all 4.
>> An interesting discussion is if there are changes that could be made
>> to make it less likely to corrupt the whole database on a JVM or machine
>> crash when the above syncs have been disabled.
>> I would have to think carefully and maybe some tests need to be
>> written, but I believe the only reasone for #3 is to insure that
>> during redo
>> crash recovery we don't run out of space.  We actually already can run
>> out of space for growing the log during undo crash recovery, but in that
>> case we just halt the boot and print that we need more space - maybe
>> something similar could be done for page allocation.  Note that the
>> change is still allocating the pages it just is not syncing before
>> inserting onto the pages.
>> #1 is probably the biggest performance win, in the case of very short
>> update transactions.  Unfortunately the jdbc standard which derby
>> implements defaults to autocommit=true - so often derby intial
>> performance results for new users looks bad when compared to other
>> databases which default to not syncing at commit time.
>> Part of this change should be to document the possible recovery
>> failures which can result by not syncing.
>> /mikem
>> Suresh Thalamati wrote:
>>> Sunitha Kambhampati (JIRA) wrote:
>>>> Add Relaxed Durability option ------------------------------
>>>>         Key: DERBY-218
>>>>         URL: http://issues.apache.org/jira/browse/DERBY-218
>>>>     Project: Derby
>>>>        Type: Improvement
>>>>  Components: Store     Versions:    Environment: all
>>>>    Reporter: Sunitha Kambhampati
>>>>     Fix For:
>>>> Dan Debrunner posted a fix to allow for relaxed durability changes
>>>> in
>>>> http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
>>>> 1) Need to add this option in Derby maybe as some property
>>>> 2) Also from discussions on the list, Mike suggested that - that the
>>>> logging system be changed to somehow
>>>> record that the database has operated in this manner, so that if
>>>> the database goes corrupt we don't waste effort trying to figure out
>>>> what when wrong.  Probably need some way to mark the log records, the
>>>> log control file and write a message to the user error log file.
>>> How about  adding  in-memory support for Derby instead of  trying to
>>> confuse the users with relaxed durabilty option.
>>> In this mode ,  will the database be  recoverable to some consistent
>>> state always with  possible loss of  some transaction data in
>>> case of a crash ?
>>>   -suresh

View raw message