db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunitha Kambhampati <ksunitha...@gmail.com>
Subject [PATCH] Derby 218 - Add relaxed durability option
Date Fri, 29 Apr 2005 23:15:25 GMT
A little background: Sometime earlier on the list, Dan posted a fix to 
make derby go faster with relaxed durability with some flags.  The post 
is at 
http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
This mode is very useful for unit testing or at development time when 
recoverability is not required. 

Basically in this mode, syncs are disabled for logs, data writes at 
checkpoint, page allocation when file is grown; - what this means is 
that data is not flushed all the way to the disk and in most cases i/o 
cost is not involved. Note,  code already exists in Derby to do this. 

So for Derby 218, This  patch addresses the following requirements:  
1) Have a single property to enable this relaxed durability mode, so it 
is easy for users  to enable it.
2) Print message to derby.log that this mode is enabled
3) A way to report boot errors that may be because of this option being 
enabled. What this maps to is - have a marker to recognize that database 
was booted in this mode, and then on subsequent boot; if errors happen 
during recovery - report a message that it could have happened because 
of this mode being set.

Changes are not much.  
svn stat
M      java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M      
java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M      java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M      java\engine\org\apache\derby\iapi\reference\MessageId.java
M      java\engine\org\apache\derby\iapi\reference\Property.java
M      java\engine\org\apache\derby\loc\messages_en.properties
A      
java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java
M      
java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A      
java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties
A      
java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out
M      
java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall

Since this is a longish email , for those who may not  read the rest of 
it, here are the testing results

Testing results:
 - I ran derbyall with 1.4.2 jvm and all tests passed.     
 - I ran the storeall suite  separately both with setting the 
derby.system.testMode property to true and without it.  
    with this property set, the storeall suite ran about 2 times as fast.
-------------------------------

Below I explain how the patch addresses the above requirements  and try 
to provide details on how I went about making those changes

0) Note
1) A single property has been added to enable this relaxed durability 
option.  The property name is derby.system.testMode and takes in value 
true or false. By default it is false. It can be set as a command line 
option to the JVM when starting the application or within the 
application or in the derby.properites file.  Property is loaded at boot 
time, so for it to have an effect you need to enable it at boot time.  
More comments added in code itself. see variable TESTMODE_NO_SYNC =  
"derby.system.testMode"; in Property.java.

(In this patch, I have used 'no_sync' as suffix  in variable names 
because I think with respect to store - that is what this mode is all 
about - disabling syncs for the cases already mentioned earlier in this 
mail)

I chose the property name derby.system.testMode for this
- represents a higher concept as to when one would use  relaxed 
durability (no sync) option.
- mostly it should be enabled only when data is not important and 
ideally maybe for testing and development purposes,
- also with the hope that the word 'testMode'  would prompt users to 
look this property up before using it in a production like environment 
where it may not necessarily be appropriate.

Code changes for this:
- Added this new property  derby.system.testMode in
java\engine\org\apache\derby\iapi\reference\Property.java

- Removed the following properties from Property.java as they are no 
longer required.
    derby.storage.dataNotSyncedAtCheckPoint=true    ( I'll refer to this 
as DNSAC   in notes below)
    derby.storage.dataNotSyncedAtAllocation=true       ( I'll refer to 
this as DNSAA  in notes below)
    derby.storage.logNotSynced=true            (I'll refer to this as 
LNS in notes below)

- Previously code existed for enabling  the DNSAC, DNSAA and LNS options 
conditional on Performance.MEASURE

So how it works is , the 2 variables in BaseDataFileFactory.java are 
used to control if syncs happen or not for data pages at allocation and 
at checkpoint.
dataNotSyncedAtAllocation  
dataNotSyncedAtCheckpoint

And  in java/engine/org/apache/derby/impl/store/raw/log/LogToFile, the 
logNotSynced variable is used to control if syncs happen for the logs 
are not.

These variables are enabled (set to true) if Performance.MEASURE is set 
to true, and if the DNSAC, and DSNAA propertie are enabled.

So all that is required now is
- remove conditional on Performance.MEASURE in these cases, instead use 
the derby.system.testMode property to enable them . This happens in the 
boot methods
- make sure to remove any conditionals on Performance.MEASURE that is 
used to disable syncs but instead control it by the variables mentioned 
above.. ie e.g. dataNotSyncedAtAllocation in BaseDataFileFactory.java ...
- remove reading of the DNSAC, DNSAA,LNS property as they are not needed 
anymore

Changes were needed to following files -  
i) 
java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java  
(check boot method)
ii) java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java
iii) java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java   
(check boot method).

2)  Write to the derby.log a warning message

Needed to add a new message.
Messages need to be translated, so I made the following changes

i) added a variable  in 
java/engine/org/apache/derby/iapi/reference/MessageId.java  to represent 
this new message.
  String STORE_TESTMODE_NO_SYNC           = "D013";

ii) Message content added to 
java/engine/org/apache/derby/loc/messages_en.properties
D013=WARNING: {0} is set to true. In this mode, there is no guarantee 
that the system will recover a system crash or JVM termination. Enable 
this property only when there is no need for recoverability \n

iii) In the boot process, rawstore boot calls logfactory boot, so the 
first time the derby.system.testMode property is read, it is in 
rawstore, so added code to print the warning message in 
BaseDataFileFactory.boot(..)

 
3)  A way to report boot errors that may be because of this option being 
enabled. What this maps to is - have a marker to recognize that database 
was booted in this mode, and then on subsequent boot if errors happen 
during recovery report a message that it could have happened because of 
this mode being set.

- To write a marker out , log.ctrl file is used, it has a few spare 
bytes so no major changes needed here. ( ie no disk format changes.... )
 Check the method readControlFile  in  
java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java

- Used the first spare byte available in log control file which will be 
used to store information about if derby.system.testMode is set or not.
Added variable TESTMODE_NO_SYNC_FLAG_OFFSET to represent this offset in 
LogToFile.java

- The value written out if this property is enabled is  
TESTMODE_NO_SYNC_TRUE = 0x1 else a 0 is written out.  
( I didnt like to put the 0 but wasnt feeling too strong to adding a new 
static byte variable either.. and  as rest of the code around it in 
writeControlFile seems to use 0 as is , I just followed the same approach).

-  Change made to readControlFile to read in the value for the testmode 
case when Logfactory is booted, the flag is read and value stored
 in  variable called wasDBInTestModeNoSync. This variable keeps track of 
if database was previously booted in the derby.system.testMode or not.  
Comes in use for error reporting.

- Also added a message to report to the derby.log incase database was 
previously booted with derby.system.testMode set to true. Change to 
messages_en.properties and MessageId.java.
L020=WARNING: The database was previously booted with {0} set to true. 
Note that this mode {0}=true does not guarantee recoverability \n

little on the flow with respect to writeControlFile .. if you look at 
the code in LogToFile for boot and recover, writeControlFile is called a 
couple of times.. to read the checkpoint instant etc..
- For my changes, I am just copy/paste the comments from 
writeControlFile(...)  , Note case4 below .. which is the interesting one.

       // A spare byte at offset TESTMODE_NO_SYNC_FLAG_OFFSET
        // is used to write out if database was in 
derby.system.testMode=true mode
        // or not.
        // This byte will have one of the two values
        // - TESTMODE_NO_SYNC_TRUE to represent database was booted
        //   in this mode
        // -  0 to represent database was not booted in this mode
        //  
        // The sole purpose of this flag in control file is to help
        // identify any recovery problems that might happen as a result 
of this
        // mode being set.
        //
        // Possible scenarios. In table below, cases are listed giving
        // the value of derby.system.testMode if it was enabled or not 
at boot time
        // ---------------------------------------------------------------
        //  Case   Previous       Current         Action , what should 
the flag be set to in log.ctrl
        //               Boot           Boot               file
        // --------------------------------------------------------------
        //   1       -             true                write 
TESTMODE_NO_SYNC_TRUE
        //   2       -                 false           write 0
        //   3       false        false               write 0
        //   4       true          false               write 
TESTMODE_NO_SYNC_TRUE
        //   5       true          true                write 
TESTMODE_NO_SYNC_TRUE
        //
        // Case 4 is the interesting case here.
        // one cannot write out the derby.system.testMode is disabled 
just yet
        // because it is possible that the database recovery might fail 
and on subsequent
        // restarts this state would be lost. So only when recovery is 
successful, will
        // we update the flag in the log control file to indicate that 
the derby.system.testMode
        // is not enabled.  (This will happen at the end of recovery , 
see recover method)
        //
        // Case 1, 2 is when database is created fresh with this 
property enabled.
        //
        // wasDBInTestModeNoSync has information if database was 
previously booted
        // in derby.system.testMode=true mode

To handle case 4, changes went into recover () method , if database 
recovery is successful only then we update the flag in the control file 
to say database is not booted with this property enabled.  Since there 
was no method available to update the log control file, added a new 
method updateTestModeFlagInCtrlFile(byte value) to do that.

So now that changes are done to write the marker to the control file and 
to read it, it is necessary to take care of case when error happens.

- added a new message  to report that cause of an error could be related 
to this property being set
 added a new variable in  MessageId.LOG_TESTMODE_NO_SYNC_ERR in   
java/engine/org/apache/derby/iapi/reference/MessageId.java
and the message to java/engine/org/apache/derby/loc/messages_en.properties
L021=WARNING: The database was either previously booted or is currently 
booted with {0} set to true. Note that this mode {0}=true does not 
guarantee recoverability, and is probably what might have caused this 
boot error \n
- added  MessageId.LOG_TESTMODE_NO_SYNC_ERR in   
java/engine/org/apache/derby/iapi/reference/MessageId.java

Added a new method logErrMsgForTestModeNoSync() in LogToFile.java to 
print out this message if any error happens when database is in the 
influence of this property.

In deciding what the best place would be to call the 
logErrMsgForTestModeNoSync, I debated between adding it in markCorrupt() 
versus logErrMsg(..).  It seemed like if error happened during boot - 
the flow was to call markCorrupt which would call logErrMsg.  Some error 
messages seem to call logErrMsg(..) directly.   So I picked the 
logErrMsg as it covered most of the errors thrown during boot/recovery 
that would be related to something with this property being set.  
-------------------------------------------

TESTING

1) I added one new  java file  per the derby test harness,   to test the 
following
  -  check if the property is enabled correctly by timing some inserts 
into a table.
  -  check if the correct flag (TESTMODE_NO_SYNC_TRUE) is written out 
into the log.ctrl file
  -  check if the derby.log has the warning message.

 Changes include
i)Added new test to   
java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
ii)Added 
java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties

TestNoSyncMode_derby.properties has
derby.system.testMode=true

This property file is needed so when test is run it picks up this property.

iii)The TestNoSyncMode_derby.properties is added to copyfiles.ant so it 
will be copied over when compiling derby tests.

iv) To enable this new  test as part of the store suite, I modified the
java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall
added store/TestNoSyncMode.java

v) To get my correct output file, first i just put an initial empty file 
in 
java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out

Ran the test  after compiling source;  java 
org.apache.derbyTesting.functionTests.harness.RunTest 
store/TestNoSyncMode.java
Then I updated the out file in the master directory, after checking if 
the TestNoSyncMode.out file was correct.

2) For the error messages on a boot error, it is difficult to put it in 
a test, so I checked this in the debugger and it works ok.

Meanwhile, Suresh Thalamati posted on the list that he is working on a 
corruptible storage factory, once that is there - I am hoping I could 
use that to have a test to check the error messages for boot errors.

As far as I can tell, this does not affect upgrade and such since we are 
not changing the disk format etc.. ..There is one spare byte  in 
log.ctrl file that is being used now as a result of this change  and 
this byte value should not have been used to represent anything useful 
by any earlier version

Testing results (already  mentioned in the beginning of this email).

Please give your comments.  

Thanks,
Sunitha.

Mime
View raw message