Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Development" <derby-dev@db.apache.org>
Received-SPF: pass (hermes.apache.org: domain of ksunithaghm@gmail.com
 designates 64.233.170.201 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:user-agent:x-accept-language:mime-version:to:subject:content-type;
        b=E9E6HJPOn0Gb7p+ayrJPGL/b21Wye9ll3CmUCd9YIdsPACh3xtpdS69D2LkWkRlq+9jJVcpIUSR6ky6OB08bTsV/5r4YbLsIGrPEzQMRxwewMtUPXIXRWFzycyYnU2CtSJ9r1iOhtuTbnUVt9o9zX+6DzKqk/mmcwrd/ty26Lik=
Message-ID: <4272C00D.5050506@gmail.com>
Date: Fri, 29 Apr 2005 16:15:25 -0700
From: Sunitha Kambhampati <ksunithaghm@gmail.com>
User-Agent: Mozilla Thunderbird 0.7.3 (Windows/20040803)
MIME-Version: 1.0
To: Derby Development <derby-dev@db.apache.org>
Subject: [PATCH] Derby 218 - Add relaxed durability option 
Content-Type: multipart/mixed;
 boundary="------------090209020407060702020106"

This is a multi-part message in MIME format.
--------------090209020407060702020106
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

A little background: Sometime earlier on the list, Dan posted a fix to 
make derby go faster with relaxed durability with some flags.  The post 
is at 
http://article.gmane.org/gmane.comp.apache.db.derby.user/681/match=relaxed+durability
This mode is very useful for unit testing or at development time when 
recoverability is not required. 

Basically in this mode, syncs are disabled for logs, data writes at 
checkpoint, page allocation when file is grown; - what this means is 
that data is not flushed all the way to the disk and in most cases i/o 
cost is not involved. Note,  code already exists in Derby to do this. 

So for Derby 218, This  patch addresses the following requirements:  
1) Have a single property to enable this relaxed durability mode, so it 
is easy for users  to enable it.
2) Print message to derby.log that this mode is enabled
3) A way to report boot errors that may be because of this option being 
enabled. What this maps to is - have a marker to recognize that database 
was booted in this mode, and then on subsequent boot; if errors happen 
during recovery - report a message that it could have happened because 
of this mode being set.

Changes are not much.  
svn stat
M      java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
M      
java\engine\org\apache\derby\impl\store\raw\data\BaseDataFileFactory.java
M      java\engine\org\apache\derby\impl\store\raw\data\RAFContainer.java
M      java\engine\org\apache\derby\iapi\reference\MessageId.java
M      java\engine\org\apache\derby\iapi\reference\Property.java
M      java\engine\org\apache\derby\loc\messages_en.properties
A      
java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode.java
M      
java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant
A      
java\testing\org\apache\derbyTesting\functionTests\tests\store\TestNoSyncMode_derby.properties
A      
java\testing\org\apache\derbyTesting\functionTests\master\TestNoSyncMode.out
M      
java\testing\org\apache\derbyTesting\functionTests\suites\storemats.runall

Since this is a longish email , for those who may not  read the rest of 
it, here are the testing results

Testing results:
 - I ran derbyall with 1.4.2 jvm and all tests passed.     
 - I ran the storeall suite  separately both with setting the 
derby.system.testMode property to true and without it.  
    with this property set, the storeall suite ran about 2 times as fast.
-------------------------------

Below I explain how the patch addresses the above requirements  and try 
to provide details on how I went about making those changes

0) Note
1) A single property has been added to enable this relaxed durability 
option.  The property name is derby.system.testMode and takes in value 
true or false. By default it is false. It can be set as a command line 
option to the JVM when starting the application or within the 
application or in the derby.properites file.  Property is loaded at boot 
time, so for it to have an effect you need to enable it at boot time.  
More comments added in code itself. see variable TESTMODE_NO_SYNC =  
"derby.system.testMode"; in Property.java.

(In this patch, I have used 'no_sync' as suffix  in variable names 
because I think with respect to store - that is what this mode is all 
about - disabling syncs for the cases already mentioned earlier in this 
mail)

I chose the property name derby.system.testMode for this
- represents a higher concept as to when one would use  relaxed 
durability (no sync) option.
- mostly it should be enabled only when data is not important and 
ideally maybe for testing and development purposes,
- also with the hope that the word 'testMode'  would prompt users to 
look this property up before using it in a production like environment 
where it may not necessarily be appropriate.

Code changes for this:
- Added this new property  derby.system.testMode in
java\engine\org\apache\derby\iapi\reference\Property.java

- Removed the following properties from Property.java as they are no 
longer required.
    derby.storage.dataNotSyncedAtCheckPoint=true    ( I'll refer to this 
as DNSAC   in notes below)
    derby.storage.dataNotSyncedAtAllocation=true       ( I'll refer to 
this as DNSAA  in notes below)
    derby.storage.logNotSynced=true            (I'll refer to this as 
LNS in notes below)

- Previously code existed for enabling  the DNSAC, DNSAA and LNS options 
conditional on Performance.MEASURE

So how it works is , the 2 variables in BaseDataFileFactory.java are 
used to control if syncs happen or not for data pages at allocation and 
at checkpoint.
dataNotSyncedAtAllocation  
dataNotSyncedAtCheckpoint

And  in java/engine/org/apache/derby/impl/store/raw/log/LogToFile, the 
logNotSynced variable is used to control if syncs happen for the logs 
are not.

These variables are enabled (set to true) if Performance.MEASURE is set 
to true, and if the DNSAC, and DSNAA propertie are enabled.

So all that is required now is
- remove conditional on Performance.MEASURE in these cases, instead use 
the derby.system.testMode property to enable them . This happens in the 
boot methods
- make sure to remove any conditionals on Performance.MEASURE that is 
used to disable syncs but instead control it by the variables mentioned 
above.. ie e.g. dataNotSyncedAtAllocation in BaseDataFileFactory.java ...
- remove reading of the DNSAC, DNSAA,LNS property as they are not needed 
anymore

Changes were needed to following files -  
i) 
java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java  
(check boot method)
ii) java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java
iii) java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java   
(check boot method).

2)  Write to the derby.log a warning message

Needed to add a new message.
Messages need to be translated, so I made the following changes

i) added a variable  in 
java/engine/org/apache/derby/iapi/reference/MessageId.java  to represent 
this new message.
  String STORE_TESTMODE_NO_SYNC           = "D013";

ii) Message content added to 
java/engine/org/apache/derby/loc/messages_en.properties
D013=WARNING: {0} is set to true. In this mode, there is no guarantee 
that the system will recover a system crash or JVM termination. Enable 
this property only when there is no need for recoverability \n

iii) In the boot process, rawstore boot calls logfactory boot, so the 
first time the derby.system.testMode property is read, it is in 
rawstore, so added code to print the warning message in 
BaseDataFileFactory.boot(..)

 
3)  A way to report boot errors that may be because of this option being 
enabled. What this maps to is - have a marker to recognize that database 
was booted in this mode, and then on subsequent boot if errors happen 
during recovery report a message that it could have happened because of 
this mode being set.

- To write a marker out , log.ctrl file is used, it has a few spare 
bytes so no major changes needed here. ( ie no disk format changes.... )
 Check the method readControlFile  in  
java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java

- Used the first spare byte available in log control file which will be 
used to store information about if derby.system.testMode is set or not.
Added variable TESTMODE_NO_SYNC_FLAG_OFFSET to represent this offset in 
LogToFile.java

- The value written out if this property is enabled is  
TESTMODE_NO_SYNC_TRUE = 0x1 else a 0 is written out.  
( I didnt like to put the 0 but wasnt feeling too strong to adding a new 
static byte variable either.. and  as rest of the code around it in 
writeControlFile seems to use 0 as is , I just followed the same approach).

-  Change made to readControlFile to read in the value for the testmode 
case when Logfactory is booted, the flag is read and value stored
 in  variable called wasDBInTestModeNoSync. This variable keeps track of 
if database was previously booted in the derby.system.testMode or not.  
Comes in use for error reporting.

- Also added a message to report to the derby.log incase database was 
previously booted with derby.system.testMode set to true. Change to 
messages_en.properties and MessageId.java.
L020=WARNING: The database was previously booted with {0} set to true. 
Note that this mode {0}=true does not guarantee recoverability \n

little on the flow with respect to writeControlFile .. if you look at 
the code in LogToFile for boot and recover, writeControlFile is called a 
couple of times.. to read the checkpoint instant etc..
- For my changes, I am just copy/paste the comments from 
writeControlFile(...)  , Note case4 below .. which is the interesting one.

       // A spare byte at offset TESTMODE_NO_SYNC_FLAG_OFFSET
        // is used to write out if database was in 
derby.system.testMode=true mode
        // or not.
        // This byte will have one of the two values
        // - TESTMODE_NO_SYNC_TRUE to represent database was booted
        //   in this mode
        // -  0 to represent database was not booted in this mode
        //  
        // The sole purpose of this flag in control file is to help
        // identify any recovery problems that might happen as a result 
of this
        // mode being set.
        //
        // Possible scenarios. In table below, cases are listed giving
        // the value of derby.system.testMode if it was enabled or not 
at boot time
        // ---------------------------------------------------------------
        //  Case   Previous       Current         Action , what should 
the flag be set to in log.ctrl
        //               Boot           Boot               file
        // --------------------------------------------------------------
        //   1       -             true                write 
TESTMODE_NO_SYNC_TRUE
        //   2       -                 false           write 0
        //   3       false        false               write 0
        //   4       true          false               write 
TESTMODE_NO_SYNC_TRUE
        //   5       true          true                write 
TESTMODE_NO_SYNC_TRUE
        //
        // Case 4 is the interesting case here.
        // one cannot write out the derby.system.testMode is disabled 
just yet
        // because it is possible that the database recovery might fail 
and on subsequent
        // restarts this state would be lost. So only when recovery is 
successful, will
        // we update the flag in the log control file to indicate that 
the derby.system.testMode
        // is not enabled.  (This will happen at the end of recovery , 
see recover method)
        //
        // Case 1, 2 is when database is created fresh with this 
property enabled.
        //
        // wasDBInTestModeNoSync has information if database was 
previously booted
        // in derby.system.testMode=true mode

To handle case 4, changes went into recover () method , if database 
recovery is successful only then we update the flag in the control file 
to say database is not booted with this property enabled.  Since there 
was no method available to update the log control file, added a new 
method updateTestModeFlagInCtrlFile(byte value) to do that.

So now that changes are done to write the marker to the control file and 
to read it, it is necessary to take care of case when error happens.

- added a new message  to report that cause of an error could be related 
to this property being set
 added a new variable in  MessageId.LOG_TESTMODE_NO_SYNC_ERR in   
java/engine/org/apache/derby/iapi/reference/MessageId.java
and the message to java/engine/org/apache/derby/loc/messages_en.properties
L021=WARNING: The database was either previously booted or is currently 
booted with {0} set to true. Note that this mode {0}=true does not 
guarantee recoverability, and is probably what might have caused this 
boot error \n
- added  MessageId.LOG_TESTMODE_NO_SYNC_ERR in   
java/engine/org/apache/derby/iapi/reference/MessageId.java

Added a new method logErrMsgForTestModeNoSync() in LogToFile.java to 
print out this message if any error happens when database is in the 
influence of this property.

In deciding what the best place would be to call the 
logErrMsgForTestModeNoSync, I debated between adding it in markCorrupt() 
versus logErrMsg(..).  It seemed like if error happened during boot - 
the flow was to call markCorrupt which would call logErrMsg.  Some error 
messages seem to call logErrMsg(..) directly.   So I picked the 
logErrMsg as it covered most of the errors thrown during boot/recovery 
that would be related to something with this property being set.  
-------------------------------------------

TESTING

1) I added one new  java file  per the derby test harness,   to test the 
following
  -  check if the property is enabled correctly by timing some inserts 
into a table.
  -  check if the correct flag (TESTMODE_NO_SYNC_TRUE) is written out 
into the log.ctrl file
  -  check if the derby.log has the warning message.

 Changes include
i)Added new test to   
java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
ii)Added 
java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties

TestNoSyncMode_derby.properties has
derby.system.testMode=true

This property file is needed so when test is run it picks up this property.

iii)The TestNoSyncMode_derby.properties is added to copyfiles.ant so it 
will be copied over when compiling derby tests.

iv) To enable this new  test as part of the store suite, I modified the
java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall
added store/TestNoSyncMode.java

v) To get my correct output file, first i just put an initial empty file 
in 
java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out

Ran the test  after compiling source;  java 
org.apache.derbyTesting.functionTests.harness.RunTest 
store/TestNoSyncMode.java
Then I updated the out file in the master directory, after checking if 
the TestNoSyncMode.out file was correct.

2) For the error messages on a boot error, it is difficult to put it in 
a test, so I checked this in the debugger and it works ok.

Meanwhile, Suresh Thalamati posted on the list that he is working on a 
corruptible storage factory, once that is there - I am hoping I could 
use that to have a test to check the error messages for boot errors.

As far as I can tell, this does not affect upgrade and such since we are 
not changing the disk format etc.. ..There is one spare byte  in 
log.ctrl file that is being used now as a result of this change  and 
this byte value should not have been used to represent anything useful 
by any earlier version

Testing results (already  mentioned in the beginning of this email).

Please give your comments.  

Thanks,
Sunitha.

--------------090209020407060702020106
Content-Type: text/plain;
 name="DerbyTestMode_Patch.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="DerbyTestMode_Patch.txt"

Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java	(revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java	(working copy)
@@ -265,7 +265,38 @@
 
 	/* Log Control file flags. */
 	private static final byte IS_BETA_FLAG = 0x1;
-
+	
+    /**
+     * When the derby.system.testMode property is enabled, the store 
+     * system will not force sync calls in the following cases
+     * - for the log file at each commit
+     * - for the log file before data page is forced to disk
+     * - for page allocation when file is grown
+     * - for data writes during checkpoint
+     * This means the recovery system may not work properly
+     * In order that we recognize this case that the database was previously
+     * booted in this mode, this value is written out
+     * into the log control file. This will help prevent us from 
+     * wasting time to resolve issues in such cases. 
+     * @see org.apache.derby.iapi.reference.Property#TESTMODE_NO_SYNC
+     */
+    private static final byte TESTMODE_NO_SYNC_TRUE = 0x1;
+	
+    /**
+     * A byte in the log control file is used to store information
+     * about if the database is booted in derby.system.testMode mode
+     * The first available spare byte that is available for this purpose is 
+     * after 5 ints ( 4bytes each), a long ( 8 bytes) and 1 byte in the log control
+     * file. Hence the offset is calculated as follows
+     */
+    private static final int TESTMODE_NO_SYNC_FLAG_OFFSET =  5*4 + 8 + 1;
+    
+    /**
+     * keeps track of if the database was booted previously with 
+     * derby.system.testMode=true 
+     */
+    private static boolean wasDBInTestModeNoSync = false;
+    
 	/* to err on the conservative side, unless otherwise set, assume log
 	 *	archive is ON 
 	 */
@@ -362,7 +393,7 @@
 	// log goes to the log subdirectory underneath the data directory
 	String logDevice;
 
-    // debug only flag - disable syncing of log file for debugging performance.
+    // disable syncing of log file when running in derby.system.testMode enabled
     private boolean logNotSynced = false;
 
 	private boolean logArchived = false;
@@ -403,6 +434,7 @@
 	
 	private CRC32 checksum = new CRC32(); // holder for the checksum
 
+    protected static int CHECKSUM_SIZE = 8; // checksum size in bytes
 
 	
 	/**
@@ -1194,8 +1226,28 @@
 			tf.resetTranId();
 		}
 
-		// done with recovery
-
+        // If we have reached here successfully, that means recovery 
+        // was successful
+        
+        // Now that database has recovered successfully and if it
+        // is booted with derby.system.testMode=false mode, there is no
+        // need to store information that database was booted previously in 
+        // derby.system.testMode=true mode. 
+        // It is safe to update the marker in log.ctrl file to say
+        // that the derby.system.testMode in not being used
+   
+        // Need to write to the log.ctrl file as well as the log mirror file
+	if( wasDBInTestModeNoSync && !logNotSynced )
+        {
+	    // write to log control file and the mirror file that the testmode 
+            // is not used now
+            if(!updateTestModeFlagInCtrlFile((byte)0))
+                throw StandardException.newException(
+                    SQLState.LOG_CONTROL_FILE, getControlFileName());
+           
+            wasDBInTestModeNoSync = false;
+        } 
+        
 		/////////////////////////////////////////////////////////////
 		// setup checktpoint daemon
 		/////////////////////////////////////////////////////////////
@@ -1207,6 +1259,99 @@
         }
 	}
 
+    /**
+     * Update the flag that represents whether 
+     * derby.system.testMode is enabled or not in both the log control file
+     * as well as the mirror control file
+     * <BR>
+     * Basically, Read the control file, 
+     * update the flag to the new passed in value 
+     * recalculate the checksum and write it out to the log control file
+     * <P> MT- synchronized by caller
+     * @param value to set
+     * @return false if the update didnt happen successfully, else true
+     * @throws IOException
+     * @throws StandardException
+     * @see #readControlFile
+     * @see #writeControlFile(StorageFile, long)
+     */
+    private boolean updateTestModeFlagInCtrlFile(byte value)
+    {
+        StorageRandomAccessFile logControlFile = null;
+        byte barray[] = null;
+        StorageFile ctrlFile = null;
+        
+        try
+        {
+            ctrlFile = getControlFileName();
+            logControlFile = privRandomAccessFile(ctrlFile, "rw");
+            if (!privCanWrite(ctrlFile))
+                return false;
+        
+            // retrieve all except the checksum bytes
+            barray = new byte[(int) logControlFile.length() - CHECKSUM_SIZE];
+            logControlFile.readFully(barray);
+               
+            if (SanityManager.DEBUG)
+            {
+                if (SanityManager.DEBUG_ON(DBG_FLAG))
+                {
+                    SanityManager.DEBUG(DBG_FLAG, 
+                        "Updating derby.system.testMode marker written out"+
+                        "to log control file: prev value =  " +
+                        barray[TESTMODE_NO_SYNC_FLAG_OFFSET] + " to "+ value); 
+                }
+            }
+            // update the byte at position that represents whether 
+            // derby.system.testMode was used
+            barray[TESTMODE_NO_SYNC_FLAG_OFFSET] = value;
+            
+            // write back into file the updated value and also compute 
+            // the newchecksum and write it to file
+            logControlFile.seek(0);
+            logControlFile.write(barray);
+            checksum.reset();
+            checksum.update(barray, 0, barray.length);
+            logControlFile.writeLong(checksum.getValue());
+            syncFile(logControlFile);
+            logControlFile.close();
+
+            // write the same data to mirror control file
+            ctrlFile = getMirrorControlFileName();
+            logControlFile =
+                    privRandomAccessFile(ctrlFile, "rw");
+            if (!privCanWrite(ctrlFile))
+                return false;
+            logControlFile.seek(0);
+            logControlFile.write(barray);
+            logControlFile.writeLong(checksum.getValue());
+            syncFile(logControlFile);
+        }
+        catch(IOException ioe)
+        {
+            logControlFile = null;
+            return false;
+        }
+        catch(StandardException se)
+        {
+       	    logControlFile = null;
+            return false;
+        }
+        finally
+        {
+           try
+           { 
+               if (logControlFile != null)
+                   logControlFile.close();
+           }
+           catch(IOException ioe2)
+           {
+               logControlFile = null;
+               return false;
+           }
+        }
+        return true;
+    }
 	/**
 		Checkpoint the rawStore.
 
@@ -2142,10 +2287,61 @@
 
 		//
 		// write some spare bytes after 2.0 we have 3 + 2(8) spare bytes.
+        // now with v10.1.0.0 we have 2 +2(8) spare bytes as one of the spare
+        // byte is used as a marker to write out if system is runinng
+        // in derby.system.testMode=true or not.
+        
 		long spare = 0;
+        
+        // A spare byte at offset TESTMODE_NO_SYNC_FLAG_OFFSET
+        // is used to write out if database was in derby.system.testMode=true mode
+        // or not. 
+        // This byte will have one of the two values
+        // - TESTMODE_NO_SYNC_TRUE to represent database was booted
+        //   in this mode 
+        // -  0 to represent database was not booted in this mode
+        //  
+        // The sole purpose of this flag in control file is to help
+        // identify any recovery problems that might happen as a result of this
+        // mode being set. 
+        // 
+        // Possible scenarios. In table below, cases are listed giving 
+        // the value of derby.system.testMode if it was enabled or not at boot time
+        // ---------------------------------------------------------------
+        //  Case   Previous   Current     Action , what should the flag be set to in log.ctrl 
+        //          Boot       Boot       file
+        // --------------------------------------------------------------
+        //   1       -         true        write TESTMODE_NO_SYNC_TRUE 
+        //   2       -         false       write 0 
+        //   3       false     false       write 0
+        //   4       true      false       write TESTMODE_NO_SYNC_TRUE 
+        //   5       true      true        write TESTMODE_NO_SYNC_TRUE
+        // 
+        // Case 4 is the interesting case here.
+        // one cannot write out the derby.system.testMode is disabled just yet
+        // because it is possible that the database recovery might fail and on subsequent
+        // restarts this state would be lost. So only when recovery is successful, will 
+        // we update the flag in the log control file to indicate that the derby.system.testMode
+        // is not enabled.  (This will happen at the end of recovery , see recover method) 
+        // 
+        // Case 1, 2 is when database is created fresh with this property enabled.
+        // 
+        // wasDBInTestModeNoSync has information if database was previously booted
+        // in derby.system.testMode=true mode
+        if( !wasDBInTestModeNoSync ) // derby.system.testMode=false
+        {
+            // in this case, no need to write out any special marker
+            // for derby.system.testMode=false case.
+            if(!logNotSynced)
+                daos.writeByte(0);  
+            else
+	        daos.writeByte(TESTMODE_NO_SYNC_TRUE); 
+        }
+        else
+            daos.writeByte(TESTMODE_NO_SYNC_TRUE);
+        
 		daos.writeByte(0);
 		daos.writeByte(0);
-		daos.writeByte(0);
 		daos.writeLong(spare);
 		daos.flush();
 		// write the checksum for the control data written
@@ -2281,7 +2477,24 @@
 				onDiskMinorVersion = dais.readInt();
 				int dbBuildNumber = dais.readInt();
 				int flags = dais.readByte();
-
+				
+		// check if the database was running previously with
+		// derby.system.testMode=true mode
+                // if database was running previously in derby.system.testMode=true
+                // and if on a restart - recovery fails we want to be able
+                // to report that the boot error could have happened
+                // because of this property
+                wasDBInTestModeNoSync =(dais.readByte() & TESTMODE_NO_SYNC_TRUE) != 0;
+                
+                if (SanityManager.DEBUG)
+                {
+                    if (SanityManager.DEBUG_ON(LogToFile.DBG_FLAG))
+                        SanityManager.DEBUG(LogToFile.DBG_FLAG, 
+                            "log control file, was derby.system.testMode enabled = " + 
+                            wasDBInTestModeNoSync);
+                }
+                    
+				
 				onDiskBeta = (flags & IS_BETA_FLAG) != 0;
 
 				if (onDiskBeta)
@@ -2794,19 +3007,15 @@
 			isWriteSynced = false;
 		}
 
-		if (Performance.MEASURE)
+		// if system is running with derby.system.testMode enabled
+                // log records and log file are not synced, so 
+		// set flag to disable log syncs
+		if (PropertyUtil.getSystemBoolean(Property.TESTMODE_NO_SYNC))
 		{
-			// debug only flag - disable syncing of log.
-			logNotSynced = 
-				PropertyUtil.getSystemBoolean(Property.STORAGE_LOG_NOT_SYNCED);
-
-			if (logNotSynced)
-			{
-				Monitor.logMessage("logNotSynced = true");
-				//if log is Not being synced;files should not be open in write sync mode
-				isWriteSynced = false;
-			}
-			
+		    // disable syncing of log.
+		    logNotSynced = true;
+  		    //if log is Not being synced;files should not be open in write sync mode
+		    isWriteSynced = false;	
 		}
 
 		// try to access the log
@@ -2830,6 +3039,18 @@
                 if (privExists(logControlFileName))
 				{
 					checkpointInstant = readControlFile(logControlFileName, startParams);
+					// in case system was running previously with derby.system.testMode=true
+                    // then print a message to the derby log
+                    if(wasDBInTestModeNoSync)
+		    {
+		        // print message stating that the database was
+			// previously running with derby.system.testMode=true
+		        // mode
+		        Monitor.logMessage(MessageService.getTextMessage(
+			            		MessageId.LOG_WAS_IN_TESTMODE_NO_SYNC,
+			            		Property.TESTMODE_NO_SYNC));
+		    }
+						
 					if (checkpointInstant == LogCounter.INVALID_LOG_INSTANT &&
 										getMirrorControlFileName().exists())
 						checkpointInstant =
@@ -2879,8 +3100,9 @@
 					// blow away the log file if possible
                     if (!privDelete(logFile) && logFileNumber == 1)
                     {
+                        logErrMsgForTestModeNoSync();
 						throw StandardException.newException(
-                            SQLState.LOG_INCOMPATIBLE_FORMAT, dataDirectory);
+                            SQLState.LOG_INCOMPATIBLE_FORMAT,dataDirectory);
                     }
 
 					// If logFileNumber > 1, we are not going to write that file just
@@ -2913,6 +3135,7 @@
 
                         if (!privDelete(logFile))
                         {
+                            logErrMsgForTestModeNoSync();
 							throw StandardException.newException(
                                     SQLState.LOG_INCOMPATIBLE_FORMAT,
                                     dataDirectory);
@@ -3584,15 +3807,8 @@
 			}
 			else
 			{
-				if (Performance.MEASURE)
-				{
-					if (!logNotSynced)
-						logOut.syncLogAccessFile();
-				}
-				else
-				{
-					logOut.syncLogAccessFile();
-				}
+				if (!logNotSynced)
+				    logOut.syncLogAccessFile();
 			}
 
 			syncSuceed = true;
@@ -3925,6 +4141,7 @@
 	*/
 	protected void logErrMsg(String msg)
 	{
+        	logErrMsgForTestModeNoSync();
 		Monitor.logTextMessage(MessageId.LOG_BEGIN_ERROR);
 		Monitor.logMessage(msg);
 		Monitor.logTextMessage(MessageId.LOG_END_ERROR);
@@ -3936,6 +4153,7 @@
 	*/
 	protected void logErrMsg(Throwable t)
 	{
+		logErrMsgForTestModeNoSync();
 		if (corrupt != null)
 		{
 			Monitor.logTextMessage(MessageId.LOG_BEGIN_CORRUPT_STACK);
@@ -3951,6 +4169,22 @@
 		}
 	}
 
+    /**
+     * In case of boot errors, and if database is either booted
+     * with derby.system.testMode being set or was previously booted in this
+     * mode, mention in the error message that the error is probably 
+     * because the derby.system.testMode was set. 
+     * Dont want to waste time to resolve issues in such
+     * cases
+     * <p>
+     * MT - not needed, informational only
+     */
+    protected void logErrMsgForTestModeNoSync()
+    {
+        if(logNotSynced || wasDBInTestModeNoSync)
+            Monitor.logTextMessage(MessageId.LOG_TESTMODE_NO_SYNC_ERR);
+    }
+
 	private void printErrorStack(Throwable t)
 	{
 		Monitor.logMessage("-------------------------\n");
Index: java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java	(revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/data/BaseDataFileFactory.java	(working copy)
@@ -179,11 +179,11 @@
 	private static final String LINE = 
 	"----------------------------------------------------------------";
 
-    // debug only flag - disable syncing of data during page allocation.
-    boolean dataNotSyncedAtAllocation;
+    // disable syncing of data during page allocation.
+    boolean dataNotSyncedAtAllocation = false;
 
-    // debug only flag - disable syncing of data during checkpoint.
-    boolean dataNotSyncedAtCheckpoint;
+    // disable syncing of data during checkpoint.
+    boolean dataNotSyncedAtCheckpoint = false;
 
 	// these fields can be accessed directly by subclasses if it needs a
 	// different set of actions
@@ -375,29 +375,24 @@
 
 		droppedTableStubInfo = new Hashtable();
 
-        if (Performance.MEASURE)
+        /**
+         * If derby.system.testMode=true then set flags to 
+         * disable sync of data pages at allocation when file is grown,
+         * disable sync of data writes during checkpoint
+         */
+        if (PropertyUtil.getSystemBoolean(Property.TESTMODE_NO_SYNC))
         {
-            // debug only flag - disable syncing of data during checkpoint.
-            dataNotSyncedAtCheckpoint = 
-                PropertyUtil.getSystemBoolean(
-                    Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT);
+            // - disable syncing of data during checkpoint.
+            dataNotSyncedAtCheckpoint = true;
 
-            // debug only flag - disable syncing of data during page allocation.
-            dataNotSyncedAtAllocation = 
-                PropertyUtil.getSystemBoolean(
-                    Property.STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION);
-
-            if (dataNotSyncedAtCheckpoint)
-                Monitor.logMessage(
-                    "Warning: " + 
-                    Property.STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT +
-                    "set to true.");
-
-            if (dataNotSyncedAtAllocation)
-                Monitor.logMessage(
-                    "Warning: " + 
-                    Property.STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION +
-                    "set to true.");
+            // - disable syncing of data during page allocation.
+            dataNotSyncedAtAllocation = true;
+            
+            // log message stating that derby.system.testMode
+            // has been enabled.  
+            Monitor.logMessage(MessageService.getTextMessage(
+            	MessageId.STORE_TESTMODE_NO_SYNC,
+            	Property.TESTMODE_NO_SYNC));
 		}
 
         fileHandler = new RFResource( this);
Index: java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java	(revision 165265)
+++ java/engine/org/apache/derby/impl/store/raw/data/RAFContainer.java	(working copy)
@@ -325,15 +325,8 @@
 				dataFactory.writeInProgress();
 				try
 				{
-                    if (Performance.MEASURE)
-                    {
-                        if (!dataFactory.dataNotSyncedAtAllocation)
-                            fileData.sync( false);
-                    }
-                    else
-                    {
+                    if (!dataFactory.dataNotSyncedAtAllocation)
                         fileData.sync( false);
-                    }
 				}
 				finally
 				{
@@ -504,16 +497,9 @@
 					dataFactory.writeInProgress();
 					inwrite = true;
 
-                    if (Performance.MEASURE)
-                    {
-                        if (!dataFactory.dataNotSyncedAtAllocation)
-                            fileData.sync( false);
-                    }
-                    else
-                    {
+                    if (!dataFactory.dataNotSyncedAtAllocation)
                         fileData.sync( false);
-                    }
-				}
+  				}
 				catch (IOException ioe)
 				{
 					// The disk may have run out of space. 
@@ -629,15 +615,9 @@
 			dataFactory.writeInProgress();
 			try
 			{
-                if (Performance.MEASURE)
-                {
-                    if (!dataFactory.dataNotSyncedAtCheckpoint)
-                        file.sync( false);
-                }
-                else
-                {
-                    file.sync( false);
-                }
+                if (!dataFactory.dataNotSyncedAtCheckpoint)
+                   file.sync( false);
+
 			}
 			finally
 			{
Index: java/engine/org/apache/derby/iapi/reference/MessageId.java
===================================================================
--- java/engine/org/apache/derby/iapi/reference/MessageId.java	(revision 165265)
+++ java/engine/org/apache/derby/iapi/reference/MessageId.java	(working copy)
@@ -58,8 +58,10 @@
     String LOG_CANNOT_CREATE_NEW_DUETO      = "L017";
     String LOG_MAYBE_INCONSISTENT           = "L018";
     String LOG_TRANS_LOG_SCAN               = "L019";
+    String LOG_WAS_IN_TESTMODE_NO_SYNC      = "L020"; // system was running in derby.system.testMode set 
+    String LOG_TESTMODE_NO_SYNC_ERR         = "L021"; // hint that error could be because 
+                                                      // derby.system.testMode was set
 
-
     /*
      * Raw Store data
      */
@@ -75,6 +77,7 @@
     String STORE_BACKUP_ABORTED             = "D010";
     String STORE_REMOVED_BACKUP             = "D011";
     String STORE_BACKUP_COMPLETED           = "D012";
+    String STORE_TESTMODE_NO_SYNC           = "D013"; // for derby.system.testMode is set
 
 
 	/*
Index: java/engine/org/apache/derby/iapi/reference/Property.java
===================================================================
--- java/engine/org/apache/derby/iapi/reference/Property.java	(revision 165265)
+++ java/engine/org/apache/derby/iapi/reference/Property.java	(working copy)
@@ -368,55 +368,47 @@
 		<P>
 	*/
 	String STORAGE_TEMP_DIRECTORY = "derby.storage.tempDirectory";
-
+	
     /**
-     * derby.storage.dataNotSyncedAtCheckPoint
+     * derby.system.testMode
      * <p>
-     * When set, the store system will not force a sync() call on the
-     * containers during a checkpoint.
+     * When set, the store system will not force sync calls in the following
+     * cases  
+     * - for the log file at each commit
+     * - for the log file before data page is forced to disk
+     * - for page allocation when file is grown
+     * - for data writes during checkpoint
      * <p>
-     * An internal debug system only flag.  The recovery system will not
-     * work properly if this flag is enabled, it is provided to do performance
-     * debugging to see whether the system is I/O bound based on checkpoint
-     * synchronous I/O.
+     * That means, when this property is set to true
+     * - a commit no longer guarantees that the transaction's modification
+     *   will survive a system crash or JVM termination
+     * - the database may not recover successfully upon restart
+     * - a near full disk at runtime may cause unexpected errors
      * <p>
-     *
-     **/
-	public static final String STORAGE_DATA_NOT_SYNCED_AT_CHECKPOINT = 
-        "derby.storage.dataNotSyncedAtCheckPoint";
-
-    /**
-     * derby.storage.dataNotSyncedAtAllocation
+     * This property is provided for performance reasons and should ideally
+     * only be used when there is no requirement for the database to recover.
+     * <BR> 
+     * One sample use would be to enable this property when using Derby as a test
+     * database, where recoverability is not a concern but performance is important.
+     * <BR>
+     * Default value is false
+     * <BR>
+     * Valid Values
+     * Valid values are true or false
+     * <BR>
+     * Example
+     * derby.system.testMode=true
+     * One can set this as a command line option to the JVM when starting the
+     * application or in the derby.properties file
+     * <BR>
+     * This property is static; if you change it while Derby is running, 
+     * the change does not take effect until you reboot.  
      * <p>
-     * When set, the store system will not force a sync() call on the
-     * containers when pages are allocated.
-     * <p>
-     * An internal debug system only flag.  The recovery system will not
-     * work properly if this flag is enabled, it is provided to do performance
-     * debugging to see whether the system is I/O bound based on page allocation
-     * synchronous I/O.
-     * <p>
      *
      **/
-	public static final String STORAGE_DATA_NOT_SYNCED_AT_ALLOCATION = 
-        "derby.storage.dataNotSyncedAtAllocation";
-
-    /**
-     * derby.storage.logNotSynced
-     * <p>
-     * When set, the store system will not force a sync() call on the log at 
-     * commit.
-     * <p>
-     * An internal debug system only flag.  The recovery system will not
-     * work properly if this flag is enabled, it is provided to do performance
-     * debugging to see whether the system is I/O bound based on log file
-     * synchronous I/O.
-     * <p>
-     *
-     **/
-	public static final String STORAGE_LOG_NOT_SYNCED = 
-        "derby.storage.logNotSynced";
-
+	public static final String TESTMODE_NO_SYNC = 
+        "derby.system.testMode";
+ 	
 	/**
      * derby.storage.fileSyncTransactionLog
      * <p>
Index: java/engine/org/apache/derby/loc/messages_en.properties
===================================================================
--- java/engine/org/apache/derby/loc/messages_en.properties	(revision 165265)
+++ java/engine/org/apache/derby/loc/messages_en.properties	(working copy)
@@ -1150,7 +1150,10 @@
 L016=cannot create a new log file {0}
 L017=cannot create a new log file {0} due to {1}
 L018=System may be in a inconsistent state, missing file {0}
+L020=WARNING: The database was previously booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability \n
+L021=WARNING: The database was either previously booted or is currently booted with {0} set to true. Note that this mode {0}=true does not guarantee recoverability, and is probably what might have caused this boot error \n
 
+
 # Class Manager
 C000=Wrote class {0} to file {1}. Please provide support with the file and the following exception information: {2}
 C001=Class {0} has a unknown certificate type in jar {1}, expecting X509 certificate.
@@ -1174,6 +1177,7 @@
 D010=Backup aborted due to error
 D011=removed old backup copy at {0}
 D012=Backup completed, log instant at {0} \n
+D013=WARNING: {0} is set to true. In this mode, there is no guarantee that the system will recover a system crash or JVM termination. Enable this property only when there is no need for recoverability \n
 
 # Connectivity 
 J004=database identity
Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java	(revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java	(revision 0)
@@ -0,0 +1,202 @@
+/*
+ 
+ Derby - Class org.apache.derbyTesting.functionTests.tests.store.TestNoSyncs
+ 
+ Copyright 2002, 2005 The Apache Software Foundation or its licensors, as applicable.
+ 
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ 
+ http://www.apache.org/licenses/LICENSE-2.0
+ 
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ 
+ */
+
+package org.apache.derbyTesting.functionTests.tests.store;
+
+import java.sql.Connection;
+import java.sql.Statement;
+import java.sql.PreparedStatement;
+import java.io.*;
+/**
+ * This program tests the system when the derby.system.testMode property is enabled
+ *
+ * When the derby.system.testMode is set, the system will not do any sync to the disk and
+ * the recovery system will not work property.
+ * It is provided for performance reasons and should ideally only be used
+ * when there is no requirement for the database to recover.
+ * <p>
+ * When set, the system will not do any syncs, the cases namely
+ * - no sync of the log file at each commit
+ * - no sync of the log file before data page is forced to disk
+ * - no sync of page allocation when file is grown
+ * - no sync of data writes during checkpoint
+ * <p>
+ * That means, when this property is set to true,
+ * - a commit no longer guarantees that the transaction's modification
+ *   will survive a system crash or JVM termination
+ * - the database may not recover successfully upon restart
+ * - a near full disk at runtime may cause unexpected errors
+ *
+ * This program tests for
+ * 1. enabling the derby.system.testMode is actually not doing the syncs
+ *    by timing inserts
+ * 2. check if a warning message exists in derby.log
+ * 3. read log.ctrl file and check if the flag is set or not
+ *
+ *
+ * @author <a href="mailto:ksunithaghm@gmail.com">Sunitha Kambhampati</a>
+ * @version 1.0
+ */
+public class TestNoSyncMode
+{
+	public static void main(String[] args)
+	{
+		Connection conn = null;
+		Statement s = null;
+		PreparedStatement ps = null;
+		try
+		{
+			// use the ij utility to read the property file and
+			// make the initial connection.
+			org.apache.derby.tools.ij.getPropertyArg(args);
+			conn = org.apache.derby.tools.ij.startJBMS();
+			
+			s = conn.createStatement();
+			s.execute("create table t1 (c1 int, c2 int)");
+			s.close();
+			
+			// Test 1
+			// this is a quick check incase someone breaks the derby.system.testMode 
+			long timeTaken = doInsertsInAutoCommit(conn);
+			conn.close();
+			
+			int approxUpperBound = 3000; // approximate upper bound in millisecond
+			if ( timeTaken > approxUpperBound )
+			{
+				report("FAIL -- derby.system.testMode mode seems to be broken.");
+				report(" Time to insert rows in test exceeded the usual limit.");
+			}
+			
+			// Test 2
+			// Check if derby.log has the warning message
+			report("Is warning message about derby.system.testMode present in derby.log ="+ isMessageInDerbyLog());
+			// Test 3
+			// Check if marker is correctly written out to database   
+			markerInControlFile();
+			
+			
+		}
+		catch (Throwable e) {
+			report("FAIL -- unexpected exception: "+e);
+			e.printStackTrace();
+		}
+		
+	}
+	
+	
+	/**
+	 * Note doing inserts in autocommit mode is probably the worst case scenario
+	 * in terms of performance as each commit will involve a flush/sync to disk
+	 * but in case of the derby.system.testMode=true mode, the syncs dont happen
+	 * This test case times the inserts and assumes that the inserts on any system
+	 * will be less than three second for 500 inserts.
+         * Note this upper bound on time is just an approximate estimation 
+	 */
+	public static long doInsertsInAutoCommit(Connection conn)
+	    throws Exception
+	{
+		PreparedStatement ps = conn.prepareStatement("insert into t1 values(?,?)");
+		long count = 0;
+		
+		long start = System.currentTimeMillis();
+		
+		for ( int i = 0; i < 500; i++ )
+		{
+			ps.setInt(1,i);
+			ps.setInt(2,i);
+			count += ps.executeUpdate();
+		}
+		
+		long end = System.currentTimeMillis();
+		report("num successful inserts = "+count);
+		
+		return ( end - start);
+	}
+	
+	/**
+     * When derby.system.testMode is set, a warning message is written out to derby.log
+	 * indicating that the property is set and that it does not guarantee recoverability
+	 * This test tests if a message is written out to derby.log or not 
+	 */
+	public static boolean isMessageInDerbyLog()
+	    throws Exception
+	{
+        BufferedReader reader = null;
+        File derbylog = null;
+        try
+        {
+        	String derbyHome = System.getProperty("derby.system.home");
+        	derbylog = new File(derbyHome,"derby.log");
+        	reader = new BufferedReader(new FileReader(derbylog));
+		
+        	String line = null;
+        	while((line = reader.readLine())!= null)
+        	{
+        		if(line.indexOf("derby.system.testMode") != -1)
+        			return true;
+			
+        	}
+        	return false;
+        }
+        finally
+        {
+            if (reader != null)
+            {
+            	reader.close();
+            }
+            derbylog = null;
+        }
+	}
+
+	/**
+     * if derby.system.testMode is set, a marker is written out into log control file
+	 * to recognize that the database was previously booted in this mode 
+ 	 * Test if the marker byte is set correctly or not
+ 	 * See comments in org.apache.derby.impl.store.log.LogToFile for IS_TESTMODE_NO_SYNC_FLAG
+ 	 */
+	public static void  markerInControlFile()
+		throws Exception
+	{
+        RandomAccessFile controlFile = null;
+        try
+        {
+        	int testModeNoSyncMarkerPosition = 29;
+        	String derbyHome = System.getProperty("derby.system.home");
+        	controlFile = new RandomAccessFile(derbyHome+"/wombat/log/log.ctrl","r");
+        	controlFile.seek(testModeNoSyncMarkerPosition);
+        	report("log.ctrl file has testMode NoSync marker value = "+ controlFile.readByte());
+        }
+        finally
+        {
+        	if(controlFile != null)
+        		controlFile.close();
+            
+        }
+	}
+	
+	public static void report(String msg)
+	{
+		System.out.println(msg);
+	}
+	
+}
+
+
+

Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode.java
___________________________________________________________________
Name: svn:eol-style
   + native

Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant	(revision 165265)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant	(working copy)
@@ -23,6 +23,7 @@
 TransactionTable.sql
 TransactionTable_app.properties
 TransactionTable_derby.properties
+TestNoSyncMode_derby.properties
 access.sql
 access_app.properties
 access_derby.properties
Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties	(revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties	(revision 0)
@@ -0,0 +1 @@
+derby.system.testMode=true

Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/TestNoSyncMode_derby.properties
___________________________________________________________________
Name: svn:eol-style
   + native

Index: java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out	(revision 0)
+++ java/testing/org/apache/derbyTesting/functionTests/master/TestNoSyncMode.out	(revision 0)
@@ -0,0 +1,3 @@
+num successful inserts = 500
+Is warning message about derby.system.testMode present in derby.log =true
+log.ctrl file has testMode NoSync marker value = 1
Index: java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall	(revision 165265)
+++ java/testing/org/apache/derbyTesting/functionTests/suites/storemats.runall	(working copy)
@@ -4,3 +4,4 @@
 store/longColumn.sql
 store/madhare.sql
 store/streamingColumn.java
+store/TestNoSyncMode.java

--------------090209020407060702020106--