Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Development" <derby-dev@db.apache.org>
Received-SPF: neutral (hermes.apache.org: local policy)
Message-ID: <42434CFD.2010301@sbcglobal.net>
Date: Thu, 24 Mar 2005 15:27:57 -0800
From: Mike Matrigali <mikem_app@sbcglobal.net>
User-Agent: Mozilla Thunderbird 0.7.2 (Windows/20040707)
MIME-Version: 1.0
To: Derby Development <derby-dev@db.apache.org>
Subject: Re: [PATCH] checksum support for transaction log to recognize out
 of order log writes during recovery (Derby-96)
References: <4241F27F.3010308@gmail.com>
In-Reply-To: <4241F27F.3010308@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I have taken a quick look and this looks like a good feature for
derby, I will look in more detail and assuming it looks ok will
look to commit sometime early next week.

I like the phased approach (similar to what army has been
doing).  I will work on committing this, if anyone has comments
or concerns - please post to the list.


Suresh Thalamati wrote:

> Attached is a first patch towards implementing checksum support for
> transaction
> log to handle out of order incomplete log writes during recovery.  This
> patch
> is based on writing a checksum log record that contain checksum
> information for a
> group of log records in the log buffers. Please refer to Derby-96 in
> JIRA for
> more details.
> 
> Testing : Ran derbyall test suite, all tests passed.
> 
> Changes in this patch addresses writing checksum information to the
> transaction
> log before the log records are written and verifying the log at recovery
> time
> using the checksum information on the disk.
> 
> Writing Checksum Log Records:
> Checksum log record contains checksum Algorithm, size of  the data  and
> the checksum value.
> Added a new class to implement this log operation.
> 
> The checksum Log record is placed before the actual log data the checksum
> record represents. This is done by reserving the space in the log
> buffers and
> in the log file then writing into reserved buffer space the checksum log
> record whenever
> buffer is full or it need to be written because of a flush request due to a
> commit. Incase of a large log records that does not fit into a single log
> buffer, the log records are written directly to the log file, in this case
> checksum log record represents only one log record and it is written to the
> log file before writing the large log record directly into the log file.
> 
> In the current system the log group information is encrypted when a
> database
> is encrypted. There is no facility to identify that a log record is
> checksum
> log record without decrypting the log record. Checksum Log Record is also
> encrypted to work correctly with the rest of the system.
> 
> changed files: LogAccessFile.java, ChecksumOperation.java, LogToFile.java
> 
> Verifying the Log:
> During recovery, while doing forward scan whenever scan finds a checksum
> record,
> it reads the amount of the data specified in the checksum record ,
> calculates
> the checksum for the data and compares it to the checksum value in the
> log record. If the on
> disk checksum value does not match with the value recalculated then that
> portion of the log is assumed as incompletely written and the record
> before the
> checksum log record becomes the last valid record for the recovery.
> 
> changed files: Scan.java
> 
> Unit Tests:
> Enhanced a existing  recovery unit test with test cases that simulate
> out of
> order incomplete writes by  corrupting the end of the log intentionally.
> 
> changed files: T_RecoveryBadLog.java
> 
> functional tests: current backup and recovery tests checks all cases where
> checksum should be valid.
> 
> To Be Done:
> 0) Handle Soft upgrade from old versions.   1) Check  for any timing
> issues between Checkpoint Thread and Checksum Log Record writes.
> 2) Check  for any timing issues between Checksum Log Record writes and
> the backup/restore.
> 3) Performance Testing
> 4) Write a  JDBC level functional test with large log records(>32k>
> incomplete log writes.
> 5) Any thing else that I find  while testing :-)
> 
> 
> Thanks
> -suresht
> 
> $ svn status
> M      java\engine\org\apache\derby\impl\store\raw\log\LogAccessFile.java
> M      java\engine\org\apache\derby\impl\store\raw\log\LogRecord.java
> A     
> java\engine\org\apache\derby\impl\store\raw\log\ChecksumOperation.java
> M      java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java
> M      java\engine\org\apache\derby\impl\store\raw\log\Scan.java
> M     
> java\engine\org\apache\derby\impl\store\raw\log\LogAccessFileBuffer.java
> M     
> java\engine\org\apache\derby\iapi\services\io\RegisteredFormatIds.java
> M      java\engine\org\apache\derby\iapi\services\io\StoredFormatIds.java
> M      java\engine\org\apache\derby\iapi\store\raw\Loggable.java
> M     
> java\testing\org\apache\derbyTesting\unitTests\store\T_RecoverBadLog.java
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLogSetup_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog2_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog4_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog6_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog1_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog3_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog5_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\tests\unit\recoverBadChecksumLog7_derby.properties
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog4.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLogSetup.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog1.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog5.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog2.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog6.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog3.out
> 
> A     
> java\testing\org\apache\derbyTesting\functionTests\master\recoverBadChecksumLog7.out
> 
> M     
> java\testing\org\apache\derbyTesting\functionTests\suites\storeunit.runall
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> Index: java/engine/org/apache/derby/impl/store/raw/log/LogAccessFile.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/LogAccessFile.java	(revision 158720)
> +++ java/engine/org/apache/derby/impl/store/raw/log/LogAccessFile.java	(working copy)
> @@ -33,7 +33,10 @@
>  import java.io.InterruptedIOException;
>  import java.util.LinkedList;
>  
> +import org.apache.derby.iapi.services.io.FormatIdOutputStream;
> +import org.apache.derby.iapi.services.io.ArrayOutputStream;
>  
> +
>  /**
>  	Wraps a RandomAccessFile file to provide buffering
>  	on log writes. Only supports the write calls
> @@ -56,8 +59,27 @@
>  	freeBuffers --> dirtyBuffers --> freeBuffers. Movement of buffers from one
>      stage to 	another stage is synchronized using	the object(this) of this class. 
>  
> +	A Checksum log record that has the checksum value for the data that is
> +    being written to the disk is generated and written 	before the actual data. 
> +	Except for the large log records that does not fit into a single buffer, 
> +    checksum is calcualted for a group of log records that are in the buffer 
> +	when buffers is switched. Checksum log record is written into the reserved
> +	space in the beginning buffer. 
> +
> +    In case of a large log record that does not fit into a bufffer, it needs to 
> +    be written directly to the disk instead of going through the log buffers. 
> +    In this case the log record write gets broken into three parts:
> +        1) Write checksum log record and LOG RECORD HEADER (length + instant) 
> +        2) Write the log record. 
> +        3) Write the trailing length of the log record. 
> +
> +	Checksum log records helps in identifying the incomplete log disk writes during 
> +    recovery. This is done by recalculating the checksum value for the data on
> +    the disk and comparing it to the the value stored in the checksum log
> +    record. 
> +
>  */
> -public class LogAccessFile extends OutputStream 
> +public class LogAccessFile 
>  {
>  
>      /**
> @@ -67,7 +89,8 @@
>       *     int   trailing length    : 4 bytes
>       **/
>      private static final int            LOG_RECORD_FIXED_OVERHEAD_SIZE = 16;
> -
> +	private static final int            LOG_RECORD_HEADER_SIZE = 12; //(length + instant)
> +	private static final int            LOG_RECORD_TRAILER_SIZE = 4; //trailing length 
>      private static final int            LOG_NUMBER_LOG_BUFFERS = 3;
>  
>  
> @@ -75,7 +98,7 @@
>  	private LinkedList    dirtyBuffers; //list of dirty buffers to flush
>  	private  LogAccessFileBuffer currentBuffer; //current active buffer
>  	private boolean flushInProgress = false;
> -
> +	
>  	private final StorageRandomAccessFile  log;
>  
>  	// log can be touched only inside synchronized block protected by
> @@ -85,9 +108,23 @@
>  	static int                      mon_numWritesToLog;
>  	static int                      mon_numBytesToLog;
>  
> -	public LogAccessFile(
> -    StorageRandomAccessFile    log, 
> -    int                 bufferSize) throws IOException 
> +
> +	//streams used to generated check sume log record ; see if there is any simpler way
> +	private ArrayOutputStream logOutputBuffer;
> +	private FormatIdOutputStream logicalOut;
> +	private boolean directWrite = false; //true when log is written directly to file.
> +	private long checksumInstant = -1;
> +	private int checksumLength;
> +	private int checksumLogRecordSize;      //checksumLength + LOG_RECORD_FIXED_OVERHEAD_SIZE
> +	private boolean writeChecksum = true;  //gets set to false incase of a soft upgrade.
> +	private ChecksumOperation checksumLogOperation;
> +	private LogRecord checksumLogRecord;
> +	private LogToFile logFactory;
> +	private boolean databaseEncrypted=false;
> +		
> +	public LogAccessFile(LogToFile logFactory,
> +						 StorageRandomAccessFile    log, 
> +						 int                 bufferSize) 
>      {
>  		if (SanityManager.DEBUG)
>  		{
> @@ -97,6 +134,7 @@
>  		
>  		this.log            = log;
>  		logFileSemaphore    = log;
> +		this.logFactory     = logFactory;
>  
>  		if (SanityManager.DEBUG)
>              SanityManager.ASSERT(LOG_NUMBER_LOG_BUFFERS >= 1);
> @@ -115,9 +153,56 @@
>  
>  		currentBuffer = (LogAccessFileBuffer) freeBuffers.removeFirst();
>  
> +		if(writeChecksum)
> +		{
> +			/**
> +			 * setup structures that are required to write the checksum log records
> +			 * for a group of log records are being written to the disk. 
> +			 */
> +			checksumLogOperation = new ChecksumOperation();
> +			checksumLogOperation.init();
> +			checksumLogRecord = new LogRecord();
> +
> +			// Note: Checksum log records are not related any particular transaction, 
> +			// they are written to store a checksum information identify
> +			// incomplete log record writes. No transacton id is set for this
> +			// log record. That is why a null argument is passed below 
> +			// setValue(..) call. 
> +			checksumLogRecord.setValue(null, checksumLogOperation);
> +
> +			checksumLength = 
> +				checksumLogRecord.getStoredSize(checksumLogOperation.group(), null) + 
> +				checksumLogOperation.getStoredSize();
> +
> +			// calculate checksum log operation length when the database is encrypted
> +			if (logFactory.databaseEncrypted())
> +			{
> +				checksumLength =  logFactory.getEncryptedDataLength(checksumLength);
> +				databaseEncrypted = true;
> +			}
> +			checksumLogRecordSize = checksumLength  + LOG_RECORD_FIXED_OVERHEAD_SIZE;
> +
> +			//streams required to convert a log record to raw byte array. 
> +			logOutputBuffer = new ArrayOutputStream(); 
> +			logicalOut = new FormatIdOutputStream(logOutputBuffer);
> +
> +			/** initialize the buffer with space reserved for checksum log record in
> +			 * the beginning of the log buffer; checksum record is written into
> +			 * this space when buffer is switched or while doing direct write to the log file.
> +			 */
> +		}else
> +		{
> +			//checksumming of transaction log feature is not in use. 
> +			checksumLogRecordSize = 0;
> +		}
> +		
> +		currentBuffer.init(checksumLogRecordSize);
>  	}
>  
>  
> +	private byte[] db = new byte[LOG_RECORD_TRAILER_SIZE]; 
> +
> +
>      /**
>       * Write a single log record to the stream.
>       * <p>
> @@ -164,34 +249,16 @@
>      {
>          int total_log_record_length = length + LOG_RECORD_FIXED_OVERHEAD_SIZE;
>  
> -        if (total_log_record_length > currentBuffer.bytes_free && 
> -            total_log_record_length <= currentBuffer.buffer.length) 
> -        {
> -            // If the whole record will fit in an empty buffer, flush this
> -            // one now and put this record into the next one.
> -            switchLogBuffer();
> -        }
> -
>  		if (total_log_record_length <= currentBuffer.bytes_free)
>          {
>              byte[] b    = currentBuffer.buffer;
>              int    p    = currentBuffer.position;
>  
>              // writeInt(length)
> -            b[p++] = (byte) ((length >>> 24) & 0xff); 
> -            b[p++] = (byte) ((length >>> 16) & 0xff); 
> -            b[p++] = (byte) ((length >>>  8) & 0xff); 
> -            b[p++] = (byte) ((length       ) & 0xff);
> +			p = writeInt(length, b, p);
>              
>              // writeLong(instant)
> -            b[p++] = (byte) (((int)(instant >>> 56)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>> 48)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>> 40)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>> 32)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>> 24)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>> 16)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant >>>  8)) & 0xff); 
> -            b[p++] = (byte) (((int)(instant       )) & 0xff); 
> +			p = writeLong(instant, b , p);
>  
>              // write(data, data_offset, length - optional_data_length)
>              int transfer_length = (length - optional_data_length);
> @@ -213,55 +280,101 @@
>              }
>  
>              // writeInt(length)
> -            b[p++] = (byte) ((length >>> 24) & 0xff); 
> -            b[p++] = (byte) ((length >>> 16) & 0xff); 
> -            b[p++] = (byte) ((length >>>  8) & 0xff); 
> -            b[p++] = (byte) ((length       ) & 0xff);
> -
> -            currentBuffer.position   = p;
> +			p = writeInt(length, b, p);
> +            
> +			currentBuffer.position   = p;
>              currentBuffer.bytes_free -= total_log_record_length;
>  		}
>          else
>          {
> -            writeInt(length);
> -            writeLong(instant);
> -            write(data, data_offset, length - optional_data_length);
> +			
> +			/** Because current log record will never fit in a single buffer
> +			 * a direct write to the log file is required instead of 
> +			 * writing the log record through  the log bufffers. 
> +			 */
> +			directWrite = true;
> +
> +			byte[] b    = currentBuffer.buffer;
> +            int    p    = currentBuffer.position;
> +
> +            // writeInt(length)
> +			p = writeInt(length , b, p);
> +            
> +            // writeLong(instant)
> +			p = writeLong(instant, b, p);
> +
> +			currentBuffer.position   = p;
> +			currentBuffer.bytes_free -= LOG_RECORD_HEADER_SIZE;
> +
> +			/** using a seperate small buffer to write the traling length
> +			 * instead of the log buffer because data portion will be 
> +			 * written directly to log file after the log buffer is 
> +			 * flushed and the trailing length should be written after that. 
> +			 */
> +
> +			// writeInt(length)
> +			writeInt(length , db, 0);
> +
> +			if(writeChecksum)
> +			{
> +				checksumLogOperation.reset();
> +				checksumLogOperation.update(b, checksumLogRecordSize, p - checksumLogRecordSize);
> +				checksumLogOperation.update(data, data_offset, length - optional_data_length);
> +				if (optional_data_length != 0)
> +				{
> +					checksumLogOperation.update(optional_data, optional_data_offset, optional_data_length);	
> +				}
> +
> +				// update the checksum to include the trailing length.
> +				checksumLogOperation.update(db, 0, LOG_RECORD_TRAILER_SIZE);
> +			
> +				// write checksum log record to the log buffer 
> +				writeChecksumLogRecord();
> +			}
> +			
> +			
> +			// now do the  writes directly to the log file. 
> +
> +			// flush all buffers before wrting directly to the log file. 
> +			flushLogAccessFile();
> +
> +			// Note:No Special Synchronization required here , 
> +			// There will be nothing to write by flushDirtyBuffers that can run
> +			// in parallel to the threads that is executing this code. Above
> +			// flush call should have written all the buffers and NO new log will 
> +			// get added until the following direct log to file call finishes. 
> +
> +
> +			// write the rest of the log directltly to the log file. 
> +            writeToLog(data, data_offset, length - optional_data_length);
>              if (optional_data_length != 0)
>              {
> -                write(
> +                writeToLog(
>                      optional_data, optional_data_offset, optional_data_length);
>              }
> -            writeInt(length);
> +
> +			// write the trailing length 
> +			writeToLog(db,0, 4);
> +			directWrite = false;
>  		}
>      }
>  
>  
> -	public void writeInt(int i) throws IOException 
> -    {
> -		if (currentBuffer.bytes_free < 4)
> -			switchLogBuffer();
>  
> -		byte[] b = currentBuffer.buffer;
> -		int p = currentBuffer.position;
> -
> +	private final int writeInt(int i , byte b[], int p)
> +	{
> +	
>          b[p++] = (byte) ((i >>> 24) & 0xff); 
>          b[p++] = (byte) ((i >>> 16) & 0xff); 
>          b[p++] = (byte) ((i >>> 8) & 0xff); 
> -        b[p++] = (byte) (i & 0xff);
> -
> -		currentBuffer.position = p;
> -		currentBuffer.bytes_free -= 4;
> +        b[p++] = (byte) (i & 0xff);	
> +		return p;
>  	}
>  
> -	public void writeLong(long l) 
> -        throws IOException 
> -    {
> -		if (currentBuffer.bytes_free < 8)
> -			switchLogBuffer();
>  
> -		byte[] b = currentBuffer.buffer;
> - 		int p = currentBuffer.position;
> -        b[p++] = (byte) (((int)(l >>> 56)) & 0xff); 
> +	private final int writeLong(long l , byte b[], int p)
> +	{
> +		b[p++] = (byte) (((int)(l >>> 56)) & 0xff); 
>          b[p++] = (byte) (((int)(l >>> 48)) & 0xff); 
>          b[p++] = (byte) (((int)(l >>> 40)) & 0xff); 
>          b[p++] = (byte) (((int)(l >>> 32)) & 0xff); 
> @@ -269,65 +382,60 @@
>          b[p++] = (byte) (((int)(l >>> 16)) & 0xff); 
>          b[p++] = (byte) (((int)(l >>> 8)) & 0xff); 
>          b[p++] = (byte) (((int)l) & 0xff); 
> -		currentBuffer.position = p;
> +		return p;
> +	}
> +
> +	public void writeInt(int i) 
> +    {
> +
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(currentBuffer.bytes_free >= 4);
> +		}
> +		
> +		currentBuffer.position = 
> +			writeInt(i , currentBuffer.buffer, currentBuffer.position);
> +		currentBuffer.bytes_free -= 4;
> +	}
> +
> +	public void writeLong(long l) 
> +    {
> +		
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(currentBuffer.bytes_free >= 8);
> +		}
> +		
> +		currentBuffer.position = 
> +			writeLong(l , currentBuffer.buffer, currentBuffer.position);
>  		currentBuffer.bytes_free -= 8;
>      }
>  
>  	public void write(int b) 
> -        throws IOException 
>      {
> -
> -		if (currentBuffer.bytes_free == 0)
> -			switchLogBuffer();
> -
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(currentBuffer.bytes_free > 0);
> +		}
> +		
>  		currentBuffer.buffer[currentBuffer.position++] = (byte) b;
>  		currentBuffer.bytes_free--;
>  	}
>  
>  
>  	public void write(byte b[], int off, int len) 
> -        throws IOException 
>      {
> -
> -		if (len <= currentBuffer.bytes_free)  
> -        {
> -			// data fits in buffer
> -			System.arraycopy(b, off, currentBuffer.buffer, currentBuffer.position, len);
> -			currentBuffer.bytes_free -= len;
> -			currentBuffer.position += len;
> -			return;
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(len <= currentBuffer.bytes_free);
>  		}
> -        else if (len <= currentBuffer.buffer.length) 
> -        {
> -            // some data will be cached
> -            System.arraycopy(b, off, currentBuffer.buffer, currentBuffer.position, currentBuffer.bytes_free);
> -            len -= currentBuffer.bytes_free;
> -            off += currentBuffer.bytes_free;
> -            currentBuffer.position += currentBuffer.bytes_free;
> -            currentBuffer.bytes_free = 0;
> -            switchLogBuffer();
> +		
> +		System.arraycopy(b, off, currentBuffer.buffer, currentBuffer.position, len);
> +		currentBuffer.bytes_free -= len;
> +		currentBuffer.position += len;
> +	}
>  
> -            System.arraycopy(b, off, currentBuffer.buffer, 0, len);
> -            currentBuffer.position = len;
> -            currentBuffer.bytes_free -= len;	
> -        }
> -        else
> -        {
> -			
> -			//data will never fit in currentBuffer.buffer, write directly to log
> -			//flush all buffers before wrting directly to the log file. 
> -			flushLogAccessFile();
>  
> -			//Note:No Special Synchronization required here , 
> -			//There will be nothing to write by flushDirtyBuffers that can run
> -			//in parallel to the threads that is executing this code. Above
> -			//flush call should have written all the buffers and NO new log will 
> -			//get added until the following direct log to file call finishes. 
> -
> -			writeToLog(b, off, len);
> -			return;
> -		}
> -	}
>      /**
>       * Write data from all dirty buffers into the log file.
>       * <p>
> @@ -427,7 +535,7 @@
>  
>  
>  	//flush all the the dirty buffers to disk
> -	public void flushLogAccessFile() throws IOException 
> +	public void flushLogAccessFile() throws IOException,  StandardException 
>  	{
>  		switchLogBuffer();
>  		flushDirtyBuffers();
> @@ -442,12 +550,25 @@
>  	 * when  flushDirtyBuffers() is invoked by  a commit call 
>  	 * or when no more free buffers are available. 
>  	 */
> -	public void switchLogBuffer() throws IOException  
> +	public void switchLogBuffer() throws IOException, StandardException  
>      {
>  
>  		synchronized(this)
>  		{
> +			// ignore empty buffer switch requests
> +			if(currentBuffer.position == checksumLogRecordSize)
> +				return;
>  
> +			// calculate the checksum for the current log buffer 
> +			// and write the record to the space reserverd in 
> +			// the beginning of the buffer. 
> +			if(writeChecksum && !directWrite)
> +			{
> +				checksumLogOperation.reset();
> +				checksumLogOperation.update(currentBuffer.buffer, checksumLogRecordSize, currentBuffer.position - checksumLogRecordSize);
> +				writeChecksumLogRecord();
> +			}
> +
>  			//add the current buffer to the flush buffer list
>  			dirtyBuffers.addLast(currentBuffer);
>  
> @@ -467,13 +588,13 @@
>  
>  			//switch over to the next log buffer, let someone else write it.
>  			currentBuffer = (LogAccessFileBuffer) freeBuffers.removeFirst();
> -			currentBuffer.init();
> -     
> +			currentBuffer.init(checksumLogRecordSize);
> +
>  			if (SanityManager.DEBUG)
>  			{
> -				SanityManager.ASSERT(currentBuffer.position == 0);
> +				SanityManager.ASSERT(currentBuffer.position == checksumLogRecordSize);
>  				SanityManager.ASSERT(
> -									 currentBuffer.bytes_free == currentBuffer.buffer.length);
> +									 currentBuffer.bytes_free == currentBuffer.length);
>                  SanityManager.ASSERT(currentBuffer.bytes_free > 0);
>  			}
>  		}
> @@ -545,11 +666,11 @@
>  		}
>  	}
>  
> -	public void close() throws IOException 
> +	public void close() throws IOException, StandardException
>      {
>  		if (SanityManager.DEBUG) 
>          {
> -			if (currentBuffer.position != 0)
> +			if (currentBuffer.position !=  checksumLogRecordSize)
>  				SanityManager.THROWASSERT(
>  				"Log file being closed with data still buffered " + 
>                  currentBuffer.position +  " " + currentBuffer.bytes_free);
> @@ -572,6 +693,7 @@
>  		{
>              if (log != null)
>              {
> +
>                  // Try to handle case where user application is throwing
>                  // random interrupts at cloudscape threads, retry in the case
>                  // of IO exceptions 5 times.  After that hope that it is 
> @@ -602,6 +724,142 @@
>  			mon_numBytesToLog += len;
>  		}
>  	}
> +
> +	/**
> +	 * reserve the space for the checksum log record in the log file. 
> +	 * @param  the length of the log record that is going to be written
> +	 * @param  logFileNumber current log file number 
> +	 * @param  currentPosition  current position in the log file. 
> +	 * @return the space that is needed to write a checksum log record.
> +	 */
> +	protected long reserveSpaceForChecksum(int length, long logFileNumber, long currentPosition )
> +		throws StandardException, IOException 
> +	{
> +		if(!writeChecksum)
> +			return 0;
> +
> +		int total_log_record_length = length + LOG_RECORD_FIXED_OVERHEAD_SIZE;
> +		boolean reserveChecksumSpace = false;
> +		
> +		/* checksum log record is calculated for a group of log 
> +		 * records that can fit in to a single buffer or for 
> +		 * a single record when it does not fit into 
> +		 * a fit into a buffer at all. When a new buffer 
> +		 * is required to write a log record, log space 
> +		 * has to be reserved before writing the log record
> +		 * becuase checksum is written in the before the 
> +		 * log records that are being checksummed. 
> +		 * What it also means is a real log instant has to be 
> +		 * reserved for writing the checksum log record in addition 
> +		 * to the log buffer space.
> +		 */
> +		
> +
> +		/* reserve checkum space for new log records if a log buffer switch had
> +		 * happened before because of a explicit log flush requests(like commit)
> +		 * or a long record write 
> +		 */
> +		if(currentBuffer.position == checksumLogRecordSize)
> +		{
> +			reserveChecksumSpace = true;
> +		}
> +		else{
> +			if (total_log_record_length > currentBuffer.bytes_free)
> +			{
> +				// the log record that is going to be written is not 
> +				// going to fit in the current buffer, switch the 
> +				// log buffer to create buffer space for it. 
> +				switchLogBuffer();
> +				reserveChecksumSpace = true;
> +			}
> +		}
> +		
> +		if(reserveChecksumSpace)
> +		{
> +			if (SanityManager.DEBUG)
> +			{
> +				// Prevoiusly reserved real checksum instant should have been
> +				// used, before an another one is generated. 
> +				SanityManager.ASSERT(checksumInstant == -1,  "CHECKSUM INSTANT IS GETTING OVER WRITTEN");
> +			}
> +			
> +			checksumInstant = LogCounter.makeLogInstantAsLong(logFileNumber, currentPosition);
> +			return  checksumLogRecordSize;
> +		}else
> +		{
> +			return 0 ;
> +		}
> +	}
> +
> +
> +	/*
> +	 * generate the checkum log record and write it into the log buffer.
> +	 */
> +	private void writeChecksumLogRecord() throws IOException, StandardException
> +	{
> +		
> +		byte[] b    = currentBuffer.buffer;
> +		int    p    = 0; //checksum is written in the beginning of the buffer
> +
> +		// writeInt(length)
> +		p = writeInt(checksumLength, b , p);
> +            
> +		// writeLong(instant)
> +		p = writeLong(checksumInstant, b , p);
> +
> +		//write the checksum log operation  
> +		logOutputBuffer.setData(b);
> +		logOutputBuffer.setPosition(p);
> +		logicalOut.writeObject(checksumLogRecord);
> +
> +		if(databaseEncrypted)
> +		{
> +			//encrypt the checksum log operation part.
> +			int len = 
> +				logFactory.encrypt(b, LOG_RECORD_HEADER_SIZE, checksumLength, 
> +								   b, LOG_RECORD_HEADER_SIZE);
> +			
> +		   
> +			if (SanityManager.DEBUG)
> +				SanityManager.ASSERT(len == checksumLength, 
> +									 "encrypted log buffer length != log buffer len");
> +		}
> +
> +		p = LOG_RECORD_HEADER_SIZE + checksumLength ;
> +
> +		// writeInt(length) trailing
> +		p = writeInt(checksumLength, b, p );
> +		
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(p == checksumLogRecordSize, "position=" + p  + "ckrecordsize=" + checksumLogRecordSize);
> +			if (SanityManager.DEBUG_ON(LogToFile.DBG_FLAG))
> +			{
> +				SanityManager.DEBUG(
> +									LogToFile.DBG_FLAG, 
> +									"Write log record: tranId=Null"  +
> +									" instant: " + LogCounter.toDebugString(checksumInstant) + " length: " +
> +									checksumLength + "\n" + checksumLogOperation + "\n");
> +			}
> +			checksumInstant = -1; 
> +		}
> +
> +	}
> +
> +
> +	protected void writeEndMarker(int marker) throws IOException, StandardException 
> +	{
> +		//flush all the buffers and then write the end marker.
> +		flushLogAccessFile();
> +		
> +		byte[] b    = currentBuffer.buffer;
> +		int    p    = 0; //end is written in the beginning of the buffer, no
> +						 //need to checksum a int write.
> +		p = writeInt(marker , b , p);
> +		writeToLog(b, 0, p);
> +	}
> +
> +	
>  }
>  
>  
> Index: java/engine/org/apache/derby/impl/store/raw/log/LogRecord.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/LogRecord.java	(revision 158720)
> +++ java/engine/org/apache/derby/impl/store/raw/log/LogRecord.java	(working copy)
> @@ -145,6 +145,21 @@
>  		return tranId.getMaxStoredSize();
>  	}
>  
> +	
> +	public static int getStoredSize(int group, TransactionId xactId)
> +	{
> +		
> +		if (SanityManager.DEBUG)
> +		{
> +			SanityManager.ASSERT(xactId == null, 
> +								 "size calculation are based on xactId being  null"); 
> +		}
> +
> +		return 	formatLength + 	CompressedNumber.sizeInt(group) + 
> +			FormatIdUtil.getFormatIdByteLength(StoredFormatIds.NULL_FORMAT_ID);
> +	}
> +
> +
>  	public TransactionId getTransactionId() 
>  		 throws IOException, ClassNotFoundException 
>  	{
> @@ -292,4 +307,7 @@
>  	}
>  
>  
> +	public boolean isChecksum()	{
> +		return ((group & Loggable.CHECKSUM) != 0);
> +	}
>  }
> Index: java/engine/org/apache/derby/impl/store/raw/log/ChecksumOperation.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/ChecksumOperation.java	(revision 0)
> +++ java/engine/org/apache/derby/impl/store/raw/log/ChecksumOperation.java	(revision 0)
> @@ -0,0 +1,266 @@
> +/*
> +
> +   Derby - Class org.apache.derby.impl.store.raw.log.ChecksumOperation
> +
> +   Copyright 1997, 2004 The Apache Software Foundation or its licensors, as applicable.
> +
> +   Licensed under the Apache License, Version 2.0 (the "License");
> +   you may not use this file except in compliance with the License.
> +   You may obtain a copy of the License at
> +
> +      http://www.apache.org/licenses/LICENSE-2.0
> +
> +   Unless required by applicable law or agreed to in writing, software
> +   distributed under the License is distributed on an "AS IS" BASIS,
> +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +   See the License for the specific language governing permissions and
> +   limitations under the License.
> +
> + */
> +
> +package org.apache.derby.impl.store.raw.log;
> +
> +import org.apache.derby.iapi.services.sanity.SanityManager;
> +import org.apache.derby.iapi.services.io.Formatable;
> +import org.apache.derby.iapi.services.io.FormatIdUtil;
> +import org.apache.derby.iapi.services.io.StoredFormatIds;
> +import org.apache.derby.catalog.UUID;
> +
> +import org.apache.derby.iapi.store.raw.Transaction;
> +import org.apache.derby.iapi.store.raw.Loggable;
> +import org.apache.derby.iapi.store.raw.log.LogInstant;
> +import org.apache.derby.iapi.store.raw.log.LogFactory;
> +import org.apache.derby.iapi.store.raw.xact.RawTransaction;
> +
> +import org.apache.derby.iapi.error.StandardException;
> +
> +import org.apache.derby.iapi.services.io.CompressedNumber;
> +import org.apache.derby.iapi.util.ByteArray;
> +
> +import java.io.Externalizable;
> +import java.io.OutputStream;
> +import java.io.InputStream;
> +import java.io.ObjectInput;
> +import java.io.ObjectOutput;
> +import java.io.IOException;
> +import org.apache.derby.iapi.services.io.LimitObjectInput;
> +
> +import java.util.zip.Checksum;
> +import java.util.zip.CRC32;
> +
> +
> +/**
> +	A Log Operation that represents a checksum for a group of log records
> +	that are written to the tranaction log file.
> +
> +	<PRE>
> +	@format_id	LOGOP_CHECKSUM
> +		the formatId is written by FormatIdOutputStream when this object is
> +		written out by writeObject
> +	@purpose  checksum one or more log records while writing to disk
> +	@upgrade
> +	@disk_layout
> +		checksumAlgo(byte)  	the checksum algorithm 
> +		checksumValue(long)     the checksum value 
> +		dataLength(int)			number of bytes that the checksum is calculated
> +	@end_format
> +	</PRE>
> +
> +	@author  Suresh Thalamati
> +	@see Loggable
> +*/
> +
> +public class ChecksumOperation implements Loggable 
> +{
> +
> +	private  byte   checksumAlgo;
> +	private  long   checksumValue;   
> +	private  int	dataLength; 
> +	private Checksum checksum; 
> +
> +	/*
> +	 * constant values for algorithm that are used to perform the checksum.
> +	 */
> +    public static final byte CRC32_ALGORITHM  = (byte) 0x1; //java.util.zip.CRC32
> +	
> +	private static final int formatLength = FormatIdUtil.getFormatIdByteLength(StoredFormatIds.LOGOP_CHECKSUM);
> +	
> +	public void init()
> +	{
> +		this.checksumAlgo = CRC32_ALGORITHM;
> +		initializeChecksumAlgo();
> +		dataLength = 0;
> +	}
> +
> +	
> +	// update the checksum
> +	protected void update(byte[] buf, int off, int len)
> +	{
> +		checksum.update(buf, off , len);
> +		dataLength += len;
> +	}
> +
> +	
> +	// reset the checksum 
> +	protected void reset()
> +	{
> +		checksum.reset();
> +		dataLength = 0;
> +	}
> +
> +
> +	private void initializeChecksumAlgo()
> +	{
> +		if(checksumAlgo == CRC32_ALGORITHM)
> +			this.checksum = new CRC32();
> +	}
> +
> +
> +	/*
> +	 * Formatable methods
> +	 */
> +
> +	// no-arg constructor, required by Formatable 
> +	public ChecksumOperation() { super();}
> +
> +	public void writeExternal(ObjectOutput out) throws IOException 
> +	{	
> +		checksumValue = checksum.getValue();
> +		out.writeByte(checksumAlgo);
> +		out.writeInt(dataLength);
> +		out.writeLong(checksumValue);
> +	}
> +
> +	public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException
> +	{
> +		checksumAlgo = (byte) in.readUnsignedByte();
> +		dataLength = in.readInt();
> +		checksumValue = in.readLong();
> +		initializeChecksumAlgo();
> +	}
> +
> +
> +	public int getStoredSize()
> +	{
> +		return formatLength + 1 + 4 + 8;
> +	}	
> +
> +
> +
> +	/**
> +		Return my format identifier.
> +	*/
> +	public int getTypeFormatId() {
> +		return StoredFormatIds.LOGOP_CHECKSUM;
> +	}
> +
> +
> +
> +
> +
> +	/**
> +		Loggable methods
> +	*/
> +
> +	/**
> +	 *	Nothing to do for the checksum log record because it does need to be
> +	 *  applied during redo. 
> +	 */
> +	public void doMe(Transaction xact, LogInstant instant, LimitObjectInput in) throws StandardException
> +	{
> +	}
> +
> +	/**
> +		the default for prepared log is always null for all the operations
> +		that don't have optionalData.  If an operation has optional data,
> +		the operation need to prepare the optional data for this method.
> +
> +		Checksum has no optional data to write out
> +
> +		@param out Where and how to write to optional data.
> +		
> +	*/
> +	public ByteArray getPreparedLog()
> +	{
> +		return (ByteArray) null;
> +	}
> +
> +	/**
> +		Checksum does not need to be redone, it is used to just verify that
> +		log records are written completely.
> +	*/
> +	public boolean needsRedo(Transaction xact)
> +	{
> +		return false;
> +	}
> +
> +
> +	/**
> +	  Checksum has no resources to release
> +	*/
> +	public void releaseResource(Transaction xact)
> +	{}
> +
> +	/**
> +		Checksum is a raw store operation
> +	*/
> +	public int group()
> +	{
> +		return Loggable.RAWSTORE | Loggable.CHECKSUM;
> +	}
> +
> +	
> +
> +
> +	/**
> +	 * Access attributes of the checksum log record
> +	 */
> +
> +	protected int getDataLength() 
> +	{
> +		return dataLength;
> +	}
> +
> +
> +	protected boolean isChecksumValid(byte[] data, int off , int length)
> +	{
> +		checksum.reset();
> +		checksum.update(data , off , length);
> +		return checksum.getValue()== checksumValue;
> +
> +	}
> +
> +
> +	/**
> +	  DEBUG: Print self.
> +	*/
> +	public String toString()
> +	{
> +		if (SanityManager.DEBUG)
> +		{
> +			StringBuffer str = new StringBuffer(200)
> +				.append("Checksum Operation ")
> +				.append(" algorithm = ")
> +				.append(checksumAlgo)
> +				.append(" value = ")
> +				.append(checksumValue)
> +				.append(" data length= ").append(dataLength);
> +
> +				return str.toString();
> +		}
> +		else
> +			return null;
> +	}
> +}
> +
> +
> +
> +
> +
> +
> +
> +
> +
> +
> +
> +
> 
> Property changes on: java/engine/org/apache/derby/impl/store/raw/log/ChecksumOperation.java
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java	(revision 158720)
> +++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java	(working copy)
> @@ -282,7 +282,7 @@
>  
>  	protected LogAccessFile logOut;		// an output stream to the log file
>  								// (access of the variable should sync on this)
> -
> +	private   StorageRandomAccessFile firstLog = null;
>  	protected long		     endPosition = -1; // end position of the current log file
>  	long					 lastFlush = 0;	// the position in the current log
>  											// file that has been flushed to disk
> @@ -565,6 +565,12 @@
>  
>  		rawStoreFactory = rsf;
>  		dataFactory     = df;
> +		
> +		// initialize the log writer only after the rawstorefactory is available, 
> +		// log writer requires encryption block size info from rawstore factory 
> +		// to encrypt checksum log records. 
> +		if (firstLog != null) 
> +			logOut = new LogAccessFile(this, firstLog, logBufferSize);
>  
>  		// we don't want to set ReadOnlyDB before recovery has a chance to look
>  		// at the latest checkpoint and determine that the database is shutdown
> @@ -1003,7 +1009,7 @@
>  				}
>  
>  				if (theLog != null)
> -					logOut = new LogAccessFile(theLog, logBufferSize);
> +					logOut = new LogAccessFile(this, theLog, logBufferSize);
>  				
>  				if(logSwitchRequired)
>  					switchLogFile();
> @@ -1850,7 +1856,8 @@
>  
>  					// write out an extra 0 at the end to mark the end of the log
>  					// file.
> -					logOut.writeInt(0);
> +					
> +					logOut.writeEndMarker(0);
>  
>  					endPosition += 4;
>  					//set that we are in log switch to prevent flusher 
> @@ -1859,6 +1866,7 @@
>  					// flush everything including the int we just wrote
>  					flush(logFileNumber, endPosition);
>  					
> +					
>  					// simulate out of log error after the switch over
>  					if (SanityManager.DEBUG)
>  					{
> @@ -1883,7 +1891,7 @@
>  						newLog.seek(endPosition);
>  					}
>  
> -					logOut = new LogAccessFile(newLog, logBufferSize);
> +					logOut = new LogAccessFile(this, newLog, logBufferSize);
>  					newLog = null;
>  
>  
> @@ -1972,7 +1980,7 @@
>  		@exception IOException Failed to flush to the log
>  	*/
>  	private void flushBuffer(long fileNumber, long wherePosition)
> -		throws IOException
> +		throws IOException, StandardException
>  	{
>  		synchronized (this) {
>  			if (fileNumber < logFileNumber)	// history
> @@ -2913,29 +2921,28 @@
>  					}
>  
>  					// don't need to try to delete it, we know it isn't there
> -                    StorageRandomAccessFile theLog = privRandomAccessFile(logFile, "rw");
> +                    firstLog = privRandomAccessFile(logFile, "rw");
>  
> -					if (!initLogFile(theLog, logFileNumber, LogCounter.INVALID_LOG_INSTANT))
> +					if (!initLogFile(firstLog, logFileNumber, LogCounter.INVALID_LOG_INSTANT))
>                      {
>  						throw StandardException.newException(
>                              SQLState.LOG_SEGMENT_NOT_EXIST, logFile.getPath());
>                      }
>  
> -					endPosition = theLog.getFilePointer();
> -					lastFlush = theLog.getFilePointer();
> +					endPosition = firstLog.getFilePointer();
> +					lastFlush = firstLog.getFilePointer();
>  
>                      //if write sync is true , prellocate the log file
>                      //and reopen the file in rws mode.
>                      if(isWriteSynced)
>                      {
>                          //extend the file by wring zeros to it
> -                        preAllocateNewLogFile(theLog);
> -                        theLog.close();
> -                        theLog=  privRandomAccessFile(logFile, "rws");
> +                        preAllocateNewLogFile(firstLog);
> +                        firstLog.close();
> +                        firstLog=  privRandomAccessFile(logFile, "rws");
>                          //postion the log at the current log end postion
> -                        theLog.seek(endPosition);
> +                        firstLog.seek(endPosition);
>                      }
> -					logOut = new LogAccessFile(theLog, logBufferSize);
>  
>  					if (SanityManager.DEBUG)
>  					{
> @@ -2949,6 +2956,7 @@
>  					// read only database
>  					ReadOnlyDB = true;
>  					logOut = null;
> +					firstLog = null;
>  				}
>  
>  				recoveryNeeded = false;
> @@ -3012,8 +3020,9 @@
>  				try {
>  					logOut.flushLogAccessFile();
>  					logOut.close();
> -				} catch (IOException ioe) {
>  				}
> +				catch (IOException ioe) {}
> +				catch(StandardException se){}
>  				logOut = null;
>  			}
>  		}
> @@ -3319,6 +3328,9 @@
>                      }
>  				}
>  
> +				//reserve the space for the checksum log record
> +				endPosition += logOut.reserveSpaceForChecksum(length, logFileNumber,endPosition);
> +
>  				// don't call currentInstant since we are already in a
>  				// synchronzied block 
>  				instant = 
> @@ -3793,7 +3805,23 @@
>  		return rawStoreFactory.getEncryptionBlockSize();
>  	}
>  
> +	/**
> +	   returns the length that will make the data to be multiple of encryption
> +	   block size based on the given length. Block cipher algorithms like DES 
> +	   and Blowfish ..etc  require their input to be an exact multiple of the block size.
> +	*/
> +	public int getEncryptedDataLength(int length)
> +	{
> +		if ((length % getEncryptionBlockSize()) != 0)
> +		{
> +			return length + getEncryptionBlockSize() - (length % getEncryptionBlockSize());
> +		}
>  
> +		return length;
> +	}
> +
> +
> +
>  	/**
>  	  Get the instant of the first record which was not
>  	  flushed.
> @@ -3939,16 +3967,23 @@
>  			Monitor.logMessage("TEST_LOG_INCOMPLETE_LOG_WRITE: writing " + bytesToWrite + 
>  				   " bytes out of " + length + " + " + LOG_RECORD_OVERHEAD + " log record");
>  
> -
> -
> -
> -			long instant = currentInstant();
> +			long instant;
>  			try
>  			{
> +								
>  				synchronized (this)
>  				{
> -						//check if the length of the records to be written is 
> -						//actually smaller than the number of bytesToWrite 
> +					// reserve the space for the checksum log record
> +					// NOTE:  bytesToWrite include the log record overhead.
> +					endPosition += 
> +						logOut.reserveSpaceForChecksum(((length + LOG_RECORD_OVERHEAD) 
> +														< bytesToWrite ? length :
> +														(bytesToWrite - LOG_RECORD_OVERHEAD)),
> +													   logFileNumber,endPosition);
> +					instant = currentInstant();
> +
> +					//check if the length of the records to be written is 
> +					//actually smaller than the number of bytesToWrite 
>  					if(length + LOG_RECORD_OVERHEAD < bytesToWrite)
>  						endPosition += (length + LOG_RECORD_OVERHEAD);
>  					else
> @@ -4073,7 +4108,26 @@
>  		}	
>  	}
>  
> +	/**
> +	 * Get the log file to Simulate a log corruption 
> +	 * FOR UNIT TESTING USAGE ONLY 
> +	*/
> +	public StorageRandomAccessFile getLogFileToSimulateCorruption(long filenum) throws IOException, StandardException
> +	{
> +		if (SanityManager.DEBUG)
> +		{
> +			//long filenum = LogCounter.getLogFileNumber(logInstant);
> +			//			long filepos = LogCounter.getLogFilePosition(logInstant);
> +			StorageFile fileName = getLogFileName(filenum);
> +			StorageRandomAccessFile log = null;
> +			return privRandomAccessFile(fileName, "rw");
> +		}
> +		
> +		return null;
>  
> +	}
> +	
> +
>  	/*********************************************************************
>  	 * Log Testing
>  	 * 
> @@ -4122,9 +4176,6 @@
>  	public static final String TEST_RECORD_TO_FILL_LOG = SanityManager.DEBUG ? "db2j.unittest.recordToFillLog" : null;
>  
>  
> -
> -
> -
>  	//enable the log archive mode
>  	public void enableLogArchiveMode() throws StandardException
>  	{
> Index: java/engine/org/apache/derby/impl/store/raw/log/Scan.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/Scan.java	(revision 158720)
> +++ java/engine/org/apache/derby/impl/store/raw/log/Scan.java	(working copy)
> @@ -39,6 +39,7 @@
>  import org.apache.derby.io.StorageRandomAccessFile;
>  
>  import java.io.IOException;
> +import org.apache.derby.iapi.store.raw.Loggable;
>  
>  /**
>  
> @@ -460,9 +461,20 @@
>  			}
>  
>  			lr = (LogRecord) input.readObject();
> -			if (groupmask != 0 || tranId != null)
> +
> +			// skip the checksum log records, there is no need to look at them 
> +			// during backward scans. They are used only in forwardscan during recovery. 
> +			if(lr.isChecksum())
>  			{
> -				if (groupmask != 0 && (groupmask & lr.group()) == 0)
> +				candidate = false; 
> +			}else if (groupmask != 0 || tranId != null)
> +			{
> +
> +				// skip the checksum log records  
> +				if(lr.isChecksum())
> +					candidate = false; 
> +
> +				if (candidate && groupmask != 0 && (groupmask & lr.group()) == 0)
>  					candidate = false; // no match, throw this log record out 
>  
>  				if (candidate && tranId != null)
> @@ -950,6 +962,93 @@
>  
>  			// the scan is now positioned just past this log record and right
>  			// at the beginning of the next log record
> +
> +
> +			/** if the current log record is a checksum log record then
> +			 * using the information available in this record validate
> +			 * that data in the log file by matching the checksum in 
> +			 * checksum log record and by recalculating the checksum for the 
> +			 * specified length of the data in the log file. cheksum values
> +			 * should match unless the right was incomplete before the crash.
> +			 */
> +			if(lr.isChecksum())
> +			{
> +				// checksum log record should not be returned to the logger recovery redo
> +				// routines, it is just used to identify the incomplete log writes.
> +
> +				candidate = false;
> +				Loggable op = lr.getLoggable(); 
> +				if (SanityManager.DEBUG)
> +                {
> +                    if (SanityManager.DEBUG_ON(LogToFile.DUMP_LOG_ONLY) ||
> +                        SanityManager.DEBUG_ON(LogToFile.DBG_FLAG))
> +
> +						SanityManager.DEBUG(LogToFile.DBG_FLAG, 
> +											"scanned " + "Null" + " : " + op + 
> +											" instant = " + 
> +											LogCounter.toDebugString(currentInstant) + 
> +											" logEnd = " +  LogCounter.toDebugString(knownGoodLogEnd));
> +				}
> +
> +				ChecksumOperation clop = (ChecksumOperation) op;
> +				int ckDataLength =  clop.getDataLength(); 
> +				// resize the buffer to be size of checksum data length if required.
> +				if (data.length < ckDataLength)
> +				{
> +					// make a new array of sufficient size and reset the arrary
> +					// in the input stream
> +					data = new byte[ckDataLength];
> +					input.setData(data);
> +					input.setLimit(0, ckDataLength);
> +				}
> +				
> +				boolean validChecksum = false;
> +				// check if the expected number of bytes by the checksum log
> +				// record actually exist in the file and then verify if checksum
> +				// is valid to identify any incomplete out of order writes.
> +				if((recordStartPosition + ckDataLength) <= currentLogFileLength)
> +				{
> +					// read the data into the buffer
> +					scan.readFully(data, 0, ckDataLength);
> +					// verify the checksum 
> +					if(clop.isChecksumValid(data, 0 , ckDataLength))
> +						validChecksum = true;
> +				}
> +
> +
> +				if(!validChecksum)
> +				{
> +					// declare that the end of the transaction log is fuzzy, checksum is invalid
> +					// only when the writes are incomplete; this can happen
> +					// only when writes at the end of the log were partially
> +					// written before the crash. 
> +
> +					if (SanityManager.DEBUG)
> +                    {
> +                        if (SanityManager.DEBUG_ON(LogToFile.DBG_FLAG))
> +                        {
> +                            SanityManager.DEBUG(
> +                                LogToFile.DBG_FLAG, 
> +                                "detected fuzzy log end on log file while doing checksum checks " + 
> +								currentLogFileNumber + 
> +                                " checksum record start position " + recordStartPosition + 
> +                                " file length " + currentLogFileLength + 
> +								" checksumDataLength=" + ckDataLength);
> +                        }
> +						
> +					}
> +					
> +					fuzzyLogEnd = true;
> +					scan.close();
> +					scan = null;
> +					return null;
> +				}
> +
> +				// reset the scan to the start of the next log record
> +				scan.seek(recordStartPosition);
> +			}
> +
> +
>  		} while (candidate == false) ;
>  
>  		return lr;
> Index: java/engine/org/apache/derby/impl/store/raw/log/LogAccessFileBuffer.java
> ===================================================================
> --- java/engine/org/apache/derby/impl/store/raw/log/LogAccessFileBuffer.java	(revision 158720)
> +++ java/engine/org/apache/derby/impl/store/raw/log/LogAccessFileBuffer.java	(working copy)
> @@ -38,6 +38,7 @@
>      protected byte[]    buffer;
>      protected int       bytes_free;
>      protected int       position;
> +	protected int       length;
>  
>      LogAccessFileBuffer next;
>      LogAccessFileBuffer prev;
> @@ -53,17 +54,18 @@
>          prev        = null;
>          next        = null;
>  
> -        init();
> +        init(0);
>      }
>  
>      /**************************************************************************
>       * Private/Protected methods of This class:
>       **************************************************************************
>       */
> -    public void init()
> +    public void init(int reserve)
>      {
> -        bytes_free  = buffer.length;
> -        position    = 0;
> +		length =  buffer.length - reserve;
> +        bytes_free  = length;
> +        position    = reserve;
>      }
>  
>      /**************************************************************************
> Index: java/engine/org/apache/derby/iapi/services/io/RegisteredFormatIds.java
> ===================================================================
> --- java/engine/org/apache/derby/iapi/services/io/RegisteredFormatIds.java	(revision 158720)
> +++ java/engine/org/apache/derby/iapi/services/io/RegisteredFormatIds.java	(working copy)
> @@ -512,8 +512,8 @@
>          /* 450 */       "org.apache.derby.iapi.types.DTSClassInfo", //InstanceGetter,
>  
>   
> -        /* 451 */   "org.apache.derby.catalog.types.RoutineAliasInfo"
> -
> -
> +        /* 451 */   "org.apache.derby.catalog.types.RoutineAliasInfo",
> +		/* 452 */   null,
> +		/* 453 */   "org.apache.derby.impl.store.raw.log.ChecksumOperation"
>  };
>  }
> Index: java/engine/org/apache/derby/iapi/services/io/StoredFormatIds.java
> ===================================================================
> --- java/engine/org/apache/derby/iapi/services/io/StoredFormatIds.java	(revision 158720)
> +++ java/engine/org/apache/derby/iapi/services/io/StoredFormatIds.java	(working copy)
> @@ -1682,7 +1682,11 @@
>          public static final int LOGOP_REMOVE_FILE =
>                  (MIN_ID_2 + 291);
>  
> +        /* org.apache.derby.impl.store.raw.log.ChecksumOperation */
> +        public static final int LOGOP_CHECKSUM =
> +                (MIN_ID_2 + 453);
>  
> +
>          /*******************************************************************
>          **
>          ** container types
> @@ -1802,7 +1806,7 @@
>           * Make sure this is updated when a new module is added
>           */
>          public static final int MAX_ID_2 =
> -                (MIN_ID_2 + 452);
> +                (MIN_ID_2 + 453);
>  
>          // DO NOT USE 4 BYTE IDS ANYMORE
>          static public final int MAX_ID_4 =
> Index: java/engine/org/apache/derby/iapi/store/raw/Loggable.java
> ===================================================================
> --- java/engine/org/apache/derby/iapi/store/raw/Loggable.java	(revision 158720)
> +++ java/engine/org/apache/derby/iapi/store/raw/Loggable.java	(working copy)
> @@ -160,7 +160,9 @@
>  
>  	public static final int RAWSTORE =		  0x100;	// a log record generated by the raw store
>  	public static final int FILE_RESOURCE =   0x400;    // related to "non-transactional" files.
> +	public static final int CHECKSUM =        0x800;    // a checksum log record 
>  
> +
>  	/**
>  		Get the loggable's group value
>  	*/
> Index: java/testing/org/apache/derbyTesting/unitTests/store/T_RecoverBadLog.java
> ===================================================================
> --- java/testing/org/apache/derbyTesting/unitTests/store/T_RecoverBadLog.java	(revision 158720)
> +++ java/testing/org/apache/derbyTesting/unitTests/store/T_RecoverBadLog.java	(working copy)
> @@ -30,15 +30,14 @@
>  import org.apache.derby.iapi.services.context.ContextManager;
>  import org.apache.derby.iapi.services.daemon.DaemonService;
>  import org.apache.derby.iapi.services.monitor.Monitor;
> -
>  import org.apache.derby.iapi.services.locks.LockFactory;
>  import org.apache.derby.iapi.services.io.Storable;
>  import org.apache.derby.iapi.services.sanity.SanityManager;
>  import org.apache.derby.iapi.reference.Property;
>  import org.apache.derby.iapi.reference.EngineType;
> -
>  import org.apache.derby.iapi.services.property.PropertyUtil;
>  import org.apache.derby.iapi.services.io.FormatableBitSet;
> +import org.apache.derby.io.StorageRandomAccessFile;
>  
>  import org.apache.derby.iapi.error.StandardException;
>  
> @@ -65,6 +64,9 @@
>  
>      Execute in order
>  
> +	To Test Bad Log due to partial write that are identified by checking the
> +	length in the beginning and end of the log record. 
> +
>  	java -DTestBadLogSetup=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
>  	java -DTestBadLog1=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
>  	java -DTestBadLog2=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> @@ -74,12 +76,27 @@
>  	java -DTestBadLog6=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
>  	java -DTestBadLog7=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
>  	java -DTestBadLog1=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +
> +	To Test Bad Log due to an incomplete out of order write that is identified
> +	by the checksum logic (simulated by	explicitly corrupting a middle of a 
> +	log record at  the  end of log file after it is written).
>  	
> +	java -DTestBadLogSetup=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog1=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog2=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog3=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog4=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog5=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog6=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog7=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	java -DTestBadLog1=true -DTestBadChecksumLog=true org.apache.derbyTesting.unitTests.harness.UnitTestMain
> +	
> +	
>  */
>  
>  public class T_RecoverBadLog extends T_Generic {
>  
> -	private static final String testService = "BadLogTest";
> +	private  String testService = "BadLogTest";
>  
>  	static final String REC_001 = "McLaren";
>  	static final String REC_002 = "Ferrari";
> @@ -102,9 +119,10 @@
>  	private boolean test5;
>  	private boolean test6;
>  	private boolean test7;
> +	private boolean checksumTest; 
> +	
> +	private  String infoPath = "extinout/T_RecoverBadLog.info";
>  
> -	private static final String infoPath = "extinout/T_RecoverBadLog.info";
> -
>  	private static final String TEST_BADLOG_SETUP = "TestBadLogSetup";
>  	private static final String TEST_BADLOG1 = "TestBadLog1";
>  	private static final String TEST_BADLOG2 = "TestBadLog2";
> @@ -114,10 +132,14 @@
>  	private static final String TEST_BADLOG6 = "TestBadLog6";
>  	private static final String TEST_BADLOG7 = "TestBadLog7";
>  
> +	private static final String TEST_BAD_CHECKSUM_LOG = "TestBadChecksumLog";
> +
>  	private static final String TEST_BADLOG_INFO = "TestBadLogInfo";
> +	private static final String TEST_BADCHECKSUMLOG_INFO = "TestBadChecksumLogInfo";
>  
>  	RawStoreFactory	factory;
>  	LockFactory  lf;
> +	LogToFile   logFactory;
>  	ContextService contextService;
>  	T_Util t_util;
>  
> @@ -163,6 +185,15 @@
>  
>  		param = PropertyUtil.getSystemProperty(TEST_BADLOG7);
>  		test7 = Boolean.valueOf(param).booleanValue();
> +		
> +		param = PropertyUtil.getSystemProperty(TEST_BAD_CHECKSUM_LOG);
> +		checksumTest = Boolean.valueOf(param).booleanValue();
> +		
> +		if(checksumTest)
> +		{
> +			infoPath = "extinout/T_RecoverBadChecksumLog.info";
> +			testService = "BadChecksumLogTest";
> +		}
>  	}
>  
>  
> @@ -183,7 +214,6 @@
>  		if (test5) tests++;
>  		if (test6) tests++;
>  		if (test7) tests++;
> -
>  		
>  		if (tests != 1)
>  			throw T_Fail.testFailMsg("One & only one of the bad log recovery test should be run");
> @@ -242,21 +272,25 @@
>  			else					// not setup, recover it
>  			{
>  				REPORT("_______________________________________________________");
> +				
> +				String message = "\n\t\tRunning bad log test ";
> +				if (checksumTest)
> +					message = "\n\t\tRunning bad checksum log test ";
>  				if (test1)
> -					REPORT("\n\t\tRunning bad log test 1");
> +					REPORT(message + " 1");
>  				if (test2)
> -					REPORT("\n\t\tRunning bad log test 2");
> +					REPORT(message + " 2");
>  				if (test3)
> -					REPORT("\n\t\tRunning bad log test 3");
> +					REPORT(message + " 3");
>  				if (test4)
> -					REPORT("\n\t\tRunning bad log test 4");
> +					REPORT(message + " 4");
>  				if (test5)
> -					REPORT("\n\t\tRunning bad log test 5");
> +					REPORT(message + " 5");
>  				if (test6)
> -					REPORT("\n\t\tRunning bad log test 6");
> +					REPORT(message + " 6");
>  				if (test7)
> -					REPORT("\n\t\tRunning bad log test 7");
> -				
> +					REPORT(message + " 7");
> +
>  				REPORT("_______________________________________________________");
>  
>  				//if external input output files does not exist ,create one
> @@ -283,7 +317,8 @@
>  					throw T_Fail.testFailMsg("Monitor didn't know how to restart service: " + testService);
>  
>  				factory = (RawStoreFactory) Monitor.findService(getModuleToTestProtocolName(), testService);
> -
> +				logFactory =(LogToFile) Monitor.findServiceModule(factory, factory.getLogFactoryModule());
> +				
>  			}
>  		} catch (StandardException mse) {
>  			throw T_Fail.exceptionFail(mse);
> @@ -302,6 +337,7 @@
>  		t_util = new T_Util(factory, lf, contextService);
>  
>  		try {
> +			
>  
>  			// these tests can be run in any order
>  			RTest1();
> @@ -464,16 +500,23 @@
>  			///////////////////////////////////////////
>  			factory.checkpoint();
>  
> -
>  			//////////////////////////////////////////////////////////
>  			// writing approx 1/2 log record to the end of the log - 
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(numcol*20));
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(numcol*20));
> +			}
> +			
> +			logFactory.flushAll();
>  
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
>  
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test1: cid = " + cid + " numcol " + numcol);
> @@ -596,10 +639,17 @@
>  			// writing approx 1/2 log record to the end of the log - 
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES,Integer.toString(numcol*20));
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES,Integer.toString(numcol*20));
> +			}
>  
> +			logFactory.flushAll();
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
> +			
> +			if(checksumTest)
> +				simulateLogFileCorruption();
>  
>  			////////////////////////////////////////////////////////
>  
> @@ -1002,9 +1052,13 @@
>  			// writing 200 bytes of the log record to the end of the log - 
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			//////////////////////////////////////////////////////////
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, "200");
> +			}
> +			logFactory.flushAll();
>  
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, "200");
>  
>  			// RESOLVE:
>  			// copy and purge actually generates 2 log records, this is
> @@ -1014,6 +1068,10 @@
>  			badPage1.copyAndPurge(badPage2, 0, i, 0);
>  
>  			t[3].resetContext();
> +
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test3: numtrans " + numtrans + " numpages " + numpages);
> @@ -1206,11 +1264,18 @@
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			// Length  4 bytes + 7(8) bytes of log record instance
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(11));
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(11));
> +			}
>  
> +			logFactory.flushAll();
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
>  
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test4: cid = " + cid + " numcol " + numcol);
> @@ -1340,11 +1405,18 @@
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			// Length  3 bytes (4) of log record length
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(3));
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(3));
> +			}
> +			logFactory.flushAll();
>  
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
>  
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test5: cid = " + cid + " numcol " + numcol);
> @@ -1472,11 +1544,17 @@
>  			// writing (1997/2 (data)+ 16(log records ov)) bytes of log record to the end of the log - 
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString((1997/2) + 16));
> -
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString((1997/2) + 16));
> +			}
> +			logFactory.flushAll();
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
>  
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test6: cid = " + cid + " numcol " + numcol);
> @@ -1605,11 +1683,18 @@
>  			//i.e: instead of (1997(data) + 16 (log records overhead)) write (1997 + 15) 
>  			// NO MORE LOG RECORD SHOULD BE WRITTEN,
>  			//////////////////////////////////////////////////////////
> -			SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> -			System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(1997+15));
> -
> +			if(!checksumTest)
> +			{
> +				SanityManager.DEBUG_SET(LogToFile.TEST_LOG_INCOMPLETE_LOG_WRITE);
> +				System.getProperties().put(LogToFile.TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES, Integer.toString(1997+15));
> +			}
> +			logFactory.flushAll();
>  			page.update(rh, bigrow.getRow(), (FormatableBitSet) null);
>  
> +			if(checksumTest)
> +				simulateLogFileCorruption();
> +
> +
>  			////////////////////////////////////////////////////////
>  
>  			REPORT("badlog test7: cid = " + cid + " numcol " + numcol);
> @@ -1678,6 +1763,54 @@
>  			t.close();
>  		}
>  	}
> +
> +
> +
> +	/*
> +	 * simulate log corruption to test the checksuming of log records. 
> +	 */
> +	private void simulateLogFileCorruption() throws T_Fail, StandardException
> +	{
> +		long filenum;
> +		long filepos;
> +		long amountOfLogWritten;
> +		LogCounter logInstant = (LogCounter)logFactory.getFirstUnflushedInstant();
> +		filenum = logInstant.getLogFileNumber();
> +		filepos = logInstant.getLogFilePosition();
> +		logFactory.flushAll();
> +		logInstant = (LogCounter)logFactory.getFirstUnflushedInstant();
> +		filenum = logInstant.getLogFileNumber();
> +		amountOfLogWritten = logInstant.getLogFilePosition() - filepos;
> +
> +		// write some random  garbage into the log file , 
> +		// purpose of doing this is to test that recovery works correctly when 
> +		// log records in the end of a log file did not get wrtten completely
> +		// and in the correct order. 
> +
> +		try{
> +			StorageRandomAccessFile log = logFactory.getLogFileToSimulateCorruption(filenum) ;
> +		
> +			int noWrites = (int) amountOfLogWritten / 512;
> +			//mess up few bytes in every block of a 512 bytes.
> +			filepos += 512;
> +			java.util.Random r = new java.util.Random();
> +			for(int i = 0 ; i < noWrites ; i++)
> +			{
> +				REPORT("corruptig log file : filenum " + filenum + " fileposition " + filepos);
> +				log.seek(filepos);
> +				log.writeInt(r.nextInt());
> +				filepos +=512;
> +
> +			}
> +			log.sync(false);
> +			log.close();
> +		}catch(IOException ie)
> +		{
> +			throw T_Fail.exceptionFail(ie);
> +		}
> +		
> +	}
> +
>  }
>  
>  
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLogSetup_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLogSetup_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLogSetup_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLogSetup=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLogSetup_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog2_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog2_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog2_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog2=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog2_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog4_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog4_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog4_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog4=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog4_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog6_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog6_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog6_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog6=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog6_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog1_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog1_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog1_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog1=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog1_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog3_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog3_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog3_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog3=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog3_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog5_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog5_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog5_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog5=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog5_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog7_derby.properties
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog7_derby.properties	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog7_derby.properties	(revision 0)
> @@ -0,0 +1,5 @@
> +derby.module.testManager=org.apache.derbyTesting.unitTests.harness.BasicUnitTestManager
> +derby.infolog.append=true
> +derby.module.test.recoverBadLog=org.apache.derbyTesting.unitTests.store.T_RecoverBadLog
> +TestBadLog7=true
> +TestBadChecksumLog=true
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/unit/recoverBadChecksumLog7_derby.properties
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog4.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog4.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog4.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog4.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLogSetup.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLogSetup.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLogSetup.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLogSetup.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog1.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog1.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog1.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog1.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog5.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog5.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog5.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog5.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog2.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog2.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog2.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog2.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog6.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog6.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog6.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog6.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog3.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog3.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog3.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog3.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog7.out
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog7.out	(revision 0)
> +++ java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog7.out	(revision 0)
> @@ -0,0 +1,2 @@
> +-- Unit Test T_RecoverBadLog starting
> +-- Unit Test T_RecoverBadLog finished
> 
> Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/recoverBadChecksumLog7.out
> ___________________________________________________________________
> Name: svn:eol-style
>    + native
> 
> Index: java/testing/org/apache/derbyTesting/functionTests/suites/storeunit.runall
> ===================================================================
> --- java/testing/org/apache/derbyTesting/functionTests/suites/storeunit.runall	(revision 158720)
> +++ java/testing/org/apache/derbyTesting/functionTests/suites/storeunit.runall	(working copy)
> @@ -1,26 +1,34 @@
> -unit/T_RawStoreFactory.unit
> -unit/T_FileSystemData.unit
> -unit/T_StreamFile.unit
> -unit/T_AccessFactory.unit
> -unit/T_Heap.unit
> -unit/T_b2i.unit
> -unit/T_SortController.unit
> -unit/TLockFactory.unit
> -unit/TUUIDFactory.unit
> -unit/recoverySetup.unit
> -unit/recoveryTest.unit
> -unit/recoverBadLogSetup.unit
> -unit/recoverBadLog1.unit
> -unit/recoverBadLog2.unit
> -unit/recoverBadLog3.unit
> -unit/recoverBadLog4.unit
> -unit/recoverBadLog5.unit
> -unit/recoverBadLog6.unit
> -unit/recoverBadLog7.unit
> -unit/recoverBadLog1.unit
> -unit/recoverBadLog2.unit
> -unit/fillLog.unit
> -unit/logSwitchFail.unit
> -unit/fullRecoveryFail.unit
> -unit/fullRecoveryFail.unit
> -unit/fullRecovery.unit
> +unit/T_RawStoreFactory.unit
> +unit/T_FileSystemData.unit
> +unit/T_StreamFile.unit
> +unit/T_AccessFactory.unit
> +unit/T_Heap.unit
> +unit/T_b2i.unit
> +unit/T_SortController.unit
> +unit/TLockFactory.unit
> +unit/TUUIDFactory.unit
> +unit/recoverySetup.unit
> +unit/recoveryTest.unit
> +unit/recoverBadLogSetup.unit
> +unit/recoverBadLog1.unit
> +unit/recoverBadLog2.unit
> +unit/recoverBadLog3.unit
> +unit/recoverBadLog4.unit
> +unit/recoverBadLog5.unit
> +unit/recoverBadLog6.unit
> +unit/recoverBadLog7.unit
> +unit/recoverBadLog1.unit
> +unit/recoverBadLog2.unit
> +unit/fillLog.unit
> +unit/logSwitchFail.unit
> +unit/fullRecoveryFail.unit
> +unit/fullRecoveryFail.unit
> +unit/fullRecovery.unit
> +unit/recoverBadChecksumLogSetup.unit
> +unit/recoverBadChecksumLog1.unit
> +unit/recoverBadChecksumLog2.unit
> +unit/recoverBadChecksumLog3.unit
> +unit/recoverBadChecksumLog4.unit
> +unit/recoverBadChecksumLog5.unit
> +unit/recoverBadChecksumLog6.unit
> +unit/recoverBadChecksumLog7.unit