Return-Path: Delivered-To: apmail-db-derby-dev-archive@www.apache.org Received: (qmail 9574 invoked from network); 18 May 2005 03:36:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 18 May 2005 03:36:56 -0000 Received: (qmail 78554 invoked by uid 500); 17 May 2005 22:49:12 -0000 Delivered-To: apmail-db-derby-dev-archive@db.apache.org Received: (qmail 78467 invoked by uid 500); 17 May 2005 22:49:11 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: List-Id: Reply-To: "Derby Development" Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 78381 invoked by uid 99); 17 May 2005 22:49:10 -0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=DNS_FROM_RFC_ABUSE,DNS_FROM_RFC_POST,SPF_HELO_FAIL X-Spam-Check-By: apache.org Received-SPF: neutral (hermes.apache.org: local policy) Received: from e34.co.us.ibm.com (HELO e34.co.us.ibm.com) (32.97.110.132) by apache.org (qpsmtpd/0.28) with ESMTP; Tue, 17 May 2005 15:49:08 -0700 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e34.co.us.ibm.com (8.12.10/8.12.9) with ESMTP id j4HMmqcu163840 for ; Tue, 17 May 2005 18:48:52 -0400 Received: from [9.72.134.65] (dyn9072134065.usca.ibm.com [9.72.134.65]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id j4HMmqnP022604 for ; Tue, 17 May 2005 16:48:52 -0600 Message-ID: <428A74DB.10902@sbcglobal.net> Date: Tue, 17 May 2005 15:48:59 -0700 From: Mike Matrigali User-Agent: Mozilla Thunderbird 0.7.2 (Windows/20040707) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Derby Development Subject: Re: [PATCH] increasinng the possible log file id's from 2^22 -1 to 2^33 -1 (derby-101) References: <428956A5.8050806@gmail.com> In-Reply-To: <428956A5.8050806@gmail.com> X-Enigmail-Version: 0.85.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N I committed this change with svn 170686. Suresh Thalamati wrote: > Attached is the patch to increase the max possible log files number from > 2^22 -1 to > 2^33 -1. Derby creates log file names in sequential order by > incrementing log file number > by one each time(like log1.dat ....logN.dat). id's are not reused, so > the last possible log file that > can be created currently is limited to 2^22 - 1 (4194303). Problem with > this limit is , > it can be possible hit on systems that creates lot of log records very > fast with the default > log file size (1MB) , for example if 1MB of log is created every > second this limit can be hit > in (4194303 / (60 * 60 * 24)) 48 days. > > Attached fix bumps the max possible log file number to 2^33 -1 by > borrowing some unused bits in the > log file number and log file size fields in the log instant. With the > same assumption as above if a > a 1MB of log file is created for every second; it will take > ((8589934591 / (60 * 60 * 24))/365) > approximately 272 years; Which I think is a reasonable limit. :-) > > I believe the new limit is a reasonable one , and not necessary to solve > this problem by > implementing a complicated solution that reuse the ids, at this moment. > > Testing : ran derbyall ; all tests passed. > New tests : Added couple of tests to test the max log file ids using > debug flags. > > svn status > M java\engine\org\apache\derby\impl\store\raw\log\LogCounter.java > M java\engine\org\apache\derby\impl\store\raw\log\LogToFile.java > M > java\testing\org\apache\derbyTesting\functionTests\tests\store\copyfiles.ant > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumberRecovery_derby.properties > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumber_app.properties > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumberRecovery_app.properties > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumber_derby.properties > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumber.java > > A > java\testing\org\apache\derbyTesting\functionTests\tests\store\MaxLogNumberRecovery.java > > A > java\testing\org\apache\derbyTesting\functionTests\master\MaxLogNumber.out > A > java\testing\org\apache\derbyTesting\functionTests\master\MaxLogNumberRecovery.out > > M > java\testing\org\apache\derbyTesting\functionTests\suites\storerecovery.runall > > $ > Thanks > -suresht > > > ------------------------------------------------------------------------ > > Index: java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java > =================================================================== > --- java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java (revision 170490) > +++ java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java (working copy) > @@ -62,13 +62,15 @@ > public static final long INVALID_LOG_INSTANT = 0; > > > - // reserve top 10 bits in log file number for future use > - public static final long MAX_LOGFILE_NUMBER = (long)0x003FFFFFL; // 4194303 > - private static final long FILE_NUMBER_SHIFT = 32; > + // max possible log file number is 2^33 -1 (8589934591) > + public static final long MAX_LOGFILE_NUMBER = (long)0x1FFFFFFFFL ; > + // lower end of 30 bits in a long type are used to store the log file postion. > + private static final long FILE_NUMBER_SHIFT = 30; > > - // reserve top 4 bits in log file size for future use > + // reserve top 2 bits in log file size for future use > public static final long MAX_LOGFILE_SIZE = (long)0x0FFFFFFFL; // 268435455 > - private static final long FILE_POSITION_MASK = (long)0x7FFFFFFFL; > + // 30 bits are used to store the log file postion > + private static final long FILE_POSITION_MASK = (long)0x3FFFFFFFL; > > private long fileNumber; > private long filePosition; > Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java > =================================================================== > --- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (revision 170490) > +++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java (working copy) > @@ -649,7 +649,7 @@ > if (beginLogFileNumber != null) > { > logFileNumber = > - Integer.valueOf(beginLogFileNumber).intValue(); > + Long.valueOf(beginLogFileNumber).longValue(); > } > else > { > @@ -2975,6 +2975,17 @@ > { > firstLogFileNumber = 1; > logFileNumber = 1; > + if (SanityManager.DEBUG) > + { > + if (SanityManager.DEBUG_ON(TEST_MAX_LOGFILE_NUMBER)) > + { > + // set the value to be two less than max possible > + // log number, test case will perform some ops to > + // hit the max number case. > + firstLogFileNumber = LogCounter.MAX_LOGFILE_NUMBER -2; > + logFileNumber = LogCounter.MAX_LOGFILE_NUMBER -2; > + } > + } > logFile = getLogFileName(logFileNumber); > > if (privExists(logFile)) > @@ -4293,7 +4304,12 @@ > */ > public static final String TEST_RECORD_TO_FILL_LOG = SanityManager.DEBUG ? "db2j.unittest.recordToFillLog" : null; > > + /** > + * Set to true if we want to simulate max possible log file number is being used. > + */ > + public static final String TEST_MAX_LOGFILE_NUMBER = SanityManager.DEBUG ? "testMaxLogFileNumber" : null; > > + > //enable the log archive mode > public void enableLogArchiveMode() throws StandardException > { > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant (revision 170490) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/copyfiles.ant (working copy) > @@ -131,3 +131,8 @@ > LogChecksumRecovery_derby.properties > LogChecksumRecovery1_app.properties > LogChecksumRecovery1_derby.properties > +MaxLogNumber_app.properties > +MaxLogNumber_derby.properties > +MaxLogNumberRecovery_app.properties > +MaxLogNumberRecovery_derby.properties > + > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_derby.properties > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_derby.properties (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_derby.properties (revision 0) > @@ -0,0 +1 @@ > +derby.debug.true=testMaxLogFileNumber > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_derby.properties > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_app.properties > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_app.properties (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_app.properties (revision 0) > @@ -0,0 +1,13 @@ > +# > +# *** DO NOT PUT PROPERTIES FOR THE DERBY SYSTEM IN THIS FILE. THEY BELONG > +# *** IN the _derby.properties file. > +# > +# It will get handed to the test on the command line in a -p > +# argument. > +# > +# This causes ij (or the GUI on ij) to load the driver and make an > +# initial connection to the database. > +# > +# > +database=jdbc:derby:maxlogwombat;create=true > + > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_app.properties > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_app.properties > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_app.properties (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_app.properties (revision 0) > @@ -0,0 +1,12 @@ > +# > +# *** DO NOT PUT PROPERTIES FOR THE DERBY SYSTEM IN THIS FILE. THEY BELONG > +# *** IN the _derby.properties file. > +# > +# It will get handed to the test on the command line in a -p > +# argument. > +# > +# This causes ij (or the GUI on ij) to load the driver and make an > +# initial connection to the database. > +# > +# > +database=jdbc:derby:maxlogwombat > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery_app.properties > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_derby.properties > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_derby.properties (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_derby.properties (revision 0) > @@ -0,0 +1 @@ > +derby.debug.true=testMaxLogFileNumber > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber_derby.properties > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber.java > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber.java (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber.java (revision 0) > @@ -0,0 +1,216 @@ > +/* > + > + Derby - Class org.apache.derbyTesting.functionTests.store.MaxLogNumber > + > + Copyright 2005 The Apache Software Foundation or its licensors, as applicable. > + > + Licensed under the Apache License, Version 2.0 (the "License"); > + you may not use this file except in compliance with the License. > + You may obtain a copy of the License at > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > + Unless required by applicable law or agreed to in writing, software > + distributed under the License is distributed on an "AS IS" BASIS, > + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + See the License for the specific language governing permissions and > + limitations under the License. > + > + */ > + > +package org.apache.derbyTesting.functionTests.tests.store; > +import java.sql.Connection; > +import java.sql.Statement; > +import java.sql.PreparedStatement; > +import java.sql.ResultSet; > +import java.sql.SQLException; > +import org.apache.derby.tools.ij; > + > +/* > + * This class tests log writes to the transaction log files with large log file > + * id's and do a setup to test recovery with large log file id's in > + * MaxLogNumberRecovery.java test. large log file id's are simulated using > + * a debug flag 'testMaxLogFileNumber' in the log factory, this is enabled > + * by setting derby.debug.true=testMaxLogFileNumber in the properties file. > + * In Non debug mode, this tests just acts as a plain log recovery test. > + * > + * @author Suresh Thalamati > + * @version 1.0 > + */ > + > +public class MaxLogNumber{ > + > + MaxLogNumber() { > + } > + > + > + private void runTest(Connection conn) throws SQLException { > + logMessage("Begin MaxLogNumber Test"); > + // perform a checkpoint otherwise recovery test will look at log1 > + // instead of the log number that gets by the testMaxLogFileNumber > + // debug flags. > + performCheckPoint(conn); > + createTable(conn); > + insert(conn, 100, COMMIT, 10); > + insert(conn, 100, ROLLBACK, 10); > + update(conn, 50, COMMIT, 10); > + update(conn, 50, ROLLBACK, 10); > + verifyData(conn, 100); > + //do some inserts that will be rolled back by recovey > + insert(conn, 2000, NOACTION, 2000); > + logMessage("End MaxLogNumber Test"); > + } > + > + void performCheckPoint(Connection conn) throws SQLException > + { > + Statement stmt = conn.createStatement(); > + //wait to make sure that checkpoint thread finished it's work > + stmt.executeUpdate("CALL SYSCS_UTIL.SYSCS_CHECKPOINT_DATABASE()"); > + stmt.close(); > + } > + > + > + /** > + * Insert some rows into the table. > + */ > + void insert(Connection conn, int rowCount, > + int txStatus, int commitCount) throws SQLException { > + > + PreparedStatement ps = conn.prepareStatement("INSERT INTO " + > + "emp" + > + " VALUES(?,?,?)"); > + for (int i = 0; i < rowCount; i++) { > + > + ps.setInt(1, i); // ID > + ps.setString(2 , "skywalker" + i); > + ps.setFloat(3, (float)(i * 2000)); > + ps.executeUpdate(); > + if ((i % commitCount) == 0) > + { > + endTransaction(conn, txStatus); > + } > + } > + > + endTransaction(conn, txStatus); > + ps.close(); > + } > + > + > + static final int COMMIT = 1; > + static final int ROLLBACK = 2; > + static final int NOACTION = 3; > + > + void endTransaction(Connection conn, int txStatus) throws SQLException > + { > + switch(txStatus){ > + case COMMIT: > + conn.commit(); > + break; > + case ROLLBACK: > + conn.rollback(); > + break; > + case NOACTION: > + //do nothing > + break; > + } > + } > + > + /** > + * update some rows in the table. > + */ > + > + void update(Connection conn, int rowCount, > + int txStatus, int commitCount) throws SQLException > + { > + > + PreparedStatement ps = conn.prepareStatement("update " + "emp" + > + " SET salary=? where id=?"); > + > + for (int i = 0; i < rowCount; i++) { > + > + ps.setFloat(1, (float)(i * 2000 * 0.08)); > + ps.setInt(2, i); // ID > + ps.executeUpdate(); > + if ((i % commitCount) == 0) > + { > + endTransaction(conn, txStatus); > + } > + } > + endTransaction(conn, txStatus); > + ps.close(); > + } > + > + > + /* > + * verify the rows in the table. > + */ > + void verifyData(Connection conn, int expectedRowCount) throws SQLException { > + > + Statement s = conn.createStatement(); > + ResultSet rs = s.executeQuery("SELECT ID, name from emp order by id" ); > + int count = 0; > + int id = 0; > + while(rs.next()) > + { > + int tid = rs.getInt(1); > + String name = rs.getString(2); > + if(name.equals("skywalker" + id) && tid!= id) > + { > + > + logMessage("DATA IN THE TABLE IS NOT AS EXPECTED"); > + logMessage("Got :ID=" + tid + " Name=:" + name); > + logMessage("Expected: ID=" + id + "Name=" + "skywalker" + id ); > + } > + > + id++; > + count++; > + } > + > + if(count != expectedRowCount) > + { > + logMessage("Expected Number Of Rows (" + > + expectedRowCount + ")" + "!=" + > + "No Of rows in the Table(" + > + count + ")"); > + } > + s.close(); > + } > + > + /* > + * create the tables that are used by this test. > + */ > + void createTable(Connection conn) throws SQLException { > + > + Statement s = conn.createStatement(); > + s.executeUpdate("CREATE TABLE " + "emp" + > + "(id INT," + > + "name CHAR(200),"+ > + "salary float)"); > + s.executeUpdate("create index emp_idx on emp(id) "); > + conn.commit(); > + s.close(); > + } > + > + void logMessage(String str) > + { > + System.out.println(str); > + } > + > + > + public static void main(String[] argv) throws Throwable { > + > + MaxLogNumber test = new MaxLogNumber(); > + ij.getPropertyArg(argv); > + Connection conn = ij.startJBMS(); > + conn.setAutoCommit(false); > + > + try { > + test.runTest(conn); > + } > + catch (SQLException sqle) { > + org.apache.derby.tools.JDBCDisplayUtil.ShowSQLException( > + System.out, sqle); > + sqle.printStackTrace(System.out); > + } > + } > +} > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumber.java > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery.java > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery.java (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery.java (revision 0) > @@ -0,0 +1,104 @@ > +/* > + > + Derby - Class org.apache.derbyTesting.functionTests.store.MaxLogNumber > + > + Copyright 2005 The Apache Software Foundation or its licensors, as applicable. > + > + Licensed under the Apache License, Version 2.0 (the "License"); > + you may not use this file except in compliance with the License. > + You may obtain a copy of the License at > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > + Unless required by applicable law or agreed to in writing, software > + distributed under the License is distributed on an "AS IS" BASIS, > + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + See the License for the specific language governing permissions and > + limitations under the License. > + > + */ > + > +package org.apache.derbyTesting.functionTests.tests.store; > +import java.sql.Connection; > +import java.sql.SQLException; > +import org.apache.derby.tools.ij; > +import org.apache.derby.iapi.services.sanity.SanityManager; > + > +/* > + * This class tests recovery logic with large log file id's and the error > + * handling logic when Max possible log file limit is reached. MaxLogNumber.java > + * test does the setup, so it should be run before this test. > + * In Non debug mode, this tests just acts as a plain log recovery test. > + * > + * @author Suresh Thalamati > + * @version 1.0 > + * @see MaxLogNumber > + */ > + > +public class MaxLogNumberRecovery extends MaxLogNumber { > + > + MaxLogNumberRecovery() { > + super(); > + } > + > + private void runTest(Connection conn) throws SQLException { > + logMessage("Begin MaxLogNumberRecovery Test"); > + verifyData(conn, 100); > + boolean hitMaxLogLimitError = false; > + try{ > + insert(conn, 110, COMMIT, 11); > + update(conn, 110, ROLLBACK, 5); > + update(conn, 110, NOACTION, 5); > + verifyData(conn, 210); > + if (SanityManager.DEBUG) > + { > + // do lot of inserts in debug mode , > + // so that actuall reach the max log file number > + // limit > + insert(conn, 11000, COMMIT, 5); > + } > + } catch(SQLException se) { > + > + SQLException ose = se; > + while (se != null) { > + if ("XSLAK".equals(se.getSQLState())) { > + hitMaxLogLimitError = true; > + break; > + } > + se = se.getNextException(); > + } > + if(!hitMaxLogLimitError) > + throw ose; > + } > + > + if (SanityManager.DEBUG) > + { > + // In the debug build mode , this test should hit the max log limit while > + // doing above DML. > + if(!hitMaxLogLimitError) > + logMessage("Expected: ERROR XSLAK:" + > + "Database has exceeded largest log file" + > + "number 8,589,934,591."); > + } > + > + logMessage("End MaxLogNumberRecovery Test"); > + } > + > + > + public static void main(String[] argv) throws Throwable { > + > + MaxLogNumberRecovery test = new MaxLogNumberRecovery(); > + ij.getPropertyArg(argv); > + Connection conn = ij.startJBMS(); > + conn.setAutoCommit(false); > + > + try { > + test.runTest(conn); > + } > + catch (SQLException sqle) { > + org.apache.derby.tools.JDBCDisplayUtil.ShowSQLException( > + System.out, sqle); > + sqle.printStackTrace(System.out); > + } > + } > +} > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/tests/store/MaxLogNumberRecovery.java > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumber.out > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumber.out (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumber.out (revision 0) > @@ -0,0 +1,2 @@ > +Begin MaxLogNumber Test > +End MaxLogNumber Test > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumber.out > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumberRecovery.out > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumberRecovery.out (revision 0) > +++ java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumberRecovery.out (revision 0) > @@ -0,0 +1,2 @@ > +Begin MaxLogNumberRecovery Test > +End MaxLogNumberRecovery Test > > Property changes on: java/testing/org/apache/derbyTesting/functionTests/master/MaxLogNumberRecovery.out > ___________________________________________________________________ > Name: svn:eol-style > + native > > Index: java/testing/org/apache/derbyTesting/functionTests/suites/storerecovery.runall > =================================================================== > --- java/testing/org/apache/derbyTesting/functionTests/suites/storerecovery.runall (revision 170490) > +++ java/testing/org/apache/derbyTesting/functionTests/suites/storerecovery.runall (working copy) > @@ -1,3 +1,5 @@ > store/LogChecksumSetup.java > store/LogChecksumRecovery.java > store/LogChecksumRecovery1.java > +store/MaxLogNumber.java > +store/MaxLogNumberRecovery.java