db-derby-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mi...@apache.org
Subject svn commit: r1307832 - in /db/derby/code/branches/10.7/java: engine/org/apache/derby/impl/store/raw/data/ engine/org/apache/derby/impl/store/raw/xact/ testing/org/apache/derbyTesting/functionTests/tests/store/ testing/org/apache/derbyTesting/junit/
Date Sat, 31 Mar 2012 15:37:52 GMT
Author: mikem
Date: Sat Mar 31 15:37:51 2012
New Revision: 1307832

URL: http://svn.apache.org/viewvc?rev=1307832&view=rev
Log:
r1307679 | mikem | 2012-03-30 18:12:49 -0700 (Fri, 30 Mar 2012) | 34 lines

DERBY-5624 System can run out of stack space while processing DropOnCommit requests.

backporting changes #1307679 from 10.8 to 10.7 branch.

Taking care of cleanup after a commit is handled by notifying all "Observers"
that an event has taken place that they might want to act on and cleanup. In
the added test case this is triggered by off line commit which effectively
drops and recreates the base table and all of its indexes after loading the
data into them.

Sometimes these Observers may execute work which adds to the Observer queue,
and that queue can "miss" them in the first pass through.

A previous fix for this problem added a recursive call to notifyObservers in
the place that could cause this addition of observers. This recursive call
was causing stack problems when the number of Observers became large. For
the checked in test case this was 1000 indexes on 1000 columns of the table.
For other users I believe the cause was a by product of sorts on large disk
based sorts for multi-gigabyte tables and indexes. 2 users were reporting
similar failed stacks for failing compresses of large tables, and one was
able to take this fix to their environment and then successfully run the
compress.

The fix was to remove the recursion and instead loop at the outermost point
until there were no Observers.

backport to largedata suite did not go smoothly, so just put the test into
the store suite.

Only run the testDERBY_5624 on windows until linux issue resolved.

Currently on linux with 1024 file descriptors per user this test fails.


Added:
    db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5624Test.java
Modified:
    db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
    db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
    db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java
    db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java

Modified: db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java?rev=1307832&r1=1307831&r2=1307832&view=diff
==============================================================================
--- db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
(original)
+++ db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
Sat Mar 31 15:37:51 2012
@@ -98,14 +98,6 @@ public class DropOnCommit extends Contai
 			}
 
 			obj.deleteObserver(this);
-
-            // DERBY-3993
-            // make sure any observer that may have been added by either
-            // dropContainer() or dropStreamContainer() is also handled.
-            // The calling notifyObservers() call from Xact.doComplete()
-            // may not "see" new observers added during processing of the
-            // initial observer list.
-            xact.notifyObservers(arg);
 		}
 	}
 }

Modified: db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java?rev=1307832&r1=1307831&r2=1307832&view=diff
==============================================================================
--- db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
(original)
+++ db/derby/code/branches/10.7/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
Sat Mar 31 15:37:51 2012
@@ -1960,20 +1960,25 @@ public class Xact extends RawTransaction
 		if (savePoints != null)
 			savePoints.removeAllElements();
 
-		// notify any of our observers that we are completing.
-		notifyObservers(commitOrAbort);
+        do
+        {
+            // notify any of our observers that we are completing.
+            notifyObservers(commitOrAbort);
 
-		checkObserverException();
+            checkObserverException();
 
-		if (SanityManager.DEBUG) 
-        {
-			if (countObservers() != 0)
-            {
-                SanityManager.THROWASSERT(
-                    "There should be 0 observers, but we still have "
-					+ countObservers() + " observers.");
-            }
-		}
+            // DERBY-3993
+            // make sure any observer that may have been added by either
+            // dropContainer() or dropStreamContainer() is also handled.
+            // The calling notifyObservers() call from Xact.doComplete()
+            // may not "see" new observers added during processing of the
+            // initial observer list.  So loop until notifyObservers()
+            // call causes countObservers() to go to 0.  This should only
+            // loop if one of the observers adds to the list as part of
+            // the notify.  Even then depending on ordering the added
+            // observer may be picked up in the first try.
+
+        } while (countObservers() > 0);
 	}
 
 	private void checkObserverException() throws StandardException {

Added: db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5624Test.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5624Test.java?rev=1307832&view=auto
==============================================================================
--- db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5624Test.java
(added)
+++ db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/Derby5624Test.java
Sat Mar 31 15:37:51 2012
@@ -0,0 +1,182 @@
+package org.apache.derbyTesting.functionTests.tests.store;
+
+import org.apache.derbyTesting.junit.BaseJDBCTestCase;
+import org.apache.derbyTesting.junit.CleanDatabaseTestSetup;
+import org.apache.derbyTesting.junit.DatabasePropertyTestSetup;
+import org.apache.derbyTesting.junit.JDBC;
+import org.apache.derbyTesting.junit.TestConfiguration;
+
+import org.apache.derby.shared.common.sanity.SanityManager;
+
+import junit.framework.Assert;
+import junit.framework.Test;
+import junit.framework.TestSuite;
+
+import java.sql.CallableStatement;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.sql.SQLException;
+
+
+/*
+Class org.apache.derbyTesting.functionTests.tests.store.Derby5624Test
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+*/
+
+
+/**
+
+Test to reproduce DERBY-5624, a recursion during DropOnCommit causes out
+of stack space for operations that generate a lot of objects to be dropped
+at commit time. 
+
+This test reproduces the problem by creating a table with 1000 columns, then
+an index on each of those columns, loads some data and then call compress
+will drop and recreate each of those indexes.  At commit time each index
+drop will have registered itself onto the Observer list for processing at
+commit time.  Before fix this would fail with out of disk space in at least
+XP, ibm16 default jvm configuration.
+
+**/
+
+public class Derby5624Test extends BaseJDBCTestCase
+{
+    /**************************************************************************
+     * Fields of the class
+     **************************************************************************
+     */
+
+    /**************************************************************************
+     * Constructors for This class:
+     **************************************************************************
+     */
+
+    /**************************************************************************
+     * Private/Protected methods of This class:
+     **************************************************************************
+     */
+
+    /**************************************************************************
+     * Public Methods of This class:
+     **************************************************************************
+     */
+
+    /**************************************************************************
+     * Public Methods of XXXX class:
+     **************************************************************************
+     */
+
+    public Derby5624Test(String name) 
+    {
+        super(name);
+    }
+
+    
+    /**
+     * DERBY-5624 test case
+     * <p>
+     *
+     **/
+    public void testDERBY_5624()
+        throws SQLException
+    {
+        Statement stmt = createStatement();
+
+        PreparedStatement insert_stmt = 
+            prepareStatement(
+                "INSERT INTO TESTBIGTABLE (col1, col2, col3) VALUES(?, ?, ?)");
+
+        for (int i = 0; i < 3000; i++)
+        {
+            insert_stmt.setInt(1, i);
+            insert_stmt.setInt(2, -(i));
+            insert_stmt.setInt(3, i);
+            insert_stmt.executeUpdate();
+        }
+
+        commit();
+
+        // create index on each column
+        for (int i = 0; i < 1000; i++)
+        {
+            stmt.executeUpdate(
+                "CREATE INDEX INDEXBIG" + i + 
+                " on TESTBIGTABLE (col" + i + ")");
+        }
+        commit();
+
+        // see if compress succeeeds 
+        stmt.executeUpdate(
+            "call SYSCS_UTIL.SYSCS_COMPRESS_TABLE('APP', 'TESTBIGTABLE', 0)");
+
+        commit();
+
+        // verify access to table after the commit, previous to fix the
+        // commit would fail with an out of memory or out of stack space error.
+        JDBC.assertUnorderedResultSet(
+            prepareStatement(
+                "select col1, col2 from TESTBIGTABLE where col1 = 10").executeQuery(),
+            new String[][] {{"10", "-10"}});
+
+    }
+
+    protected static Test baseSuite(String name) 
+    {
+        TestSuite suite = new TestSuite(name);
+        suite.addTestSuite(Derby5624Test.class);
+        return new CleanDatabaseTestSetup(
+                DatabasePropertyTestSetup.setLockTimeouts(suite, 2, 4)) 
+        {
+            /**
+             * Creates the tables used in the test cases.
+             * @exception SQLException if a database error occurs
+             */
+            protected void decorateSQL(Statement stmt) throws SQLException
+            {
+                Connection conn = stmt.getConnection();
+
+                // create table with 1000 columns
+                StringBuffer create_table_qry = new StringBuffer(10000);
+
+                create_table_qry.append("CREATE TABLE TESTBIGTABLE (col0 int");
+
+                for (int colnum = 1; colnum < 1000; colnum++)
+                {
+                    create_table_qry.append(", col" + colnum + " int");
+                }
+                create_table_qry.append(")");
+
+                // CREATE TABLE TESTBIGTABLE (
+                //     col0 int, col1 int, ... , col999 int)
+                stmt.executeUpdate(create_table_qry.toString());
+
+                conn.setAutoCommit(false);
+            }
+        };
+    }
+
+    public static Test suite() 
+    {
+        TestSuite suite = new TestSuite("Derby5624Test");
+        if (isWindowsPlatform())
+            suite.addTest(baseSuite("Derby5624Test:embedded"));
+        return suite;
+    }
+}

Modified: db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java?rev=1307832&r1=1307831&r2=1307832&view=diff
==============================================================================
--- db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java
(original)
+++ db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/functionTests/tests/store/_Suite.java
Sat Mar 31 15:37:51 2012
@@ -57,6 +57,7 @@ public class _Suite extends BaseTestCase
         suite.addTest(StreamingColumnTest.suite());
         suite.addTest(Derby3625Test.suite());
         suite.addTest(Derby4577Test.suite());
+        suite.addTest(Derby5624Test.suite());
         suite.addTest(Derby151Test.suite());
         suite.addTest(Derby4676Test.suite());
         suite.addTest(BootLockTest.suite());

Modified: db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java?rev=1307832&r1=1307831&r2=1307832&view=diff
==============================================================================
--- db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
(original)
+++ db/derby/code/branches/10.7/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
Sat Mar 31 15:37:51 2012
@@ -564,6 +564,83 @@ public abstract class BaseTestCase
                 "Oracle Corporation".equals(vendor);
     }
     
+    public static final boolean isPlatform(String osName)  {
+
+        return getSystemProperty("os.name").equals(osName);
+    }
+
+    /**
+     * Determine if platform is a Windows variant.
+     * <p>
+     * Return true if platform is a windows platform.  Just looks for
+     * os.name starting with "Windows".  The os.name property
+     * can have at least the following values (there are probably more):
+     *
+     * AIX
+     * Digital Unix
+     * FreeBSD
+     * HP UX
+     * Irix
+     * Linux
+     * Mac OS
+     * Mac OS X
+     * MPE/iX
+     * Netware 4.11
+     * OS/2
+     * Solaris
+     * Windows 2000
+     * Windows 95
+     * Windows 98
+     * Windows NT
+     * Windows Vista
+     * Windows XP
+     * <p>
+     *
+     * @return true if running on a Windows platform.
+     **/
+    public static final boolean isWindowsPlatform() {
+        return getSystemProperty("os.name").startsWith("Windows");
+    }
+    
+    /**
+     * Check if this is java 5
+     * @return true if java.version system property starts with 1.5
+     */
+    public static final boolean isJava5() {
+        return getSystemProperty("java.version").startsWith("1.5");
+    }
+   
+    /**
+     * Returns the major version of the class specification version supported
+     * by the running JVM.
+     * <ul>
+     *  <li>48 = Java 1.4</li>
+     *  <li>49 = Java 1.5</li>
+     *  <li>50 = Java 1.6</li>
+     *  <li>51 = Java 1.7</li>
+     * </ul>
+     *
+     * @return Major version of class version specification, i.e. 49 for 49.0,
+     *      or -1 if the version can't be obtained for some reason.
+     */
+    public static int getClassVersionMajor() {
+        String tmp = getSystemProperty("java.class.version");
+        if (tmp == null) {
+            println("VM doesn't have property java.class.version");
+            return -1;
+        }
+        // Is String.split safe to use by now?
+        int dot = tmp.indexOf('.');
+        int major = -1;
+        try {
+            major = Integer.parseInt(tmp.substring(0, dot));
+        } catch (NumberFormatException nfe) {
+            // Ignore, return -1.
+        }
+        return major;
+    }
+
+
     public static final boolean isIBMJVM() {
         return ("IBM Corporation".equals(
                 getSystemProperty("java.vendor")));



Mime
View raw message