db-derby-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mi...@apache.org
Subject svn commit: r1307679 - in /db/derby/code/branches/10.8/java: engine/org/apache/derby/impl/store/raw/data/ engine/org/apache/derby/impl/store/raw/xact/ testing/org/apache/derbyTesting/functionTests/tests/largedata/ testing/org/apache/derbyTesting/junit/
Date Sat, 31 Mar 2012 01:12:49 GMT
Author: mikem
Date: Sat Mar 31 01:12:49 2012
New Revision: 1307679

URL: http://svn.apache.org/viewvc?rev=1307679&view=rev
Log:
DERBY-5624 System can run out of stack space while processing DropOnCommit requests.

backporting changes #1292079, #1292096, #1292432, and #1292595 from trunk to 
10.8 branch.

Taking care of cleanup after a commit is handled by notifying all "Observers"
that an event has taken place that they might want to act on and cleanup. In
the added test case this is triggered by off line commit which effectively
drops and recreates the base table and all of its indexes after loading the
data into them.

Sometimes these Observers may execute work which adds to the Observer queue,
and that queue can "miss" them in the first pass through.

A previous fix for this problem added a recursive call to notifyObservers in
the place that could cause this addition of observers. This recursive call
was causing stack problems when the number of Observers became large. For
the checked in test case this was 1000 indexes on 1000 columns of the table.
For other users I believe the cause was a by product of sorts on large disk
based sorts for multi-gigabyte tables and indexes. 2 users were reporting
similar failed stacks for failing compresses of large tables, and one was
able to take this fix to their environment and then successfully run the
compress.

The fix was to remove the recursion and instead loop at the outermost point
until there were no Observers.

Adding the test to the largedata suite as it takes over 10 minutes to run
on my machine. 
Only run the testDERBY_5624 in largedata on windows until linux issue resolved.

Currently on linux with 1024 file descriptors per user this test fails. 


Added:
    db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java
      - copied, changed from r1292096, db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java
Modified:
    db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
    db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
    db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/_Suite.java
    db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java

Modified: db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java?rev=1307679&r1=1307678&r2=1307679&view=diff
==============================================================================
--- db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
(original)
+++ db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/data/DropOnCommit.java
Sat Mar 31 01:12:49 2012
@@ -98,14 +98,6 @@ public class DropOnCommit extends Contai
 			}
 
 			obj.deleteObserver(this);
-
-            // DERBY-3993
-            // make sure any observer that may have been added by either
-            // dropContainer() or dropStreamContainer() is also handled.
-            // The calling notifyObservers() call from Xact.doComplete()
-            // may not "see" new observers added during processing of the
-            // initial observer list.
-            xact.notifyObservers(arg);
 		}
 	}
 }

Modified: db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java?rev=1307679&r1=1307678&r2=1307679&view=diff
==============================================================================
--- db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
(original)
+++ db/derby/code/branches/10.8/java/engine/org/apache/derby/impl/store/raw/xact/Xact.java
Sat Mar 31 01:12:49 2012
@@ -1960,20 +1960,25 @@ public class Xact extends RawTransaction
 		if (savePoints != null)
 			savePoints.removeAllElements();
 
-		// notify any of our observers that we are completing.
-		notifyObservers(commitOrAbort);
+        do
+        {
+            // notify any of our observers that we are completing.
+            notifyObservers(commitOrAbort);
 
-		checkObserverException();
+            checkObserverException();
 
-		if (SanityManager.DEBUG) 
-        {
-			if (countObservers() != 0)
-            {
-                SanityManager.THROWASSERT(
-                    "There should be 0 observers, but we still have "
-					+ countObservers() + " observers.");
-            }
-		}
+            // DERBY-3993
+            // make sure any observer that may have been added by either
+            // dropContainer() or dropStreamContainer() is also handled.
+            // The calling notifyObservers() call from Xact.doComplete()
+            // may not "see" new observers added during processing of the
+            // initial observer list.  So loop until notifyObservers()
+            // call causes countObservers() to go to 0.  This should only
+            // loop if one of the observers adds to the list as part of
+            // the notify.  Even then depending on ordering the added
+            // observer may be picked up in the first try.
+
+        } while (countObservers() > 0);
 	}
 
 	private void checkObserverException() throws StandardException {

Copied: db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java
(from r1292096, db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java)
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java?p2=db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java&p1=db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java&r1=1292096&r2=1307679&rev=1307679&view=diff
==============================================================================
--- db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java
(original)
+++ db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/Derby5624Test.java
Sat Mar 31 01:12:49 2012
@@ -43,7 +43,16 @@ limitations under the License.
 
 /**
 
-Test to reproduce DERBY-5624, An expanding update fails with an nospc.U error.
+Test to reproduce DERBY-5624, a recursion during DropOnCommit causes out
+of stack space for operations that generate a lot of objects to be dropped
+at commit time. 
+
+This test reproduces the problem by creating a table with 1000 columns, then
+an index on each of those columns, loads some data and then call compress
+will drop and recreate each of those indexes.  At commit time each index
+drop will have registered itself onto the Observer list for processing at
+commit time.  Before fix this would fail with out of disk space in at least
+XP, ibm16 default jvm configuration.
 
 **/
 
@@ -119,7 +128,8 @@ public class Derby5624Test extends BaseJ
 
         commit();
 
-        // verify access to table
+        // verify access to table after the commit, previous to fix the
+        // commit would fail with an out of memory or out of stack space error.
         JDBC.assertUnorderedResultSet(
             prepareStatement(
                 "select col1, col2 from TESTBIGTABLE where col1 = 10").executeQuery(),

Modified: db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/_Suite.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/_Suite.java?rev=1307679&r1=1307678&r2=1307679&view=diff
==============================================================================
--- db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/_Suite.java
(original)
+++ db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/functionTests/tests/largedata/_Suite.java
Sat Mar 31 01:12:49 2012
@@ -32,19 +32,24 @@ public class _Suite extends BaseJDBCTest
     }
 
     /**
-     * Suite runs first the lite suite for both embedded and client with LobLimitsLiteTest
+     * Suite runs first the lite suite for both embedded and client with 
+     * LobLimitsLiteTest.
      * Then runs the full embeddded suite with LobLimitsTest 
      * Then runs the full client suite with LobLimitsClientTest.
      * The full suite may take a very long time.
      */
     public static Test suite() {
         TestSuite suite = new TestSuite("largedata suite");
+
+        // DERBY-5624, currently this runs out of file descriptors on unix
+        // systems with 1024 limit per user.  Setting to run only on windows
+        // until solution for unix is found.
+        if (isWindowsPlatform())
+            suite.addTest(Derby5624Test.suite());
+
         suite.addTest(LobLimitsLiteTest.suite());
         suite.addTest(LobLimitsTest.suite());
         suite.addTest(LobLimitsClientTest.suite());
         return suite;
-        
     }
-    
-    
 }

Modified: db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java?rev=1307679&r1=1307678&r2=1307679&view=diff
==============================================================================
--- db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
(original)
+++ db/derby/code/branches/10.8/java/testing/org/apache/derbyTesting/junit/BaseTestCase.java
Sat Mar 31 01:12:49 2012
@@ -627,6 +627,39 @@ public abstract class BaseTestCase
 
         return getSystemProperty("os.name").equals(osName);
     }
+
+    /**
+     * Determine if platform is a Windows variant.
+     * <p>
+     * Return true if platform is a windows platform.  Just looks for
+     * os.name starting with "Windows".  The os.name property
+     * can have at least the following values (there are probably more):
+     *
+     * AIX
+     * Digital Unix
+     * FreeBSD
+     * HP UX
+     * Irix
+     * Linux
+     * Mac OS
+     * Mac OS X
+     * MPE/iX
+     * Netware 4.11
+     * OS/2
+     * Solaris
+     * Windows 2000
+     * Windows 95
+     * Windows 98
+     * Windows NT
+     * Windows Vista
+     * Windows XP
+     * <p>
+     *
+     * @return true if running on a Windows platform.
+     **/
+    public static final boolean isWindowsPlatform() {
+        return getSystemProperty("os.name").startsWith("Windows");
+    }
     
     /**
      * Check if this is java 5



Mime
View raw message