Return-Path: Delivered-To: apmail-db-derby-commits-archive@www.apache.org Received: (qmail 25775 invoked from network); 2 Nov 2007 15:38:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Nov 2007 15:38:42 -0000 Received: (qmail 12999 invoked by uid 500); 2 Nov 2007 15:38:30 -0000 Delivered-To: apmail-db-derby-commits-archive@db.apache.org Received: (qmail 12979 invoked by uid 500); 2 Nov 2007 15:38:30 -0000 Mailing-List: contact derby-commits-help@db.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: Reply-To: "Derby Development" List-Id: Delivered-To: mailing list derby-commits@db.apache.org Received: (qmail 12968 invoked by uid 99); 2 Nov 2007 15:38:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Nov 2007 08:38:30 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO eris.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Nov 2007 15:38:41 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 3136E1A9832; Fri, 2 Nov 2007 08:38:21 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r591369 - in /db/derby/code/branches/10.1/java: engine/org/apache/derby/impl/services/bytecode/BCMethod.java engine/org/apache/derby/impl/services/bytecode/CodeChunk.java testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out Date: Fri, 02 Nov 2007 15:38:19 -0000 To: derby-commits@db.apache.org From: kmarsden@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20071102153821.3136E1A9832@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: kmarsden Date: Fri Nov 2 08:38:18 2007 New Revision: 591369 URL: http://svn.apache.org/viewvc?rev=591369&view=rev Log: merge from trunk revision 432856 Comment from trunk checkin: DERBY-766 DERBY-1714 Working method in CodeChunk that splits expressions out of generated methods that are too large. Bumps the number of unions supported in largeCodeGen to over 6,000 from around 800. Also increases the number of rows supported in a VALUES clause. A large number of UNION clauses still requires a large amount of memory for optimization (see DERBY-1315). A large number of rows in a VALUES clause fails at some point due to a StackOverflow. In 10.1 this completes the merge of changes for DERBY-176/DERBY-766 but I am seeing out of memory exceptions on more at 1800 unions, even when bumping memory to 512M. I will investigate some more to see if there is another change that needs to go to 10.1 to enable the large number of unions. Modified: db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/BCMethod.java db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/CodeChunk.java db/derby/code/branches/10.1/java/testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out Modified: db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/BCMethod.java URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/BCMethod.java?rev=591369&r1=591368&r2=591369&view=diff ============================================================================== --- db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/BCMethod.java (original) +++ db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/BCMethod.java Fri Nov 2 08:38:18 2007 @@ -73,7 +73,7 @@ * cause some splitting. Tested with value set to 2000. */ static final int CODE_SPLIT_LENGTH = VMOpcode.MAX_CODE_LENGTH; - + final BCClass cb; protected final ClassHolder modClass; // the class it is in (modifiable fmt) final String myReturnType; @@ -287,13 +287,14 @@ } else { - //TODO: re-start split at point left off - //split_pc = myCode.splitExpressionOut(this, modClass, - // optimalMinLength, maxStack); - - // DERBY-766 temp - don't call the split method yet. - split_pc = -1; - } + // Note the split expression does not re-start split + // at point left off by the previous split expression. + // This could be done but would require some level + // of stack depth history to be kept across calls. + split_pc = myCode.splitExpressionOut(this, modClass, + optimalMinLength, maxStack); + + } // Negative split point returned means that no split // was possible. Give up on this approach and goto Modified: db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/CodeChunk.java URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/CodeChunk.java?rev=591369&r1=591368&r2=591369&view=diff ============================================================================== --- db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/CodeChunk.java (original) +++ db/derby/code/branches/10.1/java/engine/org/apache/derby/impl/services/bytecode/CodeChunk.java Fri Nov 2 08:38:18 2007 @@ -1282,6 +1282,9 @@ */ final int splitZeroStack(BCMethod mb, ClassHolder ch, final int split_pc, final int optimalMinLength) { + + int splitMinLength = splitMinLength(mb); + int stack = 0; // maximum possible split seen that is less than @@ -1371,11 +1374,8 @@ return -1; // Decide if the earlier possible split is - // worth it. 100 is an arbitary number, - // a real low limit would be the number of - // bytes of instructions required to call - // the sub-method, four I think. - if (possibleSplitLength < 100) + // worth it. + if (possibleSplitLength <= splitMinLength) return -1; // OK go with the earlier split @@ -1715,14 +1715,16 @@ * */ - final int splitExpressionOut(BCMethod mb, ClassHolder ch, + final int splitExpressionOut(final BCMethod mb, final ClassHolder ch, final int optimalMinLength, - int maxStack) + final int maxStack) { // Save the best block we have seen for splitting out. int bestSplitPC = -1; int bestSplitBlockLength = -1; - String bestSplitRT = null; + String bestSplitRT = null; + + int splitMinLength = splitMinLength(mb); // Program counter of the earliest instruction // that the word in the current active stack @@ -1778,6 +1780,15 @@ // same stack depth. int[] cond_pcs = findConditionalPCs(pc, opcode); if (cond_pcs != null) { + + // TODO: This conditional handling was copied + // from splitZeroStack, haven't looked in detail + // to see how a conditional should be handled + // with an expression split. So for the time + // being just bail. + if (true) + return -1; + // an else block exists, skip the then block. if (cond_pcs[3] != -1) { pc = cond_pcs[3]; @@ -1917,8 +1928,10 @@ if (width == 0) { // objectref was at one more than the current depth - selfContainedBlockStart = - earliestIndepPC[stack + 1]; + // no plan to split here though, as we are only + // splitting methods that return a reference. + selfContainedBlockStart = -1; + // earliestIndepPC[stack + 1]; } else if (width == 1) { @@ -1930,47 +1943,62 @@ { // width == 2, objectref was one below the // current stack depth. - selfContainedBlockStart = earliestIndepPC[stack] = + // no plan to split here though, as we are only + // splitting methods that return a reference. + selfContainedBlockStart = -1; + + // top two words depend on the objectref + // which was at the same depth of the first word + // of the 64 bit value. + earliestIndepPC[stack] = earliestIndepPC[stack - 1]; } if (selfContainedBlockStart != -1) { int blockLength = pc - selfContainedBlockStart; - - // Only split for a method that returns - // an class reference. - int me = vmDescriptor.lastIndexOf(')'); - if (vmDescriptor.charAt(me+1) == 'L') + if (blockLength <= splitMinLength) { - String rt = vmDescriptor.substring(me + 2, - vmDescriptor.length() - 1); - - if (blockLength > (VMOpcode.MAX_CODE_LENGTH - 1)) - { - // too big to split into a single method - // (one for the return opcode) - } - else if (blockLength >= optimalMinLength) - { - // Split now! - System.out.println("NOW " + blockLength - + " @ " + selfContainedBlockStart); - BCMethod subMethod = startSubMethod(mb, - rt, selfContainedBlockStart, - blockLength); - - return splitCodeIntoSubMethod(mb, ch, subMethod, - selfContainedBlockStart, blockLength); - } - else if (blockLength > bestSplitBlockLength) + // No point splitting, too small + } + else if (blockLength > (VMOpcode.MAX_CODE_LENGTH - 1)) + { + // too big to split into a single method + // (one for the return opcode) + } + else + { + // Only split for a method that returns + // an class reference. + int me = vmDescriptor.lastIndexOf(')'); + + if (vmDescriptor.charAt(me+1) == 'L') { - // Save it, may split at this point - // if nothing better seen. - bestSplitPC = selfContainedBlockStart; - bestSplitBlockLength = blockLength; - bestSplitRT = rt; + String rt = vmDescriptor.substring(me + 2, + vmDescriptor.length() - 1); + + // convert to external format. + rt = rt.replace('/', '.'); + + if (blockLength >= optimalMinLength) + { + // Split now! + BCMethod subMethod = startSubMethod(mb, + rt, selfContainedBlockStart, + blockLength); + + return splitCodeIntoSubMethod(mb, ch, subMethod, + selfContainedBlockStart, blockLength); + } + else if (blockLength > bestSplitBlockLength) + { + // Save it, may split at this point + // if nothing better seen. + bestSplitPC = selfContainedBlockStart; + bestSplitBlockLength = blockLength; + bestSplitRT = rt; + } } } } @@ -1980,19 +2008,16 @@ } - if (bestSplitBlockLength > 100) - { - System.out.println("BEST " + bestSplitBlockLength - + " @ " + bestSplitPC); + + if (bestSplitBlockLength != -1) { BCMethod subMethod = startSubMethod(mb, bestSplitRT, bestSplitPC, bestSplitBlockLength); - + return splitCodeIntoSubMethod(mb, ch, subMethod, - bestSplitBlockLength, bestSplitBlockLength); - + bestSplitPC, bestSplitBlockLength); } - + return -1; } @@ -2017,6 +2042,38 @@ } } + /** + * Minimum split length for a sub-method. If the number of + * instructions to call the sub-method exceeds the length + * of the sub-method, then there's no point splitting. + * The number of bytes in the code stream to call + * a generated sub-method can take is based upon the number of method args. + * A method can have maximum of 255 words of arguments (section 4.10 JVM spec) + * which in the worst case would be 254 (one-word) parameters + * and this. For a sub-method the arguments will come from the + * parameters to the method, i.e. ALOAD, ILOAD etc. + *
+ * This leads to this number of instructions. + *
    + *
  • 4 - 'this' and first 3 parameters have single byte instructions + *
  • (N-4)*2 - Remaining parameters have two byte instructions + *
  • 3 for the invoke instruction. + *
+ */ + private static int splitMinLength(BCMethod mb) { + int min = 1 + 3; // For ALOAD_0 (this) and invoke instruction + + if (mb.parameters != null) { + int paramCount = mb.parameters.length; + + min += paramCount; + + if (paramCount > 3) + min += (paramCount - 3); + } + + return min; + } /* final int splitNonZeroStack(BCMethod mb, ClassHolder ch, final int codeLength, final int optimalMinLength, Modified: db/derby/code/branches/10.1/java/testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out URL: http://svn.apache.org/viewvc/db/derby/code/branches/10.1/java/testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out?rev=591369&r1=591368&r2=591369&view=diff ============================================================================== --- db/derby/code/branches/10.1/java/testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out (original) +++ db/derby/code/branches/10.1/java/testing/org/apache/derbyTesting/functionTests/master/largeCodeGen.out Fri Nov 2 08:38:18 2007 @@ -22,5 +22,21 @@ PASS: EXECUTE SELECT with 800 unions Row data check ok PASS: PREPARE: SELECT with 900 unions PASS: EXECUTE SELECT with 900 unions Row data check ok -FAILED QUERY: SELECT with 1000 unions. +PASS: PREPARE: SELECT with 1000 unions +PASS: EXECUTE SELECT with 1000 unions Row data check ok +PASS: PREPARE: SELECT with 1100 unions +PASS: EXECUTE SELECT with 1100 unions Row data check ok +PASS: PREPARE: SELECT with 1200 unions +PASS: EXECUTE SELECT with 1200 unions Row data check ok +PASS: PREPARE: SELECT with 1300 unions +PASS: EXECUTE SELECT with 1300 unions Row data check ok +PASS: PREPARE: SELECT with 1400 unions +PASS: EXECUTE SELECT with 1400 unions Row data check ok +PASS: PREPARE: SELECT with 1500 unions +PASS: EXECUTE SELECT with 1500 unions Row data check ok +PASS: PREPARE: SELECT with 1600 unions +PASS: EXECUTE SELECT with 1600 unions Row data check ok +PASS: PREPARE: SELECT with 1700 unions +PASS: EXECUTE SELECT with 1700 unions Row data check ok +FAILED QUERY: SELECT with 1800 unions. FAILED QUERY: SELECT with 10000 unions.