Return-Path: 
 <derby-commits-return-2936-apmail-db-derby-commits-archive=db.apache.org@db.apache.org>
Delivered-To: apmail-db-derby-commits-archive@www.apache.org
Received: (qmail 53073 invoked from network); 13 Apr 2006 18:22:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 13 Apr 2006 18:22:24 -0000
Received: (qmail 56756 invoked by uid 500); 13 Apr 2006 18:22:17 -0000
Delivered-To: apmail-db-derby-commits-archive@db.apache.org
Received: (qmail 56714 invoked by uid 500); 13 Apr 2006 18:22:17 -0000
Mailing-List: contact derby-commits-help@db.apache.org; run by ezmlm
Precedence: bulk
list-help: <mailto:derby-commits-help@db.apache.org>
list-unsubscribe: <mailto:derby-commits-unsubscribe@db.apache.org>
List-Post: <mailto:derby-commits@db.apache.org>
Reply-To: "Derby Development" <derby-dev@db.apache.org>
List-Id: <derby-commits.db.apache.org>
Delivered-To: mailing list derby-commits@db.apache.org
Received: (qmail 56702 invoked by uid 99); 13 Apr 2006 18:22:17 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Apr 2006 11:22:17 -0700
X-ASF-Spam-Status: No, hits=-9.4 required=10.0
	tests=ALL_TRUSTED,NO_REAL_NAME
X-Spam-Check-By: apache.org
Received: from [209.237.227.194] (HELO minotaur.apache.org) (209.237.227.194)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 13 Apr 2006 11:22:15 -0700
Received: (qmail 52792 invoked by uid 65534); 13 Apr 2006 18:21:53 -0000
Message-ID: <20060413182153.52790.qmail@minotaur.apache.org>
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: svn commit: r393880 - in
 /db/derby/code/trunk/java/engine/org/apache/derby:
 iapi/types/SQLBinary.java impl/jdbc/BinaryToRawStream.java
 impl/jdbc/EmbedResultSet.java
Date: Thu, 13 Apr 2006 18:21:52 -0000
To: derby-commits@db.apache.org
From: djd@apache.org
X-Mailer: svnmailer-1.0.7
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

Author: djd
Date: Thu Apr 13 11:21:48 2006
New Revision: 393880

URL: http://svn.apache.org/viewcvs?rev=393880&view=rev
Log:
DERBY-438 (partial) Consistwnt handling of binary streams for updateXXX, namely check
the length correctly if it is greater than the 2Gb supported by Derby. Add some comments
to SQLBinary to clarify the format. Save the length when reading a binary stream from the
store into an application, will be used in a subsequent checkin. Ensure eof handling
when reading the binary value length in the on-disk format is handled correctly.

Modified:
    db/derby/code/trunk/java/engine/org/apache/derby/iapi/types/SQLBinary.java
    db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/BinaryToRawStream.java
    db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedResultSet.java

Modified: db/derby/code/trunk/java/engine/org/apache/derby/iapi/types/SQLBinary.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/iapi/types/SQLBinary.java?rev=393880&r1=393879&r2=393880&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/iapi/types/SQLBinary.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/iapi/types/SQLBinary.java Thu Apr 13 11:21:48 2006
@@ -72,21 +72,32 @@
   <P>
   Format : <encoded length><raw data>
   <BR>
-  Length is encoded to support 5.x databases where the length was stored as the number of bits.
-  The first bit of the first byte indicates if the format is an old (5.x) style or a new 8.1 style.
-  8.1 then uses the next two bits to indicate how the length is encoded.
+  Length is encoded to support Cloudscape 5.x databases where the length was stored as the number of bits.
+  The first bit of the first byte indicates if the format is an old (Cloudscape 5.x) style or a new Derby style.
+  Derby then uses the next two bits to indicate how the length is encoded.
   <BR>
   <encoded length> is one of N styles.
   <UL>
-  <LI> (5.x format) 4 byte Java format integer value 0 - either <raw data> is 0 bytes/bits  or an unknown number of bytes.
-  <LI> (5.x format) 4 byte Java format integer value >0 (positive) - number of bits in <raw data>, number of bytes in <raw data>
+  <LI> (5.x format zero) 4 byte Java format integer value 0 - either <raw data> is 0 bytes/bits  or an unknown number of bytes.
+  <LI> (5.x format bits) 4 byte Java format integer value >0 (positive) - number of bits in <raw data>, number of bytes in <raw data>
   is the minimum number of bytes required to store the number of bits.
-  <LI> (8.1 format) 1 byte encoded length (0 <= L <= 31) - number of bytes of <raw data> - encoded = 0x80 & L
-  <LI> (8.1 format) 3 byte encoded length (32 <= L < 64k) - number of bytes of <raw data> - encoded = 0xA0 <L as Java format unsigned short>
-  <LI> (8.1 format) 5 byte encoded length (64k <= L < 2G) - number of bytes of <raw data> - encoded = 0xC0 <L as Java format integer>
+  <LI> (Derby format) 1 byte encoded length (0 <= L <= 31) - number of bytes of <raw data> - encoded = 0x80 & L
+  <LI> (Derby format) 3 byte encoded length (32 <= L < 64k) - number of bytes of <raw data> - encoded = 0xA0 <L as Java format unsigned short>
+  <LI> (Derby format) 5 byte encoded length (64k <= L < 2G) - number of bytes of <raw data> - encoded = 0xC0 <L as Java format integer>
   <LI> (future) to be determined L >= 2G - encoded 0xE0 <encoding of L to be determined>
   (0xE0 is an esacape to allow any number of arbitary encodings in the future).
   </UL>
+  <BR>
+  When the value was written from a byte array the Derby encoded byte
+  length format was always used from Derby 10.0 onwards (ie. all open
+  source versions).
+  <BR>
+  When the value was written from a stream (e.g. PreparedStatement.setBinaryStream)
+  then the Cloudscape '5.x format zero' was used by 10.0 and 10.1.
+  The was due to the class RawToBinaryFormatStream always writing
+  four zero bytes for the length before the data.
+  <BR>
+  The Cloudscape '5.x format bits' format I think was never used by Derby.
  */
 abstract class SQLBinary
 	extends DataType implements BitDataValue
@@ -325,12 +336,18 @@
 		}
 	}
 
+    /**
+     * Read the encoded length of the value from the on-disk format.
+     * 
+     * @see SQLBinary
+    */
 	private static int readBinaryLength(ObjectInput in) throws IOException {
-		int len = 0;
+		
 		int bl = in.read();
-		if (len < 0)
+		if (bl == -1)
 			throw new java.io.EOFException();
 
+        int len;
 		if ((bl & 0x80) != 0)
 		{
 			if (bl == 0xC0)
@@ -352,7 +369,7 @@
 			int v2 = in.read();
 			int v3 = in.read();
 			int v4 = in.read();
-			if (v2 < 0 || v3 < 0 || v4 < 0)
+			if (v2 == -1 || v3 == -1 || v4 == -1)
 				throw new java.io.EOFException();
             int lenInBits = (((bl & 0xff) << 24) | ((v2 & 0xff) << 16) | ((v3 & 0xff) << 8) | (v4 & 0xff));
 

Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/BinaryToRawStream.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/BinaryToRawStream.java?rev=393880&r1=393879&r2=393880&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/BinaryToRawStream.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/BinaryToRawStream.java Thu Apr 13 11:21:48 2006
@@ -29,27 +29,30 @@
 /**
 	Converts a stream containing the Cloudscape stored binary form
 	to one that just contains the application's data.
-	Simply remove the length information.
+	Simply read and save the length information.
 */
 final class BinaryToRawStream
 extends java.io.FilterInputStream
 {
+    /**
+     * Length of the value represented by this stream.
+     * Set to -1 if the length is unknown.
+     */
+    private int length;
 
     // used by caller to insure that parent can not be GC'd until this
     // stream is no longer being used.
     private Object          parent;
 
-	public BinaryToRawStream(InputStream in, Object parent) 
+	BinaryToRawStream(InputStream in, Object parent) 
         throws IOException
 	{
 		super(in);
 
 		this.parent     = parent;
 
-		// no need to calculate the actual length
-		// int len = 0;
 		int bl = in.read();
-		if (bl < 0)
+		if (bl == -1)
 			throw new java.io.EOFException();
 
 		if ((bl & 0x80) != 0)
@@ -61,9 +64,12 @@
 				int v3 = in.read();
 				int v4 = in.read();
 
-				if (v1 < 0 || v2 < 0 || v3 < 0 || v4 < 0)
+				if (v1 == -1 || v2 == -1 || v3 == -1 || v4 == -1)
 					throw new java.io.EOFException();
-                //len = (((v1 & 0xff) << 24) | ((v2 & 0xff) << 16) | ((v3 & 0xff) << 8) | (v4 & 0xff));
+                length = (((v1 & 0xff) << 24) |
+                          ((v2 & 0xff) << 16) |
+                          ((v3 & 0xff) << 8)  |
+                           (v4 & 0xff));
 
 			}
 			else if (bl == 0xA0)
@@ -71,14 +77,14 @@
 				// read an unsigned short
 				int v1 = in.read();
 				int v2 = in.read();
-				if (v1 < 0 || v2 < 0)
+				if (v1 == -1 || v2 == -1)
 					throw new java.io.EOFException();
-                //len = (((v1 & 0xff) << 8) + (v2 & 0xff));
+                length = (((v1 & 0xff) << 8) + (v2 & 0xff));
 
 			}
 			else
 			{
-				// len = bl & 0x1F;
+				length = bl & 0x1F;
 			}
 		}
 		else
@@ -87,13 +93,26 @@
 			int v2 = in.read();
 			int v3 = in.read();
 			int v4 = in.read();
-			if (v2 < 0 || v3 < 0 || v4 < 0)
+			if (v2 == -1 || v3 == -1 || v4 == -1)
 				throw new java.io.EOFException();
-            //int lenInBits = (((bl & 0xff) << 24) | ((v2 & 0xff) << 16) | ((v3 & 0xff) << 8) | (v4 & 0xff));
+            int lenInBits = (((bl & 0xff) << 24) | ((v2 & 0xff) << 16) | ((v3 & 0xff) << 8) | (v4 & 0xff));
 
-			//len = lenInBits / 8;
-			//if ((lenInBits % 8) != 0)
-			//	len++;
+			length = lenInBits / 8;
+			if ((lenInBits % 8) != 0)
+			    length++;
+            
+            // Signifies unknown length
+            if (length == 0)
+                length = -1;
 		}
 	}
+    
+    /**
+     * Return the length of the value in thie stream in bytes.
+     * If the value is unknown then -1 is returned.
+     */
+    int getLength()
+    {
+        return length;
+    }
 }

Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedResultSet.java
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedResultSet.java?rev=393880&r1=393879&r2=393880&view=diff
==============================================================================
--- db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedResultSet.java (original)
+++ db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedResultSet.java Thu Apr 13 11:21:48 2006
@@ -2658,8 +2658,6 @@
 			default:
 				throw dataTypeConversion(columnIndex, "java.io.InputStream");
 		}
-		if (length < 0) //we are doing the check here and not in updateBinaryStreamInternal becuase updateClob needs to pass -1 for length.
-			throw newSQLException(SQLState.NEGATIVE_STREAM_LENGTH);
 
 		if (x == null)
 		{
@@ -2670,12 +2668,24 @@
 		updateBinaryStreamInternal(columnIndex, x, length,"updateBinaryStream");
 	}
 
-	protected void updateBinaryStreamInternal(int columnIndex,
-						java.io.InputStream x, int length, String updateMethodName)
+	private void updateBinaryStreamInternal(int columnIndex,
+						java.io.InputStream x, long length, String updateMethodName)
 	    throws SQLException
 	{
-		try {
-			getDVDforColumnToBeUpdated(columnIndex, updateMethodName).setValue(new RawToBinaryFormatStream(x, length), length);
+        if (length < 0)
+            throw newSQLException(SQLState.NEGATIVE_STREAM_LENGTH);
+        
+        // max number of bytes that can be set to be inserted 
+        // in Derby is 2Gb-1 (ie Integer.MAX_VALUE). 
+        // (e.g into a blob column).
+        if (length > Integer.MAX_VALUE ) {
+            throw newSQLException(SQLState.LANG_OUTSIDE_RANGE_FOR_DATATYPE,
+                    getColumnSQLType(columnIndex));
+        }
+        
+        try {
+			getDVDforColumnToBeUpdated(columnIndex, updateMethodName).setValue(
+                    new RawToBinaryFormatStream(x, (int) length), (int) length);
 		} catch (StandardException t) {
 			throw noStateChangeException(t);
 		}
@@ -2719,30 +2729,35 @@
 		updateCharacterStreamInternal(columnIndex, x, length, "updateCharacterStream");
 	}
 
-    protected void updateCharacterStreamInternal(int columnIndex,
-						java.io.Reader reader, int length, String updateMethodName)
+    private void updateCharacterStreamInternal(int columnIndex,
+						java.io.Reader reader, long length, String updateMethodName)
 	    throws SQLException
 	{
 		try {
 
-            // currently the max number of chars that is allowed in update
-            // via updateCharacterStream or updateClob interface
-            // is Integer.MAX_INT ( 2Gb -1) for clobs.
-            // check for -ve length here 
-            if (length < 0) 
-                throw newSQLException(SQLState.NEGATIVE_STREAM_LENGTH);
-
             if (reader == null)
             {
                 updateNull(columnIndex);
                 return;
             }
+            
+            // check for -ve length here 
+            if (length < 0) 
+                throw newSQLException(SQLState.NEGATIVE_STREAM_LENGTH);
+
+            // max number of characters that can be set to be inserted 
+            // in Derby is 2Gb-1 (ie Integer.MAX_VALUE). 
+            // (e.g into a CLOB column).
+            if (length > Integer.MAX_VALUE ) {
+                throw newSQLException(SQLState.LANG_OUTSIDE_RANGE_FOR_DATATYPE,
+                        getColumnSQLType(columnIndex));
+            } 
 
             LimitReader limitIn = new LimitReader(reader);
             
             // length is +ve. at this point, all checks for negative
             // length has already been done
-            int usableLength = length;
+            int usableLength = (int) length;
             ReaderToUTF8Stream utfIn = null;
 
             // Currently long varchar does not allow for truncation of
@@ -2772,15 +2787,15 @@
                 // needs to be truncated, and colWidth info to give proper
                 // truncation message
                 utfIn = new ReaderToUTF8Stream(
-                            limitIn, colWidth,     length - usableLength);
+                            limitIn, colWidth,     ((int) length) - usableLength);
             } else {
                 utfIn = new ReaderToUTF8Stream(
-                            limitIn, usableLength, length - usableLength);
+                            limitIn, usableLength, ((int)length) - usableLength);
             }
 
             limitIn.setLimit(usableLength);
             getDVDforColumnToBeUpdated(columnIndex, updateMethodName).setValue(
-                    utfIn, length);
+                    utfIn, (int) usableLength);
         } catch (StandardException t) {
             throw noStateChangeException(t);
         }
@@ -3881,8 +3896,10 @@
 
         if (x == null)
             updateNull(columnIndex);
-        else
-            updateBinaryStreamInternal(columnIndex, x.getBinaryStream(), -1, "updateBlob");
+        else {
+            long length = x.length();
+            updateBinaryStreamInternal(columnIndex, x.getBinaryStream(), length, "updateBlob");
+        }
 	}
 
 	/**
@@ -3931,23 +3948,8 @@
         }
         else
         {
-
-            // 1. max number of characters that can be updated into a clob 
-            // column is 2Gb-1 which is Integer.MAX_INT.
-            // This means that we do not allow any updates of clobs where
-            // clob.length() > Integer.MAX_INT. For now, we cast the x.length()
-            // to int as a result. This will work ok for valid clob values that
-            // derby supports. If we ever decide to increase these limits for 
-            // clobs, in that case the cast of x.Length() to int would not be 
-            // appropriate.
-            //
-            // 2. Note, x.length() needs to be called before retrieving the
-            // stream using x.getCharacterStream() because EmbedClob.length()
-            // will read from the stream and drain the stream.
-            // Hence the need to declare this local variable to store the 
-            // length.  The cast from long to int, can make length -ve.  The 
-            // length will be checked later in updateCharacterStreamInternal
-            int length = (int)x.length();
+            
+            long length = x.length();
 
             updateCharacterStreamInternal(
                 columnIndex, x.getCharacterStream(),length, "updateClob");
@@ -4302,15 +4304,21 @@
 			return ((NoPutResultSet) theResults).isForUpdate();
 		return false;
 	}
+    
+    final String getColumnSQLType(int column)
+    {
+        return resultDescription.getColumnDescriptor(column)
+                       .getType().getTypeId().getSQLTypeName();
+    }
 
-	protected final SQLException dataTypeConversion(String targetType, int column) {
+	private final SQLException dataTypeConversion(String targetType, int column) {
 		return newSQLException(SQLState.LANG_DATA_TYPE_GET_MISMATCH, targetType,
-			resultDescription.getColumnDescriptor(column).getType().getTypeId().getSQLTypeName());
+                getColumnSQLType(column));
 	}
 
-	protected final SQLException dataTypeConversion(int column, String targetType) {
+	private final SQLException dataTypeConversion(int column, String targetType) {
 		return newSQLException(SQLState.LANG_DATA_TYPE_GET_MISMATCH,
-			resultDescription.getColumnDescriptor(column).getType().getTypeId().getSQLTypeName(), targetType);
+                getColumnSQLType(column), targetType);
 	}
     
     /**