Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Message-ID: <1402060436.1214603745046.JavaMail.jira@brutus>
Date: Fri, 27 Jun 2008 14:55:45 -0700 (PDT)
From: "Kathey Marsden (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Subject: [jira] Updated: (DERBY-3732) SQL Length function materializes lob
 into memory
In-Reply-To: <953379663.1213917585016.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kathey Marsden updated DERBY-3732:
----------------------------------

    Attachment: derby-3732_skip3_diff.txt

Thanks Knut, Kristian, and Mike for the reviews.

Attached is a new patch derby-3732_skip3_diff.txt which incorporates all the review comments thus far except I am not quite sure what to do about the pageCacheSize not getting set if this test is not run as the first test.   It's not an issue with suites.All since the heap is large enough and for the junit-lowmem suite it is the first and only test, so works ok there too.

The one outstanding issue is that the new test with 16MB heap runs out of memory in the client for jdk 1.4.2. It runs fine with jdk 1.5, so right now the lowmem suite does not run with jdk 1.4.2. The problem is in client and is related to retrieving the entire lob,  (not related to the length which is the focus of this issue.) I'd like to go ahead and check in this change and deal with the jdk 1.4.2 out of memory issue separately.

I'm going to go ahead and run tests and checkin unless there are more comments.


> SQL Length function materializes lob into memory
> ------------------------------------------------
>
>                 Key: DERBY-3732
>                 URL: https://issues.apache.org/jira/browse/DERBY-3732
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt, derby-3732_skip2_diff.txt, derby-3732_skip3_diff.txt, derby-3732_skip_diff.txt, LengthLargeLob.zip, LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In SQLBinary.getLength() we have 
> public final int	getLength() throws StandardException
> 	{
> 		if (stream != null) {
> 			if (streamValueLength != -1)
> 				return streamValueLength;
> 		}
> 		return (getBytes() == null) ? 0 : getBytes().length;
> 	}
> Which actually is doubly bad because we call getBytes twice and materialize it twice.
> It would be good to read the length from the stream if available and otherwise stream the value to get the length, rather than materializing it into memory.
> To reproduce, run the attached repro.
> java -Xmx16M  LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
>         at org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
>         at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
>         at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
>         at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
>         at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
>         at org.apache.derby.exe.acf81e0010x011axa317x5db8x0000003d9dc81.e1(Unknown Source)
>         at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
>         ... 2 more
> [
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.