db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-3732) SQL Length function materializes lob into memory
Date Mon, 23 Jun 2008 20:37:45 GMT

     [ https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Mike Matrigali updated DERBY-3732:

>Looking at a possible quick fix approach as opposed to reworking the type code.

>If I insert the lob using length, I should have a streamValueLength available. When we
call >getLength() stream is set but streamValueLength is not. Below is the trace when the
stream is >set. It seems to me readRecordFromArray should be able to tell the length and
pass that to >setStream(). Does that sound reasonable? 

No I don't think this will work.  store does not really know anything about the stream, it
just knows that it is a "long" column which means that it was asked to store a column longer
than a page.  And the current format for long column does not include any sort of length 
encoding for the entire column.  It splits the long column into multiple pieces and links
them and only maintains length of each link.  In the case of blob the type itself has the
length encoding in the column itself.  The description of the encoding of this length is located
in SQLBinary.java.

What I think would be easiest  is that the getlength() call in SQLBinary could read the first
few bytes and determine the length and then "reset()" the stream so that the other calls would
unaffected.  Also depending on how the stream was stored there may not be a length in
the stream so get length would have to read all the stream and then again do the reset.  The
question is whether this reset is going to mess anything else up.  Doing it this way would
only cause overhead if getlength is called, rather than always reading it.  
SQLBinary.readBinaryLength() is the call that reads the length from inside the column at the
front of the stream.

> SQL Length function materializes lob into memory
> ------------------------------------------------
>                 Key: DERBY-3732
>                 URL: https://issues.apache.org/jira/browse/DERBY-3732
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions:,,
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: LengthLargeLob.zip, LengthThruBlob.java
> Currently the SQL length function materializes the entire lob into memory. In SQLBinary.getLength()
we have 
> public final int	getLength() throws StandardException
> 	{
> 		if (stream != null) {
> 			if (streamValueLength != -1)
> 				return streamValueLength;
> 		}
> 		return (getBytes() == null) ? 0 : getBytes().length;
> 	}
> Which actually is doubly bad because we call getBytes twice and materialize it twice.
> It would be good to read the length from the stream if available and otherwise stream
the value to get the length, rather than materializing it into memory.
> To reproduce, run the attached repro.
> java -Xmx16M  LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
>         at org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
>         at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
>         at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
>         at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
>         at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
>         at org.apache.derby.exe.acf81e0010x011axa317x5db8x0000003d9dc81.e1(Unknown Source)
>         at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
>         ... 2 more
> [

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message