db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-3732) SQL Length function materializes lob into memory
Date Tue, 24 Jun 2008 00:33:45 GMT

     [ https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Mike Matrigali updated DERBY-3732:

The 2 cases I try to think about with stream/blobs/clobs are always the following:
1) The data is coming off the disk, through store as a stream, and back to user somehow
 - ie. select
2) The data is coming from user to be put into store as a stream - ie. insert.

Mostly the datatype code should not really care which of these 2 cases is in effect, but
I think Resetable is a case that matters.

The case you are testing is #1, with getLength() being called by the query itself on a stream
created by the store.  That stream is always going to be a OverflowInputStream when it comes
out of store - does not matter what type.  But type code should not use that.  I think using
instanceof Resetable is the right thing.

In case 2 embedded I think the stream may come directly from user - but I am not sure what
jdbc layer may do to it before it gets made into a SQLBinary.  A different case might be what
comes to the embedded engine after a stream gets sent in from a network client across drda
and then into the embedded engine.  In both of these cases I don't know if you can actually
get getLength() called, but I also don't think you can count on resetting the stream so seems

cleaner to leave the code as is for this case rather than have the code pretend it could handle
it.  I don't really know a lot about case 2, so may be off track.  The fact that the stream
the length encoding in it means that some derby code has already got involved and modified
the stream the user provided.

I don't know if there are any other situations with streams available in the datatype  -
 maybe triggers? 

you probably should ignore the skip comment, better to get it right than worry about optimizing
it.  Your fix should be way faster in case where length exists at front, and be way less memory
intensive in the other case so a good incremental improvement.

> SQL Length function materializes lob into memory
> ------------------------------------------------
>                 Key: DERBY-3732
>                 URL: https://issues.apache.org/jira/browse/DERBY-3732
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions:,,
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip, LengthThruBlob.java
> Currently the SQL length function materializes the entire lob into memory. In SQLBinary.getLength()
we have 
> public final int	getLength() throws StandardException
> 	{
> 		if (stream != null) {
> 			if (streamValueLength != -1)
> 				return streamValueLength;
> 		}
> 		return (getBytes() == null) ? 0 : getBytes().length;
> 	}
> Which actually is doubly bad because we call getBytes twice and materialize it twice.
> It would be good to read the length from the stream if available and otherwise stream
the value to get the length, rather than materializing it into memory.
> To reproduce, run the attached repro.
> java -Xmx16M  LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
>         at org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
>         at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
>         at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
>         at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
>         at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
>         at org.apache.derby.exe.acf81e0010x011axa317x5db8x0000003d9dc81.e1(Unknown Source)
>         at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
>         ... 2 more
> [

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message