db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-3732) SQL Length function materializes lob into memory
Date Mon, 23 Jun 2008 21:47:45 GMT

     [ https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mike Matrigali updated DERBY-3732:
----------------------------------


I think you are on right track.

What I described should work for the behavior when getLength() is called on a datatype that
has been constructed by reading it out of the store.  But I think the generic type code also
has
to worry about a datatype that is coming from a user and may be a stream.  I think that is
the "magic" of testing the type in the getValue() call.  I don't know if we can always "reset"
a
stream from a user.  I don't know off hand if it is possible to force the code through that
path.  Is it better to get an I/O error on the call, or to sometimes be able to instantiate
the
blob in memory and only get error if you run out memory?  

longer term the reading of the whole stream to determine the length could be optimized if

the work could be pushed into the stream.  In the case of reading this data from store I think
the data is being read into the page cache page array, and then into a intermediate array
to implement the stream back to the user.  And now this is adding another level.  I wonder
if skip() will work rather than reading the actual bytes?

one nit is that indentation doesn't look right for mixure of tabs/spaces.  



> SQL Length function materializes lob into memory
> ------------------------------------------------
>
>                 Key: DERBY-3732
>                 URL: https://issues.apache.org/jira/browse/DERBY-3732
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 10.3.3.0, 10.4.1.3, 10.5.0.0
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3732_proto_diff.txt, LengthLargeLob.zip, LengthThruBlob.java
>
>
> Currently the SQL length function materializes the entire lob into memory. In SQLBinary.getLength()
we have 
> public final int	getLength() throws StandardException
> 	{
> 		if (stream != null) {
> 			if (streamValueLength != -1)
> 				return streamValueLength;
> 		}
> 		return (getBytes() == null) ? 0 : getBytes().length;
> 	}
> Which actually is doubly bad because we call getBytes twice and materialize it twice.
> It would be good to read the length from the stream if available and otherwise stream
the value to get the length, rather than materializing it into memory.
> To reproduce, run the attached repro.
> java -Xmx16M  LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
>         at org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
>         at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
>         at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
>         at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
>         at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
>         at org.apache.derby.exe.acf81e0010x011axa317x5db8x0000003d9dc81.e1(Unknown Source)
>         at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
>         ... 2 more
> [
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message