db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-3732) SQL Length function materializes lob into memory
Date Thu, 26 Jun 2008 18:55:45 GMT

     [ https://issues.apache.org/jira/browse/DERBY-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Mike Matrigali updated DERBY-3732:

from review of  derby-3732_diff.txt

Code changes look fine, seems like a safe incremental bug fix.  I would say it is ok for backport
when ready.  
nit code comment - I would have broke the 2 exceptions to muliple lines to fit in 80 characters.

test code comments (nothing to stop a checkin):
o While it probably works in the case of the test it is a bad paradigm to assume any ordering
of rows in a 
table without an order by.  Even insert order in general is not the guaranteed return order.
 Simply fixed by adding
a key column to the base table and an order by.  I don't remember what we do with blobs to
the sorter, so 
adding an index may also avoid unwanted memory usage - not sure as the row count is so small
it probably
fits in memory.
o Because of other bugs in the system I was a little worried about selecting length(b), b
in same statement - I know
   there are bugs where selecting b, b results in the stream of one column affecting the other.
  It might be nice to 
   add a check that it also works if you select them separately.
o I always like to see in blob tests one other case.  You covered null, 0 length, and long.
 It would be nice to cover
   short also (ie. some blob less than the size of a page - 1k is a safe choice).  Again from
looking at code I don't
   think this is an issue but does cover the other case through the code (maybe 0 is good
enough but seems worth
   checking it as a separate case).  
o can you explain the need for reflection code added to BaseTestCase? 

> SQL Length function materializes lob into memory
> ------------------------------------------------
>                 Key: DERBY-3732
>                 URL: https://issues.apache.org/jira/browse/DERBY-3732
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions:,,
>            Reporter: Kathey Marsden
>            Priority: Minor
>         Attachments: derby-3732_diff.txt, derby-3732_proto_diff.txt, derby-3732_skip_diff.txt,
LengthLargeLob.zip, LengthThruBlob.java
> Currently the SQL length function materializes the entire lob into memory. In SQLBinary.getLength()
we have 
> public final int	getLength() throws StandardException
> 	{
> 		if (stream != null) {
> 			if (streamValueLength != -1)
> 				return streamValueLength;
> 		}
> 		return (getBytes() == null) ? 0 : getBytes().length;
> 	}
> Which actually is doubly bad because we call getBytes twice and materialize it twice.
> It would be good to read the length from the stream if available and otherwise stream
the value to get the length, rather than materializing it into memory.
> To reproduce, run the attached repro.
> java -Xmx16M  LengthLargeLob
> It gives an out of memory exception
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.derby.iapi.types.SQLBinary.readFromStream(SQLBinary.java:415)
>         at org.apache.derby.iapi.types.SQLBinary.readExternal(SQLBinary.java:318)
>         at org.apache.derby.iapi.types.SQLBinary.getValue(SQLBinary.java:220)
>         at org.apache.derby.iapi.types.SQLBinary.getBytes(SQLBinary.java:210)
>         at org.apache.derby.iapi.types.SQLBinary.getLength(SQLBinary.java:250)
>         at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
>         at org.apache.derby.exe.acf81e0010x011axa317x5db8x0000003d9dc81.e1(Unknown Source)
>         at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
>         ... 2 more
> [

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message