db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DERBY-6096) OutOfMemoryError with Clob or Blob hash join: DataTypeDescriptor.estimatedMemoryUsage() has no case for BLOB or CLOB so would underestimate memory usage for those types at zero
Date Wed, 06 Mar 2013 18:12:12 GMT

    [ https://issues.apache.org/jira/browse/DERBY-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594913#comment-13594913
] 

Mike Matrigali commented on DERBY-6096:
---------------------------------------

what estimate is proposed for estimatedMemoryUsage of blob's clob's?  Since they are variable
length objects it is hard to know what actual size they are.  
Given your repro program, it might be reasonable to use 32k and assume store will stream the
rest of each.  definitely better than 0.  

Note that with the fix we may use way less memory for the query, and for some like your repro
that will be good.  But for others that did not see errors their
queries may run slower now after the fix when we use less memory.  This may especially be
a concern if the fix is to be backported.
                
> OutOfMemoryError with Clob or Blob hash join: DataTypeDescriptor.estimatedMemoryUsage()
 has no case for BLOB or CLOB so would underestimate memory usage for those types at zero
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-6096
>                 URL: https://issues.apache.org/jira/browse/DERBY-6096
>             Project: Derby
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 10.1.3.1, 10.2.2.0, 10.3.3.0, 10.4.2.0, 10.5.3.0, 10.6.2.1, 10.7.1.1,
10.9.1.0, 10.10.0.0, 10.8.3.0
>            Reporter: Kathey Marsden
>         Attachments: D6096.java
>
>
> In discussion on derby-dev regarding how much memory is used for hash joins, Knut noted:
> I haven't verified, but I think HashJoinStrategy uses
> DataTypeDescriptor.estimatedMemoryUsage() to estimate how much memory
> the hash table will consume. That method has no case for BLOB or CLOB,
> so it looks as if it will return zero for LOB columns. If that's so, it
> will definitely overestimate how many rows fits in maxMemoryPerTable
> kilobytes if the rows contain LOBs.
> DataTypeDescriptor.estimatedMemoryUsage() should be updated to include BLOB and CLOB
and we should try verify if this theory is correct with a reproduction.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message