db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "RPost" <rp0...@pacbell.net>
Subject Re: [jira] Commented: (DERBY-106) HashJoinStrategy leads to java.lang.OutOfMemoryError
Date Fri, 17 Dec 2004 18:34:43 GMT
Gerald's last comment/request says it all:

> > If it turns out to be too complicated to make the memory consumption
more predicatble or even bounded to some configurable limit, then I need at
least as a workaround a way to turn off the HashJoin strategy completely: I
did it by patching and building my own derby.jar, but if there would be an
official solution with some kind of switch like a system property, it would
be great!
>

You'll be pleased to know its not complicated at all!!! -:)

Please refer to the 'Tuning Derby' manual. The section 'About the
Optimizer's Choice of Join Strategy' will explain why you are running out of
memory.

Also - did you notice this line in the OptimizerFactoryImpl code?
  'protected int maxMemoryPerTable = 1048576;'

Then around line 100 or so you will also notice that the 'boot' method looks
for a system property 'Optimizer.MAX_MEMORY_PER_TABLE' that can be used to
set the limit for a table.

The comment block before this code states:

  /*
  ** This property determines the maximum size of memory (in KB)
  ** the optimizer can use for each table.  If an access path takes
  ** memory larger than that size for a table, the access path is skipped.
  ** Default is 1024 (KB).
  */

Thus by combining the information found in the manual with the comments
found in the code we can conclude the following:

1. If you don't change the setting from the default of 1MB the optimizer may
choose a nested join strategy for a table.

2. If the table is actually bigger than 1MB you have fooled the optimizer
and it may well run out of memory.

My comment referring you to the tuning manual is not meant to be the type of
RTFM comment you often see in other forums. There is a great team of people
doing a lot of work on the documentation to make it even better and more
useful.

We want people to report problems like the one you reported because it may
well indeed be a problem that needs fixed. As Jack implies though, when a
problem is reported it is very helpful to have as much information as
possible: stack trace, query, system information, etc.

As Derby gets wider distribution and more people use it in different ways it
may point out the need to highlight some of the system properties that can
help people configure Derby for their use.

Thanks for highlighting this issue!



----- Original Message ----- 
From: "Jack Klebanoff (JIRA)" <derby-dev@db.apache.org>
To: <derby-dev@db.apache.org>
Sent: Friday, December 17, 2004 9:18 AM
Subject: [jira] Commented: (DERBY-106) HashJoinStrategy leads to
java.lang.OutOfMemoryError


>      [
http://nagoya.apache.org/jira/browse/DERBY-106?page=comments#action_56830 ]
>
> Jack Klebanoff commented on DERBY-106:
> --------------------------------------
>
> Do you know whether the memory overflow occurred during compilation or
execution? Do you have a stack trace from the OutOfMemoryError? Do you know
the sizes of the tables being joined?
>
> > HashJoinStrategy leads to java.lang.OutOfMemoryError
> > ----------------------------------------------------
> >
> >          Key: DERBY-106
> >          URL: http://nagoya.apache.org/jira/browse/DERBY-106
> >      Project: Derby
> >         Type: Bug
> >     Reporter: Gerald Khin
>
> >
> > My application is running out of memory: I encounterd a
java.lang.OutOfMemoryError. I used -Xmx256M. Unfortunatley, I cannot spend
an arbitrary amount of JVM memory.
> > Then, I commented out the line in class OptimizerFactoryImpl which was
adding the HashJoinStrategy to the set of Join strategies:
> > if (joinStrategySet == null)
> > {
> > // JoinStrategy[] jss = new JoinStrategy[2];
> > JoinStrategy[] jss = new JoinStrategy[1];
> > jss[0] = new NestedLoopJoinStrategy();
> > // jss[1] = new HashJoinStrategy();
> > joinStrategySet = jss;
> > }
> > And with these changes the OutOfMemoryError has gone away! And it works
even with -Xmx128M!!!
> > So I guess that there is a major memory issue with this HashJoin
strategy implementation.
> > If it turns out to be too complicated to make the memory consumption
more predicatble or even bounded to some configurable limit, then I need at
least as a workaround a way to turn off the HashJoin strategy completely: I
did it by patching and building my own derby.jar, but if there would be an
official solution with some kind of switch like a system property, it would
be great!
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
>    http://nagoya.apache.org/jira/secure/Administrators.jspa
> -
> If you want more information on JIRA, or have a bug to report see:
>    http://www.atlassian.com/software/jira
>


Mime
View raw message