db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kristian Waagan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DERBY-5387) Memory leak or unbounded consumption problem when running a utility to copy one database to another using SYSCS_EXPORT_TABLE and SYSCS_IMPORT_TABLE
Date Thu, 18 Aug 2011 11:46:27 GMT

    [ https://issues.apache.org/jira/browse/DERBY-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086970#comment-13086970

Kristian Waagan commented on DERBY-5387:

This may be somewhat tricky to investigate. To isolate the problem, are you able to first
export the tables, and then import the largest file? If that works, there must be a problem
with cleaning up / releasing resources.
If it doesn't work there's a problem in the insert/sort code. For instance, the memory figures
are wrong - causing Derby to grow the sort buffer too much (i.e. the sort fails to spill to

You could try with a debug build and enable output for the sort buffer. The output is far
from perfect, but you'll get some indications of what's going on. This will take even longer
to run, so save the output to file and go do something else... (I think this output goes to
std err by default). Enable by running the debug build with "-Dderby.debug.true=SortTuning".

If you can somehow provide a link to the heap dump that *could* also help (too large to be
attached here). **NOTE**: If the data is confidential, this is not an option!
Finally, a junk data generator and small program to actually run the import would be another
way to allow people to debug this.

On another matter, if I understand the code correctly, the dbcopy code is "leaking" connections
to the source database (line 537).

> Memory leak or unbounded consumption problem when running a utility to copy one database
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: DERBY-5387
>                 URL: https://issues.apache.org/jira/browse/DERBY-5387
>             Project: Derby
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions:
>         Environment: Solaris 10/9, Oracle Java 1.6.0_22, 1Gb heap space (also ran with
8Gb heap space with no difference other than how long it takes to run out of memory).
>            Reporter: Brett Bergquist
>            Priority: Critical
>         Attachments: dbcopy.zip, java_pid2364_Leak_Hunter.zip, java_pid2364_Leak_Suspects.zip
> I have a utility that copies one database to another by using 'dblook" to export the
schema from the first which is then uses to create the copy's schema.  The tables are exported
from the first database using the SYSCS_EXPORT_TABLE and imported into the second database
using SYSCS_IMPORT_TABLE, processing each table before moving on to the next.  The the constraints
and indexes present in the schema generated by 'dblook' are applied to the second database.
 The utility runs out of memory regardless of the amount of memory given when run on a very
large database (one table has 75 million rows in it and the total database size is 110Gb of
disk storage).  The utility does complete on a smaller database.
> I will attach the source code for the utility.  Also added the -XX:+HeapDumpOnOutOfMemory
flag and ran with -Xmx1024m heap.  I will attach the suspected leaks report generated by the
Eclipse MemoryAnalyzer tool.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message