Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Message-ID: <14067484.1157035583509.JavaMail.jira@brutus>
Date: Thu, 31 Aug 2006 07:46:23 -0700 (PDT)
From: "John H. Embretsen (JIRA)" <derby-dev@db.apache.org>
To: derby-dev@db.apache.org
Subject: [jira] Updated: (DERBY-1713) Memory do not return to the system
 after Shuting down derby 10.2.1.0, following an out of memory event
In-Reply-To: <24249054.1155810313849.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

     [ http://issues.apache.org/jira/browse/DERBY-1713?page=all ]

John H. Embretsen updated DERBY-1713:
-------------------------------------

    Attachment: Derby1713repro.java

I have not been able to try Ibrahim's Test1.java, since I am not able to decompress the database needed (my version of bzip2 tells me it (test.zip) is not a valid bzip2 archive - and I'm not using Windows (at the moment)).

Instead, I have created a fully reproducible test case which does not require having a database in advance. See attachment 'Derby1713repro.java'. To run, put derby.jar in your classpath and run:

$ java Derby1713repro

If you already have a DB to test with, you can run with the "haveDB" argument:

$ java Derby1713repro haveDB

The test case inserts 40k rows into a table (same DDL as Ibrahim described in previous comments), which results in a 30 MB database (unless "haveDB" is specified, in which case the test case only performs a query against an already existing database).

I think one important thing to do is to reduce as many variables as possible when trying to reproduce or debug a problem. With this repro, I do get an OutOfMemoryError (OOME) if the max heap size is 32 MB (-Xmx32M). This happens without using an inner class (as is done in Test1.java), and it even happens without executing a 'SELECT count(*)' statement. I have also tried commenting out the setFetchSize(60000) call, and the OOME still happens (Sun's jvm version 1.5.0_06).

The problem seems to be the huge SELECT, FROM, WHERE statement. At start-up (when using the "haveDB" option), jstat reports a tenured space usage of 512 bytes. When executing the query, tenured space usage increases to as much as 30272 bytes, which is almost the whole heap. In addition, other parts of the heap need some space, hence we get an OutOfMemoryError. Feel free to use this repro as basis for creating a minimalistic test showing the problem.

I think Mike is right when he says that 32 MB max heap is not suitable ("safe") for this kind of database and this kind of usage. I therefore recommend lowering the Priority and Urgency of this Jira issue somewhat.

I also recommend changing the summary of this issue once the real problem has been positively identified. The fact that the JVM is not able to free memory following an OutOfMemoryEvent is as far as I can tell not a bug in Derby.


> Memory do not return to the system after Shuting down derby 10.2.1.0, following an out of memory event
> ------------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-1713
>                 URL: http://issues.apache.org/jira/browse/DERBY-1713
>             Project: Derby
>          Issue Type: Bug
>          Components: Performance
>    Affects Versions: 10.2.1.0
>         Environment: Windows XP SP2
> JRE 1.6 beta2
>            Reporter: Ibrahim
>            Priority: Critical
>         Attachments: Derby1713repro.java, test.zip, Test1.java
>
>
> I face a problem when querying large tables. I run the below SQL and it stuck in this query and throws java heap exception OutOfMemory:
> SELECT count(*) FROM <table> WHERE .....
> N.B. I'm using a database of more than 90,000 records (40 MB). I set the maxHeap to 32 MB (all other settings have the default value, pageCache ... etc ). 
> Then, I shutdown the database but the memory is not returned to the system (and remain 32 MB [max threshold]). I tried to increase the maxHeap to 128 MB in which it works and releases the memory, so I think the problem is when it reaches the maxHeap then it seems to not respond to anything such as closing the connection or shutting down the database. How can I get rid of this? (because i cannot increase the maxHeap as the database increases, I want to throw an exception and release the memory)
> I'm using this to shutdown the DB:
> try{DriverManager.getConnection("jdbc:derby:;shutdown=true");}
> catch(SQLException ex){System.err.println("SQLException: " + ex.getMessage());}
> I'm using a memory Profiler for monitoring the memory usage.
> Thanks in advanced.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira