Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Message-ID: <4250674.1177076295722.JavaMail.jira@brutus>
Date: Fri, 20 Apr 2007 06:38:15 -0700 (PDT)
From: "Dag H. Wanvik (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Subject: [jira] Commented: (DERBY-1947) OutOfMemoryError after repeated
 calls to boot and shutdown a database
In-Reply-To: <2419779.1160451860731.JavaMail.root@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DERBY-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12490357 ] 

Dag H. Wanvik commented on DERBY-1947:
--------------------------------------

Thanks for looking at this Knut! 

> It could also be that gc() just makes the garbage collection start,
> possibly in another thread, so that runFinalization() is called before
> the objects are marked as unreferenced. IIRC, gc() doesn't block
> whereas runFinalization() does.

On Solaris, gc() alone manages to clear up the result set objects in
time: Not sure how that works; possibly the marking of the nested
connection objects makes it possible to also mark the result set
objects which are then effectively collected in time for the test in
#prepareToInvalidate() even though the connection objects can't yet be
finalized (blocked). As you say, since gc() doesn't block, there is no deadlock,
and it works.

> Note that the IBM 1.5 VM implemented a much more aggressive gc than
> previous versions. I remember that many tests had to be rewritten when
> people started running tests with that VM because ResultSets were

On Linux with SUN VM and with the IBM VM, gc() alone are not able to
free the result set objects in time, but this is could be timing
dependent.

> closed earlier, etc. That probably explains why you don't see the
> deadlock in runFinalization() on IBM 1.5, since the objects most
> likely are gc'ed and finalized before you come to that point.

No, I get the error with the IBM VM too, so the calls are
necessary. But they will not succeed in finalizing the nested
connection objects (unless they are collected earlier but result set
objects are not) but at least it doesn't deadlock.

I will upload a new patch which avoids synchronization in the
finalizer of nested connection objects, which solves the deadlock
problem.

> OutOfMemoryError after repeated calls to boot and shutdown a database
> ---------------------------------------------------------------------
>
>                 Key: DERBY-1947
>                 URL: https://issues.apache.org/jira/browse/DERBY-1947
>             Project: Derby
>          Issue Type: Bug
>          Components: Services
>    Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 10.2.1.6
>         Environment: Any
>            Reporter: Rajesh Kartha
>         Assigned To: Dag H. Wanvik
>         Attachments: DERBY-1947-1.diff, DERBY-1947-1.stat, DERBY-1947-2.diff, DERBY-1947-2.stat, DERBY-1947-3.diff, DERBY-1947-3.stat, HeapAnalysis_op.jpg, ProcedureInTriggerTest-threadstacks.txt, testEmbedAndClient.java
>
>
> I came across this OOM issue while running some system tests involving
> backup and restore against  Derby. The test is expected to run forever but
> using the default heap space it runs into OOM within 2 days. I earlier mentioned about this
> in my reply to the 10.2.1.6 vote - http://www.nabble.com/Re%3A--VOTE--10.2.1.6-release-p6650528.html
> Also there has been some discussions on the list on the related topic:
> http://issues.apache.org/jira/browse/DERBY-23 and
> http://www.nabble.com/question-about-shutdown-tf2300504.html
> Basic Analysis:
> --------------------
> Wrote a simple Java app (attached to this issue) that booted and shutdown the same 
> database multiple times. Depending on the heapsize the program ran into the
> OOM at some point, as expected. Some heap dump analysis using the IBM HeapAnalyzer 
> and revealed that the HashSet (allContexts) within org.apache.derby.iapi.services.context.ContextService 
> class seemed to be location of the leak (snapshot of the heapanalysis attached).
> A little bit of debugging shows that:
>  - for every:connection two ContextManager objects (say, cm1, cm2) are added to the HashSet
>  - for every shutdown a new ContextManager object (say, cm3) is added and two objects are removed
>  - the object removed are cm2 and cm3 - in that sequence
>  - but the object cm1 gets left behind
> This happens for every connect/shutdown sequence and since one of the ContextManager objects added to the 
> HashSet is not removed as a part of the cleanup, it contributes to growth in memory usage, hence
> an OOM eventually.
> For example:
> ============
> A 64M heap could allow about 1107 iterations of connect/shutdown only before running into OOM and 
> created 1108 un-used ContextManager objects in the memory.
> java -Xmx64M testEmbedAndClient
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ----Debug: remove() Size of allContexts HashSet obj= 0
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ----Debug: remove() Size of allContexts HashSet obj= 0
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ++++Debug: add() Size of allContexts HashSet obj= 2
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 3
> ----Debug: remove() Size of allContexts HashSet obj= 2
> ----Debug: remove() Size of allContexts HashSet obj= 1
> ==== Shutdown complete in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 2
> ++++Debug: add() Size of allContexts HashSet obj= 3
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 4
> ----Debug: remove() Size of allContexts HashSet obj= 3
> ----Debug: remove() Size of allContexts HashSet obj= 2
> ..
> ..
> ..
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 1109
> ----Debug: remove() Size of allContexts HashSet obj= 1108
> ----Debug: remove() Size of allContexts HashSet obj= 1107
> ==== Shutdown complete in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 1108
> ++++Debug: add() Size of allContexts HashSet obj= 1109
> ----Debug: remove() Size of allContexts HashSet obj= 1108
> java.sql.SQLException: Failed to start database 'testdb', see the next exception
>  for details.
>         at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
>         at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:89)
>         at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:95)
>         at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:174)
>         at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(EmbedConnection.java:1985)
>         at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:1649)
>         at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:223)
>         at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
>         at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:74)
>         at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:210)
> ----Debug: remove() Size of allContexts HashSet obj= 1107
>         at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:117)
>         at java.sql.DriverManager.getConnection(DriverManager.java:525)
>         at java.sql.DriverManager.getConnection(DriverManager.java:193)
>         at testEmbedAndClient.testInEmbedded(testEmbedAndClient.java:40)
>         at testEmbedAndClient.main(testEmbedAndClient.java:19)
> java.lang.OutOfMemoryError: Java heap space
> OOM happened after 1107 iterations

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.