db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dag H. Wanvik (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-1947) OutOfMemoryError after repeated calls to boot and shutdown a database
Date Sat, 24 Mar 2007 16:32:32 GMT

     [ https://issues.apache.org/jira/browse/DERBY-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dag H. Wanvik updated DERBY-1947:
---------------------------------

    Attachment: DERBY-1947-1.stat
                DERBY-1947-1.diff

My analysis of this problem:

* When EmbedConnection#close is called from testEmbedAndClient.java:50, the
  call fails to clean up the connection properly, since the call to the
  private EmbedConnection#close(StandardException) never happens.

  This is due to the fact that the public close method first checks if
  the connection is already closed via a call to
  EmbedConnection#isClosed(). If so, close just returns,

  EmbedConnection#isClosed() returns true even though the connection is still
  marked as active, because the underlying database has been shut down. This
  fact makes TransactionResourceImpl#isActive() return false.

  Now, the connection objects and the TransactionResourceImpl will eventually
  be garbage collected, but the ContextManager object owned by
  TransactionResourceImpl, will not, leading to the memory leak.  This is
  because the ContextManager object is referenced from the HashSet
  'allContexts' of the ContextService.

  Normally, the context manager object is removed from 'allContexts' when the
  EmbedConnection#close(exceptionclose) calls
  TransactionResourceImpl#cleanupOnError. However, it does not happen here
  since isClosed returns true, so EmbedConnection#close() never gets
  to call EmbedConnection#close(exceptionclose).

  Modifying the repro to never close the connection object, I also see
  the leak, since EmbedConnection#finalize similarly fails to call
  EmbedConnection#close(exceptionclose) when isClosed() returns true.

  More detail (feel free to skip ;)

  TransactionResourceImpl#cleanupOnError, in turn, calls
  ContextManager#cleanupOnError which, when popping its stack of contexts
  finally calls the stack's bottom SystemContext#cleanupOnError, which again
  makes this call on line 62:

      :
      getContextManager().owningCsf.removeContext(getContextManager());
      :

  which releases the context manager from 'allContexts' so it can be garbage
  collected.


* I have made a patch which makes the repro (and the modified repro I
  mentioned) work without running out of space: DERBY-1947-1.{diff,stat}.

  It modifies EmbedConnection#close to clean up in the case described
  above.

  It also adds a finalizer method to TransactionResourceImpl which
  calls ContextManager#cleanupOnError with an argument of
  StandardException.closeException(), if not done already.  This
  catches the case when no close is called on the connection.  This
  change would make the leak go away on its own for both repro cases,
  but I added the change to EmbedConnection#close also, since it
  purports to release database resources in its Javadoc.

  So, when the connection is closed or garbage collected, the
  ContextManager will be cleaned up as well, and the leak is plugged.

  suites.All and derbyall ran without incident on Sun JDK 1.6, Solaris
  10/x86.

* There is a related issue, DERBY-2480, for which Jeff Clary has
  produced a patch. They overlap, in that that both patches makes a
  changes to EmbedConnection#close, so we need to reconcile them for
  that method. 


> OutOfMemoryError after repeated calls to boot and shutdown a database
> ---------------------------------------------------------------------
>
>                 Key: DERBY-1947
>                 URL: https://issues.apache.org/jira/browse/DERBY-1947
>             Project: Derby
>          Issue Type: Bug
>          Components: Services
>    Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 10.2.1.6
>         Environment: Any
>            Reporter: Rajesh Kartha
>         Assigned To: Dag H. Wanvik
>         Attachments: DERBY-1947-1.diff, DERBY-1947-1.stat, HeapAnalysis_op.jpg, testEmbedAndClient.java
>
>
> I came across this OOM issue while running some system tests involving
> backup and restore against  Derby. The test is expected to run forever but
> using the default heap space it runs into OOM within 2 days. I earlier mentioned about
this
> in my reply to the 10.2.1.6 vote - http://www.nabble.com/Re%3A--VOTE--10.2.1.6-release-p6650528.html
> Also there has been some discussions on the list on the related topic:
> http://issues.apache.org/jira/browse/DERBY-23 and
> http://www.nabble.com/question-about-shutdown-tf2300504.html
> Basic Analysis:
> --------------------
> Wrote a simple Java app (attached to this issue) that booted and shutdown the same 
> database multiple times. Depending on the heapsize the program ran into the
> OOM at some point, as expected. Some heap dump analysis using the IBM HeapAnalyzer 
> and revealed that the HashSet (allContexts) within org.apache.derby.iapi.services.context.ContextService

> class seemed to be location of the leak (snapshot of the heapanalysis attached).
> A little bit of debugging shows that:
>  - for every:connection two ContextManager objects (say, cm1, cm2) are added to the HashSet
>  - for every shutdown a new ContextManager object (say, cm3) is added and two objects
are removed
>  - the object removed are cm2 and cm3 - in that sequence
>  - but the object cm1 gets left behind
> This happens for every connect/shutdown sequence and since one of the ContextManager
objects added to the 
> HashSet is not removed as a part of the cleanup, it contributes to growth in memory usage,
hence
> an OOM eventually.
> For example:
> ============
> A 64M heap could allow about 1107 iterations of connect/shutdown only before running
into OOM and 
> created 1108 un-used ContextManager objects in the memory.
> java -Xmx64M testEmbedAndClient
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ----Debug: remove() Size of allContexts HashSet obj= 0
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ----Debug: remove() Size of allContexts HashSet obj= 0
> ++++Debug: add() Size of allContexts HashSet obj= 1
> ++++Debug: add() Size of allContexts HashSet obj= 2
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 3
> ----Debug: remove() Size of allContexts HashSet obj= 2
> ----Debug: remove() Size of allContexts HashSet obj= 1
> ==== Shutdown complete in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 2
> ++++Debug: add() Size of allContexts HashSet obj= 3
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 4
> ----Debug: remove() Size of allContexts HashSet obj= 3
> ----Debug: remove() Size of allContexts HashSet obj= 2
> ..
> ..
> ..
> ==== Database booted in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 1109
> ----Debug: remove() Size of allContexts HashSet obj= 1108
> ----Debug: remove() Size of allContexts HashSet obj= 1107
> ==== Shutdown complete in embedded ====
> ++++Debug: add() Size of allContexts HashSet obj= 1108
> ++++Debug: add() Size of allContexts HashSet obj= 1109
> ----Debug: remove() Size of allContexts HashSet obj= 1108
> java.sql.SQLException: Failed to start database 'testdb', see the next exception
>  for details.
>         at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
>         at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:89)
>         at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:95)
>         at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:174)
>         at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(EmbedConnection.java:1985)
>         at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:1649)
>         at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:223)
>         at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
>         at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:74)
>         at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:210)
> ----Debug: remove() Size of allContexts HashSet obj= 1107
>         at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:117)
>         at java.sql.DriverManager.getConnection(DriverManager.java:525)
>         at java.sql.DriverManager.getConnection(DriverManager.java:193)
>         at testEmbedAndClient.testInEmbedded(testEmbedAndClient.java:40)
>         at testEmbedAndClient.main(testEmbedAndClient.java:19)
> java.lang.OutOfMemoryError: Java heap space
> OOM happened after 1107 iterations

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message