activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Harter (JIRA)" <>
Subject [jira] Created: (AMQ-2563) Memory Leak in DefaultJDBCAdapter
Date Fri, 08 Jan 2010 20:54:17 GMT
Memory Leak in DefaultJDBCAdapter

                 Key: AMQ-2563
             Project: ActiveMQ
          Issue Type: Bug
          Components: Message Store
    Affects Versions: 5.3.0
         Environment: Java 1.6.0_07-b06 on Solaris (5.10) sparcv9
            Reporter: Jim Harter

I was performing an endurance test on ActiveMQ 5.3.0.  My test consisted of a single queue
with a number of producers and consumers.  The producers send very simple text messages and
the consumers simply receive the message and do no other processing.  Using the OOTB configuration
(which uses -Xmx512m) with JDBC persistence to an Oracle database, I found the system came
to a halt at around 10.3 million messages.  Looking at the VM revealed a very full heap and
tiny gains from garbage collection.  Restarting the broker allows the system to run again.

To determine the cause of the exhausted heap, I took a series of heap dumps over time.  My
examination of the heaps showed that the number of live instances of TreeMap$Entry and Long
were increasing linearly with the number of messages.  The Longs were owned by the TreeMap$Entry
objects.  The TreeMap$Entry objects could be tracked back to the TreeSet<Long> instance
from the lastRecoveredMessageIds field in DefaultJDBCAdapter.

The only method that uses lastRecoveredMessageIds is:
public void doRecoverNextMessages(TransactionContext c, ActiveMQDestination destination, long
            int maxReturned, JDBCMessageRecoveryListener listener) throws Exception

As the listener is called to recover a message, the id is added to this set.  The id is only
removed from this set if it is encountered on future run of doRecoverNextMessages when it
is added to the cleanupIds list.  The SQL that is executed at the beginning of the method
filters messages based on having an id greater than nextSeq.  If nextSeq is always large enough,
an id is never added to cleanupIds and consequently never removed from lastRecoveredMessageIds.

I saw that the use of lastRecoveredMessageIds was introduced with AMQ-1918.  Also, AMQ-2436
synchronizes the TreeSet, but that should have no effect on this issue.

Dejan mentioned some work done on JDBC persistence and a memory leak fix in association with
AMQ-2519.  I have performed the same test on the latest snapshots of 5.3.1 and 5.4.  The same
results were observed on those as well.  Also, work done for AMQ-2519 modified JDBCMessageStore,
JDBCTopicMessageStore, and JDBCPersistenceAdapter.  I believe the issue is in DefaultJDBCAdapter.

To reproduce:
- Start a broker using JDBC persistence (sample configuration file will be attached)
- Start a number of producers and consumers using the same queue on that broker (sample WAR
file will be attached.  Modify WEB-INF/applicationContext.xml to set the queue name and broker
URL.  The war file currently uses TestQueue and tcp://localhost:61616, respectively.)
- Wait for about 20,000 messages to be processed (you can use less; it just gets easier in
the heap dumps to see after about 20,000 messages.  29 bytes are retained per message that
is processed)
- Look at the old generation size after a full garbage collection over time.  It grows slowly.
- Obtain a heap dump.  The heap dump will show a number of retained instances of TreeMap$Entry
and Long.  (sample heap dump will be attached)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message