activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Memory leak in JDBC Message Store
Date Fri, 15 May 2015 10:34:38 GMT
did you try running without the journal? ie: replace
 <persistenceFactory>
            <journalPersistenceAdapterFactory> ...

with:

 <persistenceAdapter>
      <jdbcPersistenceAdapter> ....

On 15 May 2015 at 09:17, Denary <dendromus.denary@gmail.com> wrote:
> Hello. Recently I've faced with weird problem, actually I'm not sure if it's
> a bug or misconfiguration or something else. But since my broker has
> "almost" default configuration I think this is bug.
> Brief configuration description(full configuration xml file at the end of
> issue):
> - Persistent storage: MySQL Db
> - Message type: only persistent Text Messages
> - Subscribers: only durable topic subscribers
>
> So, when I started broker I monitored its cpu/memory usage and here is what
> I noticed, memory chart below:(for servers monitoring I'm using New Relic):
> <http://activemq.2283324.n4.nabble.com/file/n4696558/amq_memory_leak_chart.png>
> As you can see usage of memory slowly but constantly growing.
> I decided to find out what exactly stored in activemq's heap, so I dumped
> heap and analyzed it with Eclipse MAT. Screenshots below:
> http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png
> <http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png>
> As we can see almost all of the memory occupied in JournalTopicMessageStore.
> Going deeper:
> http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png
> <http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png>
> We can see *completions* list in TransactionContext, which has size of 38590
> objects. This heap dump I made at 00:00.
> Below is a heap dump that I made 4 hours later:
> http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png
> <http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png>
>
> The picture is the same, but now this list contains 63770 objecs.
> Analyzed code a little bit, I noticed that this list is never cleared and
> hence its size always growth. Runnables that are stored in this list have
> link to Message object and hence this message is never will be GC'ed, even
> after ActiveMQ puts it to db. Of course I don't know probably this is a
> regular behavior that objects from this list are never removed, but it's
> looks really strange that object that contains this list is never recreated
> for 10 hours.
>
> Below I paste configuration of my broker:
> {code:title="broker configuration"|collapse}
> <beans
>         xmlns="http://www.springframework.org/schema/beans"
>         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>         xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>   http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
>     <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <value>file:${activemq.conf}/credentials.properties</value>
>         </property>
>     </bean>
>
>
>     <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
>           lazy-init="false" scope="singleton"
>           init-method="start" destroy-method="stop">
>     </bean>
>
>
>     <broker xmlns="http://activemq.apache.org/schema/core"
>             brokerName="broker"
>             persistent="true"
>             schedulerSupport="true"
>             advisorySupport="false"
>             dataDirectory="${activemq.data}">
>
>         <destinationPolicy>
>             <policyMap>
>                 <policyEntries>
>                     <policyEntry topic=">" producerFlowControl="false"
> expireMessagesPeriod="0">
>                         <deadLetterStrategy>
>                             <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
>                         </deadLetterStrategy>
>                     </policyEntry>
>                     <policyEntry queue=">" optimizedDispatch="true"
> producerFlowControl="false">
>                         <deadLetterStrategy>
>                             <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
>                         </deadLetterStrategy>
>                     </policyEntry>
>                 </policyEntries>
>             </policyMap>
>         </destinationPolicy>
>
>
>
>         <managementContext>
>             <managementContext createConnector="false"/>
>         </managementContext>
>
>
>         <persistenceFactory>
>             <journalPersistenceAdapterFactory
>                     journalLogFiles="8"
>                     journalLogFileSize="268435456"
>                     useJournal="true"
>                     useQuickJournal="false"
>                     dataSource="#mysql-ds"
>                     dataDirectory="${activemq.data}"/>
>         </persistenceFactory>
>
>
>         <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage limit="64 mb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="100 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="50 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>
>         <transportConnectors>
>
>             <transportConnector name="default"
> uri="nio://activemq.lc:61616?maximumConnections=1000&amp;connectionTimeout=600000&amp;wireFormat.maxFrameSize=104857600"/>
>         </transportConnectors>
>
>
>         <plugins>
>             <loggingBrokerPlugin logAll="true"/>
>             <redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
>                 <redeliveryPolicyMap>
>                     <redeliveryPolicyMap>
>                         <defaultEntry>
>                             <redeliveryPolicy maximumRedeliveries="4"
> initialRedeliveryDelay="5000" redeliveryDelay="10000"
> useExponentialBackOff="true" />
>                         </defaultEntry>
>                     </redeliveryPolicyMap>
>                 </redeliveryPolicyMap>
>             </redeliveryPlugin>
>         </plugins>
>
>
>         <shutdownHooks>
>             <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook"/>
>         </shutdownHooks>
>
>     </broker>
>
>     <bean id="mysql-ds"
>           class="org.apache.commons.dbcp.BasicDataSource"
>           destroy-method="close">
>         <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
>         <property name="url"
> value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&amp;autoReconnect=true"/>
>         <property name="username" value="root"/>
>         <property name="password" value=""/>
>         <property name="poolPreparedStatements" value="true"/>
>         <property name="maxActive" value="40"/>
>     </bean>
>
>
>     <import resource="jetty.xml"/>
>
> </beans>
> {code}
>
> Please help me fiund out is it configuration problem or it is really a bug,
> because I suffer with this problem for about 2 weeks.
> Thanks!
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Memory-leak-in-JDBC-Message-Store-tp4696558.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message