activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pothier, Peter" <ppoth...@CROSSBEAMSYS.COM>
Subject RE: ActiveMQ 5.3.0 Memory Usage - Connections
Date Thu, 25 Mar 2010 16:27:16 GMT
Hi,

Even with ActiveMQ 5.3.1, I see the Heap Memory Usage creeping up
over time (albeit sawtooth).    It's now committed 58MB of 63 MB max.

The MBeans show

     Queue
        CursorMemoryUsage = 0
        DequeueCount = 46886
        Dispatched Count = 46886
        Enqueue Count = 46886
        Queue Size = 0

     Subscription Durable Topic
        DequeueCounter = 1002
        Dispatched Counter = 1002
        Enqueue Counter = 1002
        Dispatched Queue Size = 0
	
     Subscription Non-Durable Queue
        DequeueCounter = 47331
        Dispatched Counter = 48002
        Enqueue Counter = 48002
        Dispatched Queue Size = 671

     Topic
        DequeueCount = 0
        Dispatched Count = 48825
        Enqueue Count = 48868
        Queue Size = 0

The Subscription Durable Topic keeps disappearing and reappearing
because the client in the unit tests disconnects and reconnects
after running each test, sometimes sending one message, sometimes
sending as many as 1000.  That's why the count is so small for the
Subscription Durable Topic.

I'm concerned that the Subscription of the Queue is showing up as
Non Durable.  Using activemq-cpp, the server subscribes for the queue
using

     consumer = session->createConsumer()

I would have liked to have created it using

     consumer = session->createDurableConsumer()

however, that method only has support for a Topic, not a Queue

     dynamic_cast<Topic *>(destination)

Could this be contributing to the growing memory?
Persistent messages are being sent.


Peter P


-----Original Message-----
From: Pothier, Peter [mailto:ppothier@CROSSBEAMSYS.COM] 
Sent: Thursday, March 25, 2010 10:42 AM
To: Rob Davies
Cc: users@activemq.apache.org
Subject: RE: ActiveMQ 5.3.0 Memory Usage - Connections

Hi Rob,

One more thing I noticed, is that QueueSize of the Top Level Topic
(the one created by server's createTopic() I believe) always grows
with the Enqueue Count.

     Topic Attributes
     DequeueCount = 0
     DispatchCount = 1004
     EnqueueCount = 1003
     QueueSize = 1003
	
I just loaded 5.3.1

     INFO | ActiveMQ 5.3.1 JMS Message Broker (amq-broker) is starting

and this behavior is different.  The Queue Size now remains zero

     DequeueCount = 0
     DispatchCount = 4244
     EnqueueCount = 4247
     QueueSize = 0

So I'm hopeful.  I'll let the tests run for a while....

Thanks!

Peter P

-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com] 
Sent: Thursday, March 25, 2010 9:57 AM
To: Pothier, Peter
Cc: users@activemq.apache.org
Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections

Hi Peter,

cursors are used by persistent messages too. The error about paged in
messages in the jira issue is a bit of a red herring - but I did see
similar behaviour to what you describe when before I resolved that issue

cheers,

Rob
On 25 Mar 2010, at 13:50, Pothier, Peter wrote:

> Hi Rob,
> 
> Sure, I'll dig up some time to try 5.3.1.
> 
> FYI, I do not see the
> 
>     ERROR | Failed to page in more queue messages
> 
> type messages.  I only see
> 
>     INFO  | Slow KahaDB access: Journal append took...
>     WARN  | KahaDB PageFile flush: XXX queued writes, latch wait
> took...
> 
> type messages in the log.  So I'm not sure it's the same issue.
> Besides, I'm using Persistent Messages.  Are the Cursers Memories
> related to non-persistent messages?
> 
> 
> What about the Enque/Deque counters.  Is there a good explanation
> somewhere?
> I'm still troubled that the Topic has no dequeues.  Should I be?
> 
> Peter P
> 
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com] 
> Sent: Thursday, March 25, 2010 9:39 AM
> To: users@activemq.apache.org
> Cc: Pothier, Peter
> Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections
> 
> This could be due to
https://issues.apache.org/activemq/browse/AMQ-2512
> - could you try 5.3.1 ?
> 
> On 25 Mar 2010, at 13:28, Pothier, Peter wrote:
> 
>> Hi,
>> 
>> I finally figured out how to use jconsole remotely (I had a
>> misunderstanding
>> of what value to use in -Djava.rmi.server.hostname=<host>, using the
>> jconsole's
>> machine's IP address instead of the target).
>> 
>> Going back to running unit tests based on both libstomp and
> activemq-cpp
>> (2.2.1),
>> I can see, using jconsole, the Heap usage continuously rise (albeit
> with
>> a sawtooth),
>> and then reach its limit.  At this point the unit tests halt.  What
>> seems
>> the most interesting of all the Memory Pools is the "Tenured Gen"
> which
>> eventually
>> plateau's when the Used=Committed=Max.  The activemq.log, which
>> periodically has
>> the KahaDB slow messages or PageFile flush messages, suddenly stops.
> No
>> interesting
>> messages.  (by the way I reduced the heap down to 64M to get it to
>> saturate quicker).
>> 
>> I'm not really sure where to look.  So I took a look at the MBeans.
> The
>> AMQ-BROKER
>> attributes shows
>> 
>>        StorePercentUsage = 56
>>        TotalDequeueCount = 214618
>>        TotalEnqueueCount = 429046
>>        TotalMessageCount = 214428
>>        MemoryLimit = 20971520
>>        StoreLimit = 104857600
>>        TotalConsumerCount=2
>> 
>> I've read a little about the Total Enqueue/Message/Dequeue counters,
> but
>> still
>> don't understand how they relate to each other.  A picture would be
>> worth a
>> thousand words.
>> 
>> The setup is fairly simple right now, sending persistent messages.
>> 
>>    Producer - - > JMS Queue - - > Server - - > JMS Durable Topic - -
>> 
>> Consumer
>> 
>> The Queue shows
>> 
>>        DequeueCount = 214618
>>        DispatchCount = 214618
>>        EnqueCount = 214618
>>        MemoryPercentUsage = 0
>> 
>> The Topic seems more interesting
>> 
>>        DequeueCount = 0
>>        DispatchCount = 214236
>>        EnqueueCount = 214428
>>        MemoryPercentUsage = 0
>> 
>> 
>> Is it strange that the Dequeue Count for the Topic is zero?  I know
> the
>> consumer of the Topic is receiving messages.  Why would the Dequeue
>> Count
>> be zero?  Is the consumer suppose to be doing something that it's
not?
>> 
>> Any other places in jconsole I should be looking to determine where
> all
>> the heap is going?
>> 
>> Thanks!
>> 
>> Peter P
>> 
>> 
>> -----Original Message-----
>> From: Peter P [mailto:ppothier@crossbeamsys.com] 
>> Sent: Wednesday, March 17, 2010 5:56 PM
>> To: users@activemq.apache.org
>> Subject: ActiveMQ 5.3.0 Memory Usage - Connections
>> 
>> 
>> Hi,
>> 
>> We are using ActiveMQ 5.3.0, with both libstomp and ActiveMQ-CPP
>> producer
>> and consumer clients,
>> sending persistent messages to both queues and topics.  We have
> noticed
>> the
>> amount of memory used
>> by ActiveMQ (reported by linux top) grows over time.
>> 
>> Trying to determine whether the memory grew in response to messages
or
>> connections, we noticed
>> using a python script based on stomppy-2.0.4-1cb, that ActiveMQ
memory
>> grew
>> rather quickly when
>> we connected and disconnected multiple times.
>> 
>> Here's the basis of the script
>> 
>>  for i in range (0,1000) :
>>     conn = stomp.Connection()
>>     conn.set_listener('', MyListener())
>>     conn.start()
>>     conn.connect()
>>     conn.disconnect()
>> 
>> Using jmap/jhat, here's the most popular Instance Counts and
Histogram
>> prior
>> to running the script
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 
>> 29892 root      19   0  823m  95m  10m S  0.0  2.4   0:07.86 java 
>> 
>> 
>> All Classes (excluding platform)
>> Class 	Instance Count 	Total Size
>> class [B 	6288 	7214080
>> class [C 	26005 	2285484
>> class [I 	4775 	916612
>> class java.lang.reflect.Method 	6971 	899259
>> class java.lang.Class 	4802 	691488
>> class java.lang.String 	25732 	514640
>> class [Ljava.util.HashMap$Entry; 	3240 	514504
>> class [S 	6384 	396498
>> class [Ljava.lang.Object; 	4263 	378984
>> class [Lorg.apache.activemq.command.DataStructure; 	2 	262160
>> class java.util.LinkedHashMap$Entry 	4471 	196724
>> 
>> 
>> Instance Counts for All Classes (including platform)
>> 26005 instances of class [C
>> 25732 instances of class java.lang.String
>> 6971 instances of class java.lang.reflect.Method
>> 6741 instances of class [Ljava.lang.Class;
>> 6384 instances of class [S
>> 6288 instances of class [B
>> 5806 instances of class java.util.HashMap$Entry
>> 4802 instances of class java.lang.Class
>> 4775 instances of class [I
>> 4471 instances of class java.util.LinkedHashMap$Entry
>> 4263 instances of class [Ljava.lang.Object;
>> 
>> 
>> 
>> 
>> and after running the script a bunch of times
>> 
>> 
>> 
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 
>> 29892 root      18   0  880m 153m  10m S  0.0  3.9   0:35.62 java
>> 
>> Heap Histogram
>> 
>> All Classes (excluding platform)
>> Class 	Instance Count 	Total Size
>> class [B 	9880 	16696961
>> class [C 	32597 	3238584
>> class [I 	4823 	2948344
>> class java.util.concurrent.ConcurrentHashMap$Segment 	65216
2086912
>> class java.util.concurrent.locks.ReentrantLock$NonfairSync 	66712
>> 1867936
>> class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; 	65216
>> 1570048
>> class [Ljava.util.HashMap$Entry; 	6575 	1022200
>> class java.lang.reflect.Method 	6895 	889455
>> class java.lang.Class 	4832 	695808
>> class java.lang.String 	31143 	622860
>> class [Ljava.util.concurrent.ConcurrentHashMap$Segment; 	4076
>> 586944
>> class org.apache.activemq.command.ActiveMQMessage 	1812 	467496
>> class [S 	6077 	370234
>> class java.util.HashMap$Entry 	11704 	327712
>> class [Ljava.lang.Object; 	5130 	318360
>> class java.net.SocksSocketImpl 	1755 	282555
>> class [Lorg.apache.activemq.command.DataStructure; 	2 	262160
>> class java.util.HashMap 	5182 	248736
>> class java.util.concurrent.ConcurrentHashMap 	4076 	228256
>> class java.util.LinkedHashMap$Entry 	4155 	182820
>> 
>> Instance Counts for All Classes (including platform)
>> 66712 instances of class
>> java.util.concurrent.locks.ReentrantLock$NonfairSync
>> 65216 instances of class
> java.util.concurrent.ConcurrentHashMap$Segment
>> 65216 instances of class
>> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
>> 32597 instances of class [C
>> 31143 instances of class java.lang.String
>> 11704 instances of class java.util.HashMap$Entry
>> 9880 instances of class [B
>> 9371 instances of class java.lang.Object
>> 6895 instances of class java.lang.reflect.Method
>> 6575 instances of class [Ljava.util.HashMap$Entry;
>> 6143 instances of class [Ljava.lang.Class;
>> 6077 instances of class [S
>> 5182 instances of class java.util.HashMap
>> 5130 instances of class [Ljava.lang.Object;
>> 4832 instances of class java.lang.Class
>> 4823 instances of class [I
>> 
>> 
>> 
>> 
>> Here are the diffs between our config file and activemq-demo.xml
>> 
>> 
>> 
>> 51c51
>> <     <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="amq-broker" useJmx="true">
>> ---
>>>   <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="amq-broker" persistent="true" useJmx="true">
>> 68c68
>> <                     <policyEntry queue=">"
> producerFlowControl="true"
>> memoryLimit="5mb"/>
>> ---
>>>                   <policyEntry queue=">" producerFlowControl="false"
>>> memoryLimit="5mb"/>
>> 76a77,79
>>>                       <messageEvictionStrategy>
>>>                            <oldestMessageEvictionStrategy/>
>>>                       </messageEvictionStrategy>
>> 81d83
>> <                             -->
>> 82a85,87
>>>                           <timedSubscriptionRecoveryPolicy
>>> recoverDuration="60000" /> 
>>>                           -->
>>>                           <fixedCountSubscriptionRecoveryPolicy 
>>> maximumSize="300" /> 
>> 83a89
>>> 
>> 88a95,96
>>> 
>>> 
>> 197c205
>> <             <!-- Create a TCP transport that is advertised on via
an
>> IP
>> multicast 
>> ---
>>>           <!-- Create a TCP transport that is NOT advertised on via
>> an
>>> IP multicast 
>> 199c207
>> <             <transportConnector name="openwire"
>> uri="tcp://localhost:61616" discoveryUri="multicast://default"/>
>> ---
>>>           <transportConnector name="openwire"
>>> 
>> 
>
uri="tcp://localhost:61616?transport.keepAliveResponseRequired=true;wire
>> Format.tcpNoDelayEnabled=true"/>
>> 204c212
>> <             <transportConnector name="stomp"
>> uri="stomp://localhost:61613"/>
>> ---
>>>           <transportConnector name="stomp"
>>> uri="stomp://localhost:61613?wireFormat.tcpNoDelayEnabled=true"/>
>> 208a217,219
>>> 
>>> 
>>> 
>> 325c336,337
>> < </beans>
>> \ No newline at end of file
>> ---
>>> 
>>> </beans>
>> 
>> 
>> Checking the ActiveMQ 5.3.1 Fixed Issues page
>> 
>>    http://issues.apache.org/activemq/browse/AMQ/fixforversion/12183
>> 
>> this sounds different from any issue.
>> 
>> We run with only a single instance of ActiveMQ.
>> 
>> Are there any configuration parameters that controls this behavior?
>> Does ActiveMQ normally grow this large, cleaning up periodically?
>> 
>> Is this normal behavior?  Is there something I should be looking for?
>> 
>> Thanks,
>> 
>> Peter P
>> 
>> 
>> -- 
>> View this message in context:
>> 
>
http://old.nabble.com/ActiveMQ-5.3.0-Memory-Usage---Connections-tp279378
>> 10p27937810.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> 
> 


Mime
View raw message