activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pothier, Peter" <ppoth...@CROSSBEAMSYS.COM>
Subject RE: ActiveMQ 5.3.0 Memory Usage - Connections
Date Thu, 25 Mar 2010 14:42:12 GMT
Hi Rob,

One more thing I noticed, is that QueueSize of the Top Level Topic
(the one created by server's createTopic() I believe) always grows
with the Enqueue Count.

     Topic Attributes
     DequeueCount = 0
     DispatchCount = 1004
     EnqueueCount = 1003
     QueueSize = 1003
	
I just loaded 5.3.1

     INFO | ActiveMQ 5.3.1 JMS Message Broker (amq-broker) is starting

and this behavior is different.  The Queue Size now remains zero

     DequeueCount = 0
     DispatchCount = 4244
     EnqueueCount = 4247
     QueueSize = 0

So I'm hopeful.  I'll let the tests run for a while....

Thanks!

Peter P

-----Original Message-----
From: Rob Davies [mailto:rajdavies@gmail.com] 
Sent: Thursday, March 25, 2010 9:57 AM
To: Pothier, Peter
Cc: users@activemq.apache.org
Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections

Hi Peter,

cursors are used by persistent messages too. The error about paged in
messages in the jira issue is a bit of a red herring - but I did see
similar behaviour to what you describe when before I resolved that issue

cheers,

Rob
On 25 Mar 2010, at 13:50, Pothier, Peter wrote:

> Hi Rob,
> 
> Sure, I'll dig up some time to try 5.3.1.
> 
> FYI, I do not see the
> 
>     ERROR | Failed to page in more queue messages
> 
> type messages.  I only see
> 
>     INFO  | Slow KahaDB access: Journal append took...
>     WARN  | KahaDB PageFile flush: XXX queued writes, latch wait
> took...
> 
> type messages in the log.  So I'm not sure it's the same issue.
> Besides, I'm using Persistent Messages.  Are the Cursers Memories
> related to non-persistent messages?
> 
> 
> What about the Enque/Deque counters.  Is there a good explanation
> somewhere?
> I'm still troubled that the Topic has no dequeues.  Should I be?
> 
> Peter P
> 
> -----Original Message-----
> From: Rob Davies [mailto:rajdavies@gmail.com] 
> Sent: Thursday, March 25, 2010 9:39 AM
> To: users@activemq.apache.org
> Cc: Pothier, Peter
> Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections
> 
> This could be due to
https://issues.apache.org/activemq/browse/AMQ-2512
> - could you try 5.3.1 ?
> 
> On 25 Mar 2010, at 13:28, Pothier, Peter wrote:
> 
>> Hi,
>> 
>> I finally figured out how to use jconsole remotely (I had a
>> misunderstanding
>> of what value to use in -Djava.rmi.server.hostname=<host>, using the
>> jconsole's
>> machine's IP address instead of the target).
>> 
>> Going back to running unit tests based on both libstomp and
> activemq-cpp
>> (2.2.1),
>> I can see, using jconsole, the Heap usage continuously rise (albeit
> with
>> a sawtooth),
>> and then reach its limit.  At this point the unit tests halt.  What
>> seems
>> the most interesting of all the Memory Pools is the "Tenured Gen"
> which
>> eventually
>> plateau's when the Used=Committed=Max.  The activemq.log, which
>> periodically has
>> the KahaDB slow messages or PageFile flush messages, suddenly stops.
> No
>> interesting
>> messages.  (by the way I reduced the heap down to 64M to get it to
>> saturate quicker).
>> 
>> I'm not really sure where to look.  So I took a look at the MBeans.
> The
>> AMQ-BROKER
>> attributes shows
>> 
>>        StorePercentUsage = 56
>>        TotalDequeueCount = 214618
>>        TotalEnqueueCount = 429046
>>        TotalMessageCount = 214428
>>        MemoryLimit = 20971520
>>        StoreLimit = 104857600
>>        TotalConsumerCount=2
>> 
>> I've read a little about the Total Enqueue/Message/Dequeue counters,
> but
>> still
>> don't understand how they relate to each other.  A picture would be
>> worth a
>> thousand words.
>> 
>> The setup is fairly simple right now, sending persistent messages.
>> 
>>    Producer - - > JMS Queue - - > Server - - > JMS Durable Topic - -
>> 
>> Consumer
>> 
>> The Queue shows
>> 
>>        DequeueCount = 214618
>>        DispatchCount = 214618
>>        EnqueCount = 214618
>>        MemoryPercentUsage = 0
>> 
>> The Topic seems more interesting
>> 
>>        DequeueCount = 0
>>        DispatchCount = 214236
>>        EnqueueCount = 214428
>>        MemoryPercentUsage = 0
>> 
>> 
>> Is it strange that the Dequeue Count for the Topic is zero?  I know
> the
>> consumer of the Topic is receiving messages.  Why would the Dequeue
>> Count
>> be zero?  Is the consumer suppose to be doing something that it's
not?
>> 
>> Any other places in jconsole I should be looking to determine where
> all
>> the heap is going?
>> 
>> Thanks!
>> 
>> Peter P
>> 
>> 
>> -----Original Message-----
>> From: Peter P [mailto:ppothier@crossbeamsys.com] 
>> Sent: Wednesday, March 17, 2010 5:56 PM
>> To: users@activemq.apache.org
>> Subject: ActiveMQ 5.3.0 Memory Usage - Connections
>> 
>> 
>> Hi,
>> 
>> We are using ActiveMQ 5.3.0, with both libstomp and ActiveMQ-CPP
>> producer
>> and consumer clients,
>> sending persistent messages to both queues and topics.  We have
> noticed
>> the
>> amount of memory used
>> by ActiveMQ (reported by linux top) grows over time.
>> 
>> Trying to determine whether the memory grew in response to messages
or
>> connections, we noticed
>> using a python script based on stomppy-2.0.4-1cb, that ActiveMQ
memory
>> grew
>> rather quickly when
>> we connected and disconnected multiple times.
>> 
>> Here's the basis of the script
>> 
>>  for i in range (0,1000) :
>>     conn = stomp.Connection()
>>     conn.set_listener('', MyListener())
>>     conn.start()
>>     conn.connect()
>>     conn.disconnect()
>> 
>> Using jmap/jhat, here's the most popular Instance Counts and
Histogram
>> prior
>> to running the script
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 
>> 29892 root      19   0  823m  95m  10m S  0.0  2.4   0:07.86 java 
>> 
>> 
>> All Classes (excluding platform)
>> Class 	Instance Count 	Total Size
>> class [B 	6288 	7214080
>> class [C 	26005 	2285484
>> class [I 	4775 	916612
>> class java.lang.reflect.Method 	6971 	899259
>> class java.lang.Class 	4802 	691488
>> class java.lang.String 	25732 	514640
>> class [Ljava.util.HashMap$Entry; 	3240 	514504
>> class [S 	6384 	396498
>> class [Ljava.lang.Object; 	4263 	378984
>> class [Lorg.apache.activemq.command.DataStructure; 	2 	262160
>> class java.util.LinkedHashMap$Entry 	4471 	196724
>> 
>> 
>> Instance Counts for All Classes (including platform)
>> 26005 instances of class [C
>> 25732 instances of class java.lang.String
>> 6971 instances of class java.lang.reflect.Method
>> 6741 instances of class [Ljava.lang.Class;
>> 6384 instances of class [S
>> 6288 instances of class [B
>> 5806 instances of class java.util.HashMap$Entry
>> 4802 instances of class java.lang.Class
>> 4775 instances of class [I
>> 4471 instances of class java.util.LinkedHashMap$Entry
>> 4263 instances of class [Ljava.lang.Object;
>> 
>> 
>> 
>> 
>> and after running the script a bunch of times
>> 
>> 
>> 
>> 
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 
>> 29892 root      18   0  880m 153m  10m S  0.0  3.9   0:35.62 java
>> 
>> Heap Histogram
>> 
>> All Classes (excluding platform)
>> Class 	Instance Count 	Total Size
>> class [B 	9880 	16696961
>> class [C 	32597 	3238584
>> class [I 	4823 	2948344
>> class java.util.concurrent.ConcurrentHashMap$Segment 	65216
2086912
>> class java.util.concurrent.locks.ReentrantLock$NonfairSync 	66712
>> 1867936
>> class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; 	65216
>> 1570048
>> class [Ljava.util.HashMap$Entry; 	6575 	1022200
>> class java.lang.reflect.Method 	6895 	889455
>> class java.lang.Class 	4832 	695808
>> class java.lang.String 	31143 	622860
>> class [Ljava.util.concurrent.ConcurrentHashMap$Segment; 	4076
>> 586944
>> class org.apache.activemq.command.ActiveMQMessage 	1812 	467496
>> class [S 	6077 	370234
>> class java.util.HashMap$Entry 	11704 	327712
>> class [Ljava.lang.Object; 	5130 	318360
>> class java.net.SocksSocketImpl 	1755 	282555
>> class [Lorg.apache.activemq.command.DataStructure; 	2 	262160
>> class java.util.HashMap 	5182 	248736
>> class java.util.concurrent.ConcurrentHashMap 	4076 	228256
>> class java.util.LinkedHashMap$Entry 	4155 	182820
>> 
>> Instance Counts for All Classes (including platform)
>> 66712 instances of class
>> java.util.concurrent.locks.ReentrantLock$NonfairSync
>> 65216 instances of class
> java.util.concurrent.ConcurrentHashMap$Segment
>> 65216 instances of class
>> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
>> 32597 instances of class [C
>> 31143 instances of class java.lang.String
>> 11704 instances of class java.util.HashMap$Entry
>> 9880 instances of class [B
>> 9371 instances of class java.lang.Object
>> 6895 instances of class java.lang.reflect.Method
>> 6575 instances of class [Ljava.util.HashMap$Entry;
>> 6143 instances of class [Ljava.lang.Class;
>> 6077 instances of class [S
>> 5182 instances of class java.util.HashMap
>> 5130 instances of class [Ljava.lang.Object;
>> 4832 instances of class java.lang.Class
>> 4823 instances of class [I
>> 
>> 
>> 
>> 
>> Here are the diffs between our config file and activemq-demo.xml
>> 
>> 
>> 
>> 51c51
>> <     <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="amq-broker" useJmx="true">
>> ---
>>>   <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="amq-broker" persistent="true" useJmx="true">
>> 68c68
>> <                     <policyEntry queue=">"
> producerFlowControl="true"
>> memoryLimit="5mb"/>
>> ---
>>>                   <policyEntry queue=">" producerFlowControl="false"
>>> memoryLimit="5mb"/>
>> 76a77,79
>>>                       <messageEvictionStrategy>
>>>                            <oldestMessageEvictionStrategy/>
>>>                       </messageEvictionStrategy>
>> 81d83
>> <                             -->
>> 82a85,87
>>>                           <timedSubscriptionRecoveryPolicy
>>> recoverDuration="60000" /> 
>>>                           -->
>>>                           <fixedCountSubscriptionRecoveryPolicy 
>>> maximumSize="300" /> 
>> 83a89
>>> 
>> 88a95,96
>>> 
>>> 
>> 197c205
>> <             <!-- Create a TCP transport that is advertised on via
an
>> IP
>> multicast 
>> ---
>>>           <!-- Create a TCP transport that is NOT advertised on via
>> an
>>> IP multicast 
>> 199c207
>> <             <transportConnector name="openwire"
>> uri="tcp://localhost:61616" discoveryUri="multicast://default"/>
>> ---
>>>           <transportConnector name="openwire"
>>> 
>> 
>
uri="tcp://localhost:61616?transport.keepAliveResponseRequired=true;wire
>> Format.tcpNoDelayEnabled=true"/>
>> 204c212
>> <             <transportConnector name="stomp"
>> uri="stomp://localhost:61613"/>
>> ---
>>>           <transportConnector name="stomp"
>>> uri="stomp://localhost:61613?wireFormat.tcpNoDelayEnabled=true"/>
>> 208a217,219
>>> 
>>> 
>>> 
>> 325c336,337
>> < </beans>
>> \ No newline at end of file
>> ---
>>> 
>>> </beans>
>> 
>> 
>> Checking the ActiveMQ 5.3.1 Fixed Issues page
>> 
>>    http://issues.apache.org/activemq/browse/AMQ/fixforversion/12183
>> 
>> this sounds different from any issue.
>> 
>> We run with only a single instance of ActiveMQ.
>> 
>> Are there any configuration parameters that controls this behavior?
>> Does ActiveMQ normally grow this large, cleaning up periodically?
>> 
>> Is this normal behavior?  Is there something I should be looking for?
>> 
>> Thanks,
>> 
>> Peter P
>> 
>> 
>> -- 
>> View this message in context:
>> 
>
http://old.nabble.com/ActiveMQ-5.3.0-Memory-Usage---Connections-tp279378
>> 10p27937810.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> 
> 


Mime
View raw message