Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 21510 invoked from network); 25 Mar 2010 13:57:48 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 25 Mar 2010 13:57:48 -0000 Received: (qmail 68182 invoked by uid 500); 25 Mar 2010 13:57:48 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 68160 invoked by uid 500); 25 Mar 2010 13:57:48 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 68152 invoked by uid 99); 25 Mar 2010 13:57:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 13:57:48 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rajdavies@gmail.com designates 209.85.221.171 as permitted sender) Received: from [209.85.221.171] (HELO mail-qy0-f171.google.com) (209.85.221.171) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 13:57:39 +0000 Received: by qyk1 with SMTP id 1so6076951qyk.0 for ; Thu, 25 Mar 2010 06:57:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=Mi70qfiGJvTm5KStWvvKxoflHIQcCVDDPrLwww3tPaM=; b=rlxc6pz1uu/pjQMJ13Qn01KFN1S3OiPNMnNNo2n+j4oMmZ1LvVqw6WhLkKouZzeF1c tFxtPR3kHG+bhfczDCj0YXB6mFHW+q/sPnsnzpNrhETn+xpYHLvq9d6RQ+4jkH0pRivC LK9JWiybBkOqCxn48qM0giMGUSwfsWNChcDb8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=vmSt7q9OaWwYx6R/65AfTJ9EcI+xMXmlC9wck3XiC+b0wKJHtj/C0uuMHiM3ec0n25 Bkn2VQjHDD0l3bAtRiULw5LX8xZQo+EyXvjw257GXXSBSurRpsbHLV4LL21uuRusZ4L7 3n04xhuv82mcyus3n78vpf/FjkeVerFlPhug8= Received: by 10.229.99.77 with SMTP id t13mr192026qcn.80.1269525438228; Thu, 25 Mar 2010 06:57:18 -0700 (PDT) Received: from nbwfhrboykin.americas.progress.com (host86-180-55-21.range86-180.btcentralplus.com [86.180.55.21]) by mx.google.com with ESMTPS id 22sm789284qyk.6.2010.03.25.06.57.16 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 25 Mar 2010 06:57:17 -0700 (PDT) Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Rob Davies In-Reply-To: <7425A3001D366C49940932CF649A53F5597990@cmail5.crossbeamsys.com> Date: Thu, 25 Mar 2010 13:57:13 +0000 Cc: Content-Transfer-Encoding: quoted-printable Message-Id: References: <27937810.post@talk.nabble.com> <7425A3001D366C49940932CF649A53F5597984@cmail5.crossbeamsys.com> <72BB3400-7E7F-43CD-AA19-0166B1BE0E8D@gmail.com> <7425A3001D366C49940932CF649A53F5597990@cmail5.crossbeamsys.com> To: "Pothier, Peter" X-Mailer: Apple Mail (2.1077) X-Virus-Checked: Checked by ClamAV on apache.org Hi Peter, cursors are used by persistent messages too. The error about paged in = messages in the jira issue is a bit of a red herring - but I did see = similar behaviour to what you describe when before I resolved that issue cheers, Rob On 25 Mar 2010, at 13:50, Pothier, Peter wrote: > Hi Rob, >=20 > Sure, I'll dig up some time to try 5.3.1. >=20 > FYI, I do not see the >=20 > ERROR | Failed to page in more queue messages >=20 > type messages. I only see >=20 > INFO | Slow KahaDB access: Journal append took... > WARN | KahaDB PageFile flush: XXX queued writes, latch wait > took... >=20 > type messages in the log. So I'm not sure it's the same issue. > Besides, I'm using Persistent Messages. Are the Cursers Memories > related to non-persistent messages? >=20 >=20 > What about the Enque/Deque counters. Is there a good explanation > somewhere? > I'm still troubled that the Topic has no dequeues. Should I be? >=20 > Peter P >=20 > -----Original Message----- > From: Rob Davies [mailto:rajdavies@gmail.com]=20 > Sent: Thursday, March 25, 2010 9:39 AM > To: users@activemq.apache.org > Cc: Pothier, Peter > Subject: Re: ActiveMQ 5.3.0 Memory Usage - Connections >=20 > This could be due to = https://issues.apache.org/activemq/browse/AMQ-2512 > - could you try 5.3.1 ? >=20 > On 25 Mar 2010, at 13:28, Pothier, Peter wrote: >=20 >> Hi, >>=20 >> I finally figured out how to use jconsole remotely (I had a >> misunderstanding >> of what value to use in -Djava.rmi.server.hostname=3D, using = the >> jconsole's >> machine's IP address instead of the target). >>=20 >> Going back to running unit tests based on both libstomp and > activemq-cpp >> (2.2.1), >> I can see, using jconsole, the Heap usage continuously rise (albeit > with >> a sawtooth), >> and then reach its limit. At this point the unit tests halt. What >> seems >> the most interesting of all the Memory Pools is the "Tenured Gen" > which >> eventually >> plateau's when the Used=3DCommitted=3DMax. The activemq.log, which >> periodically has >> the KahaDB slow messages or PageFile flush messages, suddenly stops. > No >> interesting >> messages. (by the way I reduced the heap down to 64M to get it to >> saturate quicker). >>=20 >> I'm not really sure where to look. So I took a look at the MBeans. > The >> AMQ-BROKER >> attributes shows >>=20 >> StorePercentUsage =3D 56 >> TotalDequeueCount =3D 214618 >> TotalEnqueueCount =3D 429046 >> TotalMessageCount =3D 214428 >> MemoryLimit =3D 20971520 >> StoreLimit =3D 104857600 >> TotalConsumerCount=3D2 >>=20 >> I've read a little about the Total Enqueue/Message/Dequeue counters, > but >> still >> don't understand how they relate to each other. A picture would be >> worth a >> thousand words. >>=20 >> The setup is fairly simple right now, sending persistent messages. >>=20 >> Producer - - > JMS Queue - - > Server - - > JMS Durable Topic - - >>=20 >> Consumer >>=20 >> The Queue shows >>=20 >> DequeueCount =3D 214618 >> DispatchCount =3D 214618 >> EnqueCount =3D 214618 >> MemoryPercentUsage =3D 0 >>=20 >> The Topic seems more interesting >>=20 >> DequeueCount =3D 0 >> DispatchCount =3D 214236 >> EnqueueCount =3D 214428 >> MemoryPercentUsage =3D 0 >>=20 >>=20 >> Is it strange that the Dequeue Count for the Topic is zero? I know > the >> consumer of the Topic is receiving messages. Why would the Dequeue >> Count >> be zero? Is the consumer suppose to be doing something that it's = not? >>=20 >> Any other places in jconsole I should be looking to determine where > all >> the heap is going? >>=20 >> Thanks! >>=20 >> Peter P >>=20 >>=20 >> -----Original Message----- >> From: Peter P [mailto:ppothier@crossbeamsys.com]=20 >> Sent: Wednesday, March 17, 2010 5:56 PM >> To: users@activemq.apache.org >> Subject: ActiveMQ 5.3.0 Memory Usage - Connections >>=20 >>=20 >> Hi, >>=20 >> We are using ActiveMQ 5.3.0, with both libstomp and ActiveMQ-CPP >> producer >> and consumer clients, >> sending persistent messages to both queues and topics. We have > noticed >> the >> amount of memory used >> by ActiveMQ (reported by linux top) grows over time. >>=20 >> Trying to determine whether the memory grew in response to messages = or >> connections, we noticed >> using a python script based on stomppy-2.0.4-1cb, that ActiveMQ = memory >> grew >> rather quickly when >> we connected and disconnected multiple times. >>=20 >> Here's the basis of the script >>=20 >> for i in range (0,1000) : >> conn =3D stomp.Connection() >> conn.set_listener('', MyListener()) >> conn.start() >> conn.connect() >> conn.disconnect() >>=20 >> Using jmap/jhat, here's the most popular Instance Counts and = Histogram >> prior >> to running the script >>=20 >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>=20 >> 29892 root 19 0 823m 95m 10m S 0.0 2.4 0:07.86 java=20 >>=20 >>=20 >> All Classes (excluding platform) >> Class Instance Count Total Size >> class [B 6288 7214080 >> class [C 26005 2285484 >> class [I 4775 916612 >> class java.lang.reflect.Method 6971 899259 >> class java.lang.Class 4802 691488 >> class java.lang.String 25732 514640 >> class [Ljava.util.HashMap$Entry; 3240 514504 >> class [S 6384 396498 >> class [Ljava.lang.Object; 4263 378984 >> class [Lorg.apache.activemq.command.DataStructure; 2 262160 >> class java.util.LinkedHashMap$Entry 4471 196724 >>=20 >>=20 >> Instance Counts for All Classes (including platform) >> 26005 instances of class [C >> 25732 instances of class java.lang.String >> 6971 instances of class java.lang.reflect.Method >> 6741 instances of class [Ljava.lang.Class; >> 6384 instances of class [S >> 6288 instances of class [B >> 5806 instances of class java.util.HashMap$Entry >> 4802 instances of class java.lang.Class >> 4775 instances of class [I >> 4471 instances of class java.util.LinkedHashMap$Entry >> 4263 instances of class [Ljava.lang.Object; >>=20 >>=20 >>=20 >>=20 >> and after running the script a bunch of times >>=20 >>=20 >>=20 >>=20 >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>=20 >> 29892 root 18 0 880m 153m 10m S 0.0 3.9 0:35.62 java >>=20 >> Heap Histogram >>=20 >> All Classes (excluding platform) >> Class Instance Count Total Size >> class [B 9880 16696961 >> class [C 32597 3238584 >> class [I 4823 2948344 >> class java.util.concurrent.ConcurrentHashMap$Segment 65216 = 2086912 >> class java.util.concurrent.locks.ReentrantLock$NonfairSync 66712 >> 1867936 >> class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; 65216 >> 1570048 >> class [Ljava.util.HashMap$Entry; 6575 1022200 >> class java.lang.reflect.Method 6895 889455 >> class java.lang.Class 4832 695808 >> class java.lang.String 31143 622860 >> class [Ljava.util.concurrent.ConcurrentHashMap$Segment; 4076 >> 586944 >> class org.apache.activemq.command.ActiveMQMessage 1812 467496 >> class [S 6077 370234 >> class java.util.HashMap$Entry 11704 327712 >> class [Ljava.lang.Object; 5130 318360 >> class java.net.SocksSocketImpl 1755 282555 >> class [Lorg.apache.activemq.command.DataStructure; 2 262160 >> class java.util.HashMap 5182 248736 >> class java.util.concurrent.ConcurrentHashMap 4076 228256 >> class java.util.LinkedHashMap$Entry 4155 182820 >>=20 >> Instance Counts for All Classes (including platform) >> 66712 instances of class >> java.util.concurrent.locks.ReentrantLock$NonfairSync >> 65216 instances of class > java.util.concurrent.ConcurrentHashMap$Segment >> 65216 instances of class >> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; >> 32597 instances of class [C >> 31143 instances of class java.lang.String >> 11704 instances of class java.util.HashMap$Entry >> 9880 instances of class [B >> 9371 instances of class java.lang.Object >> 6895 instances of class java.lang.reflect.Method >> 6575 instances of class [Ljava.util.HashMap$Entry; >> 6143 instances of class [Ljava.lang.Class; >> 6077 instances of class [S >> 5182 instances of class java.util.HashMap >> 5130 instances of class [Ljava.lang.Object; >> 4832 instances of class java.lang.Class >> 4823 instances of class [I >>=20 >>=20 >>=20 >>=20 >> Here are the diffs between our config file and activemq-demo.xml >>=20 >>=20 >>=20 >> 51c51 >> < > brokerName=3D"amq-broker" useJmx=3D"true"> >> --- >>> >> brokerName=3D"amq-broker" persistent=3D"true" useJmx=3D"true"> >> 68c68 >> < " > producerFlowControl=3D"true" >> memoryLimit=3D"5mb"/> >> --- >>> " = producerFlowControl=3D"false" >>> memoryLimit=3D"5mb"/> >> 76a77,79 >>> >>> >>> >> 81d83 >> < --> >> 82a85,87 >>> >> recoverDuration=3D"60000" />=20 >>> --> >>> >> maximumSize=3D"300" />=20 >> 83a89 >>>=20 >> 88a95,96 >>>=20 >>>=20 >> 197c205 >> <