Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F247D8BB for ; Wed, 18 Jul 2012 15:48:32 +0000 (UTC) Received: (qmail 30319 invoked by uid 500); 18 Jul 2012 15:48:31 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 30297 invoked by uid 500); 18 Jul 2012 15:48:31 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 30288 invoked by uid 99); 18 Jul 2012 15:48:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 15:48:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of MPilone@npr.org designates 216.32.181.182 as permitted sender) Received: from [216.32.181.182] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.182) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 15:48:21 +0000 Received: from mail44-ch1-R.bigfish.com (10.43.68.232) by CH1EHSOBE017.bigfish.com (10.43.70.67) with Microsoft SMTP Server id 14.1.225.23; Wed, 18 Jul 2012 15:47:59 +0000 Received: from mail44-ch1 (localhost [127.0.0.1]) by mail44-ch1-R.bigfish.com (Postfix) with ESMTP id 7F3DB3404F7 for ; Wed, 18 Jul 2012 15:47:59 +0000 (UTC) X-Forefront-Antispam-Report: CIP:205.153.36.34;KIP:(null);UIP:(null);IPV:NLI;H:mta-edge01.npr.org;RD:mta-edge01.npr.org;EFVD:NLI X-SpamScore: 0 X-BigFish: VS0(zzc85fhzz1202hzz8275bh8275dhz2fh54h2a8h668h839hd25hf0ah107ah34h) Received-SPF: pass (mail44-ch1: domain of npr.org designates 205.153.36.34 as permitted sender) client-ip=205.153.36.34; envelope-from=MPilone@npr.org; helo=mta-edge01.npr.org ;ge01.npr.org ; Received: from mail44-ch1 (localhost.localdomain [127.0.0.1]) by mail44-ch1 (MessageSwitch) id 134262647877910_24292; Wed, 18 Jul 2012 15:47:58 +0000 (UTC) Received: from CH1EHSMHS003.bigfish.com (snatpool3.int.messaging.microsoft.com [10.43.68.225]) by mail44-ch1.bigfish.com (Postfix) with ESMTP id 0F718240046 for ; Wed, 18 Jul 2012 15:47:58 +0000 (UTC) Received: from mta-edge01.npr.org (205.153.36.34) by CH1EHSMHS003.bigfish.com (10.43.70.3) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 18 Jul 2012 15:47:57 +0000 Received: from HQ-EXCH-HUB01.ads.npr.org (172.16.10.144) by hq-exch-edge01.ads.npr.org (172.16.10.145) with Microsoft SMTP Server (TLS) id 8.3.245.1; Wed, 18 Jul 2012 11:47:59 -0400 Received: from EXCHANGE.ads.npr.org ([169.254.2.29]) by HQ-EXCH-HUB01.ads.npr.org ([::1]) with mapi; Wed, 18 Jul 2012 11:47:56 -0400 From: Mike Pilone To: "users@activemq.apache.org" Date: Wed, 18 Jul 2012 11:47:56 -0400 Subject: Possible Memory Leak w/Temporary Queues and JMX Thread-Topic: Possible Memory Leak w/Temporary Queues and JMX Thread-Index: Ac1k/LL3cnjc90VOTfe65W+NmkZCzQ== Message-ID: <18FF9437FE96704F8A5114CF28DA2D13012BE8966464@EXCHANGE.ads.npr.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US x-kse-antivirus-interceptor-info: scan successful x-kse-antivirus-info: Clean Content-Type: multipart/mixed; boundary="_006_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_" MIME-Version: 1.0 X-OriginatorOrg: npr.org --_006_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_ Content-Type: multipart/related; boundary="_005_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_"; type="multipart/alternative" --_005_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_ Content-Type: multipart/alternative; boundary="_000_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_" --_000_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I'm seeing a slow memory leak in our broker process which appears to be rel= ated to temporary queues and JMX. We had major memory leak problems before,= but the 5.6.0 release fixed the majority of them. This leak seems to take = about 2 weeks to fill 512MB heap but ultimately it will bring down the JVM.= The configuration is as such: Java 6 ActiveMQ 5.6.0 Apache Camel 2.10.0 Network of brokers (2 nodes) The process simply runs an embeded broker which other clients connect to vi= a TCP. There are a couple static Camel routes, but nothing major in this pr= ocess. Other clients (connected via TCP) make heavy use of Camel and tempor= ary queues for request/reply messaging. I'm still looking into the issue but I've attached a MAT memory analysis to= see if anyone else has seen a related problem. It looks like the temporary= queues are getting registered in JMX and never removed but I'm just guessi= ng right now. Any input would be appreciated and let me know if you want any more informa= tion from the memory dump/MAT analysis. I can post the entire dump file som= eplace if it would be helpful. -mike [cid:image001.jpg@01CD64DB.08E39380] | Mike Pilone | Software Architect, Di= stribution | mpilone@npr.org | o: 202-513-2679 m: = 703-969-7493 --_000_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I'm seeing a slo= w memory leak in our broker process which appears to be related to temporar= y queues and JMX. We had major memory leak problems before, but the 5.6.0 r= elease fixed the majority of them. This leak seems to take about 2 weeks to= fill 512MB heap but ultimately it will bring down the JVM. The configurati= on is as such:

 

Java 6

ActiveMQ 5.6.0

Apache Camel 2.10.0

Network of brokers (2 nodes)

 

The process simply runs an emb= eded broker which other clients connect to via TCP. There are a couple stat= ic Camel routes, but nothing major in this process. Other clients (connecte= d via TCP) make heavy use of Camel and temporary queues for request/reply m= essaging.

 

I'm still looking into the issue but I've attached a MAT memor= y analysis to see if anyone else has seen a related problem. It looks like = the temporary queues are getting registered in JMX and never removed but I'= m just guessing right now.

 

Any input would be appreciated and let me know= if you want any more information from the memory dump/MAT analysis. I can = post the entire dump file someplace if it would be helpful.

<= p class=3DMsoNormal> 

-mike

 

= 3D"*" | Mike Pilone | Software Architect, Distr= ibution | mpi= lone@npr.org | o: 202-513-2679  m: 703-969-7493<= /span>

 

= --_000_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_-- --_005_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_ Content-Type: image/jpeg; name="image001.jpg" Content-Description: image001.jpg Content-Disposition: inline; filename="image001.jpg"; size=868; creation-date="Wed, 18 Jul 2012 15:47:55 GMT"; modification-date="Wed, 18 Jul 2012 15:47:55 GMT" Content-ID: Content-Transfer-Encoding: base64 /9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAoHBwgHBgoICAgLCgoLDhgQDg0NDh0VFhEYIx8lJCIf IiEmKzcvJik0KSEiMEExNDk7Pj4+JS5ESUM8SDc9Pjv/2wBDAQoLCw4NDhwQEBw7KCIoOzs7Ozs7 Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozv/wAARCAANACgDASIA AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3 ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm p6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEA AwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSEx BhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElK U1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3 uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwBNLsrK awvr2+M5S08vCQEAtuJHU1ifEGwj020ubeGRpIsRSIXGGw2CAffmrsd1NFbTW6SERT7fMXH3sHIr J8Y3U95o80txIZH/AHa5I7AgAflXmU2uZI+zxcJqnUk3pZ6fJf8ABIbPwboV2mm6f9o1BNU1LSzf Ry/IYEYBztIxuwdh5zxTvACCTT3jJwHugpx7haoat4z1VNG0zTbKX7JAumrBJsCl5Bls/PjcAfQH tV3wGxTS5nU4ZbjIPocCvewn8U+Er/AdprFtZWWoPaWnnsYJCjvKRhiPQCiqc08lzctPM++SRtzM e5or1kmkkzhbu9D/2Q== --_005_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_-- --_006_18FF9437FE96704F8A5114CF28DA2D13012BE8966464EXCHANGEads_--