Return-Path: 
 <dev-return-18811-apmail-activemq-dev-archive=activemq.apache.org@activemq.apache.org>
Delivered-To: apmail-activemq-dev-archive@www.apache.org
Received: (qmail 61686 invoked from network); 7 Feb 2010 14:06:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2010 14:06:57 -0000
Received: (qmail 2296 invoked by uid 500); 7 Feb 2010 14:06:57 -0000
Delivered-To: apmail-activemq-dev-archive@activemq.apache.org
Received: (qmail 2240 invoked by uid 500); 7 Feb 2010 14:06:56 -0000
Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@activemq.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@activemq.apache.org>
List-Post: <mailto:dev@activemq.apache.org>
List-Id: <dev.activemq.apache.org>
Reply-To: dev@activemq.apache.org
Delivered-To: mailing list dev@activemq.apache.org
Received: (qmail 2230 invoked by uid 99); 7 Feb 2010 14:06:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Feb 2010 14:06:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Feb 2010 14:06:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B0C7A234C045
	for <dev@activemq.apache.org>; Sun,  7 Feb 2010 06:06:32 -0800 (PST)
Message-ID: <2081794384.12881265551592709.JavaMail.jira@brutus.apache.org>
Date: Sun, 7 Feb 2010 14:06:32 +0000 (UTC)
From: "Mark Wuwer (JIRA)" <jira@apache.org>
To: dev@activemq.apache.org
Subject: [jira] Created: (AMQ-2596) Memory leak in network of brokers - the
 number of TopicSubscription/ConsumerInfo objects increases
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: ae95407df07c98740808b2ef9da0087c
X-Virus-Checked: Checked by ClamAV on apache.org

Memory leak in network of brokers - the number of TopicSubscription/ConsumerInfo objects increases
--------------------------------------------------------------------------------------------------

                 Key: AMQ-2596
                 URL: https://issues.apache.org/activemq/browse/AMQ-2596
             Project: ActiveMQ
          Issue Type: Bug
          Components: Broker
    Affects Versions: 5.3.0, 4.1.1
         Environment: OS: SLES10 SP2 or Ubuntu 9.10 (my desktop)
JVM: IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20081129 (JIT enabled)
            Reporter: Mark Wuwer
            Priority: Critical


I'd like to report a reproducible memory leak in network of brokers.

This is the scenario you can reproduce it (tried out with 4.1.1, 5.3.0 and 5.3.1):
1. start two brokers using attached configuration.
    Lets call one of it BE (for backend) another FE (for frontend) - accordingly: be_activemq.xml and fe_activemq.xml are attached.
    The BE broker contains following configuration: 
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>
        <networkConnectors>
           <networkConnector name="FE_connector" dynamicOnly="true" uri="static://(tcp://localhost:61617)"/>
        </networkConnectors>

    The FE broker contains just:
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61617"/>
        </transportConnectors>

2. now start a client that opens and close permanently topic subscriptions (see attached tests)
    to the FE broker

3. after a while you will see (e.g. verbose gc) that the heap usage of the BE broker will increase,
    but on FE not

If you analyze the heap dump of BE you will see that the memory leak consist of a large number of ConsumerInfo
objects. Easy way to find the objects when you use MAT is to use the OQL query e.g.:
    SELECT c.destination.physicalName.toString() FROM ".*ConsumerInfo" c WHERE (c.destination.physicalName.toString() LIKE "TEST_TOPIC")

I noticed that basically for each subscription on FE two ConsumerInfo objects are created on BE, which will never be closed.
There are however some special cases that only 4, 8 .. 12 objects per connection are created. This happens when the client just opens sessions/subscriptions without closing  (see appropriate junit test with comments in it).

We have it currently a P1 ticket at our customer site (running 3*FE + BE using 4.1.1) because of OoM that occurs periodically (in fact we now restart  every n-days the system). 

Another strange issue that happens at our customer site is something that I cannot reproduce at all.
The ConsumerInfo objects on BE contains there non empty selector strings, which makes the memory leak more critical.
In my test environment however selector strings are only set in ConsumerInfo on FE, but in BE broker the selectorin ConsumerInfo is always null!
I compared the customer configuration with my and I could not find any difference.

Any idea what could influence this behavior?

Thanks in advance for any feedback and help on this issue.
Best Regards,
Mark

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.