Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 49F4E4A68 for ; Thu, 23 Jun 2011 17:00:30 +0000 (UTC) Received: (qmail 30575 invoked by uid 500); 23 Jun 2011 17:00:29 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 30544 invoked by uid 500); 23 Jun 2011 17:00:29 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 30536 invoked by uid 99); 23 Jun 2011 17:00:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:00:29 +0000 X-ASF-Spam-Status: No, hits=4.2 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RFC_ABUSE_POST,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 216.139.236.26 is neither permitted nor denied by domain of vijayender.bandaru@gmail.com) Received: from [216.139.236.26] (HELO sam.nabble.com) (216.139.236.26) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:00:22 +0000 Received: from joe.nabble.com ([192.168.236.139]) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1QZnG5-0002u8-KE for users@activemq.apache.org; Thu, 23 Jun 2011 10:00:01 -0700 Date: Thu, 23 Jun 2011 10:00:01 -0700 (PDT) From: vbandaru To: users@activemq.apache.org Message-ID: <1308848401569-3620449.post@n4.nabble.com> Subject: ActiveMQ Master/Slave dead/zombie connections issue MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_45981_1250771.1308848401571" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_45981_1250771.1308848401571 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Never found a similar issue on this forum and hence this post. I am using AMQ 5.5.0 with pure Master/slave configuration (with and without persistence). Master/slave configuration seems to be Okay interms of : Message replication to slave while master is ON Failover to slave when master fails We are deploying our JMS client (implements JMSlistner interface) on two remote servers (for failover redundancy) JMS Client Listener1 connects to Master Broker (using tcp://masterip:61616) always on TEST.FOO Queue JMS Client Listener2 connects to Slave Broker (using tcp://masterip:61617) always on TEST.FOO Queue.This starts consuming messages only if Master fails.  Both of these listeners use Spring's DefaultMEssageListenerContainer (DMLC) with dynamic scaling. concurrency="1-6" (starts with 1 and based on load can scale up to 6 MAX). My Understanding is Master/Slave config, all the state (messages, connection info etc.) is replicated to slave. So if Listener 1 opens 6 connections/sessions to master broker, they will be replicated to Slave Broker as well which seems to be causing problem in my case. So at this point, I killed the master, failover happens to slave successfully.  My Listener2, which keep polling for Slave broker to accept connections (DMLC does it automatic connection recovery), will obtain 6 connections. On Webconsole, I verified messages arriving onTEST.FOO and I also noticed there are 12 connections present for activeconsumer section of "TEST.FOO".   Table below (on the bottom) , I am printin printing ClientID and sessionId from Active Consumers for TEST.FOO Webconsole page First 6 connections correspond to the Listener1 next 6 for Listener 2. Now  If my producer sends 100 messages to failover://(tcp://masterip:61616,tcp://slaveip:61617)?randomize=false, on Listern 2 on Message I get: ( i have log statement) received Message 7 received Message 8 received Message 9 received Message 10 received Message 11 received Message 12 received Message 19 received Message 20 received Message 21 received Message 22 received Message 23 received Message 24 received Message 31 received Message 32 : : Seeems like the 6 dead connections (from listener1)  consume half of the messages which resulting in message loss. If I click on one of the 6 Connections from Listener 1, I get  "No connection could be found for ID ID:HDC00020891-2445-1308844599481-1:1".  Is there any way we can remove the connections established to Master Broker be not replciated to Slave on Master shutdown. Thanks.     Client ID Connection ID Session ID *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *1* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *2* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *3* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *4* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *5* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* *6* *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 1 *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 2 *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 3 *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 4 *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 5 *ID:HDC00020891-2445-1308844599481-1:1 ID:HDC00020891-2445-1308844599481-0:1* 6   -- View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Master-Slave-dead-zombie-connections-issue-tp3620449p3620449.html Sent from the ActiveMQ - User mailing list archive at Nabble.com. ------=_Part_45981_1250771.1308848401571--