activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Reddy <vijayender.band...@gmail.com>
Subject ActiveMQ Master/Slave dead/zombie connections issue
Date Thu, 23 Jun 2011 17:12:21 GMT
Excuse me if you recieved this mail twice. I tried to post it on nabble site
but somehow it is not forwarding it to  users@activemq.apache.org.

Never found a similar issue on this forum and hence this post.

I am using AMQ 5.5.0 with pure Master/slave configuration (with and without
persistence).

Master/slave configuration seems to be Okay interms of :

   - Message replication to slave while master is ON
   - Failover to slave when master fails

We are deploying our JMS client (implements JMSlistner interface) on two
remote servers (for failover redundancy)

   - JMS Client Listener1 connects to Master Broker (using
   tcp://masterip:61616) always on TEST.FOO Queue
   - JMS Client Listener2 connects to Slave Broker (using
   tcp://masterip:61617) always on TEST.FOO Queue.This starts consuming
   messages only if Master fails.

Both of these listeners use Spring's DefaultMEssageListenerContainer (DMLC)
with dynamic scaling. concurrency="1-6" (starts with 1 and based on load can
scale up to 6 MAX).

My Understanding is Master/Slave config, all the state (messages, connection
info etc.) is replicated to slave. So if Listener 1 opens 6
*connections*/sessions
to master broker, they will be replicated to Slave Broker as well which
seems to be causing problem in my case.

So at this point, I killed the master, failover happens to slave
successfully.  My Listener2, which keep polling for Slave broker to accept *
connections* (DMLC does it automatic connection recovery), will obtain 6 *
connections*.

On Webconsole, I verified messages arriving onTEST.FOO and I also noticed
there are 12 *connections* present for activeconsumer section of "TEST.FOO".



Table below (on the bottom) , I am printin printing ClientID and sessionId
from Active Consumers for TEST.FOO Webconsole page First 6
*connections* correspond
to the Listener1 next 6 for Listener 2. Now

If my producer sends 100 messages
to failover://(tcp://masterip:61616,tcp://slaveip:61617)?randomize=false, on
Listern 2 on Message I get: ( i have log statement)

received Message 7

received Message 8

received Message 9

received Message 10

received Message 11

received Message 12

received Message 19

received Message 20

received Message 21

received Message 22

received Message 23

received Message 24

received Message 31

received Message 32

:

:

Seeems like the 6 *dead* *connections* (from listener1)  consume half of the
messages which resulting in message loss. If I click on one of the 6 *
Connections* from Listener 1, I get  "No connection could be found for ID
ID:HDC00020891-2445-1308844599481-1:1".

Is there any way we can remove the *connections* established to Master
Broker be not replciated to Slave on Master shutdown.

Thanks.


 Client ID
Connection IDSession ID*ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**1**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**2**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**3**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**4**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**5**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1**6**
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*1*
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*2*
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*3*
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*4*
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*5*
ID:HDC00020891-2445-1308844599481-1:1
ID:HDC00020891-2445-1308844599481-0:1*6

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message