activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stormtrooper <her...@stormpath.com>
Subject Re: Master/Slave: Message Hoarding
Date Fri, 30 Aug 2013 01:05:52 GMT
Hello,

Well, we've figured this out so I'll record it here for posterity.

Our master/slave setup was incorrectly configured. 

Originally our architecture was going to have two clusters that talked to
each other. In our phased development approach we scaled back to work with
just one initially. However, the Network Connector config node was
inadvertently/misunderstood and left in:

<networkConnector
uri="masterslave:(tcp://localhost:61616,tcp://[slavehost]:61616)"/>

What caused us to zero in on this was that every time we connected to one of
our queues, in the Admin Console we would see our consumers in the "Active
Consumers" page for the queue, but ALSO an additional consumer:
"NC_[our_master_brokerName]_inbound_[our_master_brokerName]"
This consumer would also disappear when our consumers disconnected from the
queue.

Behavior would manifest as such:
For a queue:
* Enqueue is always correct, it's the total number sent. 
* When the consumers are done consuming what they will be sent, the
inflights are 0. 
* The hoarded difference always corresponds to the # of rows in the
ACTIVEMQ_MSGS table that won't budge.
* Dequeue is also always correct, it's the total number received by the
consumers BUT that INCLUDES the NC consumers.

Example: We send 150 messages to the queue. 
* Our client consumer: Enqueue/Dequeue/Dispatch are all 50 each. 
* The NC consumer: Enqueue/Dequeue/Dispatch are all 50 each.
* Pending for queue 50 = 50 rows in the ACTIVEMQ_MSGS db

Our client and the NC consumer would alternate consuming messages until 100,
and then the last contiguous block of 50 would be hoarded.

Oddly enough, a couple of times we left the cluster up, but inactive, all
night. And when we returned to run tests, there would be no creation of NC
consumers and no hoarding. Everything worked fine.
Our tip off was that in this scenario, failover from the master to slave
worked fine, so it was clear that the NC was not needed within a single
master/slave cluster. [Not sure why this is?]

But without fail, if we restarted the cluster and started using it
relatively soon, we would see the NC consumers and the hoarding.

Whoosh.
Hopefully this might help someone else save some time,
-h




--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591p4670940.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message