activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Master/Slave: Message Hoarding
Date Fri, 30 Aug 2013 02:26:35 GMT
Thanks for taking the time to post your findings!



On Thu, Aug 29, 2013 at 6:05 PM, stormtrooper <hereen@stormpath.com> wrote:

> Hello,
>
> Well, we've figured this out so I'll record it here for posterity.
>
> Our master/slave setup was incorrectly configured.
>
> Originally our architecture was going to have two clusters that talked to
> each other. In our phased development approach we scaled back to work with
> just one initially. However, the Network Connector config node was
> inadvertently/misunderstood and left in:
>
> <networkConnector
> uri="masterslave:(tcp://localhost:61616,tcp://[slavehost]:61616)"/>
>
> What caused us to zero in on this was that every time we connected to one
> of
> our queues, in the Admin Console we would see our consumers in the "Active
> Consumers" page for the queue, but ALSO an additional consumer:
> "NC_[our_master_brokerName]_inbound_[our_master_brokerName]"
> This consumer would also disappear when our consumers disconnected from the
> queue.
>
> Behavior would manifest as such:
> For a queue:
> * Enqueue is always correct, it's the total number sent.
> * When the consumers are done consuming what they will be sent, the
> inflights are 0.
> * The hoarded difference always corresponds to the # of rows in the
> ACTIVEMQ_MSGS table that won't budge.
> * Dequeue is also always correct, it's the total number received by the
> consumers BUT that INCLUDES the NC consumers.
>
> Example: We send 150 messages to the queue.
> * Our client consumer: Enqueue/Dequeue/Dispatch are all 50 each.
> * The NC consumer: Enqueue/Dequeue/Dispatch are all 50 each.
> * Pending for queue 50 = 50 rows in the ACTIVEMQ_MSGS db
>
> Our client and the NC consumer would alternate consuming messages until
> 100,
> and then the last contiguous block of 50 would be hoarded.
>
> Oddly enough, a couple of times we left the cluster up, but inactive, all
> night. And when we returned to run tests, there would be no creation of NC
> consumers and no hoarding. Everything worked fine.
> Our tip off was that in this scenario, failover from the master to slave
> worked fine, so it was clear that the NC was not needed within a single
> master/slave cluster. [Not sure why this is?]
>
> But without fail, if we restarted the cluster and started using it
> relatively soon, we would see the NC consumers and the hoarding.
>
> Whoosh.
> Hopefully this might help someone else save some time,
> -h
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591p4670940.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message