Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of
 hereen@stormpath.com does not designate 216.139.250.139 as permitted sender)
Date: Mon, 22 Jul 2013 19:10:33 -0700 (PDT)
From: stormtrooper <hereen@stormpath.com>
To: users@activemq.apache.org
Message-ID: <1374545433035-4669591.post@n4.nabble.com>
Subject: Master/Slave: Message Hoarding
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hello all,

Using ActiveMQ 5.8.0 we are seeing a message hoarding issue that only crops
up when we use a Master/Slave configuration.

A single producer sends persistent, never expiring messages to two discrete
Queues in the Broker in spurts. Each queue has a single polling consumer (so
we when we look in the Admin Web Console we see two consumers attached to
each queue: One is our consumer, the other is automatically created for the
Slave), polling for max X messages every Y seconds, no selectors.

When we have just a plain (not a master) instance of the broker +
jdbc/mySQL, everything works as expected: messages are sent to the queues
and delivered in order, 100% throughput.

When we have the Master/Slave setup using JDBC
(http://activemq.apache.org/jdbc-master-slave.html) and mySQL, every spurt
of messages (well under the max # messages/poll) results in only some being
delivered while others are hoarded in the database. 
Example: First spurt sends 10 messages, only 7 get delivered to the
consumer. The other 3 are stored in the database (ACTIVEMQ_MSGS table). When
the next spurt comes, let's say 13 messages, only 8 of those get delivered
and 5 of them are stored in the database; so now the database has 8 total
hoarded messages. Every successive spurt results in only partial sets
delivered and more messages being hoarded. 
This is happening with both queues.

Some possible points of interest:
1) The messages selected for hoarding appear random. They are not
necessarily consecutive, they are not necessarily the first messages, nor
necessarily last messages of each spurt.
2) If we restart the consumers, nothing happens; The hoarded messages remain
in the DB, undelivered.
3) If we restart the broker, the hoarded messages ARE delivered (as expected
since they are persistent and never expire, which ostensibly is the point of
those options).
4) We get the same hoarding results whether our producer/consumer connects
to the master broker directly using tcp, or using the failover protocol with
both master and slave listed, or using the failover protocol with both
master and slave listed and the randomize=false param. (This is not
particularly surprising, just included for completeness).

Our activemq.xml configuration: http://pastebin.com/Y87hZuh5

We've been trying every permutation of configuration we can think of to see
where the culprit may lay (even swapping in KahaDB for mySQL => whole other
can of worms, but then we saw it wasn't supported anyway:
http://activemq.apache.org/kahadb-master-slave.html, a story for another
day). 

Does anyone have any suggestions/pointers of where we should look next? How
to get us back on track? 
It would be greatly appreciated,
-h


--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.