activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Les Hazlewood <lhazlew...@apache.org>
Subject Re: Master/Slave: Message Hoarding
Date Wed, 28 Aug 2013 21:10:59 GMT
bump.  Could someone take a peek at this again and see if they have any
ideas?

(we tried prefetch size = 0 on the client connection URL, and that didn't
change anything)

Thanks!

Les

On Mon, Jul 22, 2013 at 7:10 PM, stormtrooper <hereen@stormpath.com> wrote:

> Hello all,
>
> Using ActiveMQ 5.8.0 we are seeing a message hoarding issue that only crops
> up when we use a Master/Slave configuration.
>
> A single producer sends persistent, never expiring messages to two discrete
> Queues in the Broker in spurts. Each queue has a single polling consumer
> (so
> we when we look in the Admin Web Console we see two consumers attached to
> each queue: One is our consumer, the other is automatically created for the
> Slave), polling for max X messages every Y seconds, no selectors.
>
> When we have just a plain (not a master) instance of the broker +
> jdbc/mySQL, everything works as expected: messages are sent to the queues
> and delivered in order, 100% throughput.
>
> When we have the Master/Slave setup using JDBC
> (http://activemq.apache.org/jdbc-master-slave.html) and mySQL, every spurt
> of messages (well under the max # messages/poll) results in only some being
> delivered while others are hoarded in the database.
> Example: First spurt sends 10 messages, only 7 get delivered to the
> consumer. The other 3 are stored in the database (ACTIVEMQ_MSGS table).
> When
> the next spurt comes, let's say 13 messages, only 8 of those get delivered
> and 5 of them are stored in the database; so now the database has 8 total
> hoarded messages. Every successive spurt results in only partial sets
> delivered and more messages being hoarded.
> This is happening with both queues.
>
> Some possible points of interest:
> 1) The messages selected for hoarding appear random. They are not
> necessarily consecutive, they are not necessarily the first messages, nor
> necessarily last messages of each spurt.
> 2) If we restart the consumers, nothing happens; The hoarded messages
> remain
> in the DB, undelivered.
> 3) If we restart the broker, the hoarded messages ARE delivered (as
> expected
> since they are persistent and never expire, which ostensibly is the point
> of
> those options).
> 4) We get the same hoarding results whether our producer/consumer connects
> to the master broker directly using tcp, or using the failover protocol
> with
> both master and slave listed, or using the failover protocol with both
> master and slave listed and the randomize=false param. (This is not
> particularly surprising, just included for completeness).
>
> Our activemq.xml configuration: http://pastebin.com/Y87hZuh5
>
> We've been trying every permutation of configuration we can think of to see
> where the culprit may lay (even swapping in KahaDB for mySQL => whole other
> can of worms, but then we saw it wasn't supported anyway:
> http://activemq.apache.org/kahadb-master-slave.html, a story for another
> day).
>
> Does anyone have any suggestions/pointers of where we should look next? How
> to get us back on track?
> It would be greatly appreciated,
> -h
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message