Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of christian.posta@gmail.com
 designates 209.85.215.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAtvD4WK4w6OtV7wZ7Ht9DycL2HRXGx+CUU37UhTKGtvD+a6PA@mail.gmail.com>
References: <1374545433035-4669591.post@n4.nabble.com>
	<CAAtvD4WK4w6OtV7wZ7Ht9DycL2HRXGx+CUU37UhTKGtvD+a6PA@mail.gmail.com>
Date: Wed, 28 Aug 2013 16:48:54 -0700
Message-ID: 
 <CAKS7T5MAeYzqh3-zMhG8o+13zAQaMrRh9FZwhjrqOtXZ_6iHVg@mail.gmail.com>
Subject: Re: Master/Slave: Message Hoarding
From: Christian Posta <christian.posta@gmail.com>
To: "users@activemq.apache.org" <users@activemq.apache.org>
Content-Type: multipart/alternative; boundary=089e0158b878d758a404e50aa166

--089e0158b878d758a404e50aa166
Content-Type: text/plain; charset=ISO-8859-1

That's quite odd.
Can you reproduce this on KahaDB? ie, kahadb is supported for shared master
slave (http://activemq.apache.org/shared-file-system-master-slave.html) and
it might be easier to debug (without the dependency on the DB) if you can
recreate with KahaDB...

What do the broker stats say as far as dispatched, inflight, enqueue,
dequeue?


On Wed, Aug 28, 2013 at 2:10 PM, Les Hazlewood <lhazlewood@apache.org>wrote:

> bump.  Could someone take a peek at this again and see if they have any
> ideas?
>
> (we tried prefetch size = 0 on the client connection URL, and that didn't
> change anything)
>
> Thanks!
>
> Les
>
> On Mon, Jul 22, 2013 at 7:10 PM, stormtrooper <hereen@stormpath.com>
> wrote:
>
> > Hello all,
> >
> > Using ActiveMQ 5.8.0 we are seeing a message hoarding issue that only
> crops
> > up when we use a Master/Slave configuration.
> >
> > A single producer sends persistent, never expiring messages to two
> discrete
> > Queues in the Broker in spurts. Each queue has a single polling consumer
> > (so
> > we when we look in the Admin Web Console we see two consumers attached to
> > each queue: One is our consumer, the other is automatically created for
> the
> > Slave), polling for max X messages every Y seconds, no selectors.
> >
> > When we have just a plain (not a master) instance of the broker +
> > jdbc/mySQL, everything works as expected: messages are sent to the queues
> > and delivered in order, 100% throughput.
> >
> > When we have the Master/Slave setup using JDBC
> > (http://activemq.apache.org/jdbc-master-slave.html) and mySQL, every
> spurt
> > of messages (well under the max # messages/poll) results in only some
> being
> > delivered while others are hoarded in the database.
> > Example: First spurt sends 10 messages, only 7 get delivered to the
> > consumer. The other 3 are stored in the database (ACTIVEMQ_MSGS table).
> > When
> > the next spurt comes, let's say 13 messages, only 8 of those get
> delivered
> > and 5 of them are stored in the database; so now the database has 8 total
> > hoarded messages. Every successive spurt results in only partial sets
> > delivered and more messages being hoarded.
> > This is happening with both queues.
> >
> > Some possible points of interest:
> > 1) The messages selected for hoarding appear random. They are not
> > necessarily consecutive, they are not necessarily the first messages, nor
> > necessarily last messages of each spurt.
> > 2) If we restart the consumers, nothing happens; The hoarded messages
> > remain
> > in the DB, undelivered.
> > 3) If we restart the broker, the hoarded messages ARE delivered (as
> > expected
> > since they are persistent and never expire, which ostensibly is the point
> > of
> > those options).
> > 4) We get the same hoarding results whether our producer/consumer
> connects
> > to the master broker directly using tcp, or using the failover protocol
> > with
> > both master and slave listed, or using the failover protocol with both
> > master and slave listed and the randomize=false param. (This is not
> > particularly surprising, just included for completeness).
> >
> > Our activemq.xml configuration: http://pastebin.com/Y87hZuh5
> >
> > We've been trying every permutation of configuration we can think of to
> see
> > where the culprit may lay (even swapping in KahaDB for mySQL => whole
> other
> > can of worms, but then we saw it wasn't supported anyway:
> > http://activemq.apache.org/kahadb-master-slave.html, a story for another
> > day).
> >
> > Does anyone have any suggestions/pointers of where we should look next?
> How
> > to get us back on track?
> > It would be greatly appreciated,
> > -h
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

--089e0158b878d758a404e50aa166--