qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helen Kwong <helenkw...@gmail.com>
Subject Re: Single consumer across multiple queues
Date Fri, 23 Jan 2015 19:30:44 GMT
Hi Rob,

Thanks for looking at this. We'll probably continue using single-queue
consumers for now then, since the fair behavior is important to us, and try
using multiple-queue consumers again when the new threading model is
available.

Thanks,
Helen

On Tue, Jan 20, 2015 at 10:32 AM, Rob Godfrey <rob.j.godfrey@gmail.com>
wrote:

> Hi Helen,
>
> apologies for taking so long to respond
>
> On 6 January 2015 at 02:12, Helen Kwong <helenkwong@gmail.com> wrote:
>
> > Hi Rob,
> >
> > I finally got back to testing multiple queues on a consumer again, using
> > the changes you added to help with fairness. My broker jvm is running
> with
> > -Dqueue.maxAsynchronousDeliveries=1 for the number of messages delivered
> > per time slice by a queue. It has made things more fair than before,
> though
> > still significantly less so than before with single-queue consumers. Two
> > fairness tests that we ran:
> >
> > A) In the test I described before, where a single session listens to 2
> > queues starting with 100 messages each, it is now indeed more fair.
> > Sometimes there are 3 messages in a row from 1 queue, but the order is
> > mostly alternating. When the first queue is drained, on average the
> other /
> > slower queue still has 6-7 messages left. However, this effect can
> > accumulate and the number at the end can vary a lot -- I've seen up to
> 26,
> > which seems quite high compared to 100. Also ran this starting with 1000
> > messages per queue: on average the slower queue had about 22 messages
> left
> > at the end, and I've seen up to 65 messages left.
> >
> > B) We also ran tests with multiple connections / consumers listening to
> the
> > same set of queues. More specifically, we have 10 connections, each with
> 1
> > consumer listening to 100 queues, each queue starting with 500 messages.
> We
> > look at the order in which messages are processed, and track what the
> > largest difference in the number of remaining messages between any 2
> queues
> > at any given point is. Before, in the analogous test where each
> connection
> > has 100 single-queue consumers for the 100 queues, the largest difference
> > at any point is around 20. Now with multi-queue consumers, it's much
> > higher, around 140-170.
> >
> > Our questions / concerns:
> >
> > 1. One thing I'm not sure about is whether it's possible that 1 queue can
> > be favored over another queue. With this 1 consumer / 2 queues test, I
> use
> > the same 2 queues for every test run, one called BAREBONEQ0 and the other
> > BAREBONEQ1. What I saw intially when running the test was that BAREBONEQ0
> > was almost always the slower queue, over maybe 20 or so test runs. But
> > later I ran more test runs, and I don't see a pattern to who finishes
> last
> > anymore. It could be just by chance that I saw this in the beginning, but
> > just to be sure I'm not missing anything, can one queue end up being
> > favored over another at all?
> >
>
> There shouldn't be any favouring - it will come down to thread scheduling
> in the JVM at this point.  The queues are "notified" of consumer credit in
> a round robin order, with the "first" to be notified changed each time.
> The JVM or operating system may simply favour a thread that is already
> running.
>
>
> >
> > 2. With the second test, I'd often see a chunk of messages from the same
> > queue given to different consumers around the same time. E.g., might see
> 10
> > messages from one queue being processed consecutively, by the 10
> different
> > consumers. 5 to 7 messages in a row from the same queue is common. Why
> does
> > this happen even though I have maxAsynchronousDeliveries configured to 1?
> > You mentioned a queue pushes messages to a consumer when it's notified
> that
> > the consumer has room for messages -- what does this mean in terms of
> > multiple consumers/connections, and does this mean whenever a queue gets
> > notified, it may push 1 message to each consumer?
> >
> >
> So, when I ran a similar test on my laptop I saw no more than about 3 or 4
> in a row from the same queue (before I changed how the round-robin
> notification was done it was a lot worse)... again though this comes down
> to thread scheduling... I was running everything (client / broker) on my 4
> core laptop... I imagine using a different machine would potentially see
> different results.
>
>
> > 3. Overall does the behavior from these tests seem expected to you? Is
> this
> > as fair as we can be with multi-queue consumers, before the new threading
> > model allows consumers to pull from queues?
> >
> >
> I made it as fair as I could with the current threading model...  Keith and
> I are reworking the IO / threading model at the moment though I still can't
> really give an ETA for when that work will be finished - certainly not for
> the release that will be cut shortly (15.02 or whatever we decide to call
> it), possibly for the release after that though,
>
> Cheers,
> Rob
>
>
> > Thanks a lot!
> > Helen
> >
> > On Fri, Oct 31, 2014 at 11:18 AM, Helen Kwong <helenkwong@gmail.com>
> > wrote:
> >
> > > Thanks a lot for the changes and the explanation! Will try this
> > workaround
> > > for now. When the new queue threading model is available please let me
> > know.
> > >
> > > On Fri, Oct 31, 2014 at 6:38 AM, Rob Godfrey <rob.j.godfrey@gmail.com>
> > > wrote:
> > >
> > >> Hi Helen,
> > >>
> > >> so the fundamental issue here is that currently inside the broker
> queues
> > >> "push" messages to consumers rather than consumers pulling from
> queues.
> > >> When a queue is informed that a consumer has room to accept more
> > messages,
> > >> it immediately tries to start pump messages to that consumer.  In this
> > >> case
> > >> there are two queues trying to pump messages to one consumer, and the
> > >> first
> > >> one to get notified will pump in up to 80 messages before yielding the
> > >> thread.  At this point the second queue might be able to jump in and
> > pump
> > >> 80 messages, or the first queue may actually get the lock again.
> > >>
> > >> Keith and I are planning on reworking the underlying queue threading
> > model
> > >> soon to change this around so that the consumers/connections pull from
> > the
> > >> queues, at which point real fairness will be easier to implement.  In
> > the
> > >> meantime I've made a couple of changes (in QPID-6204, revision
> > >> https://svn.apache.org/r1635768) which help by a) allowing the number
> > of
> > >> messages delivered in one "time-slice" to be configured on a per queue
> > >> basis (i.e. removing the hardcoding of 80) and b) alternating which of
> > the
> > >> queues is notified first when the consumer has available credit.
> > >>
> > >> After this change, by setting the time slice to one delivery
> > >> (-Dqueue.maxAsynchronousDeliveries=1) I saw reasonably fair behaviour
> > >> (runs
> > >> of no more the 3 messages for the same queue).  Note that reducing the
> > >> timeslice probably has some negative performance impact, but you can
> > >> configure this value on a per queue basis (rather than setting it as a
> > >> system property you can set on each individual queue as a context
> > >> variable).
> > >>
> > >> When Keith gets done with what he's currently working on we'll try to
> > >> update you on our work on changing the queue threading model around.
> > >>
> > >> Cheers,
> > >> Rob
> > >>
> > >>
> > >> On 31 October 2014 01:19, Helen Kwong <helenkwong@gmail.com> wrote:
> > >>
> > >> > Hi Rob,
> > >> >
> > >> > I got around to doing some testing on the multi-queue consumer
> feature
> > >> you
> > >> > added. So far things have looked good mostly, but there is one issue
> > >> I've
> > >> > run into and would like your help on.
> > >> >
> > >> > When we had single-queue consumers, we had fair allocation behavior
> > >> across
> > >> > queues, in the sense that if I have 2 queues A and B, each with 100
> > >> > messages, and one JMS session having a listening consumer on queue
A
> > >> and a
> > >> > listener on queue B, the message processing order will be round
> robin
> > --
> > >> > i.e., M_A_1 (representing the first message on queue A), M_B_1,
> M_A_2,
> > >> > M_B_2, M_A_3, M_B_3, and so on. But now, if I run the same test with
> > the
> > >> > session having a single multi-queue consumer on A and B instead, the
> > >> order
> > >> > is, roughly, first the 100 messages on A, followed by the messages
> on
> > B
> > >> > (only a few B messages are processed before all A messages are
> done).
> > I
> > >> > enqueue the messages in the round robin order. I've also tried this
> > with
> > >> > synchronous receives from both queues instead of asynchronous
> > listening,
> > >> > and I see similar behavior.
> > >> >
> > >> > Is there any way we can mimic the "fair" behavior of single-queue
> > >> consumers
> > >> > with multi-queue consumers?
> > >> >
> > >> > Thanks,
> > >> > Helen
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message