qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Godfrey <rob.j.godf...@gmail.com>
Subject Re: Single consumer across multiple queues
Date Tue, 20 Jan 2015 18:32:24 GMT
Hi Helen,

apologies for taking so long to respond

On 6 January 2015 at 02:12, Helen Kwong <helenkwong@gmail.com> wrote:

> Hi Rob,
>
> I finally got back to testing multiple queues on a consumer again, using
> the changes you added to help with fairness. My broker jvm is running with
> -Dqueue.maxAsynchronousDeliveries=1 for the number of messages delivered
> per time slice by a queue. It has made things more fair than before, though
> still significantly less so than before with single-queue consumers. Two
> fairness tests that we ran:
>
> A) In the test I described before, where a single session listens to 2
> queues starting with 100 messages each, it is now indeed more fair.
> Sometimes there are 3 messages in a row from 1 queue, but the order is
> mostly alternating. When the first queue is drained, on average the other /
> slower queue still has 6-7 messages left. However, this effect can
> accumulate and the number at the end can vary a lot -- I've seen up to 26,
> which seems quite high compared to 100. Also ran this starting with 1000
> messages per queue: on average the slower queue had about 22 messages left
> at the end, and I've seen up to 65 messages left.
>
> B) We also ran tests with multiple connections / consumers listening to the
> same set of queues. More specifically, we have 10 connections, each with 1
> consumer listening to 100 queues, each queue starting with 500 messages. We
> look at the order in which messages are processed, and track what the
> largest difference in the number of remaining messages between any 2 queues
> at any given point is. Before, in the analogous test where each connection
> has 100 single-queue consumers for the 100 queues, the largest difference
> at any point is around 20. Now with multi-queue consumers, it's much
> higher, around 140-170.
>
> Our questions / concerns:
>
> 1. One thing I'm not sure about is whether it's possible that 1 queue can
> be favored over another queue. With this 1 consumer / 2 queues test, I use
> the same 2 queues for every test run, one called BAREBONEQ0 and the other
> BAREBONEQ1. What I saw intially when running the test was that BAREBONEQ0
> was almost always the slower queue, over maybe 20 or so test runs. But
> later I ran more test runs, and I don't see a pattern to who finishes last
> anymore. It could be just by chance that I saw this in the beginning, but
> just to be sure I'm not missing anything, can one queue end up being
> favored over another at all?
>

There shouldn't be any favouring - it will come down to thread scheduling
in the JVM at this point.  The queues are "notified" of consumer credit in
a round robin order, with the "first" to be notified changed each time.
The JVM or operating system may simply favour a thread that is already
running.


>
> 2. With the second test, I'd often see a chunk of messages from the same
> queue given to different consumers around the same time. E.g., might see 10
> messages from one queue being processed consecutively, by the 10 different
> consumers. 5 to 7 messages in a row from the same queue is common. Why does
> this happen even though I have maxAsynchronousDeliveries configured to 1?
> You mentioned a queue pushes messages to a consumer when it's notified that
> the consumer has room for messages -- what does this mean in terms of
> multiple consumers/connections, and does this mean whenever a queue gets
> notified, it may push 1 message to each consumer?
>
>
So, when I ran a similar test on my laptop I saw no more than about 3 or 4
in a row from the same queue (before I changed how the round-robin
notification was done it was a lot worse)... again though this comes down
to thread scheduling... I was running everything (client / broker) on my 4
core laptop... I imagine using a different machine would potentially see
different results.


> 3. Overall does the behavior from these tests seem expected to you? Is this
> as fair as we can be with multi-queue consumers, before the new threading
> model allows consumers to pull from queues?
>
>
I made it as fair as I could with the current threading model...  Keith and
I are reworking the IO / threading model at the moment though I still can't
really give an ETA for when that work will be finished - certainly not for
the release that will be cut shortly (15.02 or whatever we decide to call
it), possibly for the release after that though,

Cheers,
Rob


> Thanks a lot!
> Helen
>
> On Fri, Oct 31, 2014 at 11:18 AM, Helen Kwong <helenkwong@gmail.com>
> wrote:
>
> > Thanks a lot for the changes and the explanation! Will try this
> workaround
> > for now. When the new queue threading model is available please let me
> know.
> >
> > On Fri, Oct 31, 2014 at 6:38 AM, Rob Godfrey <rob.j.godfrey@gmail.com>
> > wrote:
> >
> >> Hi Helen,
> >>
> >> so the fundamental issue here is that currently inside the broker queues
> >> "push" messages to consumers rather than consumers pulling from queues.
> >> When a queue is informed that a consumer has room to accept more
> messages,
> >> it immediately tries to start pump messages to that consumer.  In this
> >> case
> >> there are two queues trying to pump messages to one consumer, and the
> >> first
> >> one to get notified will pump in up to 80 messages before yielding the
> >> thread.  At this point the second queue might be able to jump in and
> pump
> >> 80 messages, or the first queue may actually get the lock again.
> >>
> >> Keith and I are planning on reworking the underlying queue threading
> model
> >> soon to change this around so that the consumers/connections pull from
> the
> >> queues, at which point real fairness will be easier to implement.  In
> the
> >> meantime I've made a couple of changes (in QPID-6204, revision
> >> https://svn.apache.org/r1635768) which help by a) allowing the number
> of
> >> messages delivered in one "time-slice" to be configured on a per queue
> >> basis (i.e. removing the hardcoding of 80) and b) alternating which of
> the
> >> queues is notified first when the consumer has available credit.
> >>
> >> After this change, by setting the time slice to one delivery
> >> (-Dqueue.maxAsynchronousDeliveries=1) I saw reasonably fair behaviour
> >> (runs
> >> of no more the 3 messages for the same queue).  Note that reducing the
> >> timeslice probably has some negative performance impact, but you can
> >> configure this value on a per queue basis (rather than setting it as a
> >> system property you can set on each individual queue as a context
> >> variable).
> >>
> >> When Keith gets done with what he's currently working on we'll try to
> >> update you on our work on changing the queue threading model around.
> >>
> >> Cheers,
> >> Rob
> >>
> >>
> >> On 31 October 2014 01:19, Helen Kwong <helenkwong@gmail.com> wrote:
> >>
> >> > Hi Rob,
> >> >
> >> > I got around to doing some testing on the multi-queue consumer feature
> >> you
> >> > added. So far things have looked good mostly, but there is one issue
> >> I've
> >> > run into and would like your help on.
> >> >
> >> > When we had single-queue consumers, we had fair allocation behavior
> >> across
> >> > queues, in the sense that if I have 2 queues A and B, each with 100
> >> > messages, and one JMS session having a listening consumer on queue A
> >> and a
> >> > listener on queue B, the message processing order will be round robin
> --
> >> > i.e., M_A_1 (representing the first message on queue A), M_B_1, M_A_2,
> >> > M_B_2, M_A_3, M_B_3, and so on. But now, if I run the same test with
> the
> >> > session having a single multi-queue consumer on A and B instead, the
> >> order
> >> > is, roughly, first the 100 messages on A, followed by the messages on
> B
> >> > (only a few B messages are processed before all A messages are done).
> I
> >> > enqueue the messages in the round robin order. I've also tried this
> with
> >> > synchronous receives from both queues instead of asynchronous
> listening,
> >> > and I see similar behavior.
> >> >
> >> > Is there any way we can mimic the "fair" behavior of single-queue
> >> consumers
> >> > with multi-queue consumers?
> >> >
> >> > Thanks,
> >> > Helen
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message