qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helen Kwong <helenkw...@gmail.com>
Subject Re: Single consumer across multiple queues
Date Tue, 06 Jan 2015 01:12:42 GMT
Hi Rob,

I finally got back to testing multiple queues on a consumer again, using
the changes you added to help with fairness. My broker jvm is running with
-Dqueue.maxAsynchronousDeliveries=1 for the number of messages delivered
per time slice by a queue. It has made things more fair than before, though
still significantly less so than before with single-queue consumers. Two
fairness tests that we ran:

A) In the test I described before, where a single session listens to 2
queues starting with 100 messages each, it is now indeed more fair.
Sometimes there are 3 messages in a row from 1 queue, but the order is
mostly alternating. When the first queue is drained, on average the other /
slower queue still has 6-7 messages left. However, this effect can
accumulate and the number at the end can vary a lot -- I've seen up to 26,
which seems quite high compared to 100. Also ran this starting with 1000
messages per queue: on average the slower queue had about 22 messages left
at the end, and I've seen up to 65 messages left.

B) We also ran tests with multiple connections / consumers listening to the
same set of queues. More specifically, we have 10 connections, each with 1
consumer listening to 100 queues, each queue starting with 500 messages. We
look at the order in which messages are processed, and track what the
largest difference in the number of remaining messages between any 2 queues
at any given point is. Before, in the analogous test where each connection
has 100 single-queue consumers for the 100 queues, the largest difference
at any point is around 20. Now with multi-queue consumers, it's much
higher, around 140-170.

Our questions / concerns:

1. One thing I'm not sure about is whether it's possible that 1 queue can
be favored over another queue. With this 1 consumer / 2 queues test, I use
the same 2 queues for every test run, one called BAREBONEQ0 and the other
BAREBONEQ1. What I saw intially when running the test was that BAREBONEQ0
was almost always the slower queue, over maybe 20 or so test runs. But
later I ran more test runs, and I don't see a pattern to who finishes last
anymore. It could be just by chance that I saw this in the beginning, but
just to be sure I'm not missing anything, can one queue end up being
favored over another at all?

2. With the second test, I'd often see a chunk of messages from the same
queue given to different consumers around the same time. E.g., might see 10
messages from one queue being processed consecutively, by the 10 different
consumers. 5 to 7 messages in a row from the same queue is common. Why does
this happen even though I have maxAsynchronousDeliveries configured to 1?
You mentioned a queue pushes messages to a consumer when it's notified that
the consumer has room for messages -- what does this mean in terms of
multiple consumers/connections, and does this mean whenever a queue gets
notified, it may push 1 message to each consumer?

3. Overall does the behavior from these tests seem expected to you? Is this
as fair as we can be with multi-queue consumers, before the new threading
model allows consumers to pull from queues?

Thanks a lot!
Helen

On Fri, Oct 31, 2014 at 11:18 AM, Helen Kwong <helenkwong@gmail.com> wrote:

> Thanks a lot for the changes and the explanation! Will try this workaround
> for now. When the new queue threading model is available please let me know.
>
> On Fri, Oct 31, 2014 at 6:38 AM, Rob Godfrey <rob.j.godfrey@gmail.com>
> wrote:
>
>> Hi Helen,
>>
>> so the fundamental issue here is that currently inside the broker queues
>> "push" messages to consumers rather than consumers pulling from queues.
>> When a queue is informed that a consumer has room to accept more messages,
>> it immediately tries to start pump messages to that consumer.  In this
>> case
>> there are two queues trying to pump messages to one consumer, and the
>> first
>> one to get notified will pump in up to 80 messages before yielding the
>> thread.  At this point the second queue might be able to jump in and pump
>> 80 messages, or the first queue may actually get the lock again.
>>
>> Keith and I are planning on reworking the underlying queue threading model
>> soon to change this around so that the consumers/connections pull from the
>> queues, at which point real fairness will be easier to implement.  In the
>> meantime I've made a couple of changes (in QPID-6204, revision
>> https://svn.apache.org/r1635768) which help by a) allowing the number of
>> messages delivered in one "time-slice" to be configured on a per queue
>> basis (i.e. removing the hardcoding of 80) and b) alternating which of the
>> queues is notified first when the consumer has available credit.
>>
>> After this change, by setting the time slice to one delivery
>> (-Dqueue.maxAsynchronousDeliveries=1) I saw reasonably fair behaviour
>> (runs
>> of no more the 3 messages for the same queue).  Note that reducing the
>> timeslice probably has some negative performance impact, but you can
>> configure this value on a per queue basis (rather than setting it as a
>> system property you can set on each individual queue as a context
>> variable).
>>
>> When Keith gets done with what he's currently working on we'll try to
>> update you on our work on changing the queue threading model around.
>>
>> Cheers,
>> Rob
>>
>>
>> On 31 October 2014 01:19, Helen Kwong <helenkwong@gmail.com> wrote:
>>
>> > Hi Rob,
>> >
>> > I got around to doing some testing on the multi-queue consumer feature
>> you
>> > added. So far things have looked good mostly, but there is one issue
>> I've
>> > run into and would like your help on.
>> >
>> > When we had single-queue consumers, we had fair allocation behavior
>> across
>> > queues, in the sense that if I have 2 queues A and B, each with 100
>> > messages, and one JMS session having a listening consumer on queue A
>> and a
>> > listener on queue B, the message processing order will be round robin --
>> > i.e., M_A_1 (representing the first message on queue A), M_B_1, M_A_2,
>> > M_B_2, M_A_3, M_B_3, and so on. But now, if I run the same test with the
>> > session having a single multi-queue consumer on A and B instead, the
>> order
>> > is, roughly, first the 100 messages on A, followed by the messages on B
>> > (only a few B messages are processed before all A messages are done). I
>> > enqueue the messages in the round robin order. I've also tried this with
>> > synchronous receives from both queues instead of asynchronous listening,
>> > and I see similar behavior.
>> >
>> > Is there any way we can mimic the "fair" behavior of single-queue
>> consumers
>> > with multi-queue consumers?
>> >
>> > Thanks,
>> > Helen
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message