qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helen Kwong <helenkw...@gmail.com>
Subject Re: Limiting the number of concurrent consumers across multiple queues
Date Fri, 17 Jan 2014 21:38:48 GMT
Hi Gordon,

In the tests that we've run, the time it takes to dequeue messages using
selectors seems to increase with the depth of the queue. Since the number
of unprocessed messages can sometimes be quite high (e.g., >200000), if
they are all on the same queue and we use selectors, the dequeue time will
increase by a lot (e.g., 3-4 seconds if we're selecting the 200000th
message), and the performance hit is probably too much for us. Is there a
way to dequeue using selectors quickly from a high-depth queue?

Helen


On Fri, Jan 17, 2014 at 2:40 AM, Gordon Sim <gsim@redhat.com> wrote:

> On 01/16/2014 07:20 PM, Helen Kwong wrote:
>
>> Hi Qpid users / experts,
>>
>> I need to limit the number of consumers concurrently processing messages
>> considered to be in the same group, across multiple queues, and was
>> wondering if anyone has ideas about how to do it. We’re using the Java
>> broker and client, and have multiple queues, each with multiple listeners,
>> each listener’s session listening to multiple queues. Some messages are
>> associated with groups, and for a given group we want at most K listeners
>> processing messages from the group at any given time. The messages are
>> enqueued to multiple queues, and it’s possible for messages from the same
>> group to be in different queues.
>>
>> If messages in the same group can go into only one queue, then the message
>> groups feature will give us what we need (it’d work directly with K = 1
>> and
>> with K > 1 we can tweak the grouping value, e.g., hash it to one of 1 to K
>> and append the number to the grouping value). But since messages
>> considered
>> to be in the same group can be in different queues, the feature is not
>> enough for our case.
>>
>
> Instead of multiple queues, could you have one queue with different
> selectors pulling subsets of the messages?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message