qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helen Kwong <helenkw...@gmail.com>
Subject Re: Limiting the number of concurrent consumers across multiple queues
Date Tue, 21 Jan 2014 01:55:04 GMT
Hi Robbie,

Really appreciate your taking the time to look at this and making the
change to add the extra options / modes for message grouping.

Yes you're right about what I was thinking about with using selectors; I
see what Gordon was suggesting now.

More about our situation: we have many different types of messages, and for
each type we route messages into 10 different queues, based on a certain
property P that can be put into 10 different groups, because we want to
make sure one group's messages taking a long time to process won't starve
another group for a long time. (There are no concurrency requirements
between these groups in general.) For each queue we have multiple
listeners, and each listener's session has listeners to multiple queues --
this is done so that our consumer resources are flexible. A certain type T
of messages may have a concurrency requirement based on a property Q and
limit K (what Q and K are depend on T), the need being that for a given
value x, at most K consumers can be concurrently processing T messages with
Q = x. We don't know what the groups / Q values are ahead of time. And like
I mentioned before, messages with Q = x can have different values of P and
thus can be in different queues in our current setup. No particular Q
values would be more important than others, but the volumes of different
values can be very different.

Since we don't know the values of Q ahead of time, we can't have consumers
selecting on specific Q values. But we could have made messages of the same
type T to all go to 1 queue, and had some consumers selecting for the first
P group, some selecting for the second P group, etc., for all 10 groups of
P. Then we could just use message groups on the single queue. However, this
would actually involve some pretty major changes to how we set up our
queues and clients, and things we have in place to ensure "fairness" at
different levels. For example, we sometimes can have a huge number of
messages with a certain P value (say P = a), and we don't want them
starving other messages within the same P group, so we have a mechanism in
place to route messages with P = a to a separate queue when that happens,
to protect other messages in the same P group. If we are to use selectors
instead of queues, we could achieve something similar by closing the
current consumers and replace each of them with an extra selector criterion
P != a (I'm assuming that's possible), but this requires a lot of changes
to how we do things, so it's not a route we want to take.

As for message ordering, ideally we'd be able to preserve it, but it's not
required. The approach I mentioned would indeed destroy the order, but it's
something we can live with. What we need is just limit the number of
concurrently processing threads for a group (for Q).

So unfortunately at this point, it looks like we probably won't be using
message groups, with the way we have queues and fairness mechanisms set up.
If I'm missing something and you think there's another way, please let me
know. I also hope making the change you made for message groups didn't take
up too much of your time!


On Mon, Jan 20, 2014 at 10:43 AM, Robbie Gemmell

> Hi Helen,
> I think you and Gordon moight be talking about a different use of
> selectors; yours being for the case where you try to pick off particular
> messages from an existing backlog at a specific time, and Gordon's
> suggestion being more around use of selectors on all your long-lived
> consumers to consume the messages as they arrive and remove the need to
> pick out specific messages later.
> Picking individual messages off a queue with new consumers using a selector
> is never likely to be that fast, because as you say it might have to
> evaluate every message first to find a match (assuming there is one). On
> the other hand, when using [a number of] long lived consumers aimed at
> [collectively] consuming all the messages on the queue, the selectors are
> simply evaluated during the regular process of attempting to deliver each
> message to the available consumers. The overhead then moves from checking
> every message specifically until it matches, which may be dependant on the
> queue depth, to the regular competition between particular subscriptions
> for accepting the messages as they are processed, which in some ways isnt.
> I also presume that, as discussed on the other thread, you could also
> currently be seeing a hit in performance due to using 'shared groups' with
> unique'ish keys; more on this later.
> Particular distributions of selectors could be used to ensure that at most
> K consumers could ever process particular messages like X, and thus do so
> at one time, but would additionally mean that if those particular consumers
> were busy processing messages like Y which they can also consume, then it
> could be possible for other consumers to sit idle because their selectors
> indicate they cant process messages like X. It would need to be considered
> as a balance, governed by whatever it is you are looking to achieve by
> limiting the maximum number of concurrent consumers for a given type of
> message.
> The same approach could technically be used across multiple queues, which
> might also use groups (which is a little weird to write, since groups are
> usually used to prevent concurrent delivery), but doing so would add an
> additional element to the 'idle consumers' balancing problem, wherein at a
> given time some sources might have messages of interest and others might
> not.
> The above, and any other specific ideas people have, might all depend what
> your messages, groups, and queues are actually like. E.g. are some higher
> volume than others, are some more important than others, do you know what
> the groups are ahead of time, etc etc.
> If I am reading them correctly it seems like all of the original options in
> this thread allow for a message to go round in circles between the various
> queues, potentially forever. There also seems to be scope for interesting
> ordering effects from re-enqueing messages. Given that ordering is another
> key reason for using message grouping, what are your actual ordering
> requirements?
> On a related-but-not note, see the other thread for further discussion
> around improvements for queues with 'shared groups'.
> Robbie
> On 17 January 2014 21:38, Helen Kwong <helenkwong@gmail.com> wrote:
> > Hi Gordon,
> >
> > In the tests that we've run, the time it takes to dequeue messages using
> > selectors seems to increase with the depth of the queue. Since the number
> > of unprocessed messages can sometimes be quite high (e.g., >200000), if
> > they are all on the same queue and we use selectors, the dequeue time
> will
> > increase by a lot (e.g., 3-4 seconds if we're selecting the 200000th
> > message), and the performance hit is probably too much for us. Is there a
> > way to dequeue using selectors quickly from a high-depth queue?
> >
> > Helen
> >
> >
> > On Fri, Jan 17, 2014 at 2:40 AM, Gordon Sim <gsim@redhat.com> wrote:
> >
> > > On 01/16/2014 07:20 PM, Helen Kwong wrote:
> > >
> > >> Hi Qpid users / experts,
> > >>
> > >> I need to limit the number of consumers concurrently processing
> messages
> > >> considered to be in the same group, across multiple queues, and was
> > >> wondering if anyone has ideas about how to do it. We’re using the Java
> > >> broker and client, and have multiple queues, each with multiple
> > listeners,
> > >> each listener’s session listening to multiple queues. Some messages
> are
> > >> associated with groups, and for a given group we want at most K
> > listeners
> > >> processing messages from the group at any given time. The messages are
> > >> enqueued to multiple queues, and it’s possible for messages from the
> > same
> > >> group to be in different queues.
> > >>
> > >> If messages in the same group can go into only one queue, then the
> > message
> > >> groups feature will give us what we need (it’d work directly with K =
> 1
> > >> and
> > >> with K > 1 we can tweak the grouping value, e.g., hash it to one of
> > to K
> > >> and append the number to the grouping value). But since messages
> > >> considered
> > >> to be in the same group can be in different queues, the feature is not
> > >> enough for our case.
> > >>
> > >
> > > Instead of multiple queues, could you have one queue with different
> > > selectors pulling subsets of the messages?
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message