qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Java broker - message grouping in C++ compatibility mode
Date Tue, 14 Jan 2014 02:28:03 GMT
Hi Helen,

The change I mentioned is an important defect fix if you are looking to use
the shared groups, the underlying issue allowed messages to be delivered
concurrently that should not have been (in particular when using a prefetch
of 1, it was significantly less likely to ever occur at >1 prefetch) and so
to ensure your results are valid you should look to pick up the newer code
or backport the change to the old release.

Can I enquire what it is about your use case that would preclude use of a
higher prefetch? Also, you originally seemed to prefer the idea that
messages without a group header value would be not be grouped, so is there
a particular reason you are leaning towards using the shared groups
functionality which can't do that?

A WeakReference might be more viable, but could also end up doing little
good due to when its reference would be required (mostly after the target
entry has been unlinked from the queue structure), a SoftReference might be
another option....it would need looked into it in greater depth some time,
which I am somewhat short on at the moment (hence the late night emails).
That still wouldnt help with the entirely-unique-keys shared groups case
though, which might need a different approach in general than is currently
being taken.

Robbie

On 13 January 2014 20:34, Helen Kwong <helenkwong@gmail.com> wrote:

> Hi Robbie,
>
> I am actually still running version 0.16 of the broker. It will take me a
> little time to set up the trunk code and rebuild and rerun the experiments.
> Do you think your fix will likely make a difference?
>
> For the second case with a long-lived consumer being assigned the group of
> many messages at the head of the queue, I was indeed using a prefetch of 1.
> I ran it again (with version 0.16 still) with a prefetch of 2 as you
> suggested, and the dequeue time of the messages at the end was then not
> affected by the number of unprocessable messages at the beginning of the
> queue, about the same as the other test setups I ran. However, I think
> increasing prefetch to 2 might not work for our use case.
>
> For the first case with unique message groups, your explanation makes sense
> and I think I understand it now. Do you think there is still a way to
> optimize this behavior, so that we don't need to possibly traverse through
> the whole queue whenever a subscription is unassigned from a group? Since
> you mentioned that maintaining a fixed pointer to a queue entry would
> likely lead to memory retention issues, would having a weak reference be a
> possible option?
>
> Thanks,
> Helen
>
>
>
> On Sat, Jan 11, 2014 at 7:58 AM, Robbie Gemmell <robbie.gemmell@gmail.com
> >wrote:
>
> > Hi Helen,
> >
> > Can I check what version of the code you were using? I ask as the latest
> > trunk or 0.26 release branch code is going to be necessary for
> correctness
> > and representative testing of the shared groups functionality, due to the
> > defect fix I previously mentioned making recently.
> > You can find a nightly build of the trunk broker at:
> >
> >
> https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Artefact-Release/lastSuccessfulBuild/artifact/trunk/qpid/java/broker/release/and
> > I would need to build the 0.26 branch as the fix was introduced after
> > the latest RC.
> >
> > In your first case, I think the reason for the difference between the
> > default group and the unique group is also likely to be tied to the
> > 'findEarliestAssignedAvailableEntry' behaviour you mention later in your
> > mail. For the default group case, that next message always going to be a
> > message near the front of the queue. For the unique group, there isnt
> > actually going to be a message which matches, but it looks like it will
> > currently be checking every message to determine that and doing so under
> > the synchronization, and thus probably preventing other deliveries
> > occurring at the time. That isnt a problem in the non-shared case because
> > there isnt a need to synchronise the GroupManager as a whole, and even
> > going beyond that its also highly unlikely it would need to check as many
> > messages before finding a match due to the signifcant difference in how
> > groups become associated with a particular subscription in the non-shared
> > case.
> >
> > In your second case, your explanation seems likely and I think this case
> > really reduces to just being a variant of the above behaviour. The
> > particular issue is that one could argue it shouldnt need to be doing the
> > 'findEarliestAssignedAvailableEntry' task all that often in this case if
> > you have a long-lived consumer, and so your mention of this makes me
> think
> > you are using a prefetch of 1. Using a prefetch of 1 currently means that
> > the delivery state associated with the shared group effectively becomes
> > empty after each message, because messages are only fully added to the
> > group when they become acquired by a particular subscription, and they
> cant
> > be acquired until the previous message is consumed (or perhaps slightly
> > confusingly, explicitly not-consumed). If so, I expect it could be very
> > interesting to run this case again with a prefetch of 2 or more. The
> > obvious tradeoff with increasing prefetch is that a particular consumer
> > could then be assigned up to <prefetch> groups at a given point, though
> > likely not in your test case due to the large contiguous blocks of
> messages
> > for each group.
> >
> > I'm not sure that the suggestion to track the first message in the group
> > would really work currently, due to the way the underlying queue data
> > structure works. Maintaining a fixed pointer into it like that is likely
> to
> > lead to some undesirable memory retention issues, based on a related but
> > far simpler case I fixed previously in a similar structure elsewhere in
> the
> > broker. Looking at the way messages become assigned to a group in the
> > shared group case may be a more viable path to handling your second case
> > more gracefully. The unique groups from you first case would still need
> > something different though, as neither of these routes would really help
> > there.
> >
> > Robbie
> >
> > On 11 January 2014 01:03, Helen Kwong <helenkwong@gmail.com> wrote:
> >
> > > Hi Robbie,
> > >
> > > I did some more testing to see whether message grouping will work for
> us,
> > > and compared the dequeue performance of a queue using message grouping
> in
> > > default Java mode, a queue using C++ mode, and a queue not using
> message
> > > grouping. I found that when I use C++ mode, the performance can be much
> > > worse than in other comparable setups, and was wondering if you could
> > help
> > > me understand why.
> > >
> > > 1.  In one test, I have multiple listeners to a queue, enqueue 100000
> > > messages to it, and see how many messages are processed in 5 minutes. I
> > > have these different setups:
> > >
> > > - C++ mode queue with each message having a unique identifier
> > >
> > > - C++ mode queue with all messages having no grouping identifier (so
> all
> > > belong to the default group)
> > >
> > > - default mode queue with each message having a unique grouping
> > identifier
> > >
> > > - default mode queue with all messages having no grouping identifier
> > >
> > > - default mode queue with all messages having the same grouping
> > identifier
> > >
> > > - regular queue without a group header key configured
> > >
> > > All setups except for the first had about 35K - 39K messages processed,
> > but
> > > for the first setup, there were under 8000 messages processed. What
> could
> > > explain this big difference? I’ve looked at the two grouping modes’
> > > implementations of MessageGroupManager and see that C++ mode uses
> > > synchronized methods rather than a ConcurrentHashMap as in default
> mode,
> > so
> > > I’d guess there might be more contention because of that, but at the
> same
> > > time I can’t see why, in C++ mode, having a unique identifier for each
> > > message results in throughput that is so much worse than having all
> > > messages in the default group.
> > >
> > >
> > > 2. I also wanted to see the impact of having many messages at the head
> of
> > > the queue that a listener can’t process because the messages belong to
> a
> > > group assigned to another consumer. E.g., have 10000 messages of group
> A,
> > > followed by 1000 messages of group B, and listener 1 is holding on to
> the
> > > first A message for a long time -- see how long it will take listener 2
> > to
> > > process all the B messages. In this case C++ mode has performance that
> > > degrades as the number of unprocessable group A messages at the front
> of
> > > the queue increases, whereas default mode's performance is unaffected,
> > > about the same as processing 1000 messages on a regular queue.
> > >
> > > My rough guess from looking at DefinedGroupMessageGroupManager is that
> > > whenever listener 2 is done with a group B message, the state change
> > > listener triggers the Group.subtract() to reset pointers for other
> > > subscriptions and consequently findEarliestAssignedAvailableEntry().
> This
> > > then has to iterate through all the group A messages before it finds
> the
> > B
> > > message. Do you think this is the reason for the results I see?
> > >
> > > If so, is the idea here that other subscriptions of the queue could
> have
> > > skipped over the messages of a group while the group was assigned to
> some
> > > subscription S, so we need to tell them to set their pointers back? If
> > that
> > > is indeed the idea, would it be possible to optimize it such that when
> a
> > > group A is assigned to S and S gets its first message of the group, we
> > > store what that first A message / queue entry is. Then when S is done
> > with
> > > the last A message, we can tell other subscriptions to go back to that
> > > first entry, without having to iterate through the queue?
> > >
> > > Thanks a lot for your help!
> > >
> > > Helen
> > >
> > >
> > > On Tue, Jan 7, 2014 at 8:46 PM, Robbie Gemmell <
> robbie.gemmell@gmail.com
> > > >wrote:
> > >
> > > > ...and just to be super clear, though I think it it is mentioned
> > > correctly
> > > > in the docs this time, the 'default group' concept does not apply in
> > the
> > > > regular / 'non shared' grouping mode. Messages that dont specify a
> > group
> > > > key value in that mode are simply not grouped in any way.
> > > >
> > > > On 8 January 2014 04:41, Robbie Gemmell <robbie.gemmell@gmail.com>
> > > wrote:
> > > >
> > > > >
> > > > > On 8 January 2014 04:33, Helen Kwong <helenkwong@gmail.com>
wrote:
> > > > >
> > > > >> Oh I see, I thought what you meant was that I could only alter
the
> > > > default
> > > > >> group in shared-groups mode starting with 0.24.
> > > > >
> > > > >
> > > > > No, I just missed that you said 0.16 and assumed 0.24 was the
> version
> > > you
> > > > > were using . You could always change it, just in more limited ways
> in
> > > > > earlier releases.
> > > > >
> > > > > To make sure I'm
> > > > >> understanding this correctly -- changing the the default message
> > group
> > > > >> name
> > > > >> to something else in C++ mode won't change the serial processing
> > > > behavior
> > > > >> I
> > > > >> saw, right?
> > > > >
> > > > >
> > > > > Correct
> > > > >
> > > > >
> > > > >> Messages without a group identifier will still be considered
to
> > > > >> be in a group -- rather than no group -- and they cannot be
> > processed
> > > by
> > > > >> multiple consumers concurrently?
> > > > >>
> > > > >>
> > > > > Yes. In the C++/shared-groups mode every message is considered to
> be
> > > in a
> > > > > group, it is just a case of whether the message specifies that
> group
> > or
> > > > > instead gets put into the default group.
> > > > >
> > > > >
> > > > >
> > > > >> Thanks,
> > > > >> Helen
> > > > >>
> > > > >>
> > > > >> On Tue, Jan 7, 2014 at 8:22 PM, Robbie Gemmell <
> > > > robbie.gemmell@gmail.com
> > > > >> >wrote:
> > > > >>
> > > > >> > I just noticed you said you were using 0.16, somehow glossed
> over
> > it
> > > > >> > originally and only noticed the 0.24 in the doc URL (its
many
> > hours
> > > > past
> > > > >> > time I was asleep, I might be getting tired).
> > > > >> >
> > > > >> > Realising that, I should add that prior to 0.22 the only
way to
> > > alter
> > > > >> the
> > > > >> > default group in the shared-groups mode from 'qpid.no-group'
to
> > > > >> something
> > > > >> > else would have been via the 'qpid.default-message-group'
queue
> > > > declare
> > > > >> > argument when using an AMQP client to create the queue
> originally,
> > > and
> > > > >> for
> > > > >> > 0.22 itself only that and the system property approach I
> mentioned
> > > > would
> > > > >> > work.
> > > > >> >
> > > > >> > Robbie
> > > > >> >
> > > > >> > On 8 January 2014 04:03, Helen Kwong <helenkwong@gmail.com>
> > wrote:
> > > > >> >
> > > > >> > > Hi Robbie,
> > > > >> > >
> > > > >> > > I see. Thanks for the quick response and explanation!
> > > > >> > >
> > > > >> > > Helen
> > > > >> > >
> > > > >> > >
> > > > >> > > On Tue, Jan 7, 2014 at 7:43 PM, Robbie Gemmell <
> > > > >> robbie.gemmell@gmail.com
> > > > >> > > >wrote:
> > > > >> > >
> > > > >> > > > Hi Helen,
> > > > >> > > >
> > > > >> > > > The short answer to your question is that it is
the
> > > documentation
> > > > >> which
> > > > >> > > is
> > > > >> > > > incorrect, and the behaviour you are seeing is
expected.
> > > > >> > > >
> > > > >> > > > The long answer is, when that documentation was
composed a
> > > segment
> > > > >> was
> > > > >> > > > missed out indicating this, and needs to be added
to the
> docs.
> > > The
> > > > >> > > > behaviour listed for when no group is specified
is only true
> > of
> > > > the
> > > > >> > > > 'non-shared' groups supported by the Java broker,
in the
> > > > C++/shared
> > > > >> > group
> > > > >> > > > mode any messages recieved without an explicit
group value
> are
> > > all
> > > > >> > > assigned
> > > > >> > > > to a default group of 'qpid.no-group'. This is
as per the
> > > > behaviour
> > > > >> of
> > > > >> > > the
> > > > >> > > > C++ broker itself, which is explained in the C++
broker docs
> > at
> > > > the
> > > > >> end
> > > > >> > > of
> > > > >> > > > the following page
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://qpid.apache.org/releases/qpid-0.24/cpp-broker/book/Using-message-groups.html
> > > > >> > > > .
> > > > >> > > > For the 0.24 Java broker, this default shared
group can be
> > > changed
> > > > >> > > > broker-wide using the Java system property
> > > > >> > > > 'qpid.broker_default-shared-message-group', or
can be
> > overriden
> > > > for
> > > > >> an
> > > > >> > > > individual queue during creation programatically
via AMQP
> > > clients
> > > > or
> > > > >> > the
> > > > >> > > > management interfaces through use of the argument
> > > > >> > > > 'qpid.default-message-group' or 'messageGroupDefaultGroup'.
> > > > >> > > >
> > > > >> > > > I coincidentally happened to have fixed a defect
with the
> > shared
> > > > >> groups
> > > > >> > > > functionality last night on trunk. Its not yet
included in
> the
> > > > >> imminent
> > > > >> > > > 0.26 release, though I am about to request whether
that is
> > > > possible.
> > > > >> > > > https://issues.apache.org/jira/browse/QPID-5450
> > > > >> > > >
> > > > >> > > > Robbie
> > > > >> > > >
> > > > >> > > > On 8 January 2014 02:43, Helen Kwong <helenkwong@gmail.com>
> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi,
> > > > >> > > > >
> > > > >> > > > > I use the Java broker and client, version
0.16, and am
> > > > considering
> > > > >> > > using
> > > > >> > > > > the message grouping feature (
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://qpid.apache.org/releases/qpid-0.24/java-broker/book/Java-Broker-Queues.html#Java-Broker-Queues-OtherTypes-Message-Grouping
> > > > >> > > > > ).
> > > > >> > > > > From testing I've done, there seems to be
a bug with the
> C++
> > > > >> > > > compatibility
> > > > >> > > > > model, and I'm wondering if this is a known
issue.
> > > Specifically,
> > > > >> in
> > > > >> > my
> > > > >> > > > test
> > > > >> > > > > I have a queue configured to use a group
header field with
> > > > >> > > > > "qpid.group_header_key" and C++ mode with
> > > > "qpid.shared_msg_group",
> > > > >> > and
> > > > >> > > > have
> > > > >> > > > > multiple listeners to the queue. Each listener
will sleep
> > for
> > > a
> > > > >> short
> > > > >> > > > > amount of time when it receives a message
before
> returning.
> > I
> > > > then
> > > > >> > > > enqueue
> > > > >> > > > > 10 messages that do not have a value in the
group header
> > field
> > > > to
> > > > >> the
> > > > >> > > > > queue. Since the doc says that messages without
a value in
> > the
> > > > >> > grouping
> > > > >> > > > > header will be delivered to any available
consumer, the
> > > > behavior I
> > > > >> > > expect
> > > > >> > > > > is that the messages will be processed in
parallel, i.e.,
> > when
> > > > >> > > listener 1
> > > > >> > > > > is holding on to a message and sleeping,
listener 2 can
> > > receive
> > > > >> > another
> > > > >> > > > > message from the queue. But what I see is
that the
> messages
> > > are
> > > > >> > > processed
> > > > >> > > > > serially -- message 2 won't be received by
some thread
> until
> > > > >> message
> > > > >> > 1
> > > > >> > > is
> > > > >> > > > > done. When I use the default mode instead
of C++ mode,
> then
> > I
> > > > get
> > > > >> the
> > > > >> > > > > parallel processing behavior.
> > > > >> > > > >
> > > > >> > > > > Is this is a known bug, and is there a fix
for it already?
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Helen
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message