qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Godfrey <rob.j.godf...@gmail.com>
Subject Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Date Fri, 29 Sep 2017 10:47:01 GMT
On 29 September 2017 at 10:35, Adel Boutros <Adelboutros@live.com> wrote:

> Hello Rob,
>
>
> I would like to give my opinion on this.
>
>
> In our current use cases, we are configuring the brokers dynamically using
> the REST API. We would like to have this possibility in the use case of
> Olivier.


> Also, having to restart the virtual host is damaging the High Availability
> of the messaging system. This is because while the Virtual Host is
> restarting, no queues are available and the messages are inaccessible. So I
> was wondering if restarting the virtual is a must or it could be fixed in a
> Jira story?
>
>

Sorry - my wording wasn't very clear.  If you set this when you first
create the queue then you don't need to restart the vhost... however if you
have an existing queue that you want to change, then the effect won't be
seen until the vhost is restarted (basically the queue sets itself up to be
"group aware" or "not group aware" when it starts up, it can't change while
it is running).  I don't think this is really an issue for you in that when
you create your queues with the REST API you just need to set this
attribute.


>
> Regarding the 2nd point of multiple brokers and dispatch router, if all
> brokers have the same queues configured with the appropriate grouping
> config, would that really break the feature?
>
> In our current use cases, all components have the same configuration all
> the time (All brokers have same queues and same for the dispatch router).
>
> So I would imagine if a consumer wants to consume a message with the
> correct group, the dispatch router will propagate this header to any
> available broker and will only get the corresponding messages.
>
>
>
The way grouping works is that the first time a message with a particular
group id comes along, the broker will assign that group to a particular
consumer.  Each broker will do this independently.  So if you have two
brokers B1 and B2; and two consumers C1 and C2  and message M of group A
arrives on B1 whereas message N of group A arrives on B2; then B1 may
decide to associate group A with C1 whereas broker B2 decides to associate
group A with consumer C2.  So message groups with multiple brokers will not
work behind a router *unless* the router is enhanced so that it is aware of
message grouping (so all messages of the same group are sent to the same
broker) or the brokers somehow share information about the group ->
consumer assignment.

-- Rob

What do you think?
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Rob Godfrey <rob.j.godfrey@gmail.com>
> Sent: Friday, September 29, 2017 9:14:52 AM
> To: users@qpid.apache.org
> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>
> On 29 September 2017 at 01:09, Olivier Mallassi <
> olivier.mallassi@gmail.com>
> wrote:
>
> > Gentleman,
> >
> > I will need your help.
> >
> > I have a use case where I would like to guarantee "consumer affinity",
> > which is usually implemented using the JMSXGroupID (actually, I am sure
> it
> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
> >
> > My test does the "classical" case:
> > for (i ... i < 100.)
> >   Message textMsg = new TextM....
> >    textMsg.setStringProperty("JMSXGroupID", "groupA")
> >   MessageProducer.send(textMsg);
> >
> > and I start two consumers (assuming only one would work).
> >
> > What I observe is that both consumers are receiving messages, acting as
> > competing consumers. I also tried added the qpid.group_header_key
> property
> > to the queue but it does not change anything.
> >
> >
> > My setup is
> > - java broker 6.0.4 & 6.1.4
> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
> >
> > Is this the expected behaviour? Any ideas?
> >
> >
> This is a bug in the AMQP 1.0 behaviour of the broker.  The Message
> Grouping functionality was originally built around the AMQP 0-x protocols
> and is looking for a message group in the application headers of the
> message.  In AMQP 1.0 there is a dedicated property for this and the JMS
> client is (correctly from an AMQP 1.0 point of view) placing the group id
> there.
>
> As a work around instead of using JMSXGroupID you could use any other (non
> JMSX prefixed) header, e.g.:
>
>  textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>
> Note that on the broker you need to configure the queue so it knows which
> header to use; this is stored in the messageGroupKey property of the queue
> (to "qpidBugWorkaround" in this case) .  Note that after setting this
> property you will need to restart the virtual host before the change takes
> effect.
>
>
> >
> > Complementary question: let's assume now, that there is the DR between
> the
> > broker & the consumers. Does the DR "propagate" or support "grouping
> > messages"? (I assume it is depending if you route the link or messages)
> >
>
> With a single broker and link routing, yes grouping messages should
> continue to work.
>
> If there are multiple brokers sharding the queue then it would not work -
> to support this the router would need to be changed to recognize grouping
> and send all messages with the same group id to the same broker waypoint
> (also, in this case, you would need to use message routing for inbound
> messages and link routing for consuming messages).  We have talked about
> support for behaviour like this before, but there is nothing implemented
> yet as far as I know.
>
> -- Rob
>
>
> >
> >
> > thanks for your help
> >
> > Regards.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message