qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Date Sat, 30 Sep 2017 15:08:04 GMT
On 30 September 2017 at 09:23, Olivier Mallassi
<olivier.mallassi@gmail.com> wrote:
> Thanks all, it helps
> I saw this https://issues.apache.org/jira/browse/QPID-5165 while running
> the "test" but did not get the whole picture.
>

The interesting bits here are all with the servers as already
discusse, but just to be clear...the above JIRA isn't related to the
current qpid-jms-client 0.25.0 release in any way, but rather a change
for an older prototype client that was included in the Qpid 0.26
release some years ago. The current AMQP 1.0 Qpid JMS client JIRAs are
at https://issues.apache.org/jira/browse/QPIDJMS

> Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the
>  mapping group-id / consumers looks tricky (at least to me)...especially if
> you consider things like elasticity, network failures. no?
>
> On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <Adelboutros@live.com> wrote:
>
>> Thanks!
>>
>> ________________________________
>> From: Ken Giusti <kgiusti@redhat.com>
>> Sent: Friday, September 29, 2017 3:54:20 PM
>> To: users; keith.wall@gmail.com
>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>>
>> At the moment the qpid dispatch router ignores all group related
>> message properties  when computing the route for a destination.
>>
>> I've opened a corresponding feature request against the router to
>> track support for message groups:
>>
>> https://issues.apache.org/jira/browse/DISPATCH-843
>>
>> On Fri, Sep 29, 2017 at 9:23 AM, Keith W <keith.wall@gmail.com> wrote:
>> > A JIRA has been raised for this problem:
>> >
>> > https://issues.apache.org/jira/browse/QPID-7937
>> >
>> > On 29 September 2017 at 11:47, Rob Godfrey <rob.j.godfrey@gmail.com>
>> wrote:
>> >> On 29 September 2017 at 10:35, Adel Boutros <Adelboutros@live.com>
>> wrote:
>> >>
>> >>> Hello Rob,
>> >>>
>> >>>
>> >>> I would like to give my opinion on this.
>> >>>
>> >>>
>> >>> In our current use cases, we are configuring the brokers dynamically
>> using
>> >>> the REST API. We would like to have this possibility in the use case
of
>> >>> Olivier.
>> >>
>> >>
>> >>> Also, having to restart the virtual host is damaging the High
>> Availability
>> >>> of the messaging system. This is because while the Virtual Host is
>> >>> restarting, no queues are available and the messages are inaccessible.
>> So I
>> >>> was wondering if restarting the virtual is a must or it could be fixed
>> in a
>> >>> Jira story?
>> >>>
>> >>>
>> >>
>> >> Sorry - my wording wasn't very clear.  If you set this when you first
>> >> create the queue then you don't need to restart the vhost... however if
>> you
>> >> have an existing queue that you want to change, then the effect won't be
>> >> seen until the vhost is restarted (basically the queue sets itself up
>> to be
>> >> "group aware" or "not group aware" when it starts up, it can't change
>> while
>> >> it is running).  I don't think this is really an issue for you in that
>> when
>> >> you create your queues with the REST API you just need to set this
>> >> attribute.
>> >>
>> >>
>> >>>
>> >>> Regarding the 2nd point of multiple brokers and dispatch router, if
all
>> >>> brokers have the same queues configured with the appropriate grouping
>> >>> config, would that really break the feature?
>> >>>
>> >>> In our current use cases, all components have the same configuration
>> all
>> >>> the time (All brokers have same queues and same for the dispatch
>> router).
>> >>>
>> >>> So I would imagine if a consumer wants to consume a message with the
>> >>> correct group, the dispatch router will propagate this header to any
>> >>> available broker and will only get the corresponding messages.
>> >>>
>> >>>
>> >>>
>> >> The way grouping works is that the first time a message with a
>> particular
>> >> group id comes along, the broker will assign that group to a particular
>> >> consumer.  Each broker will do this independently.  So if you have two
>> >> brokers B1 and B2; and two consumers C1 and C2  and message M of group A
>> >> arrives on B1 whereas message N of group A arrives on B2; then B1 may
>> >> decide to associate group A with C1 whereas broker B2 decides to
>> associate
>> >> group A with consumer C2.  So message groups with multiple brokers will
>> not
>> >> work behind a router *unless* the router is enhanced so that it is
>> aware of
>> >> message grouping (so all messages of the same group are sent to the same
>> >> broker) or the brokers somehow share information about the group ->
>> >> consumer assignment.
>> >>
>> >> -- Rob
>> >>
>> >> What do you think?
>> >>>
>> >>>
>> >>> Regards,
>> >>>
>> >>> Adel
>> >>>
>> >>> ________________________________
>> >>> From: Rob Godfrey <rob.j.godfrey@gmail.com>
>> >>> Sent: Friday, September 29, 2017 9:14:52 AM
>> >>> To: users@qpid.apache.org
>> >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>> >>>
>> >>> On 29 September 2017 at 01:09, Olivier Mallassi <
>> >>> olivier.mallassi@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Gentleman,
>> >>> >
>> >>> > I will need your help.
>> >>> >
>> >>> > I have a use case where I would like to guarantee "consumer
>> affinity",
>> >>> > which is usually implemented using the JMSXGroupID (actually, I
am
>> sure
>> >>> it
>> >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>> >>> >
>> >>> > My test does the "classical" case:
>> >>> > for (i ... i < 100.)
>> >>> >   Message textMsg = new TextM....
>> >>> >    textMsg.setStringProperty("JMSXGroupID", "groupA")
>> >>> >   MessageProducer.send(textMsg);
>> >>> >
>> >>> > and I start two consumers (assuming only one would work).
>> >>> >
>> >>> > What I observe is that both consumers are receiving messages, acting
>> as
>> >>> > competing consumers. I also tried added the qpid.group_header_key
>> >>> property
>> >>> > to the queue but it does not change anything.
>> >>> >
>> >>> >
>> >>> > My setup is
>> >>> > - java broker 6.0.4 & 6.1.4
>> >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>> >>> >
>> >>> > Is this the expected behaviour? Any ideas?
>> >>> >
>> >>> >
>> >>> This is a bug in the AMQP 1.0 behaviour of the broker.  The Message
>> >>> Grouping functionality was originally built around the AMQP 0-x
>> protocols
>> >>> and is looking for a message group in the application headers of the
>> >>> message.  In AMQP 1.0 there is a dedicated property for this and the
>> JMS
>> >>> client is (correctly from an AMQP 1.0 point of view) placing the group
>> id
>> >>> there.
>> >>>
>> >>> As a work around instead of using JMSXGroupID you could use any other
>> (non
>> >>> JMSX prefixed) header, e.g.:
>> >>>
>> >>>  textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>> >>>
>> >>> Note that on the broker you need to configure the queue so it knows
>> which
>> >>> header to use; this is stored in the messageGroupKey property of the
>> queue
>> >>> (to "qpidBugWorkaround" in this case) .  Note that after setting this
>> >>> property you will need to restart the virtual host before the change
>> takes
>> >>> effect.
>> >>>
>> >>>
>> >>> >
>> >>> > Complementary question: let's assume now, that there is the DR
>> between
>> >>> the
>> >>> > broker & the consumers. Does the DR "propagate" or support
"grouping
>> >>> > messages"? (I assume it is depending if you route the link or
>> messages)
>> >>> >
>> >>>
>> >>> With a single broker and link routing, yes grouping messages should
>> >>> continue to work.
>> >>>
>> >>> If there are multiple brokers sharding the queue then it would not
>> work -
>> >>> to support this the router would need to be changed to recognize
>> grouping
>> >>> and send all messages with the same group id to the same broker
>> waypoint
>> >>> (also, in this case, you would need to use message routing for inbound
>> >>> messages and link routing for consuming messages).  We have talked
>> about
>> >>> support for behaviour like this before, but there is nothing
>> implemented
>> >>> yet as far as I know.
>> >>>
>> >>> -- Rob
>> >>>
>> >>>
>> >>> >
>> >>> >
>> >>> > thanks for your help
>> >>> >
>> >>> > Regards.
>> >>> >
>> >>>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> > For additional commands, e-mail: users-help@qpid.apache.org
>> >
>>
>>
>>
>> --
>> -K
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message