qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Message API - Real world usage issue
Date Wed, 12 Feb 2014 02:25:32 GMT
On 11 February 2014 21:57, CLIVE <clive@ckjltd.co.uk> wrote:

> Robbie,
>
> Thanks for the response.
>
> You are confirming what Fraze has said, so I obviously need to take this
> on board and rethink my understanding of the Receiver concept.
>
> In my mind I had a Receiver as an entity that received the messages
> specified by the address string, but in fact I need to just look at it as a
> conduit to a queue that has one or more bindings associated with it.
>
>
It is basically able to be both of those things, but in this case you are
using it strictly as the latter.

The main issue is simply that the x-bindings declarations you are using
(e.g. x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: 'tim'}] )
aren't really related to the consumer but rather with the bindings between
the exchange and the queue they are receiving from. I believe its really
only conveyed in the address string as a form of extension point offering
some ability to leverage the AMQP 0-10 bind commands in a way that doesn't
require the addition of a very protocol-specific API method such as those
the older client API presumably had. I expect thats why it has a name
beginning "x-", though admit I dont know for sure, perhaps those with
deeper knowledge of the messaging API can speak to that.


>
>
> On 11/02/2014 21:35, Robbie Gemmell wrote:
>
>> On 11 February 2014 19:34, CLIVE <clive@ckjltd.co.uk> wrote:
>>
>>  Ted,
>>>
>>> Thanks for the response and your comments.
>>>
>>> I have had to handle the case of multiple Receivers attaching to the same
>>> queue on several occasions; primarily because the customer has felt that
>>> it
>>> was easier to handle one queue with multiple bindings (up to 100), rather
>>> than having a hundred queues with single bindings; message order was
>>> also a
>>> contributing factor.
>>>
>>> The point of the post was just to raise it as a possible issue for future
>>> improvement..
>>>
>>> I carried out a straw poll of 10 developers today at work. I gave them
>>> the
>>> two examples previously described and asked what they would expect to
>>> happen for the case where multiple Receivers were created for the same
>>> queue. They all expected the correct Receiver to be returned from the
>>> nextReceiver method, not the undeterministic behaviour that they would
>>> see.
>>>
>>>  As has been mentioned by others, the behaviour you are seeing is
>> expected
>> because it is exactly what you are really asking the client and broker to
>> do currently: one queue which can receive messages via multiple binding
>> keys that have been added, and distribute them to any of the completely
>> equal multiple consumers receiving from it.
>>
>> As Fraser has also beaten me to saying, if you really want to make
>> particular consumers only get particular messages from a shared queue,
>> then
>> you will likely need to look at using selectors so that they can in fact
>> only receive those messages.
>>
>>
>>  I wouldn't have thought that it would take that much code/effort to add
>>> some additional functionality in the messaging API Implementation to
>>> support the behavior that, it would appear, most developers would expect
>>> to
>>> see. If I find some time I will take a look and see how it could be done.
>>>
>>>
>>>  As Fraser mentioned, I think there is some confusion as to what your
>> reciever creation calls are actually doing, but even if removing that
>> confusion from the equation the situation is not necessarily as simple as
>> it may seem. Suppose two receivers add the same binding key, which is the
>> 'correct' receiver to get the single message? Suppose wildcard matching is
>> in use on the bindings and multiple bindings then match a particular
>> message published, which receiver gets the single message? Imagine
>> selectors are also in use, but mutliple consumers selectors match the
>> message, which reciever gets the message? The list goes on...
>>
>> You are effectively talking about turning the client into a sort of
>> broker,
>> and since you already have one of those its probably easier to just ask it
>> to do what you actually want.
>>
>> Robbie
>>
>>
>>
>>> On 10/02/2014 22:17, Ted Ross wrote:
>>>
>>>  Clive,
>>>>
>>>> What you are observing is what I expect:  In the second scenario where
>>>> you use the same queue for each of the three receivers, the receiver
>>>> that
>>>> receives any particular message will be non-deterministic.
>>>>
>>>> This is because the binding key is applied between the exchange and the
>>>> queue (i.e. it is used to determine which queue(s) the message should be
>>>> enqueued on).  Multiple receivers on a queue will receive messages from
>>>> the
>>>> queue in an undetermined order, but no message shall be delivered to
>>>> more
>>>> than one receiver.  In the second case, all of the messages are placed
>>>> on
>>>> the same queue in the order in which they arrive.  The queue acts as a
>>>> buffer between the routing rule that matched the message and the
>>>> receiver
>>>> that provided the routing rule.
>>>>
>>>> It would be simpler to do the following:
>>>>
>>>>    Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
>>>>    Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
>>>>    Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
>>>>
>>>> This will give you the determinism you want.  This will cause the
>>>> creation of a temporary queue for each receiver that will receive the
>>>> messages that match the topic key (following the slash in the address).
>>>>
>>>> -Ted
>>>>
>>>>
>>>> On 02/10/2014 04:39 PM, CLIVE wrote:
>>>>
>>>>  Fraser,
>>>>>
>>>>> Thanks for the response. The real problem is that the behavior of a
>>>>> Receiver is different depending on the multiplicity of the binding
>>>>> strategy
>>>>> used. If you use a single queue with a single binding then messages
>>>>> will
>>>>> get delivered to the required receiver. If you use multiple Receivers
>>>>> bound
>>>>> to the same queue, the Receiver called by the messaging API, when
>>>>> delivering a message to your application, may not be the one that you
>>>>> think!!
>>>>>
>>>>> So if I create three Receivers in the same application, with the
>>>>> following bindings (note unique queue names)
>>>>>
>>>>>     Rxer 1 - "queue1; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'bill'}]}}"
>>>>>     Rxer 2 - "queue2; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'ben'}]}}"
>>>>>     Rxer 3 - "queue3; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'tim'}]}}"
>>>>>
>>>>> And then send a message on the amq.topic exchange with a subject of
>>>>> 'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on the
>>>>> associated Session object.
>>>>>
>>>>> But if I change the bindings so they related to the same queue
>>>>>
>>>>>     Rxer 1 - "queue1; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'bill'}]}}"
>>>>>     Rxer 2 - "queue1; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'ben'}]}}"
>>>>>     Rxer 3 - "queue1; {create: receiver, node:
>>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic',
>>>>> queue:
>>>>> 'queue1', key: 'tim'}]}}"
>>>>>
>>>>> And send the same message again, Which Receiver would you expect to get
>>>>> returned from the sessions nextReceiver method?
>>>>>
>>>>> I would expect the same result as in the first example, Rxer 3. But
>>>>> this
>>>>> does not happen, anyone of the three receivers might get called.
>>>>>
>>>>> This doesn't seem right to me and as a result you have to produce quite
>>>>> a bit of application level logic to handle this scenario; especially
>>>>> when
>>>>> your bindings are being passed down to you dynamically by several
>>>>> client
>>>>> applications.
>>>>>
>>>>> Hope this explains it a bit better than my last attempt.
>>>>>
>>>>> Clive
>>>>>
>>>>>
>>>>> On 07/02/2014 10:03, Fraser Adams wrote:
>>>>>
>>>>>  On 06/02/14 19:07, CLIVE wrote:
>>>>>>
>>>>>>  Hi,
>>>>>>>
>>>>>>> [snip]
>>>>>>>
>>>>>>> The first use case requires the dynamic creation of Receivers,
but
>>>>>>> before creating a new receiver, I would like to know if I already
>>>>>>> have a
>>>>>>> receiver that would match the required binding. This is not possible
>>>>>>> at the
>>>>>>> moment because the binding matching algorithms are hidden from
>>>>>>> public view;
>>>>>>> they are buried deep inside the Brokers Exchange Implementation
code.
>>>>>>>
>>>>>>>  You know that you can get the binding information from QMF don't
you
>>>>>> Clive? I guess I'm missing what you're looking for if it's something
>>>>>> different than that. And I guess to be fair to get the information
>>>>>> via QMF
>>>>>> you'd need a bit of code, but I'd have thought that this would be
the
>>>>>> most
>>>>>> appropriate way to get the information.
>>>>>>
>>>>>>
>>>>>> Out of curiosity why do you need to know if you already have a
>>>>>> receiver
>>>>>> that would match the binding?
>>>>>>
>>>>>> One thing that's worth mentioning, I'm suspecting that (like me)
>>>>>> you've
>>>>>> mainly been using AMQP 0.10 - If I'm reading you correctly you sound
>>>>>> like
>>>>>> you are dynamically creating queue nodes and passing x-bindings.
>>>>>>
>>>>>> I've been doing that for a few years, but a few weeks back I started
>>>>>> looking at AMQP 1.0 and that primarily takes a perspective of
>>>>>> addressing
>>>>>> the topic like exchanges and the queues end up being subscription
>>>>>> queues
>>>>>> and all of the stuff that relates to binding and the like ends up
in
>>>>>> the
>>>>>> link (not node) config.
>>>>>>
>>>>>> For me at any rate that was quite a different perspective on things
(I
>>>>>> wrote up what I was up to in the "A write up of some AMQP 1.0
>>>>>> Experiments"
>>>>>> post) previously I've been focussing on the queues, so I was
>>>>>> dynamically
>>>>>> creating queue nodes and passing x-bindings in AMQP 0.10, but in
AMQP
>>>>>> 1.0
>>>>>> I've been addressing the exchanges (topic type nodes) and using the
>>>>>> link to
>>>>>> specify what I need. For me it took a bit of getting used to because
>>>>>> I was
>>>>>> so ingrained doing it the other way, but I think I'm getting it now.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  The second use case in question requires a client application to
>>>>>>> dynamically create multiple receivers for the same queue, but
with
>>>>>>> slightly
>>>>>>> different binding keys bound to an exchange. When a message from
an
>>>>>>> exchange gets put in the queue and delivered to the client (via
a
>>>>>>> receiver)
>>>>>>>
>>>>>>>  I'm not sure if I'm correctly interpreting what you are saying
>>>>>> here, so
>>>>>> you want a client that has a single queue, but each receiver adds
>>>>>> different
>>>>>> binding keys right? You do know that this will result in what amounts
>>>>>> to an
>>>>>> OR condition - both keys will be bound and a message will be put
on
>>>>>> the
>>>>>> queue if either match so consumer A of the queue would receive
>>>>>> messages due
>>>>>> to consumer B's key - is that what you mean.
>>>>>>
>>>>>> The following AMQP 1.0 consumers will do what you seem to be saying,
>>>>>> there's a single shared subscription queue called queue1, the first
>>>>>> consumer binds *.news the second *.weather
>>>>>>
>>>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>>>> "amq.topic/*.news; {node: {capabilities: [shared]}, link: {name:
>>>>>> queue1}}"
>>>>>>
>>>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>>>> "amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name:
>>>>>> queue1}}"
>>>>>>
>>>>>> qpid-config -r queues gives
>>>>>>
>>>>>> Queue 'queue1'
>>>>>>      bind [queue1] => ''
>>>>>>      bind [*.news] => amq.topic
>>>>>>      bind [*.weather] => amq.topic
>>>>>>
>>>>>>
>>>>>> For AMQP 0.10 the following would create a similar effect (not sure
if
>>>>>> you want auto delete or not, if not remove the x-declare below and
>>>>>> for the
>>>>>> AMQP 1.0 example above add reliability: at-least-once to the link
Map)
>>>>>>
>>>>>> ./drain -b localhost -f \
>>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>>> '*.news'}]}}"
>>>>>>
>>>>>> ./drain -b localhost -f \
>>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>>> '*.weather'}]}}"
>>>>>>
>>>>>>
>>>>>> The following also works for AMQP 0.10
>>>>>>
>>>>>> ./drain -b localhost -f \
>>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>>> '*.news'}]}}"
>>>>>>
>>>>>> ./drain -b localhost -f \
>>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>>> '*.weather'}]}}"
>>>>>>
>>>>>> Don't know if this is what you are looking for.
>>>>>>
>>>>>>
>>>>>> Note that in none of the cases above have I worked out how to remove
a
>>>>>> binding other than by removing the queue so if you add the first
then
>>>>>> the
>>>>>> second then delete the second both bindings remain in place - I did
>>>>>> wonder
>>>>>> about putting the x-declare/auto delete stuff on the link in the
>>>>>> second
>>>>>> AMQP 0.10 example, but that doesn't seem to remove the binding, so
>>>>>> I'm not
>>>>>> sure if that's possible.
>>>>>>
>>>>>>
>>>>>>   I need to route the message to the correct application level
>>>>>>
>>>>>>> destination(s). To do this I need to undertake a matching operation
>>>>>>> between
>>>>>>> the routing key of the message and the binding key(s) of the
created
>>>>>>> receivers; qpid does not deliver the message to the receiver
with
>>>>>>> the most
>>>>>>> exact binding key match.
>>>>>>>
>>>>>>>  I guess than I'm not understanding you here. As far as I'm aware
if
>>>>>> you've got multiple bindings between an exchange and a queue then
the
>>>>>> message will be delivered on to the queue if either binding matches,
>>>>>> so it
>>>>>> behaves like a logical OR. In your scenario if the first receiver
adds
>>>>>> *.news then the second adds *.weather then from that point on they
>>>>>> will
>>>>>> *both* start to receive (*.news OR *.weather)
>>>>>>
>>>>>>
>>>>>>   So basically the receivers, and their bindings, enable the required
>>>>>>
>>>>>>> messages to get delivered to the required client, but I then
need to
>>>>>>> undertake application level routing to route the message to one
or
>>>>>>> more
>>>>>>> application level classes, based on message routing key/ receiver
>>>>>>> binding
>>>>>>> key matches.
>>>>>>>
>>>>>>>  So I'm still totally baffled why you want to send them to the
same
>>>>>> queue if you are then demultiplexing at the application level. Surely
>>>>>> (for
>>>>>> example) you'd be better having a news queue for the *.news messages
>>>>>> and a
>>>>>> weather queue for the *.weather messages. If you force them down
the
>>>>>> same
>>>>>> queue then you are going to have to do application level
>>>>>> demultiplexing,
>>>>>> which it sounds like you don't want to do, but why use a single queue.
>>>>>>
>>>>>> What's actually driving the single queue requirement? That sounds
like
>>>>>> the root of your problems, without knowing the nuance of your
>>>>>> scenario it
>>>>>> feels like your approaching the problem from the wrong angle and
>>>>>> fighting
>>>>>> the middleware rather than letting it work for you. I'm sure I've
>>>>>> missed
>>>>>> something subtle in your use case.
>>>>>>
>>>>>>
>>>>>>
>>>>>>  Unfortunately in both cases the messaging API does not provide
>>>>>>> visibility of the bind matching algorithms and so I have to create
>>>>>>> several
>>>>>>> utility classes to support this functionality.
>>>>>>>
>>>>>>> Would it be possible to create a Binding.h class in the messaging
API
>>>>>>> to support matching of bindings from all the supported exchange
>>>>>>> types?
>>>>>>>
>>>>>>>  I'm not actually sure what you are asking for here. Are you
asking
>>>>>> for
>>>>>> a client side filtering API?
>>>>>>
>>>>>> As I say I'm having trouble getting under the skin of your use case.
>>>>>> If
>>>>>> I'm reading it correctly it sounds like you are wanting to have a
>>>>>> single
>>>>>> queue but have multiple bindings between an exchange and that queue,
>>>>>> which
>>>>>> will result in messages for both bindings making their way on to
the
>>>>>> queue
>>>>>> and then, to get around that, to apply a client side filter to
>>>>>> deliver the
>>>>>> right message to the right receiver - is that correct?
>>>>>>
>>>>>> I'm afraid that I'm still not clear why you want to do that on the
>>>>>> client rather than on the broker??
>>>>>>
>>>>>>
>>>>>>
>>>>>> Other's might have a better view, but I'm not sure that client side
>>>>>> filtering fits into the qpid::messaging API per se (and binding
>>>>>> probably
>>>>>> wouldn't be a good idea anyway as it's a legacy AMQP 0.10 concept).
>>>>>>
>>>>>>
>>>>>> One thought moving forward (and I'm far from an expert) might be
to
>>>>>> think in terms of AMQP 1.0, so the Qpid Broker may be viewed as
>>>>>> essentially
>>>>>> an AMQP 1.0 container and it has a whole bunch of capabilities,
>>>>>> including
>>>>>> the ability to filter (the traditional bindings plus - really cool
-
>>>>>> message selectors). The qpid::messaging API is about interacting
with
>>>>>> nodes
>>>>>> on a container and attaching links with specified properties.
>>>>>>
>>>>>> As it happens though an AMQP 1.0 client application can also be
>>>>>> thought
>>>>>> of as a container, so an interesting thought might be a client
>>>>>> application
>>>>>> containing its own addressable node. In this scenario you'd establish
>>>>>> all
>>>>>> the stuff previously discussed with the broker and the consumer client
>>>>>> would have all messages delivered to the node on the client, you
>>>>>> could then
>>>>>> (in theory 'cause none of this exists) create AMQP links (on the
>>>>>> client) to
>>>>>> the node (on the client) passing filter properties on attachment
>>>>>> (such as a
>>>>>> selector).
>>>>>>
>>>>>> As I say none of this exists at the moment (except on the broker)
but
>>>>>> it might be interesting to consider if it would be possible to
>>>>>> modularise
>>>>>> things such that some of these fairly general purpose AMQP 1.0
>>>>>> "services"
>>>>>> could be extracted from the broker and made available as a toolkit
for
>>>>>> creating general purpose AMQP 1.0 containers.
>>>>>>
>>>>>> As I say I'm no expert and tentatively finding my feet with AMQP
1.0,
>>>>>> Gordon Sim would be far better placed than I to say whether that
a)
>>>>>> makes
>>>>>> sense from an AMQP 1.0 perspective b) how feasible it is and c) how
>>>>>> likely
>>>>>> it is to happen :-)
>>>>>>
>>>>>> Hope I've managed to be at least some help Clive,
>>>>>> Cheers,
>>>>>> Frase
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>  ------------------------------------------------------------
>>>>> ---------
>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>> .
>>>>
>>>>
>>>>  ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message