qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CLIVE <cl...@ckjltd.co.uk>
Subject Re: Message API - Real world usage issue
Date Tue, 11 Feb 2014 21:57:33 GMT
Robbie,

Thanks for the response.

You are confirming what Fraze has said, so I obviously need to take this 
on board and rethink my understanding of the Receiver concept.

In my mind I had a Receiver as an entity that received the messages 
specified by the address string, but in fact I need to just look at it 
as a conduit to a queue that has one or more bindings associated with it.

Clive

On 11/02/2014 21:35, Robbie Gemmell wrote:
> On 11 February 2014 19:34, CLIVE <clive@ckjltd.co.uk> wrote:
>
>> Ted,
>>
>> Thanks for the response and your comments.
>>
>> I have had to handle the case of multiple Receivers attaching to the same
>> queue on several occasions; primarily because the customer has felt that it
>> was easier to handle one queue with multiple bindings (up to 100), rather
>> than having a hundred queues with single bindings; message order was also a
>> contributing factor.
>>
>> The point of the post was just to raise it as a possible issue for future
>> improvement..
>>
>> I carried out a straw poll of 10 developers today at work. I gave them the
>> two examples previously described and asked what they would expect to
>> happen for the case where multiple Receivers were created for the same
>> queue. They all expected the correct Receiver to be returned from the
>> nextReceiver method, not the undeterministic behaviour that they would see.
>>
> As has been mentioned by others, the behaviour you are seeing is expected
> because it is exactly what you are really asking the client and broker to
> do currently: one queue which can receive messages via multiple binding
> keys that have been added, and distribute them to any of the completely
> equal multiple consumers receiving from it.
>
> As Fraser has also beaten me to saying, if you really want to make
> particular consumers only get particular messages from a shared queue, then
> you will likely need to look at using selectors so that they can in fact
> only receive those messages.
>
>
>> I wouldn't have thought that it would take that much code/effort to add
>> some additional functionality in the messaging API Implementation to
>> support the behavior that, it would appear, most developers would expect to
>> see. If I find some time I will take a look and see how it could be done.
>>
>>
> As Fraser mentioned, I think there is some confusion as to what your
> reciever creation calls are actually doing, but even if removing that
> confusion from the equation the situation is not necessarily as simple as
> it may seem. Suppose two receivers add the same binding key, which is the
> 'correct' receiver to get the single message? Suppose wildcard matching is
> in use on the bindings and multiple bindings then match a particular
> message published, which receiver gets the single message? Imagine
> selectors are also in use, but mutliple consumers selectors match the
> message, which reciever gets the message? The list goes on...
>
> You are effectively talking about turning the client into a sort of broker,
> and since you already have one of those its probably easier to just ask it
> to do what you actually want.
>
> Robbie
>
>
>>
>> On 10/02/2014 22:17, Ted Ross wrote:
>>
>>> Clive,
>>>
>>> What you are observing is what I expect:  In the second scenario where
>>> you use the same queue for each of the three receivers, the receiver that
>>> receives any particular message will be non-deterministic.
>>>
>>> This is because the binding key is applied between the exchange and the
>>> queue (i.e. it is used to determine which queue(s) the message should be
>>> enqueued on).  Multiple receivers on a queue will receive messages from the
>>> queue in an undetermined order, but no message shall be delivered to more
>>> than one receiver.  In the second case, all of the messages are placed on
>>> the same queue in the order in which they arrive.  The queue acts as a
>>> buffer between the routing rule that matched the message and the receiver
>>> that provided the routing rule.
>>>
>>> It would be simpler to do the following:
>>>
>>>    Rxer 1 - "amq.topic/bill; {link: {x-declare: {auto-delete:true}}}"
>>>    Rxer 2 - "amq.topic/ben; {link: {x-declare: {auto-delete:true}}}"
>>>    Rxer 3 - "amq.topic/tim; {link: {x-declare: {auto-delete:true}}}"
>>>
>>> This will give you the determinism you want.  This will cause the
>>> creation of a temporary queue for each receiver that will receive the
>>> messages that match the topic key (following the slash in the address).
>>>
>>> -Ted
>>>
>>>
>>> On 02/10/2014 04:39 PM, CLIVE wrote:
>>>
>>>> Fraser,
>>>>
>>>> Thanks for the response. The real problem is that the behavior of a
>>>> Receiver is different depending on the multiplicity of the binding strategy
>>>> used. If you use a single queue with a single binding then messages will
>>>> get delivered to the required receiver. If you use multiple Receivers bound
>>>> to the same queue, the Receiver called by the messaging API, when
>>>> delivering a message to your application, may not be the one that you
>>>> think!!
>>>>
>>>> So if I create three Receivers in the same application, with the
>>>> following bindings (note unique queue names)
>>>>
>>>>     Rxer 1 - "queue1; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'bill'}]}}"
>>>>     Rxer 2 - "queue2; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'ben'}]}}"
>>>>     Rxer 3 - "queue3; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'tim'}]}}"
>>>>
>>>> And then send a message on the amq.topic exchange with a subject of
>>>> 'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on the
>>>> associated Session object.
>>>>
>>>> But if I change the bindings so they related to the same queue
>>>>
>>>>     Rxer 1 - "queue1; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'bill'}]}}"
>>>>     Rxer 2 - "queue1; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'ben'}]}}"
>>>>     Rxer 3 - "queue1; {create: receiver, node:
>>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>>> 'queue1', key: 'tim'}]}}"
>>>>
>>>> And send the same message again, Which Receiver would you expect to get
>>>> returned from the sessions nextReceiver method?
>>>>
>>>> I would expect the same result as in the first example, Rxer 3. But this
>>>> does not happen, anyone of the three receivers might get called.
>>>>
>>>> This doesn't seem right to me and as a result you have to produce quite
>>>> a bit of application level logic to handle this scenario; especially when
>>>> your bindings are being passed down to you dynamically by several client
>>>> applications.
>>>>
>>>> Hope this explains it a bit better than my last attempt.
>>>>
>>>> Clive
>>>>
>>>>
>>>> On 07/02/2014 10:03, Fraser Adams wrote:
>>>>
>>>>> On 06/02/14 19:07, CLIVE wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>> The first use case requires the dynamic creation of Receivers, but
>>>>>> before creating a new receiver, I would like to know if I already
have a
>>>>>> receiver that would match the required binding. This is not possible
at the
>>>>>> moment because the binding matching algorithms are hidden from public
view;
>>>>>> they are buried deep inside the Brokers Exchange Implementation code.
>>>>>>
>>>>> You know that you can get the binding information from QMF don't you
>>>>> Clive? I guess I'm missing what you're looking for if it's something
>>>>> different than that. And I guess to be fair to get the information via
QMF
>>>>> you'd need a bit of code, but I'd have thought that this would be the
most
>>>>> appropriate way to get the information.
>>>>>
>>>>>
>>>>> Out of curiosity why do you need to know if you already have a receiver
>>>>> that would match the binding?
>>>>>
>>>>> One thing that's worth mentioning, I'm suspecting that (like me) you've
>>>>> mainly been using AMQP 0.10 - If I'm reading you correctly you sound
like
>>>>> you are dynamically creating queue nodes and passing x-bindings.
>>>>>
>>>>> I've been doing that for a few years, but a few weeks back I started
>>>>> looking at AMQP 1.0 and that primarily takes a perspective of addressing
>>>>> the topic like exchanges and the queues end up being subscription queues
>>>>> and all of the stuff that relates to binding and the like ends up in
the
>>>>> link (not node) config.
>>>>>
>>>>> For me at any rate that was quite a different perspective on things (I
>>>>> wrote up what I was up to in the "A write up of some AMQP 1.0 Experiments"
>>>>> post) previously I've been focussing on the queues, so I was dynamically
>>>>> creating queue nodes and passing x-bindings in AMQP 0.10, but in AMQP
1.0
>>>>> I've been addressing the exchanges (topic type nodes) and using the link
to
>>>>> specify what I need. For me it took a bit of getting used to because
I was
>>>>> so ingrained doing it the other way, but I think I'm getting it now.
>>>>>
>>>>>
>>>>>
>>>>>> The second use case in question requires a client application to
>>>>>> dynamically create multiple receivers for the same queue, but with
slightly
>>>>>> different binding keys bound to an exchange. When a message from
an
>>>>>> exchange gets put in the queue and delivered to the client (via a
receiver)
>>>>>>
>>>>> I'm not sure if I'm correctly interpreting what you are saying here,
so
>>>>> you want a client that has a single queue, but each receiver adds different
>>>>> binding keys right? You do know that this will result in what amounts
to an
>>>>> OR condition - both keys will be bound and a message will be put on the
>>>>> queue if either match so consumer A of the queue would receive messages
due
>>>>> to consumer B's key - is that what you mean.
>>>>>
>>>>> The following AMQP 1.0 consumers will do what you seem to be saying,
>>>>> there's a single shared subscription queue called queue1, the first
>>>>> consumer binds *.news the second *.weather
>>>>>
>>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>>> "amq.topic/*.news; {node: {capabilities: [shared]}, link: {name:
>>>>> queue1}}"
>>>>>
>>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>>> "amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name:
>>>>> queue1}}"
>>>>>
>>>>> qpid-config -r queues gives
>>>>>
>>>>> Queue 'queue1'
>>>>>      bind [queue1] => ''
>>>>>      bind [*.news] => amq.topic
>>>>>      bind [*.weather] => amq.topic
>>>>>
>>>>>
>>>>> For AMQP 0.10 the following would create a similar effect (not sure if
>>>>> you want auto delete or not, if not remove the x-declare below and for
the
>>>>> AMQP 1.0 example above add reliability: at-least-once to the link Map)
>>>>>
>>>>> ./drain -b localhost -f \
>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"
>>>>>
>>>>> ./drain -b localhost -f \
>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"
>>>>>
>>>>>
>>>>> The following also works for AMQP 0.10
>>>>>
>>>>> ./drain -b localhost -f \
>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>> '*.news'}]}}"
>>>>>
>>>>> ./drain -b localhost -f \
>>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>>> '*.weather'}]}}"
>>>>>
>>>>> Don't know if this is what you are looking for.
>>>>>
>>>>>
>>>>> Note that in none of the cases above have I worked out how to remove
a
>>>>> binding other than by removing the queue so if you add the first then
the
>>>>> second then delete the second both bindings remain in place - I did wonder
>>>>> about putting the x-declare/auto delete stuff on the link in the second
>>>>> AMQP 0.10 example, but that doesn't seem to remove the binding, so I'm
not
>>>>> sure if that's possible.
>>>>>
>>>>>
>>>>>   I need to route the message to the correct application level
>>>>>> destination(s). To do this I need to undertake a matching operation
between
>>>>>> the routing key of the message and the binding key(s) of the created
>>>>>> receivers; qpid does not deliver the message to the receiver with
the most
>>>>>> exact binding key match.
>>>>>>
>>>>> I guess than I'm not understanding you here. As far as I'm aware if
>>>>> you've got multiple bindings between an exchange and a queue then the
>>>>> message will be delivered on to the queue if either binding matches,
so it
>>>>> behaves like a logical OR. In your scenario if the first receiver adds
>>>>> *.news then the second adds *.weather then from that point on they will
>>>>> *both* start to receive (*.news OR *.weather)
>>>>>
>>>>>
>>>>>   So basically the receivers, and their bindings, enable the required
>>>>>> messages to get delivered to the required client, but I then need
to
>>>>>> undertake application level routing to route the message to one or
more
>>>>>> application level classes, based on message routing key/ receiver
binding
>>>>>> key matches.
>>>>>>
>>>>> So I'm still totally baffled why you want to send them to the same
>>>>> queue if you are then demultiplexing at the application level. Surely
(for
>>>>> example) you'd be better having a news queue for the *.news messages
and a
>>>>> weather queue for the *.weather messages. If you force them down the
same
>>>>> queue then you are going to have to do application level demultiplexing,
>>>>> which it sounds like you don't want to do, but why use a single queue.
>>>>>
>>>>> What's actually driving the single queue requirement? That sounds like
>>>>> the root of your problems, without knowing the nuance of your scenario
it
>>>>> feels like your approaching the problem from the wrong angle and fighting
>>>>> the middleware rather than letting it work for you. I'm sure I've missed
>>>>> something subtle in your use case.
>>>>>
>>>>>
>>>>>
>>>>>> Unfortunately in both cases the messaging API does not provide
>>>>>> visibility of the bind matching algorithms and so I have to create
several
>>>>>> utility classes to support this functionality.
>>>>>>
>>>>>> Would it be possible to create a Binding.h class in the messaging
API
>>>>>> to support matching of bindings from all the supported exchange types?
>>>>>>
>>>>> I'm not actually sure what you are asking for here. Are you asking for
>>>>> a client side filtering API?
>>>>>
>>>>> As I say I'm having trouble getting under the skin of your use case.
If
>>>>> I'm reading it correctly it sounds like you are wanting to have a single
>>>>> queue but have multiple bindings between an exchange and that queue,
which
>>>>> will result in messages for both bindings making their way on to the
queue
>>>>> and then, to get around that, to apply a client side filter to deliver
the
>>>>> right message to the right receiver - is that correct?
>>>>>
>>>>> I'm afraid that I'm still not clear why you want to do that on the
>>>>> client rather than on the broker??
>>>>>
>>>>>
>>>>>
>>>>> Other's might have a better view, but I'm not sure that client side
>>>>> filtering fits into the qpid::messaging API per se (and binding probably
>>>>> wouldn't be a good idea anyway as it's a legacy AMQP 0.10 concept).
>>>>>
>>>>>
>>>>> One thought moving forward (and I'm far from an expert) might be to
>>>>> think in terms of AMQP 1.0, so the Qpid Broker may be viewed as essentially
>>>>> an AMQP 1.0 container and it has a whole bunch of capabilities, including
>>>>> the ability to filter (the traditional bindings plus - really cool -
>>>>> message selectors). The qpid::messaging API is about interacting with
nodes
>>>>> on a container and attaching links with specified properties.
>>>>>
>>>>> As it happens though an AMQP 1.0 client application can also be thought
>>>>> of as a container, so an interesting thought might be a client application
>>>>> containing its own addressable node. In this scenario you'd establish
all
>>>>> the stuff previously discussed with the broker and the consumer client
>>>>> would have all messages delivered to the node on the client, you could
then
>>>>> (in theory 'cause none of this exists) create AMQP links (on the client)
to
>>>>> the node (on the client) passing filter properties on attachment (such
as a
>>>>> selector).
>>>>>
>>>>> As I say none of this exists at the moment (except on the broker) but
>>>>> it might be interesting to consider if it would be possible to modularise
>>>>> things such that some of these fairly general purpose AMQP 1.0 "services"
>>>>> could be extracted from the broker and made available as a toolkit for
>>>>> creating general purpose AMQP 1.0 containers.
>>>>>
>>>>> As I say I'm no expert and tentatively finding my feet with AMQP 1.0,
>>>>> Gordon Sim would be far better placed than I to say whether that a) makes
>>>>> sense from an AMQP 1.0 perspective b) how feasible it is and c) how likely
>>>>> it is to happen :-)
>>>>>
>>>>> Hope I've managed to be at least some help Clive,
>>>>> Cheers,
>>>>> Frase
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>
>>>>> .
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>> .
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message