qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robbie Gemmell <robbie.gemm...@gmail.com>
Subject Re: Message API - Real world usage issue
Date Tue, 11 Feb 2014 21:53:25 GMT
On 11 February 2014 21:26, CLIVE <clive@ckjltd.co.uk> wrote:

> On 11/02/2014 19:35, Fraser Adams wrote:
>
>> Clive,
>> In your second scenario I don't see why you'd expect the message to get
>> delivered to Rxer 3.
>>
> I suppose because the Receiver I used with that address had the 'tim'
> binding, and this matches the subject of the message that was sent.
>
>
>> What you've done is deliberately create three bindings to the same queue,
>> so you'd have a match bill OR ben OR tim. If you sent a message with a
>> subject of tim it would simply get added do the queue (that all receivers
>> are consuming from) so it would simply arrive at the next available
>> receiver.
>>
>
> Yes, I realize that the bindings are essentially being OR'ed together. But
> when the message gets pulled off the queue and delivered to the client, why
> make an arbitrary choice about which Receiver to send it to, why not, for a
> bit of extra code, send it to the Receiver that provided the matching
> binding (Although I have to admit that this does raise the question of what
> to do if two or more bindings match??)
>
>
>> Basically the three receivers have just been configured to receive
>> exactly the same thing. If you do qpid-config -r queues you'll see queue1
>> with three bindings.
>>
> No, the receivers are not configured to receiver the same thing, they are
> configured to receive off the same queue.


The receivers are configured to recieve the same thing, everything on the
shared queue they are all receiving from. They also happen to be confgured
to create bindings upon their creation, but those concepts are entirely
distinct.

What you are describing would seem to be message selection, such that a
particular consumer can only receive particular messages on the shared
queue. You aren't using that functionality, but you potentially could in
order to achieve the behaviour you desire.


>
>> I think that you are confusing yourself because it might look like the
>> x-bindings relate to the receiver given how the Address string looks, but
>> as I say you're establishing bindings between amq.topic and queue1 and
>> doing what you've done creates three ORed bindings.
>>
> I don't think I am confusing myself, I understand exactly what is going
> on. In my mind I am creating three high level Receivers, not three ORed
> bindings. The straw poll I carried out at work also seemed to suggest that
> what I would like to see is also what a lot of other developers would
> expect as well.
>
>
>> I'm still struggling to see why you want to use a single queue, when it
>> seems like you actually want to demultiplex.
>>
> As I said customer wanted the single queue to help with message ordering
> and reduce perceived maintenance associated with lots of queues
>
>
>> Can you explain a bit more about what you are actually trying to achieve?
>>
>
> Lets say you provide the following interface method on a public C++ API,
> that shields the User Application from the underlying QPID implementation.
>
> class Messaging {
>     public:
>       static void registerUserCallback(const std::string& address, const
> boost::function< void (const std::string&) >& callback);
> };
>
> So at run time you have no way of knowing what address strings are going
> to be passed to you.
>
> The User application then makes the following calls on your API
>
>   Messaging.registerUserCallback( "queue1; {create: receiver, node:
> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
> 'queue1', key: 'bill'}]}}" ,usrCallback1);
>   Messaging.registerUserCallback( "queue2; {create: receiver, node:
> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
> 'queue1', key: 'ben'}]}}" ,usrCallback2);
>
> The User application might not make a call with an address string that
> references the same queue, but as a good API designer, you need to be able
> to handle it.
>
> If the QPID messaging implementation routed messages to the correct
> Receiver, then the implementation is pretty trivial. As it doesn't, then
> the implementation becomes a lot harder.
>

The client is delivering the message to the receiver associated with the
underlying consumer that the broker gave the message to. It isnt really
free to hand it to anything else. The main choices would appear to be to
configure the receivers (and/or queues) such that they can only get the
messages you want them to, or perform application-level distribution, as it
sounds like you have previously done.


>
> Anyway I hope this helps to explain the scenario a little bit better.
>
> Clive
>
>
>  Frase
>>
>>
>> On 10/02/14 21:39, CLIVE wrote:
>>
>>> Fraser,
>>>
>>> Thanks for the response. The real problem is that the behavior of a
>>> Receiver is different depending on the multiplicity of the binding strategy
>>> used. If you use a single queue with a single binding then messages will
>>> get delivered to the required receiver. If you use multiple Receivers bound
>>> to the same queue, the Receiver called by the messaging API, when
>>> delivering a message to your application, may not be the one that you
>>> think!!
>>>
>>> So if I create three Receivers in the same application, with the
>>> following bindings (note unique queue names)
>>>
>>>    Rxer 1 - "queue1; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'bill'}]}}"
>>>    Rxer 2 - "queue2; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'ben'}]}}"
>>>    Rxer 3 - "queue3; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'tim'}]}}"
>>>
>>> And then send a message on the amq.topic exchange with a subject of
>>> 'tim'. Then Rxer3 will get returned by the 'nextReceiver' method on the
>>> associated Session object.
>>>
>>> But if I change the bindings so they related to the same queue
>>>
>>>    Rxer 1 - "queue1; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'bill'}]}}"
>>>    Rxer 2 - "queue1; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'ben'}]}}"
>>>    Rxer 3 - "queue1; {create: receiver, node:
>>> {x-declare:{auto-delete:true}, x-bindings: [{exchange: 'amq.topic', queue:
>>> 'queue1', key: 'tim'}]}}"
>>>
>>> And send the same message again, Which Receiver would you expect to get
>>> returned from the sessions nextReceiver method?
>>>
>>> I would expect the same result as in the first example, Rxer 3. But this
>>> does not happen, anyone of the three receivers might get called.
>>>
>>> This doesn't seem right to me and as a result you have to produce quite
>>> a bit of application level logic to handle this scenario; especially when
>>> your bindings are being passed down to you dynamically by several client
>>> applications.
>>>
>>> Hope this explains it a bit better than my last attempt.
>>>
>>> Clive
>>>
>>>
>>> On 07/02/2014 10:03, Fraser Adams wrote:
>>>
>>>> On 06/02/14 19:07, CLIVE wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> [snip]
>>>>>
>>>>> The first use case requires the dynamic creation of Receivers, but
>>>>> before creating a new receiver, I would like to know if I already have
a
>>>>> receiver that would match the required binding. This is not possible
at the
>>>>> moment because the binding matching algorithms are hidden from public
view;
>>>>> they are buried deep inside the Brokers Exchange Implementation code.
>>>>>
>>>> You know that you can get the binding information from QMF don't you
>>>> Clive? I guess I'm missing what you're looking for if it's something
>>>> different than that. And I guess to be fair to get the information via QMF
>>>> you'd need a bit of code, but I'd have thought that this would be the most
>>>> appropriate way to get the information.
>>>>
>>>>
>>>> Out of curiosity why do you need to know if you already have a receiver
>>>> that would match the binding?
>>>>
>>>> One thing that's worth mentioning, I'm suspecting that (like me) you've
>>>> mainly been using AMQP 0.10 - If I'm reading you correctly you sound like
>>>> you are dynamically creating queue nodes and passing x-bindings.
>>>>
>>>> I've been doing that for a few years, but a few weeks back I started
>>>> looking at AMQP 1.0 and that primarily takes a perspective of addressing
>>>> the topic like exchanges and the queues end up being subscription queues
>>>> and all of the stuff that relates to binding and the like ends up in the
>>>> link (not node) config.
>>>>
>>>> For me at any rate that was quite a different perspective on things (I
>>>> wrote up what I was up to in the "A write up of some AMQP 1.0 Experiments"
>>>> post) previously I've been focussing on the queues, so I was dynamically
>>>> creating queue nodes and passing x-bindings in AMQP 0.10, but in AMQP 1.0
>>>> I've been addressing the exchanges (topic type nodes) and using the link
to
>>>> specify what I need. For me it took a bit of getting used to because I was
>>>> so ingrained doing it the other way, but I think I'm getting it now.
>>>>
>>>>
>>>>
>>>>> The second use case in question requires a client application to
>>>>> dynamically create multiple receivers for the same queue, but with slightly
>>>>> different binding keys bound to an exchange. When a message from an
>>>>> exchange gets put in the queue and delivered to the client (via a receiver)
>>>>>
>>>> I'm not sure if I'm correctly interpreting what you are saying here, so
>>>> you want a client that has a single queue, but each receiver adds different
>>>> binding keys right? You do know that this will result in what amounts to
an
>>>> OR condition - both keys will be bound and a message will be put on the
>>>> queue if either match so consumer A of the queue would receive messages due
>>>> to consumer B's key - is that what you mean.
>>>>
>>>> The following AMQP 1.0 consumers will do what you seem to be saying,
>>>> there's a single shared subscription queue called queue1, the first
>>>> consumer binds *.news the second *.weather
>>>>
>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>> "amq.topic/*.news; {node: {capabilities: [shared]}, link: {name:
>>>> queue1}}"
>>>>
>>>> ./drain --connection-options {protocol:amqp1.0} -b localhost -f \
>>>> "amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name:
>>>> queue1}}"
>>>>
>>>> qpid-config -r queues gives
>>>>
>>>> Queue 'queue1'
>>>>     bind [queue1] => ''
>>>>     bind [*.news] => amq.topic
>>>>     bind [*.weather] => amq.topic
>>>>
>>>>
>>>> For AMQP 0.10 the following would create a similar effect (not sure if
>>>> you want auto delete or not, if not remove the x-declare below and for the
>>>> AMQP 1.0 example above add reliability: at-least-once to the link Map)
>>>>
>>>> ./drain -b localhost -f \
>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"
>>>>
>>>> ./drain -b localhost -f \
>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True},
>>>> x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"
>>>>
>>>>
>>>> The following also works for AMQP 0.10
>>>>
>>>> ./drain -b localhost -f \
>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>> '*.news'}]}}"
>>>>
>>>> ./drain -b localhost -f \
>>>> "queue1; {create: receiver, node: {x-declare:{auto-delete:True}},
>>>> link: {x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key:
>>>> '*.weather'}]}}"
>>>>
>>>> Don't know if this is what you are looking for.
>>>>
>>>>
>>>> Note that in none of the cases above have I worked out how to remove a
>>>> binding other than by removing the queue so if you add the first then the
>>>> second then delete the second both bindings remain in place - I did wonder
>>>> about putting the x-declare/auto delete stuff on the link in the second
>>>> AMQP 0.10 example, but that doesn't seem to remove the binding, so I'm not
>>>> sure if that's possible.
>>>>
>>>>
>>>>  I need to route the message to the correct application level
>>>>> destination(s). To do this I need to undertake a matching operation between
>>>>> the routing key of the message and the binding key(s) of the created
>>>>> receivers; qpid does not deliver the message to the receiver with the
most
>>>>> exact binding key match.
>>>>>
>>>> I guess than I'm not understanding you here. As far as I'm aware if
>>>> you've got multiple bindings between an exchange and a queue then the
>>>> message will be delivered on to the queue if either binding matches, so it
>>>> behaves like a logical OR. In your scenario if the first receiver adds
>>>> *.news then the second adds *.weather then from that point on they will
>>>> *both* start to receive (*.news OR *.weather)
>>>>
>>>>
>>>>  So basically the receivers, and their bindings, enable the required
>>>>> messages to get delivered to the required client, but I then need to
>>>>> undertake application level routing to route the message to one or more
>>>>> application level classes, based on message routing key/ receiver binding
>>>>> key matches.
>>>>>
>>>> So I'm still totally baffled why you want to send them to the same
>>>> queue if you are then demultiplexing at the application level. Surely (for
>>>> example) you'd be better having a news queue for the *.news messages and
a
>>>> weather queue for the *.weather messages. If you force them down the same
>>>> queue then you are going to have to do application level demultiplexing,
>>>> which it sounds like you don't want to do, but why use a single queue.
>>>>
>>>> What's actually driving the single queue requirement? That sounds like
>>>> the root of your problems, without knowing the nuance of your scenario it
>>>> feels like your approaching the problem from the wrong angle and fighting
>>>> the middleware rather than letting it work for you. I'm sure I've missed
>>>> something subtle in your use case.
>>>>
>>>>
>>>>
>>>>> Unfortunately in both cases the messaging API does not provide
>>>>> visibility of the bind matching algorithms and so I have to create several
>>>>> utility classes to support this functionality.
>>>>>
>>>>> Would it be possible to create a Binding.h class in the messaging API
>>>>> to support matching of bindings from all the supported exchange types?
>>>>>
>>>>
>>>> I'm not actually sure what you are asking for here. Are you asking for
>>>> a client side filtering API?
>>>>
>>>> As I say I'm having trouble getting under the skin of your use case. If
>>>> I'm reading it correctly it sounds like you are wanting to have a single
>>>> queue but have multiple bindings between an exchange and that queue, which
>>>> will result in messages for both bindings making their way on to the queue
>>>> and then, to get around that, to apply a client side filter to deliver the
>>>> right message to the right receiver - is that correct?
>>>>
>>>> I'm afraid that I'm still not clear why you want to do that on the
>>>> client rather than on the broker??
>>>>
>>>>
>>>>
>>>> Other's might have a better view, but I'm not sure that client side
>>>> filtering fits into the qpid::messaging API per se (and binding probably
>>>> wouldn't be a good idea anyway as it's a legacy AMQP 0.10 concept).
>>>>
>>>>
>>>> One thought moving forward (and I'm far from an expert) might be to
>>>> think in terms of AMQP 1.0, so the Qpid Broker may be viewed as essentially
>>>> an AMQP 1.0 container and it has a whole bunch of capabilities, including
>>>> the ability to filter (the traditional bindings plus - really cool -
>>>> message selectors). The qpid::messaging API is about interacting with nodes
>>>> on a container and attaching links with specified properties.
>>>>
>>>> As it happens though an AMQP 1.0 client application can also be thought
>>>> of as a container, so an interesting thought might be a client application
>>>> containing its own addressable node. In this scenario you'd establish all
>>>> the stuff previously discussed with the broker and the consumer client
>>>> would have all messages delivered to the node on the client, you could then
>>>> (in theory 'cause none of this exists) create AMQP links (on the client)
to
>>>> the node (on the client) passing filter properties on attachment (such as
a
>>>> selector).
>>>>
>>>> As I say none of this exists at the moment (except on the broker) but
>>>> it might be interesting to consider if it would be possible to modularise
>>>> things such that some of these fairly general purpose AMQP 1.0 "services"
>>>> could be extracted from the broker and made available as a toolkit for
>>>> creating general purpose AMQP 1.0 containers.
>>>>
>>>> As I say I'm no expert and tentatively finding my feet with AMQP 1.0,
>>>> Gordon Sim would be far better placed than I to say whether that a) makes
>>>> sense from an AMQP 1.0 perspective b) how feasible it is and c) how likely
>>>> it is to happen :-)
>>>>
>>>> Hope I've managed to be at least some help Clive,
>>>> Cheers,
>>>> Frase
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>
>>>> .
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>> .
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message