qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Message API - Real world usage issue
Date Fri, 07 Feb 2014 10:03:26 GMT
On 06/02/14 19:07, CLIVE wrote:
> Hi,
> [snip]
> The first use case requires the dynamic creation of Receivers, but 
> before creating a new receiver, I would like to know if I already have 
> a receiver that would match the required binding. This is not possible 
> at the moment because the binding matching algorithms are hidden from 
> public view; they are buried deep inside the Brokers Exchange 
> Implementation code.
You know that you can get the binding information from QMF don't you 
Clive? I guess I'm missing what you're looking for if it's something 
different than that. And I guess to be fair to get the information via 
QMF you'd need a bit of code, but I'd have thought that this would be 
the most appropriate way to get the information.

Out of curiosity why do you need to know if you already have a receiver 
that would match the binding?

One thing that's worth mentioning, I'm suspecting that (like me) you've 
mainly been using AMQP 0.10 - If I'm reading you correctly you sound 
like you are dynamically creating queue nodes and passing x-bindings.

I've been doing that for a few years, but a few weeks back I started 
looking at AMQP 1.0 and that primarily takes a perspective of addressing 
the topic like exchanges and the queues end up being subscription queues 
and all of the stuff that relates to binding and the like ends up in the 
link (not node) config.

For me at any rate that was quite a different perspective on things (I 
wrote up what I was up to in the "A write up of some AMQP 1.0 
Experiments" post) previously I've been focussing on the queues, so I 
was dynamically creating queue nodes and passing x-bindings in AMQP 
0.10, but in AMQP 1.0 I've been addressing the exchanges (topic type 
nodes) and using the link to specify what I need. For me it took a bit 
of getting used to because I was so ingrained doing it the other way, 
but I think I'm getting it now.

> The second use case in question requires a client application to 
> dynamically create multiple receivers for the same queue, but with 
> slightly different binding keys bound to an exchange. When a message 
> from an exchange gets put in the queue and delivered to the client 
> (via a receiver)
I'm not sure if I'm correctly interpreting what you are saying here, so 
you want a client that has a single queue, but each receiver adds 
different binding keys right? You do know that this will result in what 
amounts to an OR condition - both keys will be bound and a message will 
be put on the queue if either match so consumer A of the queue would 
receive messages due to consumer B's key - is that what you mean.

The following AMQP 1.0 consumers will do what you seem to be saying, 
there's a single shared subscription queue called queue1, the first 
consumer binds *.news the second *.weather

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.news; {node: {capabilities: [shared]}, link: {name: queue1}}"

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.topic/*.weather; {node: {capabilities: [shared]}, link: {name: 

qpid-config -r queues gives

Queue 'queue1'
     bind [queue1] => ''
     bind [*.news] => amq.topic
     bind [*.weather] => amq.topic

For AMQP 0.10 the following would create a similar effect (not sure if 
you want auto delete or not, if not remove the x-declare below and for 
the AMQP 1.0 example above add reliability: at-least-once to the link Map)

./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}, 
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"

./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}, 
x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"

The following also works for AMQP 0.10

./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}}, link: 
{x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.news'}]}}"

./drain -b localhost -f \
"queue1; {create: receiver, node: {x-declare:{auto-delete:True}}, link: 
{x-bindings: [{exchange: 'amq.topic', queue: 'queue1', key: '*.weather'}]}}"

Don't know if this is what you are looking for.

Note that in none of the cases above have I worked out how to remove a 
binding other than by removing the queue so if you add the first then 
the second then delete the second both bindings remain in place - I did 
wonder about putting the x-declare/auto delete stuff on the link in the 
second AMQP 0.10 example, but that doesn't seem to remove the binding, 
so I'm not sure if that's possible.

> I need to route the message to the correct application level 
> destination(s). To do this I need to undertake a matching operation 
> between the routing key of the message and the binding key(s) of the 
> created receivers; qpid does not deliver the message to the receiver 
> with the most exact binding key match.
I guess than I'm not understanding you here. As far as I'm aware if 
you've got multiple bindings between an exchange and a queue then the 
message will be delivered on to the queue if either binding matches, so 
it behaves like a logical OR. In your scenario if the first receiver 
adds *.news then the second adds *.weather then from that point on they 
will *both* start to receive (*.news OR *.weather)

> So basically the receivers, and their bindings, enable the required 
> messages to get delivered to the required client, but I then need to 
> undertake application level routing to route the message to one or 
> more application level classes, based on message routing key/ receiver 
> binding key matches.
So I'm still totally baffled why you want to send them to the same queue 
if you are then demultiplexing at the application level. Surely (for 
example) you'd be better having a news queue for the *.news messages and 
a weather queue for the *.weather messages. If you force them down the 
same queue then you are going to have to do application level 
demultiplexing, which it sounds like you don't want to do, but why use a 
single queue.

What's actually driving the single queue requirement? That sounds like 
the root of your problems, without knowing the nuance of your scenario 
it feels like your approaching the problem from the wrong angle and 
fighting the middleware rather than letting it work for you. I'm sure 
I've missed something subtle in your use case.

> Unfortunately in both cases the messaging API does not provide 
> visibility of the bind matching algorithms and so I have to create 
> several utility classes to support this functionality.
> Would it be possible to create a Binding.h class in the messaging API 
> to support matching of bindings from all the supported exchange types?

I'm not actually sure what you are asking for here. Are you asking for a 
client side filtering API?

As I say I'm having trouble getting under the skin of your use case. If 
I'm reading it correctly it sounds like you are wanting to have a single 
queue but have multiple bindings between an exchange and that queue, 
which will result in messages for both bindings making their way on to 
the queue and then, to get around that, to apply a client side filter to 
deliver the right message to the right receiver - is that correct?

I'm afraid that I'm still not clear why you want to do that on the 
client rather than on the broker??

Other's might have a better view, but I'm not sure that client side 
filtering fits into the qpid::messaging API per se (and binding probably 
wouldn't be a good idea anyway as it's a legacy AMQP 0.10 concept).

One thought moving forward (and I'm far from an expert) might be to 
think in terms of AMQP 1.0, so the Qpid Broker may be viewed as 
essentially an AMQP 1.0 container and it has a whole bunch of 
capabilities, including the ability to filter (the traditional bindings 
plus - really cool - message selectors). The qpid::messaging API is 
about interacting with nodes on a container and attaching links with 
specified properties.

As it happens though an AMQP 1.0 client application can also be thought 
of as a container, so an interesting thought might be a client 
application containing its own addressable node. In this scenario you'd 
establish all the stuff previously discussed with the broker and the 
consumer client would have all messages delivered to the node on the 
client, you could then (in theory 'cause none of this exists) create 
AMQP links (on the client) to the node (on the client) passing filter 
properties on attachment (such as a selector).

As I say none of this exists at the moment (except on the broker) but it 
might be interesting to consider if it would be possible to modularise 
things such that some of these fairly general purpose AMQP 1.0 
"services" could be extracted from the broker and made available as a 
toolkit for creating general purpose AMQP 1.0 containers.

As I say I'm no expert and tentatively finding my feet with AMQP 1.0, 
Gordon Sim would be far better placed than I to say whether that a) 
makes sense from an AMQP 1.0 perspective b) how feasible it is and c) 
how likely it is to happen :-)

Hope I've managed to be at least some help Clive,

To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

View raw message