qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: A write up of some AMQP 1.0 Experiments
Date Mon, 03 Feb 2014 19:53:22 GMT
On 03/02/14 15:38, Gordon Sim wrote:
> Thanks for taking the time to write this up and share it with the 
> list! Feedback from users is what drives everything forward.
No worries, I've been meaning to find the time to properly explore the 
AMQP 1.0 capabilities for ages. Part of my problem has been that there's 
not a lot of concrete examples so it put me off somewhat, so I thought 
that writing this might help someone else in the same position. I'll try 
to tidy it up and put it into something like a PDF, the current flat 
text isn't as readable as it could be. It'd be good to try and massage 
some of this stuff into proper documentation.

> * AMQP 1.0 specific message properties
> The x-amqp convention is indeed a qpid::messaging specific approach to 
> handling fields that do not have explicit accessors in the Message 
> class. It is described in the AMQP_1.0 readme. 
Yeah - to be honest the AMQP 1.0 readme started to make a whole lot more 
sense *after* I started actually trying things out :-D I found myself 
reading it, messing around for a while, then re-reading it. Some of the 
stuff I've written down is me replying what's there as a way of trying 
to externalise it in my own mind.

> They should be set for outgoing messages as of 0.24[1]. What version 
> were you using?
I was actually using 0.27 off trunk built a couple of weeks ago.

> As you note, the 'to' field can be set explicitly using the 
> 'x-amqp-to' property name. There is also a connection level option 
> that controls whether that should be automatically populated with the 
> same address as used in the target of the attach (set_to_on_send, 
> which takes a boolean value).
I didn't notice that. Looking again in ConnectionOptions.cpp I can see 
it now - though to be fair there aren't any comments that explain what 
that option actually means, and I didn't notice it in the Doxygen 

> The 0-10 to 1.0 conversion currently maps the 0-10 message-transfer's 
> 'destination' onto to the 'to' field. At the time I did it it seemed 
> logical enough, but I'm happy to remove that if it is confusing (as it 
> probably is).
It's a difficult call, I personally suspect that it's potentially 
confusing. When I was trying this stuff out I wasn't sure if it was 
something that AMQP 0.10 qpid::messaging did or was an artefact of the 
mapping. It's probably safest to expect it to be explicitly added like 
it is on Messenger, might be worth a wider discussion though.

> * Text v. binary content
> The translation from 0-10 messages to 1.0 equivalents only handles 
> content types for lists and maps at present. It should be improved to 
> check for text also. I've created a JIRA to track that[2].
Great, thanks!

> * Subscription queues
> I would say (loosely) that the subscription queue is part of the 
> terminus. I've added a sentence to the 1.0 readme describing how qpidd 
> constructs the subscription queue name.
What I couldn't figure was the relationship/association between node and 
terminus. The AMQP 1.0 specification says "A link attaches to a node at 
a terminus. There are two kinds of terminus: sources and targets" but 
what's not especially clear in the specification is the multiplicity 
between node and source or node and target. I think from what you are 
saying here that there is a 1..* relationship. I knew it couldn't be 
part of the link because of the ability to support shared subscriptions 
(thank goodness!!).

> I added the incoming and outgoing link entities as they seemed to be 
> useful when thinking in terms of the 1.0 model. The source and target 
> are as they appear in the attach received. 
Yeah those seem to have potential, I stumbled across the "domain" and 
"incoming"/"outgoing" stuff I've not had a chance to play yet but the 
ability to establish links with arbitrary AMQP 1.0 containers seems 
really cool. Is this stuff ultimately intended to replace the 
"traditional" link/bridge federation model?

> Though there is no QMF 'object reference' to the subscription queue 
> for an outgoing link from an exchange, the naming policy for 
> subscription queues does allow them to be mapped to their 
> corresponding outgoing links.
In QMF there isn't an object reference between an exchange and a queue 
anyway, the association is via the bindings. It's really interesting to 
look at this stuff through the GUI, what I see if I do:

./drain --connection-options {protocol:amqp1.0} -b localhost -f "amq.fanout"

Is a queue called <UUID>_amq.fanout_<UUID>
and if I look at the bindings I see
bind[] -> amq.fanout (and one to default direct)

So the exchange can be navigated to the queue and vice versa, but what 
is missing is the QMF subscription object and without the subscription 
object it's actually navigating between the subscription queue and the 
connection (and vice versa) that's impossible - and I often find it 
useful to see what Connection IPs are using which queues etc.

> That said, I would agree that the situation around QMF and 1.0 links 
> can likely be improved upon, especially for people used to dealing 
> with the 0-10 model. I've raised a JIRA to ensure that doesn't get 
> lost[3].
So as above it's the missing subscription object that is the main issue 
I was flagging, but it would indeed be really nice to have the 
associations with incoming/outgoing links be navigable.

> * Reply-to
> The qpid::messaging library doesn't particularly restrict the value of 
> the reply-to in the message itself[4].
> If it can be interpreted as of the form <name>/<subject>, then the 
> address object returned will have both the name and subject set to the 
> respective substrings. Otherwise the name will contain the full reply-to.
> If the application takes that address and uses it directly as an 
> address for creating a sender, the name is used as the source or 
> target address for the link and if a subject is specified it will be 
> used to set a filter.
> I believe a common usage will have reply-to simply contain a node name 
> (often for a temporary queue). That pattern, essentially the 
> client-server example, works against several brokers and also the 
> dispatch router[5]. It is also familiar from JMS.
> However if some other specific scheme/format is used, the application 
> would responsible for interpreting the address and e.g. establishing 
> separate connections etc if that behaviour is desired.
I'll need to have more of a play with Reply-To but the thing I was 
really trying to get at was that given the peer to peer model of AMQP 
1.0 it is (presumably) possible to set a Reply-To to the complete unique 
address of the sender (including host:port part) and the receiver could 
presumably directly address the sender without necessarily returning via 
intermediaries. That scenario seemed awkward for a connection oriented 
paradigm such as qpid::messaging or JMS because the host:port part might 
be an entirely different connection to the one it has used to connect to 
say a broker.

You are probably right that having the reply to refer to a node name is 
likely to be a common usage pattern for client/server style applications 
I was mainly musing about what a "fully qualified" address might mean 
for connection oriented paradigms - if you see what I mean.

> * Shared subscriptions
> The 'shared' subscription capability is described in the AMQP 1.0 readme. 
Yeah I did notice it there, my problem was working out what the syntax I 
needed to use was.

> Note that it is a qpidd specific extension. At present the 
> subscription queue will be autodeleted unless you set the link to be 
> durable or reliable.
Ah I didn't realise that the reliable flag would help me. So would that 
be something like reliability:|||at-least-once| in the link properties?

So are you saying that other AMQP 1.0 brokers generally don't support 
shared subscriptions? TBH that seems unfortunate I know that I find them 
incredibly useful to distribute workload between physical consumers.

> Supporting an expiry policy of 'never' should be possible as well. 
> I'll update the test to note the convention around queue name.
What I was getting at in that section of my post was "I couldn't see 
anything in the qpid::messaging code related to expiry" in other words 
there was no reference to pn_terminus_set_expiry_policy so (I think) 
it's additional code that's needed - have I missed something?

> The auto-delete value specified on a topic will now be correctly used 
> for subscription queues created for that topic[6]. 
Great, thanks - at the very least it will make things "feel" consistent 
having that ability.

> You can also specify a lifetime-policy e.g. to delete only when not 
> used *and* empty. This is a nice option as it will be recreated if 
> ever needed, but won't sit around taking up resources if not.
Ahh I didn't realise that you could set the lifetime-policy for topics, 
I assumed that it was specific to node creation.

> * The 'create' option in qpid::messaging addresses
> Using this option with an explicit node name results in non-standard 
> AMQP and is discouraged where it can be avoided.
Yeah I realised that , I was just playing  :-) Is there a reason for 
that (other than it being what the AMQP 1.0 specification says) not sure 
why the spec. chose to say this?

> A better alternative is to configure node policies. These are patterns 
> that an attaching links address are checked against if it doesn't 
> resolve to any existing node. If a matching pattern is found, the node 
> will be auto-created with the properties of that policy.
Ahhh, again that's something that I hadn't come across, I noticed 
QueuePolicy/TopicPolicy in the QMF managemment-schema.xml and it was 
bugging me what they were for.

> * Filters
> In the case of the headers exchange, the key could be taken to be the 
> name of the filter, as the key is defined simply to be an identifier 
> for that exchange. However for other exchanges this would not be the 
> case (e.g. for topic, direct and even the qpidd specific xml 
> exchanges, the key affects matching and is therefore in my view 
> logically part of the filters value).
Yeah I'd agree for topic and direct the key relates to the value, but in 
the case of headers exchange there does need to be a way of setting the 
binding key and it seems (well to me) that the filter name is probably 
the most convenient thing.

As I say in my post I have scenarios where I might have multiple 
bindings between the headers exchange and a queue and these all need 
different keys. At the moment the binding is [] so the last specified 
filter is the only one that actually gets applied.

> Note also that in AMQP 1.0 a message must match *all* the specified 
> filters, so you can't emulate OR with a set of different filters.
So that's slightly worrying I hadn't clocked that. What I'm not clear 
about then is how I'd go about mapping the scenario I had with:

x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'key1', 
arguments: {x-match: all, data-service: amqp-delivery, item-owner: 
fadams}}, {exchange: 'amq.match', queue: 'testqueue', key: 'key2', 
arguments: {x-match: all, data-service: http-delivery, item-owner: fadams}}]

In this AMQP 0.10 case I've got two separate bindings added between 
amq.match and "testqueue" so messages will be delivered if either of 
them match.

I (naively) thought that multiple filters was the way (though on 
reflection I can see that all filters must match - how else would my 
topic plus selector example work the way it did d'Oh).

Can you think of a way how the use case I've suggested might be 
supported (aside from selectors - they are cool, but it'd be good to be 
able to replicate the legacy headers stuff completely - especially for 
migration/integration purposes).

> For the xml exchange the binding key is in fact an additional 
> filtering element that must match the subject. It is not just an 
> identifier, as it is in the case of the headers exchange. You can set 
> the key using the 'subject' in the address, e.g. xml/weather.
Ahhh I think I'm getting you now, I've just tried:

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"xml/weather; {link: {name: test-link, filter: {name: 'weather', 
descriptor: 'apache.org:xquery-filter:string', value: \" \
./weather/station = 'Raleigh-Durham International Airport (KRDU)' \
and ./weather/temperature_f > 50 \
and ./weather/temperature_f - ./weather/dewpoint > 5 \
and ./weather/wind_speed_mph > 7 \
and ./weather/wind_speed_mph < 20 \

and that does indeed set weather as the binding key, thanks.

Fair play though I thought that I did pretty well getting as far as I 
did with the filter stuff :-D

The version in the "Programming" book says:

./drain -f "xml; {link:{x-bindings:[{key:'weather',
	      arguments:{xquery:\"$(cat rdu.xquery )\"}}]}}"

So, setting the key rather than the subject.

To be fair I actually think using the subject is clearer, but you can 
hardly blame me for getting a bit confused.

> Although AMQP itself does not place any restrictions on application 
> property names, the selector syntax for AMQP 1.0 is that of JMS 
> selectors[7], where as you noted, data-service is not a valid name.  
> The extension of the selector explicitly states "JMS header names 
> should be translated to amqp.<field_name> where <field_name> is the 
> appropriate AMQP 1.0 field named in the table above,  with the hyphen 
> replaced by an underscore."
> However we could probably look at ways to (optionally?) make this more 
> lenient. It certainly is also something that should be highlighted 
> more prominently in documentation somehow.
It would be *really really good* to be able to make this more lenient 
(would quoted field names be a possibility e.g. 

I've used hyphened property names quite a lot and not being able to use 
them would be a bit of a show stopper for me due to the integration 
issues (I've got a lot of producers and they certainly couldn't all be 
changed simultaneously) and given the apparent support at the moment for 
multiple headers bindings between amq.match and a given queue I'm a bit 
stuck if I want to migrate.

Re: "The extension of the selector explicitly states "JMS header names 
should be translated to amqp.<field_name> where <field_name> is the 
appropriate AMQP 1.0 field named in the table above,  with the hyphen 
replaced by an underscore."

True, but that could be interpreted to mean the *standard* JMS header 
names, and being slightly flippant it also says "The "properties" of the 
JMS message are equivalent to the AMQP application-properties section" - 
which simply says string.

It's a bit moot, I'm just having a bit of fun around how one can 
interpret things, but I really would be grateful if you guys could have 
a think about being able to support hyphened property names (clearly 
there's a potential conflict with the minus sign which is why I was 
wondering if quoting the property name was an option).

As I say for my current set of scenarios it makes things rather awkward :-/

> I believe the selector filters are supported by a number of brokers 
> outside Qpid (ActiveMQ, HornetQ, SwiftMQ). 
That's useful to know, have you tried many interoperability scenarios?

> However the legacy-amqp-filters are only supported by the two Qpid 
> brokers at present as far as I am aware.
That's not such a big deal at the moment, I'd definitely look to move to 
a subject plus selectors model if I can, that last example I did was 
really cool and selectors are so expressive. I'd have given my *eye 
teeth* to have had selectors when I first started using Qpid - when I 
was migrating data flows from the previous system we had having a "NOT" 
would have been soooooo useful.

> [1] https://issues.apache.org/jira/browse/QPID-4707
> [2] https://issues.apache.org/jira/browse/QPID-5536
> [3] https://issues.apache.org/jira/browse/QPID-5537
> [4] https://issues.apache.org/jira/browse/QPID-5168
> [5] https://issues.apache.org/jira/browse/DISPATCH-1 and/or the 
> set_to_on_send connection option
> [6] https://issues.apache.org/jira/browse/QPID-5469
> [7] "The selector-filter uses the selector as defined by JMS."
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org

BTW were my observations that Proton Messenger isn't currently able to 
support the more sophisticated subscription patterns correct? If so is 
this something that is likely to be addressed (no pun intended) :-)

Many thanks for reading through this and responding so quickly.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message