qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: A write up of some AMQP 1.0 Experiments
Date Tue, 04 Feb 2014 19:56:59 GMT
Thanks again for another update Gordon. Some responses inline below.


On 03/02/14 21:08, Gordon Sim wrote:
> On 02/03/2014 07:53 PM, Fraser Adams wrote:
>> On 03/02/14 15:38, Gordon Sim wrote:
>>> They should be set for outgoing messages as of 0.24[1]. What version
>>> were you using?
>> I was actually using 0.27 off trunk built a couple of weeks ago.
>
> Which properties didn't work for you using the x-amqp-* name?

I think that we may be speaking at cross purposes on this one, when you 
said "They should be set for outgoing messages as of 0.24[1]. What 
version were you using? " I had assumed that you were referring to the 
comments around the "to" being observed by the receiver if sent from 
AMQP 0.10 but not from AMQP 1.0 and the comment around that happening in 
the AMQP 0.10 -> AMQP 1.0 code. Re "Which properties didn't work for you 
using the x-amqp-* name" I'm not aware of any issue there (though I've 
not tried many) certainly it works fine for x-amqp-to as in the example:

./spout --connection-options {protocol:amqp1.0} -b localhost -P 
x-amqp-to=amqp://localhost/amq.fanout --content "Hello World" "amq.fanout"





>
> [...]
>>> The 0-10 to 1.0 conversion currently maps the 0-10 message-transfer's
>>> 'destination' onto to the 'to' field. At the time I did it it seemed
>>> logical enough, but I'm happy to remove that if it is confusing (as it
>>> probably is).
>> It's a difficult call, I personally suspect that it's potentially
>> confusing. When I was trying this stuff out I wasn't sure if it was
>> something that AMQP 0.10 qpid::messaging did or was an artefact of the
>> mapping. It's probably safest to expect it to be explicitly added like
>> it is on Messenger, might be worth a wider discussion though.
>
> In translating from 0-10 to 1.0 it can't really be explicitly added, 
> unless you mean as a special entry in the 0-10 application-properties? 
> (On 1.0 the default is indeed to require iut to be explicitly set).

I think that you've misinterpreted what I was meaning by "explicitly 
added" here, what I was simply meaning was expecting x-amqp-to to be set 
by the producer. If as you say that on 1.0 the default is to require it 
to be explicitly set the perhaps it's not unreasonable to expect 
applications (even if still using 0.10 on the producer side) to actually 
set it (x-amqp-to will just look like an ordinary property to 0.10 won't 
it).

I'm kind of easy on this I guess as long as the pure 1.0 to 1.0 behaves 
in a consistent manner when compared to say Messenger then it's probably 
a coin toss as to what to do when translating.




>
> [...]
>>> I added the incoming and outgoing link entities as they seemed to be
>>> useful when thinking in terms of the 1.0 model. The source and target
>>> are as they appear in the attach received.
>> Yeah those seem to have potential, I stumbled across the "domain" and
>> "incoming"/"outgoing" stuff I've not had a chance to play yet but the
>> ability to establish links with arbitrary AMQP 1.0 containers seems
>> really cool. Is this stuff ultimately intended to replace the
>> "traditional" link/bridge federation model?
>
> Listing the incoming and outgoing entities is useful even for a 
> standalone broker. Creating them offers a mechanism similar to the 
> links/bridges in 0-10.

Yeah I can see why listing the incoming/outgoing entities would be 
useful for a standalone broker - presumably being able to navigate from 
a subscription queue to a link would enable me to see say the 
filters/selectors and other properties of the link (I'd be really nice 
to be able to see what filters/selectors were present).

But my question was more about technical direction, as I say for AMQP 
1.0 systems is this ultimately intended to become the primary way to 
enable federation? TBH from what I've seen it looks quite neat, but I'll 
need to play a bit more to compare and contrast that approach with 
"traditional" federation.



>
>>> * Reply-to
>>>
>>> The qpid::messaging library doesn't particularly restrict the value of
>>> the reply-to in the message itself[4].
>>>
>>> If it can be interpreted as of the form <name>/<subject>, then the
>>> address object returned will have both the name and subject set to the
>>> respective substrings. Otherwise the name will contain the full 
>>> reply-to.
>>>
>>> If the application takes that address and uses it directly as an
>>> address for creating a sender, the name is used as the source or
>>> target address for the link and if a subject is specified it will be
>>> used to set a filter.
>>>
>>> I believe a common usage will have reply-to simply contain a node name
>>> (often for a temporary queue). That pattern, essentially the
>>> client-server example, works against several brokers and also the
>>> dispatch router[5]. It is also familiar from JMS.
>>>
>>> However if some other specific scheme/format is used, the application
>>> would responsible for interpreting the address and e.g. establishing
>>> separate connections etc if that behaviour is desired.
>> I'll need to have more of a play with Reply-To but the thing I was
>> really trying to get at was that given the peer to peer model of AMQP
>> 1.0 it is (presumably) possible to set a Reply-To to the complete unique
>> address of the sender (including host:port part) and the receiver could
>> presumably directly address the sender without necessarily returning via
>> intermediaries. That scenario seemed awkward for a connection oriented
>> paradigm such as qpid::messaging or JMS because the host:port part might
>> be an entirely different connection to the one it has used to connect to
>> say a broker.
>
> With qpid::messaging you can set a reply-to of almost any form on a 
> message as well as getting it off a received message.
>
> However passing the reply-to directly back to createReceiver() will 
> result in an attach being sent for a receiver link with the source set 
> to the node name in the address. It can't sensibly do anything else.
>
> If you want to use other schemes, such as something that identifies a 
> different process to connect to or whatever, then the application 
> needs to handle the reply to according to whatever scheme is in use.

So if I'm interpreting you correctly here; if say I'm a sender on 
localhost on port 5673 I might want to set a fully qualified reply to 
AMQP address of say:

amqp://localhost:5673/response

Now a consumer could in theory happily send a response directly to that 
address because of the peer to peer nature of AMQP 1.0, but I think what 
you are saying is that in qpid::messaging if such a ReplyTo was send 
there is no automagic mechanism given such an Address that if you pass 
it to createReceiver() it will cleverly create a Connection/Session. 
Similarly then I guess that if you got such an address on a 
|*getJMSReplyTo 
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#getJMSReplyTo%28%29>*()|

you wouldn't simply be able to do |*send 
<http://docs.oracle.com/javaee/5/api/javax/jms/MessageProducer.html#send%28javax.jms.Destination,%20javax.jms.Message%29>*(Destination

<http://docs.oracle.com/javaee/5/api/javax/jms/Destination.html> destination, 
Message 
<http://docs.oracle.com/javaee/5/api/javax/jms/Message.html> message)| 
on a MessageProducer. So what you'd have to do would be to parse the 
ReplyTo to see if it had an amqp://<host>:<port> part and if it did 
create the necessary connection/session stuff and use the node name part 
for the actual qpid::messaging Address.

So that makes some level of sense, but that's what I was meaning when I 
said that this scenario (which seems quite reasonable given the peer to 
peer nature of AMQP 1.0) was slightly *awkward* for connection oriented 
APIs. Presumably if say Proton Messenger was the consumer you'd simply 
be able to take the ReplyTo in this case and use that as as the address 
when you do pn_message_set_address(message, address); So I think that 
there is some scope for confusion/complexity as to *exactly* what an 
address means as the interpretation is subtly different between 
connection oriented and message oriented paradigms.


I guess that if you're always using node names on the container that the 
consumer is connected to then everything is OK, but if not isn't there 
scope for things getting a bit interesting - surely a consumer is 
decoupled from a producer so it doesn't necessarily know how it has set 
its reply to (as a node or a fully qualified address).



>
> [...]
>>> * Shared subscriptions
>>>
>>> The 'shared' subscription capability is described in the AMQP 1.0 
>>> readme.
>> Yeah I did notice it there, my problem was working out what the syntax I
>> needed to use was.
>>
>>> Note that it is a qpidd specific extension. At present the
>>> subscription queue will be autodeleted unless you set the link to be
>>> durable or reliable.
>> Ah I didn't realise that the reliable flag would help me. So would that
>> be something like reliability:|||at-least-once| in the link properties?
>
> Yes

Just to confirm that doing:

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.fanout; {node: {capabilities: [shared]}, link: {name: test-link, 
reliability: at-least-once}}"

Does indeed set autoDelete to false on the subscription queue.

However that raises another question in my mind (sorry for being a pain 
in the butt!!!)

In AMQP 0.10 as far as I was aware the default reliability was actually 
at-least-once, or at any rate I don't believe that the default was 
unreliable. The reason I'm thinking this was because I always had to 
explicitly do link: {reliability: unreliable} for a couple of use cases 
where I didn't want my consumers to have to acknowledge the messages.

This stuff above *seems* to point to the default reliability in AMQP 1.0 
as being unreliable - is that observation correct, if so there's clearly 
an inconsistency between AMQP 0.10 and 1.0



>
>> So are you saying that other AMQP 1.0 brokers generally don't support
>> shared subscriptions? TBH that seems unfortunate I know that I find them
>> incredibly useful to distribute workload between physical consumers.
>
> They do likely support them, but they either won't expose that 
> capability over AMQP 1.0 yet or they will do so in a different way (I 
> haven't seen any other approach documented or I would have attempted 
> to follow it).
Just because it's not documented doesn't mean it doesn't exist ;-p
If that were the case........




>
>>> Supporting an expiry policy of 'never' should be possible as well.
>>> I'll update the test to note the convention around queue name.
>> What I was getting at in that section of my post was "I couldn't see
>> anything in the qpid::messaging code related to expiry" in other words
>> there was no reference to pn_terminus_set_expiry_policy so (I think)
>> it's additional code that's needed - have I missed something?
>
> Right, there is no support as yet for controlling the expiration 
> policy (right now link scope is assumed). Adding 'never' should not be 
> difficult.
Cool, I guess expiry policy is probably the most *obvious* way do do 
this from the spec (well to me anyway, which probably doesn't count for 
much).




>
> [...]
>> So that's slightly worrying I hadn't clocked that. What I'm not clear
>> about then is how I'd go about mapping the scenario I had with:
>>
>> x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'key1',
>> arguments: {x-match: all, data-service: amqp-delivery, item-owner:
>> fadams}}, {exchange: 'amq.match', queue: 'testqueue', key: 'key2',
>> arguments: {x-match: all, data-service: http-delivery, item-owner:
>> fadams}}]
>>
>> In this AMQP 0.10 case I've got two separate bindings added between
>> amq.match and "testqueue" so messages will be delivered if either of
>> them match.
>>
>> I (naively) thought that multiple filters was the way (though on
>> reflection I can see that all filters must match - how else would my
>> topic plus selector example work the way it did d'Oh).
>>
>> Can you think of a way how the use case I've suggested might be
>> supported (aside from selectors - they are cool, but it'd be good to be
>> able to replicate the legacy headers stuff completely - especially for
>> migration/integration purposes).
>
> Not really, I guess a new filter type of some kind would be one path. 
> However I do think that using selectors is better and should work well 
> even for migration/integration. You could have a headers filter that 
> matched everything (e.g. just amq.match:any) and then specify the rest 
> of the matching logic as a selector and your receiver would behave the 
> same as if it had two headers bindings.

I was thinking about this. So I agree that ultimately using selectors is 
better (and I did do exactly what you say here in my "Bringing it all 
together" example albeit being even more fancy with the addition of 
topic filtering too). So don't get me wrong I really do like message 
selectors, but I also think that the legacy headers stuff really ought 
to be able to support everything the current AMQP 0.10 model can, 
especially as it's *nearly* there. So at the moment the headers stuff 
with a single x-bindings value looks like this.

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: key, descriptor: 
'apache.org:legacy-amqp-headers-binding:map', value: {x-match: all, 
data-service: amqp-delivery, item-owner: fadams}}}}"

Looking in AddressHelper.cpp from what I can make out filters get added 
in AddressHelper::configure and in there there's a block that iterates 
through the filters and for each calls write(filter, i->value);

Unless I'm mistaken that value is a Variant::

Now in the example above it is clearly a Map, but I'm wondering about 
the possibility of extending what gets accepted for
apache.org:legacy-amqp-headers-binding to either:
a) Include another variant of this such as 
apache.org:legacy-amqp-headers-binding:list such that you could have a 
subscription like:

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor: 
'apache.org:legacy-amqp-headers-binding:list', value: [{x-match: all, 
data-service: amqp-delivery, item-owner: fadams}, {x-match: all, 
data-service: http-delivery, item-owner: fadams}]}}}"

In other words the value part is a list of maps. Alternatively

b) For the apache.org:legacy-amqp-headers-binding:map use an x- property 
(perhaps even x-bindings) whose value is a list e.g. something like:

./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor: 
'apache.org:legacy-amqp-headers-binding:map', value: {x-bindings: 
[{x-match: all, data-service: amqp-delivery, item-owner: fadams}, 
{x-match: all, data-service: http-delivery, item-owner: fadams}]}}}}"

You could possibly even use the original x-bindings as the property of 
the filter's "value" Map though the exchange and queue parts wouldn't 
make sense, but it would give a handy place for the key - so, for example:


./drain --connection-options {protocol:amqp1.0} -b localhost -f \
"amq.match; {link: {name: test-link, filter: {name: headers, descriptor: 
'apache.org:legacy-amqp-headers-binding:map', value: {x-bindings: [{key: 
'key1', arguments: {x-match: all, data-service: amqp-delivery, item-owner:
fadams}}, { key: 'key2', arguments: {x-match: all, data-service: 
http-delivery, item-owner:
fadams}}]}}}}"

So as far as I can see that technically doesn't change the semantic of 
the filter as the type is a map, the existing approach for a single 
binding should still work but by interpreting x-bindings you'd 
essentially have the ability to do exactly what you can currently do, 
which seems kind of cool.

I've no idea how much work this would be though - as I say, sorry for 
being a pain :-)




>
> [...]
>>> However we could probably look at ways to (optionally?) make this more
>>> lenient. It certainly is also something that should be highlighted
>>> more prominently in documentation somehow.
>> It would be *really really good* to be able to make this more lenient
>> (would quoted field names be a possibility e.g.
>> 'data-service'='amqp-delivery')?
>
> I've not looked at the parsing code, but I would think something like 
> that should be possible and I agree that it would be useful since AMQP 
> itself places no restrictions on property names.

Pleeeeeaaaaaassssseee :-)

Seriously, finding a way to work through the hyphen issue would be awesome.




>
> [...]
>>> I believe the selector filters are supported by a number of brokers
>>> outside Qpid (ActiveMQ, HornetQ, SwiftMQ).
>> That's useful to know, have you tried many interoperability scenarios?
>
> I've successfully tried selectors against ActiveMQ. The only quirk is 
> that it doesn't actually recognise the filters descriptor, but just 
> recognises the filter name as used by the current qpid JMS over 1.0 
> client. SO the selector shorthand doesn't work, but a fully specified 
> filter will.
>
I'm not quite sure I understand what you mean here, as I understand it 
"selector: " is a qpid shortand, but I *thought* that over the wire that 
got translated to a filter with a descriptor 
apache.org:selector-filter:string so I *thought*

selector: \"data-service='amqp-delivery' and item-owner='fadams'\"}

was shorthand for

filter: {name: selector, descriptor: 
'apache.org:selector-filter:string', value: 
\"data-service='amqp-delivery' and item-owner='fadams'\"}

So are you saying that it accepts

filter: {name: selector, descriptor: 'x-filter-jms-selector', value: 
\"data-service='amqp-delivery' and item-owner='fadams'\"}

Or simply link : {x-filter-jms-selector: \"data-service='amqp-delivery' 
and item-owner='fadams'\"}

I *thought* that in AMQP 0.10 it went something like:

{link:{x-subscribe:{arguments:{x-filter-jms-selector:\"data-service='amqp-delivery' 
and item-owner='fadams'\"}}}}

But that won't work in AMQP 1.0 because the x-subscribe is forbidden.


Actually that reminds me - so in JMS I could specify selectors on the 
link as with qpid::messaging, but the JMS API also allows selectors to 
be specified from the Session |*createConsumer 
<http://docs.oracle.com/javaee/5/api/javax/jms/Session.html#createConsumer%28javax.jms.Destination,%20java.lang.String%29>*(Destination

<http://docs.oracle.com/javaee/5/api/javax/jms/Destination.html> destination, 
String 
<http://java.sun.com/j2se/1.5/docs/api/java/lang/String.html> messageSelector)| 


I probably shouldn't as this, but..... are you suggesting that at the 
moment with Qpid JMS specifying it from the Session will result in a 
different syntax on the link attach? If so will they both work with the 
Qpid brokers?

Sorry for all the crazy questions, there's a lot of subtlety in all of this.

Cheers,
Frase







Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message