Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
Received-SPF: pass (nike.apache.org: domain of gsim@redhat.com designates
 209.132.183.28 as permitted sender)
Message-ID: <520509C0.7050304@redhat.com>
Date: Fri, 09 Aug 2013 16:24:48 +0100
From: Gordon Sim <gsim@redhat.com>
Organization: Red Hat UK Ltd, Registered in England and Wales under Company
 Registration No. 3798903, Directors: Michael Cunningham (USA), Mark Hegarty
 (Ireland), Matt Parsons (USA), Charlie Peters (USA)
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130625 Thunderbird/17.0.7
MIME-Version: 1.0
To: users@qpid.apache.org
Subject: Updated notes for AMQP 1.0 support in qpidd and qpid::messaging (was
 Re: Creating a queue and bindings from an address in qpid.messaging / AMQP
 1.0)
References: 
 <CAOhBXQ5+ZiaDAz7G0ww4TuYaPfbtfnePrMLHyvh2Cfo5LqoB3A@mail.gmail.com>
 <520266D8.1030301@redhat.com>
 <CAOhBXQ4DWmgYrvQnLdJGGo5UzXKnMwYBQ=-tVx+TORKyvDHCfg@mail.gmail.com>
 <52035A82.60209@redhat.com>
 <CAOhBXQ4U9923Ut+zBiw1X_oSEWdzGyD6TizN3CBp+_hHdcBgvQ@mail.gmail.com>
 <52038626.6090202@redhat.com>
 <CAOhBXQ7TsSWYknq3T6zhVXZ=SSe+V5CZ8Ci1VK-jg4XPE0auuA@mail.gmail.com>
 <5203AC72.9090408@redhat.com>
 <CAOhBXQ5N+aK20WOawJsNp2G6D-NpWq1o12K5htz7+m7y+9FvkQ@mail.gmail.com>
 <5203B95D.7080705@redhat.com>
 <CAOhBXQ4F-cX6rzqpudW9uVX+MP2CZdtczMzunJETxU7Ur_LOTg@mail.gmail.com>
 <5203C3E0.6020606@redhat.com>
 <CAOhBXQ7HVP_9FU-ppGV4ToBgXb5C=Z5WvarGu5M7Z=W9Eagj2A@mail.gmail.com>
In-Reply-To: 
 <CAOhBXQ7HVP_9FU-ppGV4ToBgXb5C=Z5WvarGu5M7Z=W9Eagj2A@mail.gmail.com>
Content-Type: multipart/mixed;
 boundary="------------090107070907030207000809"

--------------090107070907030207000809
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 08/08/2013 05:46 PM, Jakub Scholz wrote:
> Are these AMQP 1.0 related changes documented somewhere?

Not properly yet. I've been sending out some rough notes to the user 
list (updated version attached). I need to spend some time figuring out 
where and how to make the information more accessible.

Perhaps while I'm trying to find that time, I should just check it in 
alongside the other READMEs in the cpp tree? Would anyone object?


--------------090107070907030207000809
Content-Type: text/plain; charset=UTF-8;
 name="AMQP-1.0.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="AMQP-1.0.txt"

AMQP 1.0 support for the qpid::messaging API
--------------------------------------------

The guiding principle has been to allow applications written to the
qpid::messaging API to speak AMQP 1.0 in a clear and natural way, to
avoid tying its use to any particular broker. The 0-10 support will of
course remain unaltered.

The API is itself fairly simple. It is in the address syntax and
specifically the more detailed options that much of the complexity of
the mapping lies.

Reply-To addresses and temporary queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There has been one minor change to the way the API itself works over
1.0. This does not affect existing 0-10 use however. The change
involves the creation of temporary queues (or topics), for retrieving
replies in a request-response pattern for example.

Over 0-10, the Address will convert a node name starting with a '#'
character by inserting a UUID. This works well for 0-10 where the name
is chose by clients and must be unique. This transformation of the
name is done when constructing an Address from a single address string
(rather than from its constituent parts). The modified name could then
be accessed via Address::getName().

Over 1.0 however the name for such nodes is determined by the
server. In this case the name assigned needs to be communicated back
to the application when the attach succeeds. To handle that a new
accessor - getAddress() - has been added to both Sender and Receiver.

In order to keep backward compatibility for 0-10, the Address
constructor still does the transformation, but applications that want
to be able to switch to 1.0 should use these new accessors to obtain
the correct address for setting reply-to on any request messages they
send. (This new approach will work for both 0-10 and 1.0).

Connections, Session and Links
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The protocol used is selected at runtime via the 'protocol' connection
property. The recognised values are 'amqp1.0' and 'amqp0-10'. AMQP
0-10 is still the default and the 1.0 support is only available if the
required module is loaded. There is no failover support for 1.0
connections yet[1].

The SASL negotiation is optional in AMQP 1.0. If no SASL layer is
desired, the sasl_mechanisms connection option can be set to NONE.

AMQP 1.0 can be used over SSL, however the messaging client does not
at this stage use an AMQP negotiated security layer for that
prupose. Peers must expect SSL on the port being used (either
exclusively or by being able to detect an SSL header).

The container id that the client advertises when establishing the
connection can be set through the connection-id/connection_id property
on the connection. If not set a UUID will be used.

Transactional sessions are not yet supported[2].

The creation of senders or receivers results in the attaching of a
link to the peer. The details of the attach, in particular the source
and/or target, are controlled through the address string.

Addresses
~~~~~~~~~

The name specified in the address supplied when creating a sender or
receiver is used to set the address of the target or source
respectively.

If the subject is specified for a sender it is used the default
subject for messages sent without an explicit subject set.

If the subject is specified for a receiver it is interpreted as a
filter on the set of messages of interest. If it includes a wildcard
(i.e. a '*' or a '#') it is sent as a legacy-amqp-topic-binding, if
not it is sent as a legacy-amqp-direct-binding.

When the name of the address is (or starts with) '#', the dynamic flag
is set on the corresponding source or target and the
dynamic-node-properties are populated based on the node
properties. Note that when the dynamic flag is set the address should
not be specified. However due to PROTON-277[3], I have to set the
address to something in order to work at all against another proton-c
based peer, such as qpidd (so I set it to '.'). This can be resolved
as soon as the proton bug is fixed.

As mentioned above in discussing the changes around reply-to
addresses, AMQP 1.0 doesn't allow on demand creation of nodes with a
client specified name. However, I have defined a special extension
capability for the c++ broker that will allow 'create' behaviour that
is similar to that supported over 0-10. That is, it will create a node
with the name specified by the client if it does not already exist. I
see this as a temporary measure to help transition situations that
rely on create policy at present. It is non-standard however and the
recommendation would be to look for ways to avoid relying on
that. Hopefully at some point a standard management mechanism will
provide better alternatives.

If the addressed node is to be created on demand - either through use
of '#' as the name, or through the create policy - the node properties
are sent as dynamic-node-properties on the source or target. These can
be specified in a nested map within the node. Additionally the durable
and type properties in the node map itself are sent if
specified. There is at present also a translation from the 0-10 style
x-declare in the node whereby all the fields specified in it are
included as if they had been listed in properties, the arguments map
is flattened into this.

The x-bindings property is not currently supported for AMQP 1.0 in
nodes or links. This has really been a question of priorities rather
than ruling out any mapping. Though I don't think a generic binding
solution is appropriate for the 1.0 implementation, I'm eager to here
of use cases that would be affected here and see how best to address
them.

For receivers, where the node is an exchange, the binding can be
controlled through the filters for the link. These can be specified
through the filter property in the link properties specified in the
address. The value of this should be a list of maps, with each map
specifying a filter through key-value pairs for name, descriptor (can
be specified as numeric or symbolic) and a value.

The subject in the address and the selector property provide shorter
ways of specifying the most common use cases.

Subscriptions can also be shared between receivers. To do this, the
'shared' capability should be requested and the receivers should all
specify the same link name. Note that due to fact that link and
container id combination must be unique, a subscription cannot be
shared between receivers on the same connection (a problem for JMS 2.0
for example). This will likely be rectified in the future by allowing
the link to be identified via the link properties instead of the name.

The delete policy is not supported over 1.0 at all. There isn't really
any obvious way to map this. It was never terribly well thought
through even for AMQP 0-10 in my view. I believe what is more
important is expanding/revising the scope of nodes.

The 1.0 specification defines the following policies in connection
with nodes created in response to establishing the link:
delete-on-close, delete-on-no-links, delete-on-no-messages or
delete-on-no-links-or-messages. For dynamically created nodes -
i.e. those with name '#', which will result in the dynamic flag being
set on the source or target, the default is delete-on-close but the
others can be chosen by specifying a lifetime-policy in the properties
map in the node options. The valid values are delete-on-close,
delete-if-unused, delete-if-empty and delete-if-empty-and-unused and
these are sent as the AMQP 1.0 specified policy types (as above).

There is some support for the assert option, though it is not entirely
equivalent to the 0-10 based mechanism. Over AMQP 1.0 it uses the
source or target capabilities. The client will set the capabilities it
desires, the broker (or other peer) will set the capabilities it can
offer and if the assert option is on, then the client will ensure all
the capabilities it requested are indeed supported. If assert sis
specified it will also ensure that any requested filters are actually
in place.

The specific capabilities are yet to be standardised. The capabilities
sent by the client can be controlled through a nested list within the
node map. Note that capabilities are simply strings (symbols in 1.0 to
be precise), not name value pairs.

If durable is set in the node properties, then a capability of
'durable' will be requested, meaning the node will not lose messages
if its volatile memory is lost, and requested if the durable property
within the node map is present in the address.

If the type is set, then that will also be passed a requested
capability e.g. 'queue' meaning the node supports queue-like
characteristics (stores messages until consumers claim them and
allocates messages between competing consumers), 'topic' meaning the
node supports classic pub-sub characteristics.

The 'browse' mode is not yet supported due to PROTON-139[4]

The Message class and the AMQP 1.0 message format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The message-id, correlation-id, user-id, subject, reply-to and
content-type fields in the properties section of a 1.0 message can all
be set or retreived via accessors of the same name on the Message
instance. Likewise for the durable, priority and ttl fields in the
header section.

An AMQP 1.0 message has a delivery-count field (within the header
section) for which there is no direct accessor yet on the Message
class. However if the value is greater than 0, then the
Message::getRedelivered() method will return true. If
Message::setRedelivered() is called with a value of true, then the
delivery count will be set to 1, else it will be set to 0. The
delivery count can also be retrieved or set through the
'x-amqp-delivery-count' pseudo-property.

The application-properties section of a 1.0 message is made available
via the properties map of the Message class. Likewise on sending a
message the properties map is used to poplulate the
application-properties section.

There are other fields defined in the AMQP 1.0 message format that as
yet do not have direct accessors on the Message class. These are made
available on received messages via special keys into the properties
map, and can be controlled for outgoing messages by setting the
properties with the same keys.

The format for the keys in general is x-amqp-<field-name>.  The keys
currently in use are: x-amqp-first-acquirer and x-amqp-delivery-count
for the header section, and x-amqp-to, x-amqp-absolute-expiry-time,
x-amqp-creation-time, x-amqp-group-id, x-amqp-qroup-sequence and
x-amqp-reply-to-group-id for the properties section.

In addition the delivery- and message- annotations sections are made
available via a nested maps with key x-amqp-delivery-annotations and
x-amqp-message-annotations respectively.

AMQP 1.0 support in qpidd
-------------------------

To enable 1.0 support in qpidd, the amqp module must be loaded. This
allows the broker to recognise the 1.0 protocol header alongside the
0-10 one.

The broker can accept connections with or without and underlying SASL
security layer as defined by the 1.0 specification. However if
authentication is turned on -- i.e. auth=yes -- then a SASL security
layer MUST be used.

There is as yet no AMQP 1.0 specific integration with the ACL. This
means for example that you cannot prevent an authenticated user for
subscribing to a particular queue, or limit the number of 1.0 based
connections opened up by such a user. This will be rectified in a
future release[5].

The broker allows links in both directions to be attached to queues or
exchanges. The address in the source or target is resolved by checking
whether it matches the name of a queue or an exchange. If there exists
a queue and an exchange of the same name, the queue is used but a
warning is logged.

If the node is an exchange, then an outgoing link (i.e. messages to
travel out from broker) will cause a temporary, link-scoped queue to
be created on the broker and bound to the exchange.

Outgoing links may have a filter set on their source. The filters
currently supported by the broker are 'legacy-amqp-direct-binding',
'legacy-amqp-topic-binding', 'legacy-amqp-headers-binding',
'selector-filter' and 'xquery-filter' as defined in the registered
extensions.

The 'selector-filter' is supported for all nodes.

The 'legacy-amqp-direct-binding' and 'legacy-amqp-topic-binding'
filters are supported when the node is a topic, direct or xml exchange
- where it is used as the binding key when binding the subscription
queue to the exchange - and when the node is a queue, where the
subscription will then skip over messages that do not match that
subject. In the case where 'legacy-amqp-topic-binding' was requested
for a direct- or xml- exchange it will be interpreted as a direct
binding (and this will be indicated via the filters set on the attach
sent by the broker).

The 'legacy-amqp-headers-binding' is supported where the node is a
headers exchange and implemented as a binding to the exchange.

The 'xquery-filter' is supported where the node is an xml exchange and
provides the xquery to use in the binding. This can be used in
conjunction with a 'legacy-amqp-direct-binding' in order to control
the binding ley used for the binding.

If the dynamic flag is set on the source or target, then the
dynamic-node-properties are inspected to determine the characteristics
of the node to be created. The properties that are recognised are the
same as accepted via the create qmf method; the 0-10 defined options
durable, auto-delete, alternate-exchange, exchange-type and then any
qpidd specific options (e.g. qpid.max-count). The broker supports all
standard values for the standard lifetime-policy property also.

The supported-dist-modes property as defined by the 1.0 specification
is used to determine whether a queue or exchange is desired (where the
create method uses the 'type' property). If 'move' is specified a
queue will be created, if 'copy' is specified an exchange will be
created. If this property is not set, then a queue is assumed.

Messages sent over AMQP 0-10 will be converted by the broker for
sending over AMQP 1.0 and vice versa.

The message-id, correlation-id, userid, content-type and
content-encoding all map fairly simply in both directions between the
properties section in 1.0 and the message-properties in an 0-10
header. Note however that in 0-10 a message-id must be a UUID, and in
translating to 0-10 from 1.0 this field will be skipped unless it is a
valid UUID.

Likewise the priority directly between the field in the header section
of a 1.0 message and the corresponding field in the
delivery-properties of an 0-10 message. The durable header in a 1.0
message is equivalent to the delivery-mode in the delivery-properties
of an 0-10 message, with a value of true in the former being
equivalent to a value of 2 in the latter and a value of false in the
former equivalent to 1 in the latter.

When converting from 0-10 to 1.0, the reply-to will be simply the
routing-key if the exchange is not set, or if it is the reply-to
address for 1.0 will be composed of the exchange and if specified the
routing key (separated by a forward slash).

When converting from 1.0 to 0-10, if the address contains a forward
slash it is assumed to be of the form exchange/routing key. If not it
is assumed to be a simple node name. If that name matches an existing
queue, then the resulting 0-10 reply to will have the exchange empty
and the routing key populated with the queue name. If not, but the
name matches and exchange, then the reply to will have the exchange
populated with the node name and the routing key left empty. If the
node refers to neither a known queue nor exchange then the resulting
reply to will be empty.

When converting from 0-10 to 1.0, if the 0-10 message has a non-empty
destination, then the subject field in the properties of the 1.0
message is set to the value of the routing-key from the
message-properties of the 0-10 message. In the reverse direction, the
subject field of the properties section of the 1.0 message is used to
populate the routing-key in the message-properties of the 0-10
message.

The destination of a 0-10 message is used to populate the 'to' field
of the properties section when converting to 1.0, but the reverse
translation is not done (as the destination for messages sent out by
the broker is controlled by the subscription in 0-10).

The application-properties section of a 1.0 message is converted to
the application-headers field in the message-properties of an 0-10
message and vice versa.

No translation of the message body is done at this point[6].

The broker recognises particular capabilities for source and
targets. It will only include a given capability in the attach it
sends if that has been 'requested' in the attach sent by the peer to
trigger the establishment of the link. This gives the peer some means
of ensuring their expectations will be met.

The 'shared' capability allows subscriptions from an exchange to be
shared by multiple receivers.

The 'durable' capability will be added if the queue or exchange
refered to by the source or target is durable. The 'queue' capability
will be added if the source or target references a queue. The 'topic'
capability will be added if the source or target references an
exchange. If the source or target references a queue or a direct
exchange the 'legacy-amqp-direct-binding' will be added. If it
references a queue or a topic exchange, 'legacy-amqp-topic-binding'
will be added.

The 'create-on-demand' capability is currently offered as a temporary
extension offering some continuity from the commonly used 'create'
policy in the messaging client. If set in the client and the named
node does not exist, it will be created using the
dynamic-node-properties as when the dynamic flag is set (see above).

[1] https://issues.apache.org/jira/browse/QPID-4708

[2] https://issues.apache.org/jira/browse/QPID-4710

[3] https://issues.apache.org/jira/browse/PROTON-277

[4] https://issues.apache.org/jira/browse/PROTON-139

[5] https://issues.apache.org/jira/browse/QPID-4712

[6] https://issues.apache.org/jira/browse/QPID-4711


--------------090107070907030207000809
Content-Type: text/plain; charset=us-ascii


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
--------------090107070907030207000809--