qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Godfrey <rob.j.godf...@gmail.com>
Subject Re: AMQP 1.0 Message Encoding - using AMQP typed data vs. Mime content-type / content-encoding (Was (unintentionally I assume): P)
Date Wed, 10 Sep 2014 19:50:57 GMT
Looking at a slightly different aspect of this... recently I added into the
Java Broker/Client the ability to automatically compress messages if they
are above a given size.  Where people send ridiculously large messages this
trade off of CPU time instead of network bandwidth makes sense.

The (de)compression occurs transparently to the end user if they are using
the Java Client.  For this I used the content-type field in 0-9/0-10 to
indicate that the message data was compressed.  The content-type remained
unchanged.  This is the same as things work in HTTP.

I like all the above - it looks exactly like it does in HTTP... it's non
ambiguous... interoperability should be a doddle... etc.

Now, in AMQP 1.0 where the body is a data section I could do exactly the
same... but what if the section is an amqp-value?  I can't just compress
the data because then it would no longer be the same type...  I suppose I
could compress the whole section, and stick that in as data but then in
your case where you had sent a amqp-value with a string but a content-type
of "application/json" how am I going to disambiguate that from the case
where the data was originally opaque.  I suppose I could futz with either
the content-type or content-encoding to start to use some non-IANA defined
values or something... but then I'm pretty much losing the advantage of
interoperability that using the content-type/encoding style gives me...

My thinking (such as it is) was that in the case there was a large message
that I wanted to compress but that it was in an amqp-value section, then I
would compress the encoding then wrap it in a described (with descriptor
org.apache.qpid.gzip-value or something similar) binary.  Not particularly
nice, but at least non-ambiguous. (There's another way, which is to add a
completely new message format, but I'm going to ignore that for now).

To be fair, the amqp type system encoding does have some advantages -
especially when you want to wrap / chain multiple transformations.
 Thinking of message encryption and signing, S/MIME is particularly
horrible and would probably be much more elegantly expressed using the AMQP
type system... though whether that makes sense from an interoperability
point of view is perhaps doubtful (though S/MIME itself isn't quite what
we'd really want anyway). Mutli-part messages are also hideous in MIME
style and would be more elegant as an AMQP sequence.

Anyway this is probably all a bit off topic.

What is probably most needed is to have guidelines for API authours about
how to encode / decode application messages to/from AMQP, and what
behaviour should be provided by the library versus that which one might
expect the user application to perform.

Part of the issue here, I think, is that we are not terribly (at all?)
clear on what each API might do with a particular input from the client
application, and how that might be treated by the library on the receiving

-- Rob

On 10 September 2014 21:02, Rob Godfrey <rob.j.godfrey@gmail.com> wrote:

> On 10 September 2014 20:54, Fraser Adams <fraser.adams@blueyonder.co.uk>
> wrote:
>> Sorry I missed this one,
>> Comments inline.
>> On 10/09/14 18:08, Rob Godfrey wrote:
>>> I think it's inconsistent to use a mechanism (content type) intended for
>>> opaque data on data which is not opaque... only in a couple of special
>>> cases (an amqp-value section containing a string or binary value) does it
>>> make sense.
>> Hmm I could interpret opacity in a number of ways. For example the data
>> that we are talking about is JSON, now that happens to be using a String as
>> a container format but the fact that it is JSON is *entirely opaque* to the
>> AMQP type system hence why IMHO it entirely makes sense to pass additional
>> metadata to detail how it should be interpreted.
>> Now described might be OK, but I assert that following common design
>> patterns is a very good idea and the pattern that I suspect 99 per cent of
>> the population would instinctively choose would be the pattern employed by
>> HTTP.
> OK, to reveal my personal preference - in this case I would avoid using
> the AMQP type system, and I would be using the content-type and sticking
> the json string in a data section.  I think this is much more sensible than
> using a described type and trying to invent yet another way of encoding
> things.
>>    In general the question is who is the field aimed at - the
>>> AMQP library or the client application? I think the content-type should
>>> actually be interpreted by the AMQP library - in the same way that I
>>> would
>>> expect the AMQP types to be.
>> I'd probably have assumed it would be more the client application, but no
>> matter I think that if you expect the AMQP library you actually strengthen
>> my argument. If I want to do what I've suggested I can have my JavaScript
>> library take a JavaScript object, set the content-type and use that to tell
>> the recipient how to deserialise it transparently passing a JavaScript
>> Object to the consumer client.
>> That's fine and grand except that you seem to be telling me that
>> content-type can't actually be used to do *anything useful*!!!
> No - I'm telling you you should use the data section type and not the
> amqp-value section type :-)
>>> I think it "appears" to be more convenient to set the content-type on the
>>> properties field for a string value because we perceive that the client
>>> library would be able to understand AmqpValue(String("{ }")) but not
>>> AmqpValue(Described(Symbol("org.json"),String("{ }"))) and that the
>>> content-type would be made available to the end user of the API for them
>>> to
>>> then be able to understand that this is Json, and not intercepted by the
>>> client library.
>> No what I'm saying is that it "appears" to be more convenient because
>> there's about half a billion AJAX based web-pages that set
>> content-type=application/json and send the JSON string in the HTTP Message
>> Body.
>> As it happens my JavaScript Library can do this fairly trivially with
>> message.body = new proton.Data.Described({"some": "json"},
>> 'application/json');
>> And it's roughly as easy to add some code to the serialisation of
>> Described to call toJSON() when it encounters a descriptor of
>> "application/json" as it would be to add something around content-type
>> (though other APIs might have more fun :-)) but although in implementation
>> terms it's roughly as easy it seems an awful lot weirder and less intuitive
>> than
>> message.setContentType('application/json');
>> message.body = {"some": "json"};
> And I'm saying I agree - but that this should go in a data section not an
> amqp-value section containing a amqp string object.  I've no idea how
> Messenger / Proton API works around these things, but in general I would
> say the 90% use case should be the data section rather than using the amqp
> type system to encode your data.  I agree on using the more widely used
> HTTP / MIME style for describing content.
>>     However that's not necessarily the case (the current JMS
>>> client doesn't expose the content type IIRC, and may use it for switching
>>> behaviours on opaque data - e.g. for object messages).
>> That's a reasonable point, though in that case I'd suggest that the
>> content type should be an extension type "application/x-amqp-map" and
>> shouldn't be interfering with any IANA registered content types.
>>> More generally I think we collectively (in Qpid and in the wider AMQP
>>> world) need to decide when/if we use the AMQP type system to encode data,
>>> versus using MIME types.
>> For the most part I'm fairly happy with the AMQP type system, though I
>> can recall Gordon and Rafi had quite an exciting debate about the relative
>> merits of JSON versus the AMQP type system.
>> I wasn't specifically looking to rekindle that in this discussion, this
>> was more about using content-type in a manner that I would assert the *vast
>> majority* of the broader technical community would intuitively expect it to
>> be used.
> Basically I'm saying we should use things consistently.  If you want to
> use HTTP / MIME style (which I think is better for pretty much everything
> except for maybe a few cases) then just stick your message body data in a
> data section.  Only where there is some specific advantage to be gained
> over the HTTP / MIME encoding style would I consider using the AMQP type
> system for your message data.
>>> If I want to start compressing data as I generally send large messages,
>>> should I send my data with the content-encoding as gzip, or should I
>>> define
>>> a new described type for compressed data.  If the former - what if the
>>> large message I want to send is a map?
>> well I think that I'd probably follow an Internet style pattern along the
>> lines of:
>> message.setContentType('application/x-amqp-map');
>> message.setContentEncoding('gzip');
> Yep - which would all go in a data section... which is in compliance with
> the spec.
>>> My general advice is to choose one mechanism or the other and not to
>>> confuse the two.
>>> Of course you have an inconsistency to resolve if somebody sets an AMQP
>>>> string body with content-type='application/json; charset="utf-16"'
>>>> Could we say: You MAY use content-type with an AMQP string body but it
>>>> MUST NOT not have a charset and MUST be ignored by the sender. It MAY be
>>>> used as a decoding hint on the receiver.
>>>>  The spec is the spec, and now ratified at ISO, so the wording there is
>>> not
>>> going to change.  We could interpret the spec more loosely, but - as
>>> above
>>> - i think that ultimately that will lead to more confusion.
>>> -- Rob
>> Yes the spec is the spec, but in my opening paragraph I asserted that
>> opacity is in the eye of the beer-holder and the fact that this stuff is
>> JSON is very definitely opaque from the perspective of AMQP in the
>> scenarion that I've been describing.
> Yep - so the appropriate section type is "data"...  :-)
> -- Rob
>> Frase
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message