qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: [c++] Status of the AMQP 1.0 work
Date Wed, 10 Apr 2013 11:38:17 GMT
Hi Gordon,
Comments inline below.

Frase

On 09/04/13 19:07, Gordon Sim wrote:
> On 6 April 2013 10:40, Fraser Adams <fraser.adams@blueyonder.co.uk> wrote:
>>
>> "
>> The x-bindings property is not currently supported for AMQP 1.0 in
>> nodes or links. This has really been a question of priorities rather
>> than ruling out any mapping. Though I don't think a generic binding
>> solution is appropriate for the 1.0 implementation, I'm eager to here
>> of use cases that would be affected here and see how best to address
>> them.
>> "
>>
>> I use the headers exchange almost
>> exclusively in my operational system and my Addresses are of the form:
>>
>> string address = "testqueue; {create: receiver, node: {x-declare:
>> {arguments: {'qpid.policy_type': ring, 'qpid.max_size': 500000000}},
>> x-bindings: [{exchange: 'amq.match', queue: 'testqueue', key: 'data1',
>> arguments: {x-match: all, data-service: amqp-delivery, item-owner:
>> fadams}}]}}";

> Does each consumer create its own queue?
Sort of. Each "logical" consumer does, but a logical consumer may 
comprise several physical nodes, they all use the same queue which 
allows distribution of messages across the physical nodes giving the 
ability to scale out quite easily.
>   I actually think this will be both
> simpler and more powerful over 1.0 as the plan is to allow you to subscribe
> to an exchange (which you can already do) using a selector.
"Subscribing to an exchange" is a term that makes me shudder from past 
experience.

I'll try to explain, my system is perhaps a little unusual. I've got a 
pretty large federated topology with lots of *real time* C++ producers 
at one end of the topology and something of a mixed economy of mainly 
Java and C++ (and a few Python) consumers mostly at the "right hand 
side" having passed through some intermediate "data consolidating" 
brokers. I've also got a few higher rate consumers subscribing directly 
off the intermediate brokers. The mean message size tends to be 
reasonably large, something like 50K but the tails are long and some 
messages might be tens of MB.

Hopefully you noticed what I had in the x-declare of the example 
Address. I'm using circular queues for a reason - high rate real time 
producer, and we generally actually use 2GB queues to give moderate 
elastic buffering given data rates and message size.

I've got nothing against the logical "subscribing to an exchange" 
pattern and it is pretty common to see addresses simply expressing the 
topic, but as you are aware "under the hood" this creates an autodelete, 
exclusive, reject policy, default size queue!! I can't live with any of 
those characteristics let alone all of them :-)

Indeed when I said it makes me shudder I'm actually thinking of our past 
federation experiences, as you know static and dynamic routes are more 
explicitly described as static and dynamic /exchange/ routes, because 
they logically allow exchanges to be connected together. This is 
conceptually very much like one exchange subscribing to another 
exchange, but again they transparently create a queue almost completely 
unsuitable for my purpose :-( We've had to use queue routes simply 
because I can control the queue config. You wouldn't believe how fast 
default size reject queues cause links to "blow" if one of my consumers 
misbehaves, which clearly results in total data loss to all my consumers 
not just the naughty one - not ideal.

I'd have *really* liked to be able to combine the rather elegant dynamic 
binding propagation of dynamic routes with the ability to specify the 
configuration of the queue/buffering used by the link with a great deal 
more precision than the defaults.

Hopefully that background give some insight into my "subscribing to an 
exchange" nerves :-/



Beyond that again if you notice from the example Address I use the 
headers exchange pretty much universally. To be fair my architecture is 
pretty much a publish/subscribe architecture and in the early days I 
started with topics, but to be honest the largely static "data model" 
and fairly inflexible wild-carding caused us to move to the headers 
exchange pretty quickly. It's a lot easier to add new "data types" for 
subscribing to with the headers exchange.

It hadn't escaped my notice that the bindings stanza was "x-bindings" 
but as far as I've been able to work out that's the only way to create 
bindings between the headers exchange and a named queue, indeed I 
thought that x-bindings was the only way to create bindings between say 
a user created topic exchange and a queue, though I may be wrong there.

If you can suggest how to say subscribe to a headers exchange without 
using x-bindings that'd be neat, it'll be something I've totally missed.

Re "using a selector " I have to say that I'm definitely a big fan of 
message selectors. It's worth pointing a few things out though. I 
suspect that had message selectors been available when I started out 
with my system I might well have used the topic exchange (with a named 
queue!) plus message selectors. I've used JMS a lot in the past and 
that's at pattern that worked well and was very flexible, however as I 
said earlier the topic exchange on its own didn't cut the mustard for my 
needs so my producers describe their data purely by means of 
properties/headers.

I guess you know this already but message selectors tend to behave as 
filters, so as I understand it messages would pass from exchange to 
queue and the selector would apply a filter to what gets passed off the 
queue to a consumer. I guess that to my way of thinking selectors 
provide most benefit in the realm of "fine grained" filtering in that 
they allow quite sophisticated boolean expressions to be applied.

Now in my scenario quite clearly message selectors will be a boon and 
offer some really useful fine grained control (I've wanted to be able to 
do "not" for ages - you might just about recall one of my very early 
posts on the headers exchange asking about a not operator). However I 
think that message selectors are best applied *in addition to* a 
subscription, without that loads of messages get delivered onto the 
queue which the consumer isn't interested in, which a) fills up the 
queue unnecessarily and b) overworks the message selector logic (which I 
suspect is more expensive than the exchange selection logic). In my case 
in particular producers deliver quite a diverse range of data at very 
high rate so filling up consumer queues unnecessarily would be pretty 
bad news.

So in précis while I really like message selectors I'd want them *in 
addition to* headers bindings and good granularity of queue control (I 
don't ask for much).

Another point about selectors. As I say I'm excited about them, but they 
are a new feature in the C++ broker (should be in 0.22 is that 
correct?). Andrew has done some great work so far, but he has very much 
said that the initial priority is on features and not interoperability. 
I'm fine with that initially, but you will undoubtedly have gathered 
from many of my posts that I'm really keen on maximum interoperability 
and cohesion. I'd really like to see the C++ message selectors 
ultimately become completely interoperable with JMS Message selectors. 
I've got no problem with an "x-selector" heading in the Address string 
but I'm really meaning at an API level too. I'd like to see JMS message 
selectors from JMS Clients propagate to the C++ and Java brokers and I'd 
like to see qpid::messaging extended to allow a JMS style message 
selector string to by passed to the createReceiver() call on Session. 
Clearly it's early days for C++ broker message selectors, but if that 
strategy wasn't part of the roadmap I'd be pretty uncomfortable.

>   The key then
> (if I understand the usecase) would be to ensure that the subscrption had
> the right lifecycle (i.e. you could reconnect to it without loss of
> messages while disconnected),
Well the lifecycle aspects are certainly part of it, hopefully my 
comments above have explained the other aspects of the set of use cases 
that are fairly critical to me.


>> One thing that's not clear from your write up is whether the x-bindings
>> property you are talking about is in the messaging client, or whether it is
>> the underlying x-bindings property that gets passed to the broker. I'm
>> guessing the latter which would also affect AddressStrings created by JMS
>> consumers and via QMF? (Although I use the C++ broker most of my consumers
>> use JMS).
>>
> It is in the messaging client. The x-bindings (and x-declare, x-subscribe),
> map directly to 0-10 commands. There is no analogous mechanism for
> *generic* binding operations in 1.0 - i.e. the ability to bind almost
> anything to anything, regardless of the node you are sending to or
> receiving from. However when you 'subscribe' to an exchange node, the
> broker can infer bindings from the link attachment, generally through the
> specification of a filter. In your case it sounds like that might be the
> applicable pattern over 1.0 and given the matching is based on examination
> of headers it could even be a selector.
>
I guess I have to admit that I don't understand what you are saying 
entirely, to be honest one common theme that seems to be building in my 
mind is that AMQP 1.0 seems somewhat to be a bit of a retrograde step 
over AMQP 0.10. I'm sure my current lack of understanding is making my 
initial assumption a tad unfair but would you be able to summarise the 
"compelling features" that AMQP 1.0 offers over AMQP 0.10 (aside from 
AMQP 1.0 having been ratified by OASIS - that's probably a good thing, 
but how much "design by committee" goes on at OASIS? is it closer to 
IETF or closer to ITU-T?)

So knee jerk comments aside :-) Re "It is in the messaging client. " 
does that statement imply then that it remains possible to create the 
queues and bindings to the headers exchange via QMF "create" calls to 
the Broker? If so then that's something positive, but I'm then confused 
by the statement "There is no analogous mechanism for *generic* binding 
operations in 1.0 - i.e. the ability to bind almost anything to 
anything, regardless of the node you are sending to or receiving from ", 
if it is possible to do this by QMF "create" surely that remains a 
mechanism for generic binding. Sorry I'm sure I'm being dumb, I really 
need to find the time to read up on the AMQP 1.0 spec.

I guess in simple terms what I'm asking is how on earth do you set up a 
subscription to a headers exchange in AMQP 1.0 along with the ability to 
control the "buffering" between producer and consumer - pretty much what 
my Address example above is doing.


> One thing I should make clear is that 0-10 is not going away any time soon.
> The intention is not to force anyone onto 1.0, but simply to make it
> available (and indeed I hope eventually the default for new developments).
Yeah I realise that Gordon, perhaps *panic* is too strong a term for my 
feelings :-) There are clearly issues though, now that AMQP 1.0 has been 
ratified as you say it does certainly seem to be something we ultimately 
need to migrate to even if not immediately. I've mentioned to you and 
Rob before that I'd really like to see some published roadmaps for Qpid. 
So "not going away any time soon" is one thing but a statement that we 
intend to support AMQP 0.10 indefinitely, or until December 2015 (random 
date!!) or whatever would give people confidence or conversely an 
opportunity to plan their own architectural changes well in advance.

I think one good thing about AMQP 1.0 being a ratified standard is that 
it opens up opportunities for integration. A couple of our consumers 
actually use ActiveMQ internally (and currently use a Camel based bridge 
IIRC), it'd be much more elegant to federate directly from qpidd to 
ActiveMQ - I'm assuming that having a ratified standard would make that 
sort of thing more of a possibility ultimately.

As an aside Qpid remains at a 0.x version. I guess that I had sort of 
assumed that there would be an intention to uplift to Qpid 1.x when AMQP 
1.0 support had stabilised and was proved to be as reliable as the AMQP 
0.10 support, but TBH I've not seen any traffic at all relating to the 
version? Any thoughts? I have to be honest I'd bet that the 0.x version 
structure puts off a whole bunch of people who would really benefit from 
using Qpid. To get on my "branding" hobby horse again we need to 
consider just quite how conservative many large Enterprises actually 
are, it's hard enough to get some of them to consider Open Source, but 
version 0.x to many implies "unreliable and unstable" which Qpid is 
anything but. I think we're doing ourselves a great dis-service if we 
don't put some consideration into what it would take to become version 1.0


> The x- options were always going to be quite 0-10 specific. In some places
> they were unavoidable.
Yeah I realise that, but the headers binding was a standard exchange in 
AMQP 0.10 (don't know about AMQP 1.0). To be fair I guess the whole 
Address String is Qpid specific :-)
> I'm hoping that for 1.0, most cases can be met much
> more simply and effectively. But again, I wouldn't want anyone to be
> alarmed by my mail as 0-10 is unaffected and will continue to be fully
> supported.
Thanks again, I'm not *too* alarmed :-) I guess it's good in that it 
prompted me to flag up one of my most critical use cases. I hope that 
the additional explanation that I've added to this mail helps clarify 
things even more.

Funnily enough I was just recently in an off-list conversation with Rob 
and Robbie on the subject of Ring Queues, cause I couldn't fathom why 
the Java broker didn't support circular queuing :-) I suppose that the 
moral is not to presuppose peoples critical use cases. I personally 
couldn't give a damn about transactions for example (though I can 
*really* understand why some cases need them :-) ) from my perspective 
I'm using Qpid as a very high throughput (rate/message size and volume) 
low latency data delivery fabric and have real time producers and 
multiple consumers who care about wildly different data types. I 
*really* don't want any dodgy consumer interfering with data delivery to 
any of the other consumers and I *really really* don't want any 
consumers interfering with my producers (I've not figured out a way of 
slowing down time yet :-)) so basically I need a very decoupled pub/sub 
system.

>
> Also, I do want to get feedback exactly like this. Highlighting things that
> concerns existing users, especially if they would like to move to 1.0.
Cool, hope this slightly lengthy tome is useful too.
> I
> can't guarantee 100% fidelity for all 0-10 address string options (some
> simply cannot be sensibly mapped and would in any case tie you to qpidd so
> tightly that there would be little benefit to using 1.0!). I do however
> want to do what I can to ease the transition. So if I need to do some more
> thinking on x-bindings, I'm happy to do that. Thanks for he feedback, its
> very helpful!
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message