qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Godfrey <rob.j.godf...@gmail.com>
Subject Re: Java broker exchange questions
Date Sun, 10 Mar 2013 13:15:40 GMT
On 10 March 2013 12:28, Fraser Adams <fraser.adams@blueyonder.co.uk> wrote:
> On 10/03/13 09:51, Rob Godfrey wrote:
>> On 10 March 2013 09:42, Fraser Adams <fraser.adams@blueyonder.co.uk>
>> wrote:
>>> I'd really like to understand more about how this is currently working in
>>> AMQP 1.0/Proton do you have some example Addresses that are being used -
>>> I'm
>>> particularly interested in how an existing AddressString containing a
>>> headers exchange and bindings to that might work.
>> So, what the Java Broker does under the covers is actually to create a
>> temporary subscription queue.
> That's interesting, that sounds fairly analogous to the federation in the
> C++ broker for things like dynamic exchange route and static exchange routes
> where *logically* exchanges are being linked together, but at the
> implementation level a temporary queue gets created.
> I guess from your previous response about what AMQP 1.0 does (and does not
> :-)) specify an awful lot is potentially at the level of "implementation
> detail". That feels like a rich seam of confusion that we're likely to be
> mining for quite some time I suspect :-D

There definitely has to be a clearer understanding of the distinction
between the protocol and the capabilities of any given intermediary
(such as a broker).

As an end user I want to be able to "subscribe to a topic".  In AMQP
1.0 that means setting up a link to receive messages from a source
which acts as a topic.  How to configure that address to behave like a
topic is not part of the core protocol.

The definition of standard node types and how to configure them will
be encouraged through the management definitions... however the 80%
use case where the client applications do not need to reconfigure the
internal wiring of the intermediary is covered by the core protocol.
In some ways the scope of the core protocol can be thought of as very
similar to the scope of the JMS API (albeit at a wire level rather
than an API level).

>>   One could actually implement
>> subscriptions to exchanges such that published messages are not
>> acknowledged until they have been flowed to all recipients... but that
>> would have been a bigger piece of work.
> That's interesting. My gut feeling is that it might be pretty dangerous.
> There's already a lot of scope for memory issues. I've been bitten in the
> past by quirks related to prefetch, so for example messages get sent to the
> prefetch queue on say slow consumers and end up building up on the broker
> because it can't remove the references until the client has acknowledged.
> When one actually thinks about what's going on it all makes sense, but it
> took me an age to get under the skin of this sort of behaviour and I wonder
> if some of these higher level abstractions and implementation details could
> leave people scratching their heads. I had enough problems thinking "my
> messages are hitting a queue with ring policy why on earth is the broker
> memory growing - and growing". If it was only *logically* hitting an
> exchange I suspect that it would *really* have melted my mind (and I've got
> a tenuous enough grip on sanity as it is :->)

Yes - though I think by building better abstractions we actually have
a chance to make things less complex.  Shoehorning everything into the
exchange->queue model with prefetch made certain scenarios (especially
with regard to topics) really mind-bending to try to understand from
an operational point of view.  Looking at the broker you'll just see
hundreds of temporary queues... which "topic" are they subscribing
too... which "topics" are growing... etc...

>> Ultimately I want to just add "topic" nodes rather than abusing
>> exchanges for this purpose (I had a 10 line patch that did this that
>> is probably still sitting around somewhere). The primary
>> implementation difference is that  messages at the head of a "topic"
>> get silently dropped if there are no subscriptions that have not yet
>> passed that message.
> Are you suggesting here that your "topic" node implements an internal queue
> ("messages at the head of a "topic" get silently dropped" is suggesting that
> to me).
> I have some issues with that and again it's analogous to federation.
> Ultimately I've been forced into using queue routes for federation due to
> the limitations of the "internal queues" that have been implemented for the
> "exchange routes" ultimately having non-configurable internal queues is a
> pain in the butt - I'd really have liked to be able to specify the size and
> policy of the internal queue.

So my "topic" implementation was of a standard queue (which can then
have a policy such as ring queue behaviour layered on top), where
"consumers" act more like browsers (they are non-destructive
consumers).  If we think of each consumer as actually just being a
pointer into the queue, then when all active consumers have "past" a
given message on a "topic" then it can be discarded.  (There is also a
use case for a topic which discards based on message age and allows
subscribers to join the topic as if it were some past point in time).

> I'm concerned that some of these thoughts around AMQP 1.0 might a) become
> inconsistent between the C++ and Java brokers and b) cause problems like I
> suggest above where the may not be as configurable as users need.

Certainly I want to reduce inconsistencies between the brokers where
possible (we're never going to count number of mallocs in the Java
Broker :) ). I'd also hope on the Java side at least to make things
more configurable (and for that configuration to be more easy to
understand).  In general I'd like to see all of Qpid aiming to track
the AMQP specifications much more closely as a way to drive
functionality, and drive standardisation where it is not yet present.

>>> Perhaps, but it's probably a good topic to start, the AMQP 1.0 work has
>>> been
>>> quietly going on in dev land and I think this is probably a good time to
>>> start sharing the love a bit on the user list.
>> +1.  We definitely need to talk more about AMQP 1.0 on this list as
>> well as better explaining what things are AMQP 1.0 and what things are
>> "Qpid".
> Cool I'm glad that you agree. It's definitely an interesting time but there
> may be some pain ahead for users as things move forward and good
> communication (and documentation) seems to me to be the best way to mitigate
> this.

Yeah... I think we need to make the transition to 1.0 as easy as
possible for existing users.  Things such as the qpid messaging API,
(or using JMS as an API) will make this easier.  Overall I think AMQP
1.0 will lead to new and better ways of doing things.

-- Rob

> BTW you asked previously "I don't suppose you work for a firm that is
> already an OASIS member, do you?" unfortunately not. As it happens all of my
> Qpid work is done in my own time at weekends
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org

To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

View raw message