qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gordon Sim <g...@redhat.com>
Subject Re: flow control - "resource-limit-exceeded"
Date Tue, 14 Jul 2009 08:43:31 GMT
Andy Li wrote:
> Hi,
> What is the best way for the producer to recover (after a pause) and
> continue sending messages after "resource-limit-exceeded" exception, when
> the queue on the broker fills up? Is it necessary to close/reopen
> connection, as it seems? 

Creating a new session should be sufficient.

> With async sending, how should we pick a point in
> the message sequence where we start re-sending messages, to guarantee no
> message loss? Can we do better than the last successfull "sync()" point? Is
> there a way to restart at the exact interruption spot, to avoid duplicate
> message delivery? 

Not at present, no. Note that you can also flush instead of syncing; 
this requests completion information to be sent but doesn't require 
halting the published stream until it arrives (and therefore might allow 
at least a smaller in-doubt window). (There is a utility, 
qpid::client::MessageReplayTracker used in the failover example that 
does this).

> There is a message number in the exception sting returned
> from the broker. Could it be used to correlate back to the first discarded
> message? 

There is a command-id, which should correspond back to the message (but 
other AMQP 'commands', queueDeclare, messageAccept etc would be in the 
same sequence).

> This number does not reset after reconnection, and seems to be
> per-exchange rather than per-session. And it seems to be vary between X+1
> and X+2, where X is 0 -base number of the last message that the subscribers
> receive.
> In general, would this mechanism be safe/stable to use with some kind of
> exponential backoff mechanism? 

I'm not clear what you are proposing here.

> Or could it cause corruption of the exchange
> or broker crash?
> Currently, we are using a fanout exchange, with some queue-size-limit and
> the "reject" policy. The subscribers have unlimited "prefetch" size for the
> LocalQueue (no flow control quota/window) and ACCEPT_MODE_NONE - to free up
> broker resources ASAP. However, the queue still overflows after a
> sufficiently long publishing burst due to 10X fanout and the resulting
> bandwidth disparity per-client between the publisher and each subscriber.
> (You can easily see this in the vanilla "perftest" with --mode fanout
> --nsubs 10, if you leave "default-queue-limit" setting in the broker, and
> run perftest on a separate machine).
> Are there any alternative methods of resource-usage-driven publisher flow
> control available? 

Not yet,  but we definitely want to add this.

> For example, in ActiveMQ the async publisher is
> automatically throttled through the flow control of the underlying TCP
> transport.
> I've read about the "max-session-rate" option. However, it's static. It
> would have to be set up for the worst possible case (all potential
> publishers sending in a burst concurrently) if we cannot handle queue
> overflow. It would limit the normal-case performance (single publisher
> sending a short burst that can fit in the queue until delivered). If we
> could throttle the publisher only if the buffer overflows, it seems we would
> achieve much better throughput.
> Here's our current publisher code:
> for(size_t i = 0; i < MAX; i++)
> {
> //set up msg, stamping with "i"
> try
> {
>   async(session).messageTransfer(arg::content=msg,
> arg::destination="amq.fanout");
>   if(i % syncInterval == 0)
>     session.sync();
> }
> catch (const qpid::framing::ResourceLimitExceededException& e) {
>   //wait some time to allow some queue space to free, then restart
> connection
>   //where to roll back "i"?
> }
> }
> Any help you can provide will be greatly appreciated.
> Thanks,
> Andy

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

View raw message