qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Li <andr.deve...@gmail.com>
Subject flow control - "resource-limit-exceeded"
Date Tue, 14 Jul 2009 00:46:42 GMT
Hi,

What is the best way for the producer to recover (after a pause) and
continue sending messages after "resource-limit-exceeded" exception, when
the queue on the broker fills up? Is it necessary to close/reopen
connection, as it seems? With async sending, how should we pick a point in
the message sequence where we start re-sending messages, to guarantee no
message loss? Can we do better than the last successfull "sync()" point? Is
there a way to restart at the exact interruption spot, to avoid duplicate
message delivery? There is a message number in the exception sting returned
from the broker. Could it be used to correlate back to the first discarded
message? This number does not reset after reconnection, and seems to be
per-exchange rather than per-session. And it seems to be vary between X+1
and X+2, where X is 0 -base number of the last message that the subscribers
receive.

In general, would this mechanism be safe/stable to use with some kind of
exponential backoff mechanism? Or could it cause corruption of the exchange
or broker crash?

Currently, we are using a fanout exchange, with some queue-size-limit and
the "reject" policy. The subscribers have unlimited "prefetch" size for the
LocalQueue (no flow control quota/window) and ACCEPT_MODE_NONE - to free up
broker resources ASAP. However, the queue still overflows after a
sufficiently long publishing burst due to 10X fanout and the resulting
bandwidth disparity per-client between the publisher and each subscriber.
(You can easily see this in the vanilla "perftest" with --mode fanout
--nsubs 10, if you leave "default-queue-limit" setting in the broker, and
run perftest on a separate machine).

Are there any alternative methods of resource-usage-driven publisher flow
control available? For example, in ActiveMQ the async publisher is
automatically throttled through the flow control of the underlying TCP
transport.

I've read about the "max-session-rate" option. However, it's static. It
would have to be set up for the worst possible case (all potential
publishers sending in a burst concurrently) if we cannot handle queue
overflow. It would limit the normal-case performance (single publisher
sending a short burst that can fit in the queue until delivered). If we
could throttle the publisher only if the buffer overflows, it seems we would
achieve much better throughput.

Here's our current publisher code:

for(size_t i = 0; i < MAX; i++)
{
//set up msg, stamping with "i"
try
{
  async(session).messageTransfer(arg::content=msg,
arg::destination="amq.fanout");

  if(i % syncInterval == 0)
    session.sync();
}
catch (const qpid::framing::ResourceLimitExceededException& e) {
  //wait some time to allow some queue space to free, then restart
connection
  //where to roll back "i"?
}
}

Any help you can provide will be greatly appreciated.
Thanks,
Andy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message