qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Jones" <jimmyjon...@gmx.co.uk>
Subject Handling queue overflows
Date Mon, 15 Jul 2013 13:01:29 GMT

I've got a system which can sometimes be a bit bursty, which would exhaust system memory if
the queues were left unchecked. Therefore I've been using ring queues, which solve the problem
quite nicely, apart from what happens to the "excess" messages. Ideally I'd like to buffer
them to disk and process them at a later, quieter time. I've been digging around and can see
a few options:

1) 0.24 will have flow to disk, which would be perfect but sometimes my messages are quite
big (eg. 10MB) and this requires messages to be smaller than a page. Is this limitation likely
to be removed soon?
2) 0.24 allows a "backup engine" to take over a loaded queue (QPID-4650), but this looks like
it'd require a fair bit of legwork to implement said engine.
3) alternate-exchanges. These look pretty good for my needs, but I can't seem to get them
to work! From reading some documentation, I thought they'd good with a limit policy of reject
- MRG 2 Installation & Configuration guide, 4.8.2 says for an alternate exchange specified
for a queue: "Messages that are acquired and then rejected by a message consumer". However
if I run the test below, messages only get routed to the alternate exchange when the queue
is destroyed while containing messages, and not when messages are rejected because the queue
is full. Presumably calling Session::reject would cause it to go to the alternate exchange,
but should a limit policy of reject be the same?

Any ideas very welcome!



qpid-config add exchange headers test1
qpid-config add exchange headers test1-overflow

# drain for messages in normal case
./drain -f "normal; { create: receiver, node: {type: queue, x-declare: {exclusive: True, alternate-exchange:
'test1-overflow', arguments: {'qpid.max_size': 1024, 'qpid.policy_type': 'reject'}}, x-bindings:
[{exchange: test1, arguments:{x-match:any, data-format: xyz}}]}}"

# drain for messages in overflow case
./drain -f "overflow; { create: receiver, node: {type: queue, x-declare: {exclusive: True,
arguments: {'qpid.max_size': 1024000, 'qpid.policy_type': 'ring'}}, x-bindings: [{exchange:
test1-overflow, arguments:{x-match:any, data-format: xyz}}]}}"

./spout --content test -c 5 --property data-format=xyz test1
# works as expected, messages received by normal drain
./spout --content test -c 5 --property data-format=xyz test1-overflow
# works as expected, messages received by overflow drain

# Now ctrl-c normal drain, and queue will remain
# Send loads of messages, fills up q1
./spout --content test -c 500 --property data-format=xyz test1
# Blocks... and no messages sent to overflow drain

qpid-config del queue normal --force
# now messages appear in overflow drain

To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

View raw message