qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aleksey Vorona <voron...@gmail.com>
Subject 100K queues question
Date Fri, 06 Oct 2017 05:25:26 GMT
Dear All,

We were fairly happy users of RabbitMQ until we got a use case which
demands for a large number of queues. Large being in the hundreds of
thousands queues range (not counting DLQ).

It looks like we can have a cluster of Qpid brokers, each with a sane
number of queues, and use qpid-dispatch to route the messages to their
brokers.

However, I am concerned about the size of the qpid-dispatch config we
would end up writing. Is there anything that may help us automate this
config management? My main concern is a new queue creation. Ideally we
would want to find the least busy broker and create a new queue there.

Would such a large routing table affect qpid-dispatch performance?

I will work on setting up a Proof-of-Concept Qpid cluster with 100K
queues shortly. If you have any tips or have ran Qpid in a similar
mode, I would appreciate the information.

P.S. The need for a large number of queues comes out of a fair
consumption requirement which we are doing this way:
https://nithril.github.io/amqp/2015/07/05/fair-consuming-with-rabbitmq/
If there is a Qpid way of achieving this without 100K queues, I would
love to hear about it.

-- Aleksey

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message