qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajith Attapattu <rajit...@gmail.com>
Subject Re: Total Ordering Guarantee?
Date Fri, 26 Feb 2010 21:47:50 GMT
On Fri, Feb 26, 2010 at 3:29 PM, jerdavis <jerdavis@speakeasy.net> wrote:
> In short I am using a persistent queue as a way to stretch scalability of a
> database:
> My initial use case is this:
> I would like to queue 1million persistent&durable messages per minute.
> This 1 million represents 1 million separate conversations(Queues?) at 1 msg
> per minute (or 16.6KTransactions per second)

I million queues (around 200k per broker) is likely not going to be
efficient no matter which vendor you use.
All though each queue would occupy very little memory, the
housekeeping and look up will be a bit expensive.
Also since you want to use durable messaging, each queue will have
it's own file.
Since you are planning an exclusive consumer per queue, you are
talking about a large number of connections.
On linux the default file handle limit per process is 1024, all though
this can be changed, I don't know what the maximum is.
Either way your file handles will be shared by the socket and
persistent storage.
So I doubt that a particular instance could handle such a large no of queues.
(Each socket, file handle structure will also take a little bit of
memory and they all add up).

I understand that the conversations are separate from each other.
But finding a way to represent them with a lesser number of queues
would be better.

I would assume you plan to route messages based on some sort of a
conversation ID?
So how about you add a message header called conversation ID to each message?
And then a single queue could take messages intended for more than one
conversation (say X conversations).
Then the consumer (Worker in your terminology) could take each message
and distribute it among X workers who does the processing as you
Also you could add a sequence number header for each message in a conversation.
So in case redelivery happens and messages gets out of order you
consumer has a reasonable way of handling out-of-order messages.

You could argue that this solution is offloading part of the problem
from the broker to the consumer, but it allows you to relieve the
strain on the broker.
Also more than one conversation on a queue allows you to balance the
load more easily, when you have busy conversations and not so busy

> ... My plan is to Federate 5
> Brokers to get ~3KTPS per broker which is reasonable for most disks. (I
> assume I can also Federate 5 Clusters of 2 brokers for redundant copies of
> data?)

OK I didn't quite understand why you need to federate (or connect) the
5 broker entities?
As I understood your conversations are distinct and I don't see what
federation buys you.
Why not use 5 distinct broker instances with the load equally spread across.
You can certainly replicate each broker to have a standby.  So you
have 5 pairs of distinct clusters.

> A Worker (that is an exclusive consumer of a given Queue) will take each
> message in a conversation and roll it up in to an hourly product that will
> be persisted to a DB. I will use 2PC to simultaneously commit the hourly
> product to a DB and delete the hours worth of messages out of the Queue.
> Concurrently with every new message arriving at the worker, I will be
> updating the product (in memory only) and make it available via
> caching(memcached/etc).
> In this way processing 1 Million Msgs per minute is only writing to a DB
> 1M/3600~~280TPS (assuming perfectly staggered), also real time products are
> available, and there is always a persistent copy in case of failure.
> So does this sort of thing seem reasonable?
I think if you find a reasonable way of using a  smaller number of
queues to handle your 1 million conversations you could certainly do
The rates you quote are certainly doable by the Qpid product.

> --
> View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4641639.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org


Rajith Attapattu
Red Hat

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

View raw message