From users-return-2617-apmail-qpid-users-archive=qpid.apache.org@qpid.apache.org Fri Feb 26 20:30:14 2010 Return-Path: Delivered-To: apmail-qpid-users-archive@www.apache.org Received: (qmail 51968 invoked from network); 26 Feb 2010 20:30:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 26 Feb 2010 20:30:14 -0000 Received: (qmail 19138 invoked by uid 500); 26 Feb 2010 20:30:14 -0000 Delivered-To: apmail-qpid-users-archive@qpid.apache.org Received: (qmail 19120 invoked by uid 500); 26 Feb 2010 20:30:14 -0000 Mailing-List: contact users-help@qpid.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@qpid.apache.org Delivered-To: mailing list users@qpid.apache.org Received: (qmail 19112 invoked by uid 99); 26 Feb 2010 20:30:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 20:30:14 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_HELO_PASS,SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of jerdavis@speakeasy.net does not designate 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 20:30:05 +0000 Received: from jim.nabble.com ([192.168.236.80]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1Nl6oh-0005Lr-VP for users@qpid.apache.org; Fri, 26 Feb 2010 12:29:43 -0800 Date: Fri, 26 Feb 2010 12:29:43 -0800 (PST) From: jerdavis To: users@qpid.apache.org Message-ID: <1267216183960-4641639.post@n2.nabble.com> In-Reply-To: References: <1267146954571-4636556.post@n2.nabble.com> <1267155013698-4636971.post@n2.nabble.com> Subject: Re: Total Ordering Guarantee? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org In short I am using a persistent queue as a way to stretch scalability of a database: My initial use case is this: I would like to queue 1million persistent&durable messages per minute. This 1 million represents 1 million separate conversations(Queues?) at 1 msg per minute (or 16.6KTransactions per second)... My plan is to Federate 5 Brokers to get ~3KTPS per broker which is reasonable for most disks. (I assume I can also Federate 5 Clusters of 2 brokers for redundant copies of data?) A Worker (that is an exclusive consumer of a given Queue) will take each message in a conversation and roll it up in to an hourly product that will be persisted to a DB. I will use 2PC to simultaneously commit the hourly product to a DB and delete the hours worth of messages out of the Queue. Concurrently with every new message arriving at the worker, I will be updating the product (in memory only) and make it available via caching(memcached/etc). In this way processing 1 Million Msgs per minute is only writing to a DB 1M/3600~~280TPS (assuming perfectly staggered), also real time products are available, and there is always a persistent copy in case of failure. So does this sort of thing seem reasonable? -- View this message in context: http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4641639.html Sent from the Apache Qpid users mailing list archive at Nabble.com. --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscribe@qpid.apache.org