Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of boneill42@gmail.com
 designates 209.85.216.44 as permitted sender)
MIME-Version: 1.0
Sender: boneill42@gmail.com
In-Reply-To: <1344017886.28730.YahooMailClassic@web125704.mail.ne1.yahoo.com>
References: <1344017886.28730.YahooMailClassic@web125704.mail.ne1.yahoo.com>
Date: Fri, 3 Aug 2012 14:27:10 -0400
Message-ID: 
 <CAJHHpg3obMfKLJGZbno1e6bdpR_sN3x99SUHcS3dvpf4pWqPsQ@mail.gmail.com>
Subject: Re: How to process new rows in parallel?
From: "Brian O'Neill" <bone@alumni.brown.edu>
To: user@cassandra.apache.org, philipomailbox-cass@yahoo.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you are deleting the messages after processing, it sounds like you
are using Cassandra as a work queue.

Here are some links for implementing a distributed queue in Cassandra:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Distribute=
d-work-queues-td5226248.html
http://comments.gmane.org/gmane.comp.db.cassandra.user/16633

There is a placeholder on the use cases wiki for this, but no info:
http://wiki.apache.org/cassandra/UseCases#A_distributed_Priority_Job_Queue

We were looking to do the same thing, but in the end decided to go with Kaf=
ka.
Given your throughput requirements, Kafka might be a good option for
you as well.

-brian


On Fri, Aug 3, 2012 at 2:18 PM, Philip Nelson
<philipomailbox-cass@yahoo.com> wrote:
> Hello,
>
> I am using a Column Family in Cassandra to store incoming messages, which=
 arrive at a high rate (100s of thousands per second). I then have a proces=
s wake up periodically to work on those messages, and then delete them. I'd=
 like to understand how I could have multiple processes running, each pulli=
ng off a bunch of messages in parallel. It would be nice to be able to add =
processes dynamically, and not have to explicitly assign message ranges to =
various processes.
>
> Any suggestions on how to ensure that each process pulls off a different =
bunch of messages? Any recommended design patterns? I was going to look at =
qsandra too, for inspiration. Would this be worthwhile?
>
> If this was a relational database, I would have the processes lock the ta=
ble (or perhaps a row), set flags on a row indicating that it's being "proc=
essed", and then unlock. Processes would choose messages by SELECTing on un=
flagged messages. I'm not sure how this might map to Cassandra. I realise i=
t may not. Even if I configure the cluster such that seting a flag on a row=
 requires all nodes to be written, two processes could still race setting t=
hat flag, right?
>
> I am open to the idea that it might help to store the messages in wide ro=
ws, if that helps.
>
> Thanks,
>
> Philip


--=20
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/