cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dnalls...@taz.qinetiq.com
Subject Priority queue in a single row - performance falls over time
Date Wed, 25 May 2011 15:43:15 GMT


Hi all,

I'm trying to implement a priority queue for holding a large number (millions)
of items that need to be processed in time order. My solution works - but gets
slower and slower until performance is unacceptable - even with a small number
of items.

Each item essentially needs to be popped off the queue (some arbitrary work is
then done) and then the item is returned to the queue with a new timestamp
indicating when it should be processed again. We thus cycle through all work
items eventually, but some may come around more frequently than others.

I am implementing this as a single Cassandra row, in a CF with a TimeUUID
comparator.

Each column name is a TimeUUID, with an arbitrary column value describing the
work item; the columns are thus sorted in time order.

To pop items, I do a get() such as:

 cf.get(row_key, column_finish=now, column_start=yesterday, column_count=1000)

to get all the items at the head of the queue (if any) whose time exceeds the
current system time.

For each item retrieved, I do a delete to remove the old column, then an insert
with a fresh TimeUUID column name (system time + arbitrary increment), thus
putting the item back somewhere in the queue (currently, the back of the queue)

I do a batch_mutate for all these deletes and inserts, with a queue size of
2000. These are currently interleaved i.e. delete1-insert1-delete2-insert2...

This all appears to work correctly, but the performance starts at around 8000
cycles/sec, falls to around 1800/sec over the first 250K cycles, and continues
to fall over time, down to about 150/sec, after a few million cycles. This
happens regardless of the overall size of the row (I have tried sizes from 1000
to 100,000 items). My target performance is 1000 cycles/sec (but my data store
will need to handle other work concurrently).

I am currently using just a single node running on localhost, using a pycassa
client. 4 core, 4GB machine, Fedora 14.

Is this expected behaviour (is there just too much churn for a single row to
perform well), or am I doing something wrong?

Would https://issues.apache.org/jira/browse/CASSANDRA-2583 in version 0.8.1 fix
this problem (I am using version 0.7.6)?

Thanks!

David.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
If you are not the intended recipient of this email, you must neither
take any action based upon its contents, nor copy or show it to anyone.
Please contact the sender if you believe you have received this email in
error. QinetiQ may monitor email traffic data and also the content of
email for the purposes of security. QinetiQ Limited (Registered in
England & Wales: Company Number: 3796233) Registered office: Cody Technology 
Park, Ively Road, Farnborough, Hampshire, GU14 0LX http://www.qinetiq.com.

Mime
View raw message