Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Th2rfbYOObGmawdYqXjdnhB7A561n9OBxatVmHVvByAXGLedPtV+ieJBHLheCpYkp2
         uRErxZmUwqF5PDGBHpcXzo11/eKeEq6y8nukxFeGcS0nyT7ugqTOWwVJ13oSKjG5A9mD
         R7JvflSu5xE37EfppFRVuD/KawYQ53II7Xmx4=
MIME-Version: 1.0
In-Reply-To: <1306338195.4ddd2393d9015@itchen.qinetiq.com>
References: <1306338195.4ddd2393d9015@itchen.qinetiq.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 25 May 2011 14:56:37 -0500
Message-ID: <BANLkTintG_TvP0d-hSVHX2UoRvKygxi90Q@mail.gmail.com>
Subject: Re: Priority queue in a single row - performance falls over time
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You're basically intentionally inflicting the worst case scenario on
the Cassandra storage engine:
http://wiki.apache.org/cassandra/DistributedDeletes

You could play around with reducing gc_grace_seconds but a PQ with
"millions" of items is something you should probably just do in memory
these days.

On Wed, May 25, 2011 at 10:43 AM,  <dnallsopp@taz.qinetiq.com> wrote:
>
>
> Hi all,
>
> I'm trying to implement a priority queue for holding a large number (mill=
ions)
> of items that need to be processed in time order. My solution works - but=
 gets
> slower and slower until performance is unacceptable - even with a small n=
umber
> of items.
>
> Each item essentially needs to be popped off the queue (some arbitrary wo=
rk is
> then done) and then the item is returned to the queue with a new timestam=
p
> indicating when it should be processed again. We thus cycle through all w=
ork
> items eventually, but some may come around more frequently than others.
>
> I am implementing this as a single Cassandra row, in a CF with a TimeUUID
> comparator.
>
> Each column name is a TimeUUID, with an arbitrary column value describing=
 the
> work item; the columns are thus sorted in time order.
>
> To pop items, I do a get() such as:
>
> =A0cf.get(row_key, column_finish=3Dnow, column_start=3Dyesterday, column_=
count=3D1000)
>
> to get all the items at the head of the queue (if any) whose time exceeds=
 the
> current system time.
>
> For each item retrieved, I do a delete to remove the old column, then an =
insert
> with a fresh TimeUUID column name (system time + arbitrary increment), th=
us
> putting the item back somewhere in the queue (currently, the back of the =
queue)
>
> I do a batch_mutate for all these deletes and inserts, with a queue size =
of
> 2000. These are currently interleaved i.e. delete1-insert1-delete2-insert=
2...
>
> This all appears to work correctly, but the performance starts at around =
8000
> cycles/sec, falls to around 1800/sec over the first 250K cycles, and cont=
inues
> to fall over time, down to about 150/sec, after a few million cycles. Thi=
s
> happens regardless of the overall size of the row (I have tried sizes fro=
m 1000
> to 100,000 items). My target performance is 1000 cycles/sec (but my data =
store
> will need to handle other work concurrently).
>
> I am currently using just a single node running on localhost, using a pyc=
assa
> client. 4 core, 4GB machine, Fedora 14.
>
> Is this expected behaviour (is there just too much churn for a single row=
 to
> perform well), or am I doing something wrong?
>
> Would https://issues.apache.org/jira/browse/CASSANDRA-2583 in version 0.8=
.1 fix
> this problem (I am using version 0.7.6)?
>
> Thanks!
>
> David.
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> If you are not the intended recipient of this email, you must neither
> take any action based upon its contents, nor copy or show it to anyone.
> Please contact the sender if you believe you have received this email in
> error. QinetiQ may monitor email traffic data and also the content of
> email for the purposes of security. QinetiQ Limited (Registered in
> England & Wales: Company Number: 3796233) Registered office: Cody Technol=
ogy
> Park, Ively Road, Farnborough, Hampshire, GU14 0LX http://www.qinetiq.com=
.
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com