Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 3550 invoked from network); 29 Mar 2010 15:01:10 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 29 Mar 2010 15:01:10 -0000 Received: (qmail 65943 invoked by uid 500); 29 Mar 2010 15:01:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 65926 invoked by uid 500); 29 Mar 2010 15:01:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 65918 invoked by uid 99); 29 Mar 2010 15:01:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 15:01:10 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.82.44] (HELO mail-ww0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 15:01:03 +0000 Received: by wwb31 with SMTP id 31so1042602wwb.31 for ; Mon, 29 Mar 2010 08:00:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.51.18 with HTTP; Mon, 29 Mar 2010 08:00:41 -0700 (PDT) In-Reply-To: References: Date: Mon, 29 Mar 2010 09:00:41 -0600 Received: by 10.216.184.131 with SMTP id s3mr3116188wem.56.1269874841939; Mon, 29 Mar 2010 08:00:41 -0700 (PDT) Message-ID: <10e230a81003290800tbe871b0x55b8b555f4feccb4@mail.gmail.com> Subject: Re: Range scan performance in 0.6.0 beta2 From: Mike Malone To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e65681dadcee940482f1c631 --0016e65681dadcee940482f1c631 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Mar 29, 2010 at 7:13 AM, Henrik Schr=F6der wrot= e: > On Mon, Mar 29, 2010 at 14:15, Jonathan Ellis wrote: > >> On Mon, Mar 29, 2010 at 4:06 AM, Henrik Schr=F6der >> wrote: >> > On Fri, Mar 26, 2010 at 14:47, Jonathan Ellis >> wrote: >> >> It's a unique index then? And you're trying to read things ordered b= y >> >> the index, not just "give me keys with that have a column with this >> >> value?" >> > >> > Yes, because if we have more than one column per row, there's no way o= f >> > (easily) limiting the result. >> >> That's exactly what the count parameter of SliceRange is for... ? >> > > I thought that only limited the number of columns per key? > > We're using the get_range_slices method, which takes both a SlicePredicat= e > (which contains a range, which contains a count) and a KeyRange (which al= so > contains a count). Say that we have a bunch of keys that each contain 10 > columns, and we do a get_range_slices over those, how do we get the first= 25 > columns? If we put it in the SliceRange count, we'll get all matching row= s, > and the 25 first columns of each, right? And if we put it in the KeyRange > count, we'll get the 25 first rows that match, and all their columns, rig= ht? > > But if we have only one column per row, then we can limit the results the > way we want to. Or have we misunderstood the api somehow? > We've run into the same issue and have a patch that limits the _total_ number of columns returned instead of limiting on number of rows / number o= f columns per row. This makes it convenient to do a two dimensional index - first key is the row key, second is the column name, column value is the thing you're indexing. Then you do a get_range_slice on the two keys, limiting on total columns returned. We haven't run any real performance metrics yet. I don't think this query i= s particularly performant, but it's certainly faster than doing the same operation on the client side. Another thing to keep in mind is that rows must fit in memory because they're serialized / deserialized into memory from time to time. I believe this happens during SSTable serialization. Feel free to verify/correct me o= n this. If people are interested I can probably get that patch pushed back upstream soon. We're in crunch mode right now for launch though so, unfortunately, it'll probably be a week or so before we can finish it up and properly vet it. Mike --0016e65681dadcee940482f1c631 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Mon, Mar 29, 2010 at 7:13 AM, Henrik = Schr=F6der <skrol= le@gmail.com> wrote:
On Mon, Mar 29, 2010 at 14:15, Jonathan E= llis <jbellis@gmail.com> wrote:
On Mon, Mar 29, 2010 at 4:06 AM, Henrik Schr=F6der <skrolle@gmail.com> wrote: > On Fri, Mar 26, 2010 at 14:47, Jonathan Ellis <jbellis@gmail.com> wrote:
>> It's a unique index then? =A0And you're trying = to read things ordered by
>> the index, not just "give me keys with that have a column wit= h this
>> value?"
>
> Yes, because if we have more than one column per row, there's no w= ay of
> (easily) limiting the result.

That's exactly what the count parameter of SliceRange is for... ?=

I thought that only limited the numbe= r of columns per key?

We're using the get_range_slices method, w= hich takes both a SlicePredicate (which contains a range, which contains a = count) and a KeyRange (which also contains a count). Say that we have a bun= ch of keys that each contain 10 columns, and we do a get_range_slices over = those, how do we get the first 25 columns? If we put it in the SliceRange c= ount, we'll get all matching rows, and the 25 first columns of each, ri= ght? And if we put it in the KeyRange count, we'll get the 25 first row= s that match, and all their columns, right?

But if we have only one column per row, then we can limit the results t= he way we want to. Or have we misunderstood the api somehow?

We've run into the same issue and hav= e a patch that limits the _total_ number of columns returned instead of lim= iting on number of rows / number of columns per row. This makes it convenie= nt to do a two dimensional index - first key is the row key, second is the = column name, column value is the thing you're indexing. Then you do a g= et_range_slice on the two keys, limiting on total columns returned.

We haven't run any real performance metrics yet. I = don't think this query is particularly performant, but it's certain= ly faster than doing the same operation on the client side.

Another thing to keep in mind is that rows must fit in memory be= cause they're serialized / deserialized into memory from time to time. = I believe this happens during SSTable serialization. Feel free to verify/co= rrect me on this.

If people are interested I can probably get that patch = pushed back upstream soon. We're in crunch mode right now for launch th= ough so, unfortunately, it'll probably be a week or so before we can fi= nish it up and properly vet it.

Mike

--0016e65681dadcee940482f1c631--