Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CB4F1D829 for ; Mon, 17 Sep 2012 14:45:15 +0000 (UTC) Received: (qmail 57552 invoked by uid 500); 17 Sep 2012 14:45:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 57514 invoked by uid 500); 17 Sep 2012 14:45:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 57476 invoked by uid 99); 17 Sep 2012 14:45:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Sep 2012 14:45:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ravikumar.govindarajan@gmail.com designates 209.85.219.44 as permitted sender) Received: from [209.85.219.44] (HELO mail-oa0-f44.google.com) (209.85.219.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Sep 2012 14:45:06 +0000 Received: by oagk14 with SMTP id k14so5389781oag.31 for ; Mon, 17 Sep 2012 07:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=XaVO7B2+zjFIZmWQurzCCg2CrpnyKme8FXHYn/zfyXU=; b=lJJpIK34Mk8aRRG3wjuCr+/3GgyVqV4vYkSuXEwsmqG5C6rgKbVsdvx11pP+YF+Jn2 /1UdlapT+NVTYJnRmeFhBj0FGAS0SLoSyQymO27LzvfG1JbjU+c3mmOMVsJueDGNQm+d ASQ9n4CmFjuqiZ/Yt1rz9rSP5E2VII89aThFwY5yFN6FEXG/fERbtmyGo27vvVUgTuKe QKQvVt6Voh5FbPDAGe/61rZoS/ysnzzIO27c9OOwD22myrXVKVK+b6aR82XsS4tAGYYl /gTtQtABxcFRRqGJVDFiqMSZAzySvldldcnzZBxS5KuN2k1TcbbwYMWuOSy3KkodQEVV QZow== MIME-Version: 1.0 Received: by 10.182.95.142 with SMTP id dk14mr12084165obb.2.1347893084953; Mon, 17 Sep 2012 07:44:44 -0700 (PDT) Received: by 10.182.90.137 with HTTP; Mon, 17 Sep 2012 07:44:44 -0700 (PDT) In-Reply-To: References: Date: Mon, 17 Sep 2012 20:14:44 +0530 Message-ID: Subject: Re: Composite Column Types Storage From: Ravikumar Govindarajan To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=14dae93b5d2a861b5604c9e6d085 --14dae93b5d2a861b5604c9e6d085 Content-Type: text/plain; charset=ISO-8859-1 Yes Aaron, I was not clear about Bloom Filters. I was thinking about the column bloom filters when I specify an absolute value for Part1 of the composite column and a start/end value for Part2 of the composite column It is slowly dawning on me that I need a super-column to use column blooms effectively and at the same time don't want the entire sub-column list deserialized. In fact, for my use-case I also do not need a column sampling index. Rather I would much prefer a multi-level skip-list Is there a way to customize how cassandra writes/reads it's key/column indexes to SSTables. Any hooks/API that is available as of now should be greatly helpful On Fri, Sep 14, 2012 at 10:33 AM, aaron morton wrote: > Range queries do not use bloom filters. > > Are you talking about row range queries ? Or a slice of columns in a row ? > > If you are getting a slice of columns from a single row, a bloom filter is > used to locate the row. > If you are getting a slice of columns from a range of rows, the bloom > filter is used to locate the first row. After that is a scan. > > There are also row level bloom filters for columns on a row. These are > used when you columns by names. If you are doing a slice with a start the > bloom filter is not used, instead the row level column index is used (if > present). > > Hope that helps. > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 13/09/2012, at 2:30 AM, Ravikumar Govindarajan < > ravikumar.govindarajan@gmail.com> wrote: > > Thanks for the clarification. Even though compression solves disk space > issue, we might still have Memtable bloat right? > > There is another issue to be handled for us. The queries are always going > to be range queries with absolute match on part1 and range on part 2 of the > composite columns > > Ex: Query > > Range queries do not use bloom filters. It holds good for > composite-columns also right? I believe I will end up writing BF bytes only > to skip it later. > > If sharing had been possible, then alone could have gone > into the bloom-filter, speeding up my queries really effectively. > > But as I understand, there are many levels of nesting possible in a > composite type and casing at every level is a big task > > May be casing for the top-level or the first-part should be a good start? > > -- > Ravi > > On Wed, Sep 12, 2012 at 5:46 PM, Sylvain Lebresne wrote: > >> > Is every / combination stored separately in disk >> >> Yes, each combination is stored separately on disk (the storage engine >> itself doesn't have special casing for composite column, at least not >> yet). But as far as disk space is concerned, I suspect that sstable >> compression makes this largely a non issue. >> >> -- >> Sylvain >> > > > --14dae93b5d2a861b5604c9e6d085 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Yes Aaron, I was not clear about Bloom Filters. I was thinking about the co= lumn bloom filters when I specify an absolute value for Part1 of the compos= ite column and a start/end value for Part2 of the composite column

It is slowly dawning on me that I need a super-column to use col= umn blooms effectively and at the same time don't want the entire sub-c= olumn list deserialized.=A0

In fact, for my use-ca= se I also do not need a column sampling index. Rather I would much prefer a= multi-level skip-list

Is there a way to customize how cassandra writes/reads = it's key/column indexes to SSTables. Any hooks/API that is available as= of now should be greatly helpful

On Fri, Sep 14, 2012 at 10:33 AM, aaron morton <aaron@thelastpickle.= com> wrote:
Range queries do not use bloom filters.=A0
Are you t= alking about row range queries ? Or a slice of columns in a row ?=A0
If you are getting a slice of columns from a single row, a bloom filte= r is used to locate the row.=A0
If you are getting a slice of col= umns from a range of rows, the bloom filter is used to locate the first row= . After that is a scan.=A0

There are also row level bloom filters for columns on a= row. These are used when you columns by names. If you are doing a slice wi= th a start the bloom filter is not used, instead the row level column index= is used (if present).=A0

Hope that helps.=A0


<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 13/09/2012, at 2:30 AM, Ravikumar Govindarajan <ravikumar.go= vindarajan@gmail.com> wrote:

Than= ks for the clarification. Even though compression solves disk space issue, = we might still have Memtable bloat right?

There is another issue to be handled for us. The queries are= always going to be range queries with absolute match on part1 and range on= part 2 of the composite columns

Ex: Query <some-key> <Column-part-1> <St= art-Id-part-2> <Limit>=A0

Range quer= ies do not use bloom filters. It holds good for composite-columns also righ= t? I believe I will end up writing BF bytes only to skip it later.

If sharing had been possible, then <Column-part-1>= ; alone could have gone into the bloom-filter, speeding up my queries reall= y effectively.

But as I understand, there are many= levels of nesting possible in a composite type and casing at every level i= s a big task

May be casing for the top-level or the first-part shoul= d be a good start?

--
Ravi
On Wed, Sep 12, 2012 at 5:46 PM, Sylvain Lebres= ne <sylvain@datastax.com> wrote:
> Is every <string>/<id>= combination stored separately in disk

Yes, each combination is stored separately on disk (the storage engin= e
itself doesn't have special casing for composite column, at least not yet). But as far as disk space is concerned, I suspect that sstable
compression makes this largely a non issue.

--
Sylvain



--14dae93b5d2a861b5604c9e6d085--