Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A377711AF9 for ; Wed, 2 Jul 2014 01:53:26 +0000 (UTC) Received: (qmail 56890 invoked by uid 500); 2 Jul 2014 01:53:24 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56854 invoked by uid 500); 2 Jul 2014 01:53:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56839 invoked by uid 99); 2 Jul 2014 01:53:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jul 2014 01:53:24 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of burtonator2011@gmail.com designates 209.85.217.178 as permitted sender) Received: from [209.85.217.178] (HELO mail-lb0-f178.google.com) (209.85.217.178) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jul 2014 01:53:22 +0000 Received: by mail-lb0-f178.google.com with SMTP id 10so7594499lbg.9 for ; Tue, 01 Jul 2014 18:52:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=4jTq8EPaL8MLgWLIhuEo0a/Ir10nvU88UQYW1APNeus=; b=cQxjafW0LrUP93BNm7IpVmMRKvSUeF7M8/hNZVFsi//kkYg/qx2nWc8v39VysTWBL8 UeTkuMGr7qiidu+FPippJLkybCwpMGA3NPcvyb4mei8cInLagN72kx+TtS7ijeoX2rW9 Nqw14Qvyn/F2+D67wGfQMl4/qTFYRytw6Yaax6Xv0hFrfzpySScxnYSVBJDbI7SQEbNf YH4rE7fCvGsNI25xYDu5dZnTY1z+qwb+JgbVQB8Ny/DYreKktjN2Cmw31c2DtC3uj1+J CuaVNGTWy9hX24xQpdjeLr1kKhHz1pGSD0D/pMHVn/SbBZSeTrJdwPB0F2vIps0aOber rpGw== X-Received: by 10.112.142.33 with SMTP id rt1mr36519973lbb.45.1404265978171; Tue, 01 Jul 2014 18:52:58 -0700 (PDT) MIME-Version: 1.0 Sender: burtonator2011@gmail.com Received: by 10.112.219.6 with HTTP; Tue, 1 Jul 2014 18:52:38 -0700 (PDT) In-Reply-To: References: From: Kevin Burton Date: Tue, 1 Jul 2014 18:52:38 -0700 X-Google-Sender-Auth: k7T9AvEX7vSvZZ5GT2HtgLR3oNk Message-ID: Subject: Re: Fetching ONE cell with a row cache hit takes 1 second on an idle box? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c36b72cc99d504fd2c26fa X-Virus-Checked: Checked by ClamAV on apache.org --001a11c36b72cc99d504fd2c26fa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable A work around for this, is the VFS page cache.. basically, disabling compression, and then allowing the VFS page cache to keep your data in memory. The only downside is the per column overhead. But if you can store everything in a 'blob' which is optionally compressed, you're generally going to be ok. Kevin On Tue, Jul 1, 2014 at 6:50 PM, Kevin Burton wrote: > so.. caching the *queries* ? > > it seems like a better mechanism would be to cache the actually logical > row=E2=80=A6, not the physical row. > > Query caches just don't work in production, If you re-word your query, o= r > structure it a different way, you get a miss=E2=80=A6 > > In my experience.. query caches have a 0% hit rate. > > > On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli wrote: > >> On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burton wrote: >> >>> you know.. one thing I failed to mention.. .is that this is going into = a >>> "bucket" and while it's a logical row, the physical row is like 500MB = =E2=80=A6 >>> according to compaction logs. >>> >>> is the ENTIRE physical row going into the cache as one unit? That's >>> definitely going to be a problem in this model. 500MB is a big atomic = unit. >>> >> >> Yes, the row cache is a row cache. It caches what the storage engine >> calls rows, which CQL calls "partitions." [1] Rows have to be assembled >> from all of their row fragments in Memtables/SSTables. >> >> This is a big part of why the "off-heap" row cache's behavior of >> invalidation on write is so bad for its overall performance. Updating a >> single column in your 500MB row invalidates it and forces you to assembl= e >> the entire 500MB row from disk. >> >> The only valid use case for the current off-heap row cache seems to be : >> very small, very uniform in size, very hot, and very rarely modified. >> >> https://issues.apache.org/jira/browse/CASSANDRA-5357 >> >> Is the ticket for replacing the row cache and its unexpected >> characteristics with something more like an actual query cache. >> >> also.. I assume it's having to do a binary search within the physical ro= w >>> ? >> >> >> Since the column level bloom filter's removal in 1.2, the only way it ca= n >> get to specific columns is via the index. >> >> =3DRob >> [1] https://issues.apache.org/jira/browse/CASSANDRA-6632 >> > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > =E2=80=A6 or check out my Google+ profile > > > > --=20 Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com =E2=80=A6 or check out my Google+ profile --001a11c36b72cc99d504fd2c26fa Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
A work around for this, is the VFS page cache.. basically,= disabling compression, and then allowing the VFS page cache to keep your d= ata in memory.

The only downside is the per column overh= ead. =C2=A0But if you can store everything in a 'blob' which is opt= ionally compressed, you're generally going to be ok.

Kevin


On Tue, Jul 1, 2014 at 6:50 PM, Kevin Burton <bur= ton@spinn3r.com> wrote:
so.. caching the *queries* = ?

it seems like a better mechanism would be to cache the= actually logical row=E2=80=A6, not the physical row. =C2=A0

Query caches just don't work in production, =C2=A0I= f you re-word your query, or structure it a different way, you get a miss= =E2=80=A6

In my experience.. query caches have a 0% hit rate.


On Tue, Jul 1, 2014 at 6:40 PM, Robert Coli <rcoli@ev= entbrite.com> wrote:
=
On Tue, Jul 1, 2014 at 6:06 PM, Kevin Burto= n <burton@spinn3r.com> wrote:
you know.. one thing I failed to mention.= . .is that this is going into a "bucket" and while it's a log= ical row, the physical row is like 500MB =E2=80=A6 according to compaction = logs.

is the ENTIRE physical row going into the cache as one unit? =C2=A0That'= ;s definitely going to be a problem in this model. =C2=A0500MB is a big ato= mic unit.

Yes, the row ca= che is a row cache. It caches what the storage engine calls rows, which CQL= calls "partitions." [1] Rows have to be assembled from all of th= eir row fragments in Memtables/SSTables.

This is a big part of why the "off-heap" row = cache's behavior of invalidation on write is so bad for its overall per= formance. Updating a single column in your 500MB row invalidates it and for= ces you to assemble the entire 500MB row from disk.=C2=A0

The only valid use case for the current off-heap row ca= che seems to be : very small, very uniform in size, very hot, and very rare= ly modified.


Is the ticket for replacing the row cache and its unexp= ected characteristics with something more like an actual query cache.
=

also.. I assume it's having to do a binary search within the physical r= ow ?=C2=A0

Since the column level blo= om filter's removal in 1.2, the only way it can get to specific columns= is via the index.

=3DRob



--

Founder/CEO=C2=A0<= a href=3D"http://Spinn3r.com" target=3D"_blank">Spinn3r.com
Location:=C2=A0San Francisco, CA
=E2=80=A6 or check out my Google+ profile




--

Founder/CEO=C2=A0Spinn3r.com
Location:=C2=A0San Francisco, CA
=E2=80=A6 or check out my Google+ profile

--001a11c36b72cc99d504fd2c26fa--