Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 21278 invoked from network); 27 May 2010 05:26:16 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 27 May 2010 05:26:16 -0000 Received: (qmail 40409 invoked by uid 500); 27 May 2010 05:26:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 40177 invoked by uid 500); 27 May 2010 05:26:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 40169 invoked by uid 99); 27 May 2010 05:26:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 05:26:14 +0000 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rantav@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 05:26:09 +0000 Received: by yxn35 with SMTP id 35so100339yxn.31 for ; Wed, 26 May 2010 22:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=D1YxQVicoKpjD31Lz1CfYnKAuD2vQs22cXfe8hgOLEU=; b=Ig2vE5v6YziSbFxZQedEUlpU2noLPc6eTppMwh90SlWAYrq/Ifdc1DI8x3a/1KDZ/x 45CvXcpcHBIhnx85i6BOZHK+m3KholI/ywOK5lBe3go+/e9XxVFdD1i0qiGCZA3oRNdy YEvwAglJDYrCpH0ijFrl/Dxd0eV/X87u9SzQU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=Q3Ywl2C/mQgLjQ2bUoEcwE//jYz5W1UaaJGnssCA3ZtYs0+KgsDLiA59kzxZOQszQL +2nHV8Pw1MNI/BSjqMdlsOxR+GxEac/PknDF9wpIt0HMkf+5+KynQwbOcE2RQslqte15 eY+Vk9zeDCtDpob4G5wQCdVNKMc4l49PH0c0A= Received: by 10.231.120.144 with SMTP id d16mr8871867ibr.94.1274937948201; Wed, 26 May 2010 22:25:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.172.79 with HTTP; Wed, 26 May 2010 22:25:28 -0700 (PDT) In-Reply-To: References: From: Ran Tavory Date: Thu, 27 May 2010 08:25:28 +0300 Message-ID: Subject: Re: Error reporting Key cache hit rate with cfstats or with JMX To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e64b90748364b104878c9fe9 --0016e64b90748364b104878c9fe9 Content-Type: text/plain; charset=UTF-8 so the row cache contains both rows and keys and if I have large enough row cache (in particular if row cache size equals key cache size) then it's just wasteful to keep another key cache and I should eliminate the key-cache, correct? On Thu, May 27, 2010 at 1:21 AM, Jonathan Ellis wrote: > It sure sounds like you're seeing the "my row cache contains the > entire hot data set, so the key cache only gets the cold reads" > effect. > > On Wed, May 26, 2010 at 2:54 PM, Ran Tavory wrote: > > If I disable row cache the numbers look good - key cache hit rate is > 0, > so > > it seems to be related to row cache. > > Interestingly, after running for a really long time and with both row and > > keys caches I do start to see Key cache hit rate > 0 but the numbers are > so > > small that it doesn't make sense. > > I have capacity for 10M keys and 10M rows, the number of cached keys is > ~5M > > and very similarly the number of cached rows is also ~5M, however the hit > > rates are very different, 0.7 for rows and 0.006 for keys. I'd expect the > > keys hit rate to be identical since none of them reached the limit yet. > > Key cache capacity: 10000000 > > Key cache size: 5044097 > > Key cache hit rate: 0.0062089764058896576 > > Row cache capacity: 10000000 > > Row cache size: 5057231 > > Row cache hit rate: 0.7361241352465543 > > > > > > On Tue, May 25, 2010 at 3:43 PM, Jonathan Ellis > wrote: > >> > >> What happens if you disable row cache? > >> > >> On Tue, May 25, 2010 at 4:53 AM, Ran Tavory wrote: > >> > It seems there's an error reporting the Key cache hit rate. The value > is > >> > always 0.0 and I have a feeling it's incorrect. This is seen both by > >> > using > >> > notetool cfstats as well as accessing JMX directly > >> > > >> > > (org.apache.cassandra.db:type=Caches,keyspace=outbrain_kvdb,cache=KvAdsKeyCache > >> > RecentHitRate) > >> > >> > RowsCached="10000000" > >> > KeysCached="10000000"/> > >> > Column Family: KvAds > >> > SSTable count: 7 > >> > Space used (live): 1288942061 > >> > Space used (total): 1559831566 > >> > Memtable Columns Count: 73698 > >> > Memtable Data Size: 17121092 > >> > Memtable Switch Count: 33 > >> > Read Count: 3614433 > >> > Read Latency: 0.068 ms. > >> > Write Count: 3503269 > >> > Write Latency: 0.024 ms. > >> > Pending Tasks: 0 > >> > Key cache capacity: 10000000 > >> > Key cache size: 619624 > >> > Key cache hit rate: 0.0 > >> > Row cache capacity: 10000000 > >> > Row cache size: 447154 > >> > Row cache hit rate: 0.8460295730014572 > >> > Compacted row minimum size: 387 > >> > Compacted row maximum size: 31430 > >> > Compacted row mean size: 631 > >> > The Row cache hit rate looks good, 0.8 but Key cache hit rate always > >> > seems > >> > to be 0.0 while the number of unique keys stays about 619624 for quite > a > >> > while. > >> > Is it a real caching problem or just a reporting glitch? > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com > --0016e64b90748364b104878c9fe9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
so the row cache contains both rows and keys and if I have= large enough row cache (in particular if row cache size equals key cache s= ize) then it's just wasteful to keep another key cache and I should eli= minate the key-cache, correct?=C2=A0

On Thu, May 27, 2010 at 1:21 AM, Jonathan El= lis <jbellis@gmai= l.com> wrote:
It sure sounds like you're seeing the "my row cache contains the entire hot data set, so the key cache only gets the cold reads"
effect.

On Wed, May 26, 2010 at 2:54 PM, Ran Tavory <rantav@gmail.com> wrote:
> If I disable row cache the numbers look good - key cache hit rate is &= gt; 0, so
> it seems to be related to row cache.
> Interestingly, after running for a really long time and with both row = and
> keys caches I do start to see Key cache hit rate > 0 but the number= s are so
> small that it doesn't make sense.
> I have capacity for 10M keys and 10M rows, the number of cached keys i= s ~5M
> and very similarly the number of cached rows is also ~5M, however the = hit
> rates are very different, 0.7 for rows and 0.006 for keys. I'd exp= ect the
> keys hit rate to be identical since none of them reached the limit yet= .
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Key cache= capacity: 10000000
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Key cache= size: 5044097
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Key cache= hit rate: 0.0062089764058896576
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Row cache= capacity: 10000000
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Row cache= size: 5057231
> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Row cache= hit rate: 0.7361241352465543
>
>
> On Tue, May 25, 2010 at 3:43 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> What happens if you disable row cache?
>>
>> On Tue, May 25, 2010 at 4:53 AM, Ran Tavory <rantav@gmail.com> wrote:
>> > It seems there's an error reporting the=C2=A0Key cache hi= t rate. The value is
>> > always 0.0 and I have a feeling it's incorrect. This is s= een both by
>> > using
>> > notetool cfstats as well as accessing JMX directly
>> >
>> > (org.apache.cassandra.db:type=3DCaches,keyspace=3Doutbrain_kv= db,cache=3DKvAdsKeyCache
>> > RecentHitRate)
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0<ColumnFamily CompareWith=3D&quo= t;BytesType" Name=3D"KvAds"
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0RowsCached=3D"10000000"
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0KeysCached=3D"10000000"/>
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Column Family: KvAds
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= SSTable count: 7
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Space used (live): 1288942061
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Space used (total): 1559831566
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Memtable Columns Count: 73698
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Memtable Data Size: 17121092
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Memtable Switch Count: 33
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Read Count: 3614433
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Read Latency: 0.068 ms.
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Write Count: 3503269
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Write Latency: 0.024 ms.
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Pending Tasks: 0
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Key cache capacity: 10000000
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Key cache size: 619624
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Key cache hit rate: 0.0
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Row cache capacity: 10000000
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Row cache size: 447154
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Row cache hit rate: 0.8460295730014572
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Compacted row minimum size: 387
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Compacted row maximum size: 31430
>> > =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Compacted row mean size: 631
>> > The=C2=A0Row cache hit rate looks good, 0.8 but Key cache hit= rate always
>> > seems
>> > to be 0.0 while the number of unique keys stays about 619624 = for quite a
>> > while.
>> > Is it a real caching problem or just a reporting glitch?
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra suppo= rt
>> http://riptano.co= m
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

--0016e64b90748364b104878c9fe9--