Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 672FFCC36 for ; Thu, 16 Feb 2012 21:53:08 +0000 (UTC) Received: (qmail 21460 invoked by uid 500); 16 Feb 2012 21:53:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 21443 invoked by uid 500); 16 Feb 2012 21:53:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 21435 invoked by uid 99); 16 Feb 2012 21:53:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Feb 2012 21:53:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of eran.chinthaka@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-tul01m020-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Feb 2012 21:52:59 +0000 Received: by obbwd15 with SMTP id wd15so4092858obb.31 for ; Thu, 16 Feb 2012 13:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=uiX7uY6FA/wCI6KWO/zY1wRTuVK3nxN7/AqP+yogrPA=; b=BCc6nc9ObrSmrlD9bseX9Ui7qOICi8bCnIZZv4o3RMZ+z9yZHSumx/N5TQ00Kf2z6T YKSY6gYW3rd9l0rGpEqLLgWKY1VUtvWKW6o16EbIIfJNpZf9/ljgBDRQOS/MVEfWS14L GNFwha1stlrkbCf2F+o/Cn2dTgNZjJl/kVTgU= MIME-Version: 1.0 Received: by 10.182.41.6 with SMTP id b6mr3220829obl.10.1329429158463; Thu, 16 Feb 2012 13:52:38 -0800 (PST) Received: by 10.60.1.168 with HTTP; Thu, 16 Feb 2012 13:52:38 -0800 (PST) In-Reply-To: References: Date: Thu, 16 Feb 2012 13:52:38 -0800 Message-ID: Subject: Re: Key cache hit rate issue From: Eran Chinthaka Withana To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d044517a3be8a8904b91bd836 X-Virus-Checked: Checked by ClamAV on apache.org --f46d044517a3be8a8904b91bd836 Content-Type: text/plain; charset=ISO-8859-1 Hi Jonathan, Thanks for the reply. Yes there is a possibility that the keys can be distributed in multiple SSTables, but my data access patterns are such that I always read/write the whole row. So I expect all the data to be in the same SSTable (please correct me if I'm wrong). For some reason 16637958 (the keys cached) has become a golden number and I don't see key cache increasing beyond that. I also checked memory and I have about 4GB left in JVM memory and didn't see any issues on logs. Thanks, Eran Chinthaka Withana On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis wrote: > So, you have roughly 1/6 of your (physical) row keys cached and about > 1/4 cache hit rate, which doesn't sound unreasonable to me. Remember, > each logical key may be spread across multiple physical sstables -- > each (key, sstable) pair is one entry in the key cache. > > On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana > wrote: > > Hi Aaron, > > > > Here it is. > > > > Keyspace: XXXX > > Read Count: 1123637972 > > Read Latency: 5.757938114343114 ms. > > Write Count: 128201833 > > Write Latency: 0.0682576607387509 ms. > > Pending Tasks: 0 > > Column Family: YY > > SSTable count: 18 > > Space used (live): 103318720685 > > Space used (total): 103318720685 > > Number of Keys (estimate): 92404992 > > Memtable Columns Count: 1425580 > > Memtable Data Size: 359655747 > > Memtable Switch Count: 2522 > > Read Count: 1123637972 > > Read Latency: 14.731 ms. > > Write Count: 128201833 > > Write Latency: NaN ms. > > Pending Tasks: 0 > > Bloom Filter False Postives: 1488 > > Bloom Filter False Ratio: 0.00000 > > Bloom Filter Space Used: 331522920 > > Key cache capacity: 16637958 > > Key cache size: 16637958 > > Key cache hit rate: 0.2708333333333333 > > Row cache: disabled > > Compacted row minimum size: 51 > > Compacted row maximum size: 6866 > > Compacted row mean size: 2560 > > > > Thanks, > > Eran Chinthaka Withana > > > > > > > > On Thu, Feb 16, 2012 at 12:30 AM, aaron morton > > wrote: > >> > >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 > is > >> bit high. Is there a way to fix/improve it? > >> > >> Sorry I don't understand what you mean. But if the ratio is 0.0 all is > >> good. > >> > >> Could you include the full output from cfstats for the CF you are > looking > >> at ? > >> > >> Cheers > >> > >> ----------------- > >> Aaron Morton > >> Freelance Developer > >> @aaronmorton > >> http://www.thelastpickle.com > >> > >> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote: > >> > >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 > is > >> bit high. Is there a way to fix/improve it? > >> > >> Thanks, > >> Eran Chinthaka Withana > >> > >> > >> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton > >> wrote: > >>> > >>> Out of interest what does cfstats say about the bloom filter stats ? A > >>> high false positive could lead to a low key cache hit rate. > >>> > >>> Also, is there a way to warm start the key cache, meaning pre-load the > >>> amount of keys I set as keys_cached? > >>> > >>> See key_cache_save_period when creating the CF. > >>> > >>> Cheers > >>> > >>> > >>> ----------------- > >>> Aaron Morton > >>> Freelance Developer > >>> @aaronmorton > >>> http://www.thelastpickle.com > >>> > >>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote: > >>> > >>> Hi, > >>> > >>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% > >>> (using the numerical values). This is visible in cfstats too. But I'm > >>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well, > the > >>> data access pattern is not the issue here as I know they are > retrieving the > >>> same row multiple times. I'm using hector client with dynamic load > balancing > >>> policy with consistency ONE for both reads and writes. Any ideas on > how to > >>> find the issue and fix this? > >>> > >>> Here is what I see on cfstats. > >>> > >>> Key cache capacity: 16637958 > >>> Key cache size: 16637958 > >>> Key cache hit rate: 0.045454545454545456 > >>> > >>> Also, is there a way to warm start the key cache, meaning pre-load the > >>> amount of keys I set as keys_cached? > >>> > >>> Thanks, > >>> Eran > >>> > >>> > >> > >> > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > --f46d044517a3be8a8904b91bd836 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Jonathan,

Thanks for the reply.=A0Yes there is a poss= ibility that the keys can be distributed in multiple SSTables, but my data = access patterns are such that I always read/write the whole row. So I expec= t all the data to be in the same SSTable (please correct me if I'm wron= g).=A0

For some reason=A016637958 (the keys cached) has become= a golden number and I don't see key cache increasing beyond that. I al= so checked memory and I have about 4GB left in JVM memory and didn't se= e any issues on logs.=A0

Thanks,
Eran Chinthaka Withana


On Thu, Feb 16, 2012 at 1:20 PM, Jonatha= n Ellis <jbellis@= gmail.com> wrote:
So, you have roughly 1/6 of your (physical) row keys cached and about
1/4 cache hit rate, which doesn't sound unreasonable to me. =A0Remember= ,
each logical key may be spread across multiple physical sstables --
each (key, sstable) pair is one entry in the key cache.

On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
<eran.chinthaka@gmail.com> wrote:
> Hi Aaron,
>
> Here it is.
>
> Keyspace: XXXX
> Read Count: 1123637972
> Read Latency: 5.757938114343114 ms.
> Write Count: 128201833
> Write Latency: 0.0682576607387509 ms.
> Pending Tasks: 0
> Column Family: YY
> SSTable count: 18
> Space used (live): 103318720685
> Space used (total): 103318720685
> Number of Keys (estimate): 92404992
> Memtable Columns Count: 1425580
> Memtable Data Size: 359655747
> Memtable Switch Count: 2522
> Read Count: 1123637972
> Read Latency: 14.731 ms.
> Write Count: 128201833
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 1488
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 331522920
> Key cache capacity: 16637958
> Key cache size: 16637958
> Key cache hit rate: 0.2708333333333333
> Row cache: disabled
> Compacted row minimum size: 51
> Compacted row maximum size: 6866
> Compacted row mean size: 2560
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aaron@thelastpickle.com>
> wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess= 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Sorry I don't understand what you mean. But if the ratio is 0.= 0 all is
>> good.
>>
>> Could you include the full output from cfstats for the CF you are = looking
>> at ?
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://= www.thelastpickle.com
>>
>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess= 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>
>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aaron@thelastpickle.com>
>> wrote:
>>>
>>> Out of interest what does cfstats say about the bloom filter s= tats ? A
>>> high false positive could lead to a low key cache hit rate. >>>
>>> Also, is there a way to warm start the key cache, meaning pre-= load the
>>> amount of keys I set as keys_cached?
>>>
>>> See=A0key_cache_save_period when creating the CF.
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> htt= p://www.thelastpickle.com
>>>
>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>
>>> Hi,
>>>
>>> I'm using Cassandra 1.0.7 and I've set the keys_cached= to about 80%
>>> (using the numerical values). This is visible in cfstats too. = But I'm
>>> getting less than 20% (or sometimes even 0%) key cache hit rat= e. Well, the
>>> data access pattern is not the issue here as I know they are r= etrieving the
>>> same row multiple times. I'm using hector client with dyna= mic load balancing
>>> policy with consistency ONE for both reads and writes. Any ide= as on how to
>>> find the issue and fix this?
>>>
>>> Here is what I see on cfstats.
>>>
>>> Key cache capacity: 16637958
>>> Key cache size: 16637958
>>> Key cache hit rate: 0.045454545454545456
>>>
>>> Also, is there a way to warm start the key cache, meaning pre-= load the
>>> amount of keys I set as keys_cached?
>>>
>>> Thanks,
>>> Eran
>>>
>>>
>>
>>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.c= om

--f46d044517a3be8a8904b91bd836--