Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 90242 invoked from network); 18 Oct 2010 20:22:12 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Oct 2010 20:22:12 -0000 Received: (qmail 36312 invoked by uid 500); 18 Oct 2010 20:22:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 36195 invoked by uid 500); 18 Oct 2010 20:22:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 36187 invoked by uid 99); 18 Oct 2010 20:22:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 20:22:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a44.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 20:22:04 +0000 Received: from homiemail-a44.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTP id 4685B11806C for ; Mon, 18 Oct 2010 13:21:42 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=to:from :subject:date:message-id:content-type:mime-version:in-reply-to; q=dns; s=thelastpickle.com; b=i06HSS9aUjk8OOvSqBDeVnkeNYb0f3RSU r8cQZciDr/+BkiFC88PESYxfRcK5rSKGJbeBnnU7L2yyQHM1+gdEEI2d35dB7fl6 zFkZTH1RGrntTlLqvjmun9PCgHVqRPB60nlfDe1CTZoM2mRUQ5m7Mln5XrCdtMtH Rj29VTvDL8= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=to :from:subject:date:message-id:content-type:mime-version: in-reply-to; s=thelastpickle.com; bh=HuL0bZGglucHCqIcX2JBvz7a5Cc =; b=k9nqM413JMbakdBO2jPfCZ3e4tXrIOZ4yDbQnhT1WIgjnQEZKM1Z4uZIwba 549/4O5OI1NwI0isdRzO/X812gMxiDXSrH++gR5goSQTVsXrUfp6SyYs4LPA8PSW esQrgqvKv5m72YhG2ehFPBFGuaZvIZY8yRKtPI6L7J5TER64= Received: from localhost (webms.mac.com [17.148.16.119]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTPSA id 38249118069 for ; Mon, 18 Oct 2010 13:21:42 -0700 (PDT) To: user@cassandra.apache.org From: Aaron Morton Subject: Re: KeysCached - clarification and recommendations on settings Date: Mon, 18 Oct 2010 20:21:41 GMT X-Mailer: MobileMe Mail (1C3203) Message-id: <5dae4f67-72f0-6b8a-bcdd-bde34b97ef57@me.com> Content-Type: multipart/alternative; boundary=Apple-Webmail-42--8aa13589-7513-613b-b40b-f6ff6815a060 MIME-Version: 1.0 In-Reply-To: <316547ca-adea-e7ff-94cf-c6cd0d8c398e@me.com> X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Webmail-42--8aa13589-7513-613b-b40b-f6ff6815a060 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Also have a read of this slide deck from Ben Black, caches are talked abou= t from slide 35 on=0A=0Ahttp://www.slideshare.net/driftx/cassandra-summit-= 2010-performance-tuning=0A=0AThere is also a video linked here=A0http://ww= w.riptano.com/blog/slides-and-videos-cassandra-summit-2010=0A=0AAaron=0A=0A= =0AOn 19 Oct, 2010,at 09:10 AM, Aaron Morton wro= te:=0A=0AAFAIK the caches do not expire entries under memory pressure, the= y just hold the number of entries you specify or are unbound in the case o= f 100%. The caches is based on the ConcurrentLinkedHashMap (http://code.go= ogle.com/p/concurrentlinkedhashmap/) and uses the SecondChance=A0eviction=A0= policy.=A0=0A=0A"=A0to keep all keys=A0in memory on all boxes in the clust= er" - Not sure exactly what your are saying, but only the keys and rows st= ored on the node are kept in the nodes caches.=A0=0A=0AIs your entire keys= et active? If not set a sane starting point (default for key cache is 200,= 000=A0http://wiki.apache.org/cassandra/StorageConfiguration=A0) =A0and see= what the cache hit's are like. How many keys do you have? What was=A0your= =A0hit rate with 100% key cache?=A0=0A=0AThere is a discussion here about = cache and heap sizing=A0=0Ahttp://www.mail-archive.com/user@cassandra.apac= he.org/msg04704.html=A0With 16GB on the box, 3GB Heap seems a bit small.=A0= =0A=0AThe cache settings are an easy way to shot yourself in foot. The bes= t approach is to start small and increase as needed. Using 100% will mean = your shoe budget will soon be cut in half :)=0A=0AHope that helps.=A0=0AAa= ron=0A=0A=0A=0A=0AOn 19 Oct, 2010,at 02:02 AM, Jedd Rashbrooke wrote:=0A=0AGreetings,=0A=0AI would like to check my un= derstanding is accurate on how=0AKeysCached is understood by Cassandra (0.= 6.5), and then=0Aget some suggestions on settings / OS FS cache interplay.= =0A=0AFirst - my initial understanding was that if you set KeysCached=0Ato= 100%, Cassandra would do a best effort to keep all keys=0Ain memory on al= l boxes in the cluster - but it would let some=0Aexpire if it was going to= run out of memory. *Now* I understand=0Athat it actually just exhausts al= l its memory and falls over if you=0Aset this value unrealistically high. = Is this right?=0A=0AAssuming this is right ... I'm curious on the optimum = settings=0Afor KeysCached. Previously we've had problems here with=0Aboxes= doing massive STW GC's, among other things - and=0Awe've been working tow= ards a configuration that is a bit more=0Astable and predictable, if not a= s blazingly fast as a brand new=0Acluster unburdened by much data but with= a configuration that=0Ameans it'll drive itself into oblivion after sever= al days. ;)=0A=0AWith that in mind, we're trying to keep JVM heap small - = about=0A3GB is our sweet spot so far - after experimenting with numbers=0A= as big as 30GB. Even at 30GB we'd be unable to have 100%=0AKeysCached anyw= ay - so it was always going to require a=0Atradeoff - and now we're trying= to guess at the best number.=0A=0AWe're using boxes with 16GB, and have a= n expectation that=0Athe OS will cache a lot of this stuff pretty intellig= ently - but I=0Afear that compactions and other activities will mean that = the=0Akeys won't always have priority for FS cache. Short of catting=0Athe= index files to /dev/null periodically (not a serious suggestion)=0Ahas an= yone gleaned some insights on the best tradeoffs for=0AKeysCached where th= e raw size of your Keys is going to be=0Aat least 10x the size of your ava= ilable memory? Do you go=0Asmall and hope the OS saves you, or do you try = to go as big=0A(but finite) as your JVM will usefully let you and hope=0AC= assandra caches the best set of keys for your usage profile?=0A=0Ataa,=0AJ= edd.=0A --Apple-Webmail-42--8aa13589-7513-613b-b40b-f6ff6815a060 Content-Type: multipart/related; type="text/html"; boundary=Apple-Webmail-86--8aa13589-7513-613b-b40b-f6ff6815a060 --Apple-Webmail-86--8aa13589-7513-613b-b40b-f6ff6815a060 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1;
Also have a read of this slide deck from Ben Black, caches are talked= about from slide 35 on

http://www.slideshare.net= /driftx/cassandra-summit-2010-performance-tuning


<= div>Aaron


On 19 Oct, 2010,at 09:10 AM, Aaron Morton <= aaron@thelastpickle.com> wrote:

AFAIK the caches do not expire entries under memory pressur= e, they just hold the number of entries you specify or are unbound in the = case of 100%. The caches is based on the ConcurrentLinkedHashMap (http://code.google.com/p/concurr= entlinkedhashmap/) and uses the SecondChance eviction policy=  

" to keep all keys&nbs= p;in memory on all boxes in the cluster" - Not sure = exactly what your are saying, but only the keys and rows stored on the nod= e are kept in the nodes caches. 

=
Is your entire keyset active? If not set a sane starting= point (default for key cache is 200,000 http://wiki.apache.org/cassandra/StorageCo= nfiguration )  and see what the cache hit's are like. How ma= ny keys do you have? What was your hit rate with 100% key cache?=  

There is a discussion here about cache and= heap sizing 
http://www.mail-archive.com/use= r@cassandra.apache.org/msg04704.html With 16GB on the box, 3GB He= ap seems a bit small. 

The cache settings ar= e an easy way to shot yourself in foot. The best approach is to start smal= l and increase as needed. Using 100% will mean your shoe budget will soon = be cut in half :)

Hope that helps. 
Aaron




On 19 O= ct, 2010,at 02:02 AM, Jedd Rashbrooke <jedd.rashbrooke@imagini.net> = wrote:

Greetings,
=0A
=0A I would like to check my understanding is = accurate on how
=0A KeysCached is understood by Cassandra (0.6.5), and = then
=0A get some suggestions on settings / OS FS cache interplay.
=0A=
=0A First - my initial understanding was that if you set KeysCached=0A to 100%, Cassandra would do a best effort to keep all keys
=0A in = memory on all boxes in the cluster - but it would let some
=0A expire i= f it was going to run out of memory. *Now* I understand
=0A that it ac= tually just exhausts all its memory and falls over if you
=0A set this = value unrealistically high. Is this right?
=0A
=0A Assuming this is= right ... I'm curious on the optimum settings
=0A for KeysCached. Pre= viously we've had problems here with
=0A boxes doing massive STW GC's, = among other things - and
=0A we've been working towards a configuration= that is a bit more
=0A stable and predictable, if not as blazingly fas= t as a brand new
=0A cluster unburdened by much data but with a configu= ration that
=0A means it'll drive itself into oblivion after several da= ys. ;)
=0A
=0A With that in mind, we're trying to keep JVM heap sma= ll - about
=0A 3GB is our sweet spot so far - after experimenting with = numbers
=0A as big as 30GB. Even at 30GB we'd be unable to have 100%<= br>=0A KeysCached anyway - so it was always going to require a
=0A trad= eoff - and now we're trying to guess at the best number.
=0A
=0A We'= re using boxes with 16GB, and have an expectation that
=0A the OS will = cache a lot of this stuff pretty intelligently - but I
=0A fear that co= mpactions and other activities will mean that the
=0A keys won't always= have priority for FS cache. Short of catting
=0A the index files to /= dev/null periodically (not a serious suggestion)
=0A has anyone gleaned= some insights on the best tradeoffs for
=0A KeysCached where the raw s= ize of your Keys is going to be
=0A at least 10x the size of your avail= able memory? Do you go
=0A small and hope the OS saves you, or do you = try to go as big
=0A (but finite) as your JVM will usefully let you and= hope
=0A Cassandra caches the best set of keys for your usage profile?=
=0A
=0A taa,
=0A Jedd.
=0A
--Apple-Webmail-86--8aa13589-7513-613b-b40b-f6ff6815a060-- --Apple-Webmail-42--8aa13589-7513-613b-b40b-f6ff6815a060--