Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 31C0110E93 for ; Mon, 2 Jun 2014 00:06:30 +0000 (UTC) Received: (qmail 56228 invoked by uid 500); 2 Jun 2014 00:06:27 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56185 invoked by uid 500); 2 Jun 2014 00:06:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56177 invoked by uid 99); 2 Jun 2014 00:06:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jun 2014 00:06:27 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.213.177] (HELO mail-ig0-f177.google.com) (209.85.213.177) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jun 2014 00:06:23 +0000 Received: by mail-ig0-f177.google.com with SMTP id l13so2782893iga.4 for ; Sun, 01 Jun 2014 17:06:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:references:from:content-type:in-reply-to :message-id:date:to:content-transfer-encoding:mime-version; bh=22/0QJyWAt7PY5iqXh5AZoCFq/R/I1J/hxeSXrDtPis=; b=k4JfZRVyzZuw7isBDJpoWgFUDLt/8CiUm8tXHd6Kd26rcIOQbKEm2tI5RI97dBzMrF A6b/vvtd5tGlxcQT5ewCI2f6WaSVXhqWZzpziMTQDo5uy55oTtVhkpD/YrgtkyM3Ci+E 19nu5mmp+uEbXQ5QxiV8EqTZ5rafXi31ks2LME2XtbSJitHeYQIjr1CSxxQ67YqiBPMj O0usxP31wyfh5w8r40Kwo0snAzRmJl3qmzcdHW+udadFH9qa50HDUvzzHSi7SLQiJj7x dk50HJsre63tD0AehBD5Jw4qkMFeyJOyar6MMOclcgTap4Kg4Hi9D+OI/UTEwwcTiqpv yPmg== X-Gm-Message-State: ALoCoQnKkCMQEDYv+Qr0+CgNjUOylCdm50fOb2jXGeGal8NYzT9JiECT6ywmWbXBVzT8rBPJoBWj X-Received: by 10.43.160.137 with SMTP id mc9mr20096074icc.4.1401667562053; Sun, 01 Jun 2014 17:06:02 -0700 (PDT) Received: from [192.168.1.54] (173-23-245-196.client.mchsi.com. [173.23.245.196]) by mx.google.com with ESMTPSA id qn9sm24573438igb.5.2014.06.01.17.06.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 01 Jun 2014 17:06:01 -0700 (PDT) Subject: Re: Tune cache MB settings per table. References: <905DA450-14C9-4D07-957C-9C87F3C0C7E2@gmail.com> From: Colin Content-Type: multipart/alternative; boundary=Apple-Mail-D92130EF-BA54-4096-9F7E-8E5A88907A0F X-Mailer: iPad Mail (11D201) In-Reply-To: Message-Id: <854D6CEB-68C1-4E30-98FB-F0272B461245@clark.ws> Date: Sun, 1 Jun 2014 19:05:59 -0500 To: "user@cassandra.apache.org" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-D92130EF-BA54-4096-9F7E-8E5A88907A0F Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Your data model will most likely be the far most important component of your= migration. Get that right, and the rest is easy. -- Colin Clark=20 +1-320-221-9531 =20 > On Jun 1, 2014, at 7:01 PM, Kevin Burton wrote: >=20 > Good question. still migrating.. but we don't want to paint ourselves into= a corner. >=20 > There's an interesting line between premature optimization and painting yo= urself into a corner ;) >=20 > Best to get it right in between both extremes. >=20 >=20 >> On Sun, Jun 1, 2014 at 4:30 PM, Colin wrote: >> Have you been unable to achieve your SLA's using Cassandra out of the box= so far? >>=20 >> Based upon my experience, trying to tune Cassandra before the app is done= and without simulating real world load patterns, you might actually be doin= g yourself a disservice. >>=20 >> -- >> Colin >> 320-221-9531 >>=20 >>=20 >>> On Jun 1, 2014, at 6:08 PM, Kevin Burton wrote: >>>=20 >>> Not in our experience=E2=80=A6 We've been using fadvise don't need to pu= rge pages that aren't necessary any longer. >>>=20 >>> Of course YMMV based on your usage. I tend to like to control everythin= g explicitly instead of having magic. >>>=20 >>> That's worked out very well for us in the past so it would be nice to st= ill have this on cassandra. >>>=20 >>>=20 >>>> On Sun, Jun 1, 2014 at 12:53 PM, Colin wrote: >>>> The OS should handle this really well as long as your on v3 linux kerne= l.... =20 >>>>=20 >>>> -- >>>> Colin Clark=20 >>>> +1-320-221-9531 >>>> =20 >>>>=20 >>>>> On Jun 1, 2014, at 2:49 PM, Kevin Burton wrote: >>>>>=20 >>>>> It's possible to set caching to: >>>>>=20 >>>>> all, keys_only, rows_only, or none >>>>>=20 >>>>> .. for a given table. >>>>>=20 >>>>> But we have one table which is MASSIVE and we only need the most recen= t 4-8 hours in memory. =20 >>>>>=20 >>>>> Anything older than that can go to disk as the queries there are very r= are. >>>>>=20 >>>>> =E2=80=A6 but I don't think cassandra can do this (which is a shame). >>>>>=20 >>>>> Another option is to partition our tables per hour=E2=80=A6 then tell t= he older tables to cache 'none'=E2=80=A6=20 >>>>>=20 >>>>> I hate this option though. A smarter mechanism would be to have a com= paction strategy that created an SSTable for every hour and then had custom c= aching settings for that table. >>>>>=20 >>>>> The additional upside for this is that TTLs would just drop the older d= ata in the compactor..=20 >>>>>=20 >>>>> --=20 >>>>> Founder/CEO Spinn3r.com >>>>> Location: San Francisco, CA >>>>> Skype: burtonator >>>>> blog: http://burtonator.wordpress.com >>>>> =E2=80=A6 or check out my Google+ profile >>>>>=20 >>>>> War is peace. Freedom is slavery. Ignorance is strength. Corporations a= re people. >>>=20 >>>=20 >>>=20 >>> --=20 >>> Founder/CEO Spinn3r.com >>> Location: San Francisco, CA >>> Skype: burtonator >>> blog: http://burtonator.wordpress.com >>> =E2=80=A6 or check out my Google+ profile >>>=20 >>> War is peace. Freedom is slavery. Ignorance is strength. Corporations ar= e people. >=20 >=20 >=20 > --=20 > Founder/CEO Spinn3r.com > Location: San Francisco, CA > Skype: burtonator > blog: http://burtonator.wordpress.com > =E2=80=A6 or check out my Google+ profile >=20 > War is peace. Freedom is slavery. Ignorance is strength. Corporations are p= eople. --Apple-Mail-D92130EF-BA54-4096-9F7E-8E5A88907A0F Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Your data model will most likely be th= e far most important component of your migration.  Get that right, and t= he rest is easy.

--
<= b>Colin Clark 
= +1-320-221-9531
 

On Jun 1, 2014, at 7:01 PM, Kevin Burton <burton@spinn3r.com> wrote:

Good question. still migrating.. but w= e don't want to paint ourselves into a corner.

There's an= interesting line between premature optimization and painting yourself into a= corner ;)

Best to get it right in between both extremes.


On Sun, Jun 1= , 2014 at 4:30 PM, Colin <colpclark@gmail.com> wrote:
Have you been unable to= achieve your SLA's using Cassandra out of the box so far?

Based upon my experience, trying to tune Cassandra before the app= is done and without simulating real world load patterns, you might actually= be doing yourself a disservice.

--
Colin

On Jun 1, 2014, at 6:08= PM, Kevin Burton <burton@spinn3r.com> wrote:

<= div>
Not in our experience=E2=80=A6 We've been using fadvise don= 't need to purge pages that aren't necessary any longer.

= Of course YMMV based on your usage.  I tend to like to control everythi= ng explicitly instead of having magic.

That's worked out very well for us in the past so it wou= ld be nice to still have this on cassandra.


On Sun, Jun 1, 2014 at 12:53 PM, Co= lin <colin@clark.ws> wrote:
The OS should handle th= is really well as long as your on v3 linux kernel....  

--
<= div style=3D"margin:0cm 0cm 0.0001pt"> Colin Clark 
 

On Jun 1, 2014, at 2:49 PM, Kevin B= urton <burton@spi= nn3r.com> wrote:

It's possible t= o set caching to:
all, keys_only, rows_only, or none


But we have one table which is MASSIVE and we only need the m= ost recent 4-8 hours in memory.  

Anything older than that can go to disk as the queries there a= re very rare.

=E2=80=A6 but I don't think cassand= ra can do this (which is a shame).

Another option i= s to partition our tables per hour=E2=80=A6 then tell the older tables to ca= che 'none'=E2=80=A6 

I hate this option though.  A smarter mechanism wou= ld be to have a compaction strategy that created an SSTable for every hour a= nd then had custom caching settings for that table.

The additional upside for this is that TTLs would just drop the older data i= n the compactor.. 

--

Fo= under/CEO Spinn3r.com<= /a>
=E2=80=A6 or check out my Google+ profile
War is peace. Freedom i= s slavery. Ignorance is strength. Corporations are people.




--

Founder/CEO Spinn3r.com
Location: San Francisco, CA
Skype: burtona= tor
=E2=80=A6 or check out my Google+ profile
War is peace. Freedom i= s slavery. Ignorance is strength. Corporations are people.




--

Founder/CEO Spinn3r.com
Location: San Francisco, CA
Skype: burtona= tor
=E2=80=A6 or check out my Google+ profile
War is peace. Freedom i= s slavery. Ignorance is strength. Corporations are people.

= --Apple-Mail-D92130EF-BA54-4096-9F7E-8E5A88907A0F--