Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 86DECF5E5 for ; Thu, 28 Mar 2013 09:30:50 +0000 (UTC) Received: (qmail 92162 invoked by uid 500); 28 Mar 2013 09:29:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 87938 invoked by uid 500); 28 Mar 2013 09:29:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 74986 invoked by uid 99); 28 Mar 2013 09:19:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 09:19:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arodrime@gmail.com designates 209.85.215.53 as permitted sender) Received: from [209.85.215.53] (HELO mail-la0-f53.google.com) (209.85.215.53) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 09:18:55 +0000 Received: by mail-la0-f53.google.com with SMTP id fr10so16974094lab.26 for ; Thu, 28 Mar 2013 02:18:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=DseTqbdoQy71LnRgjxQAI3uDr+MDUE8o0o8MTtl/3+U=; b=WQ0VxRe2HEvqQp+m6N4T0Fwrm3VqbLAMSW+LlyjOSyrC24LWz0PffXSZ9UmBlXU+pS q0WKqj31EKfZZW+/DXVRvMTeMk/UG6D2GgTRdnD8idbexbgHeV+zRsC7KpebKjsTlgpI KlN9NKHExCNHr79Ysnp1P2ZJBW4lRsmhMCpU0i14pPVjL0oylwsrj2AhaBAb1dSvz91Y +Z+TwmR7gnNEtLoHonMY47epmN7/rqpTYs1FaSz5t30r9MyFZFjV/9siqsXcQe6CS4bq 5PE3OTeL/Nq57yXH5A1wz3+e+TVOwIs6VdtsRjdG8pqbbfp4JH0xhpnsuh7BrPqzJ8g3 o6gQ== X-Received: by 10.112.26.202 with SMTP id n10mr12011063lbg.15.1364462314721; Thu, 28 Mar 2013 02:18:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.44.230 with HTTP; Thu, 28 Mar 2013 02:18:14 -0700 (PDT) In-Reply-To: <00B2CB04-D2B7-48F2-A62D-9EB22D61B9D3@thelastpickle.com> References: <1364407229.62804.GenericBBA@web160901.mail.bf1.yahoo.com> <00B2CB04-D2B7-48F2-A62D-9EB22D61B9D3@thelastpickle.com> From: Alain RODRIGUEZ Date: Thu, 28 Mar 2013 10:18:14 +0100 Message-ID: Subject: Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec55555fe941aa304d8f8a332 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec55555fe941aa304d8f8a332 Content-Type: text/plain; charset=ISO-8859-1 "remember is used more IO than STS" Are you meaning during compactions ? Because I thought that LCS should decrease the number of disks reads (since 90% of the data aren't spread across multiple sstables and C* needs to read only a file to find the entire row) while not compacting right ? 2013/3/28 aaron morton > You nailed it. A significant number of reads are done from hundreds of > sstables ( I have to add, compaction is apparently constantly 6000-7000 > tasks behind and the vast majority of the reads access recently written > data ) > > So that's not good. > If IO is saturated then maybe LCS is not for you, remember is used more IO > than STS. > Otherwise look at the compaction yaml settings to see if you can make it > go faster but watch out that you don't hurt normal requests. > > CHeers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 28/03/2013, at 7:00 AM, Wei Zhu wrote: > > Welcome to the wonderland of SSTableSize of LCS. There is some discussion > around it, but no guidelines yet. > > I asked the people in the IRC, someone is running as high as 128M on the > production with no problem. I guess you have to test it on your system and > see how it performs. > > Attached is the related thread for your reference. > > -Wei > > ----- Original Message ----- > From: "Andras Szerdahelyi" > To: user@cassandra.apache.org > Sent: Wednesday, March 27, 2013 1:19:06 AM > Subject: Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01 > > > Aaron, > > > > > What version are you using ? > > > 1.1.9 > > > > > > Have you changed the bf_ chance ? The sstables need to be rebuilt for it > to take affect. > > > I did ( several times ) and I ran upgradesstables after > > > > > > Not sure what this means. > Are you saying it's in a boat on a river, with tangerine trees and > marmalade skies ? > > > You nailed it. A significant number of reads are done from hundreds of > sstables ( I have to add, compaction is apparently constantly 6000-7000 > tasks behind and the vast majority of the reads access recently written > data ) > > > > > > Take a look at the nodetool cfhistograms to get a better idea of the row > size and use that info when consdiering the sstable size. > > > It's around 1-20K, what should I optimise the LCS sstable size for? I > suppose "I want to fit as many complete rows as possible in to a single > sstable to keep file count down while avoiding compactions of oversized ( > double digit gigabytes? ) sstables at higher levels ? " > Do I have to run a major compaction after a change to sstable_size_in_mb ? > The larger sstable size wouldn't really affect sstables on levels above L0 > , would it? > > > > > > > Thanks!! > Andras > > > > > > > From: aaron morton < aaron@thelastpickle.com > > Reply-To: " user@cassandra.apache.org " < user@cassandra.apache.org > > Date: Tuesday 26 March 2013 21:46 > To: " user@cassandra.apache.org " < user@cassandra.apache.org > > Subject: Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01 > > > > > What version are you using ? > 1.2.0 allowed a null bf chance, and I think it returned .1 for LCS and .01 > for STS compaction. > Have you changed the bf_ chance ? The sstables need to be rebuilt for it > to take affect. > > > > > > and sstables read is in the skies Not sure what this means. > Are you saying it's in a boat on a river, with tangerine trees and > marmalade skies ? > > > > > > SSTable count: 22682 > > Lots of files there, I imagine this would dilute the effectiveness of the > key cache. It's caching (sstable, key) tuples. > You may want to look at increasing the sstable_size with LCS. > > > > > > Compacted row minimum size: 104 > Compacted row maximum size: 263210 > > > Compacted row mean size: 3041 > Take a look at the nodetool cfhistograms to get a better idea of the row > size and use that info when consdiering the sstable size. > > > Cheers > > > > > > > > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > > @aaronmorton > http://www.thelastpickle.com > > > On 26/03/2013, at 6:16 AM, Andras Szerdahelyi < > andras.szerdahelyi@ignitionone.com > wrote: > > > > > Hello list, > > > Could anyone shed some light on how an FP chance of 0.01 coexist with a > measured FP ratio of .. 0.98 ? Am I reading this wrong or are 98% of the > requests hitting the bloom filter create a false positive while the > "target" false ratio is 0.01? > ( Also key cache hit ratio is around 0.001 and sstables read is in the > skies ( non-exponential (non-) drop off for LCS ) but that should be filed > under "effect" and not "cause"? ) > > > > [default@unknown] use KS; > Authenticated to keyspace: KS > [default@KS] describe CF; > ColumnFamily: CF > Key Validation Class: org.apache.cassandra.db.marshal.BytesType > Default column value validator: org.apache.cassandra.db.marshal.BytesType > Columns sorted by: org.apache.cassandra.db.marshal.BytesType > GC grace seconds: 691200 > Compaction min/max thresholds: 4/32 > Read repair chance: 0.1 > DC Local Read repair chance: 0.0 > Replicate on write: true > Caching: ALL > Bloom Filter FP chance: 0.01 > Built indexes: [] > Compaction Strategy: > org.apache.cassandra.db.compaction.LeveledCompactionStrategy > Compaction Strategy Options: > sstable_size_in_mb: 5 > Compression Options: > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor > > > > Keyspace: KS > Read Count: 628950 > Read Latency: 93.19921121869784 ms. > Write Count: 1219021 > Write Latency: 0.14352380885973254 ms. > Pending Tasks: 0 > Column Family: CF > SSTable count: 22682 > Space used (live): 119771434915 > Space used (total): 119771434915 > Number of Keys (estimate): 203837952 > Memtable Columns Count: 13125 > Memtable Data Size: 33212827 > Memtable Switch Count: 15 > Read Count: 629009 > Read Latency: 88.434 ms. > Write Count: 1219038 > Write Latency: 0.095 ms. > Pending Tasks: 0 > Bloom Filter False Positives: 37939419 > Bloom Filter False Ratio: 0.97928 > Bloom Filter Space Used: 261572784 > Compacted row minimum size: 104 > Compacted row maximum size: 263210 > Compacted row mean size: 3041 > > > I upgraded sstables after changing the FP chance > > > Thanks! > Andras > > > > --bcaec55555fe941aa304d8f8a332 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
"remember is used more IO than STS"

Are you mean= ing during compactions ? Because I thought that LCS should decrease the num= ber of disks reads (since 90% of the data aren't spread across multiple= sstables and C* needs to read only a file to find the entire row) while no= t compacting right ?


2013/3/= 28 aaron morton <aaron@thelastpickle.com>
You nail= ed it. A significant number of reads are done from hundreds of sstables ( I= have to add, compaction is apparently constantly 6000-7000 tasks behind an= d the vast majority of the reads access recently written data )
So that's not good.=A0
If IO is saturated then m= aybe LCS is not for you, remember is used more IO than STS.=A0
=
Otherwise look at the compaction yaml se= ttings to see if you can make it go faster but watch out that you don't= hurt normal requests.=A0

CHeers

-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 28/03/2013, at 7:00 AM, Wei Z= hu <wz1975@yahoo.c= om> wrote:

Welcome to the wonderland of SSTableSize of LCS. There is some discussion a= round it, but no guidelines yet.

I asked the people in the IRC, som= eone is running as high as 128M on the production with no problem. I guess = you have to test it on your system and see how it performs.

Attached is the related thread for your reference.

-Wei

-= ---- Original Message -----
From: "Andras Szerdahelyi" <andras= .szerdahelyi@ignitionone.com>
To: user@cas= sandra.apache.org
Sent: Wednesday, March 27, 2013 1:19:06 AM
Subj= ect: Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01


Aa= ron,




What version are you using ?


1.1.9


=


Have you changed the bf_ chance ? The sstables need to be rebui= lt for it to take affect.


I did ( several times ) and I ran upg= radesstables after





Not sure what this means.
Are you saying it's i= n a boat on a river, with tangerine trees and marmalade skies ?

You nailed it. A significant number of reads are done from hundreds of sst= ables ( I have to add, compaction is apparently constantly 6000-7000 tasks = behind and the vast majority of the reads access recently written data )




Take a look at the nodetool cfhistograms to get a bette= r idea of the row size and use that info when consdiering the sstable size.=


It's around 1-20K, what should I optimise the LCS sstable = size for? I suppose "I want to fit as many complete rows as possible i= n to a single sstable to keep file count down while avoiding compactions of= oversized ( double digit gigabytes? ) sstables at higher levels ? " <= br> Do I have to run a major compaction after a change to sstable_size_in_mb ? = The larger sstable size wouldn't really affect sstables on levels above= L0 , would it?






Thanks!!
Andras






From: aaron morton < aaron@thelastpickle.com >
Reply-To= : " use= r@cassandra.apache.org " < user@cassandra.apache.org >
Date: Tuesday 26 March 2013 21:46
To: " user@cassandra.apache.org " &= lt; user@cas= sandra.apache.org >
Subject: Re: bloom filter fp ratio of 0.98 with fp_chance of 0.01

<= br>

What version are you using ?
1.2.0 allowed a null bf chance,= and I think it returned .1 for LCS and .01 for STS compaction.
Have yo= u changed the bf_ chance ? The sstables need to be rebuilt for it to take a= ffect.





and sstables read is in the skies Not sure what this me= ans.
Are you saying it's in a boat on a river, with tangerine trees= and marmalade skies ?





SSTable count: 22682

Lots of files there, I imagine this would dilute the effectiveness of t= he key cache. It's caching (sstable, key) tuples.
You may want to l= ook at increasing the sstable_size with LCS.





Compac= ted row minimum size: 104
Compacted row maximum size: 263210


Compacted row mean size: 304= 1
Take a look at the nodetool cfhistograms to get a better idea of the = row size and use that info when consdiering the sstable size.


Cheers








-----------------
Aaron Mort= on
Freelance Cassandra Consultant
New Zealand


@aaronmor= ton
http://w= ww.thelastpickle.com


On 26/03/2013, at 6:16 AM, Andras Szerdahelyi < andras.szerdahelyi@= ignitionone.com > wrote:




Hello list,

Could anyone shed some light on how an FP chance of 0.01 coexist with a mea= sured FP ratio of .. 0.98 ? Am I reading this wrong or are 98% of the reque= sts hitting the bloom filter create a false positive while the "target= " false ratio is 0.01?
( Also key cache hit ratio is around 0.001 and sstables read is in the skie= s ( non-exponential (non-) drop off for LCS ) but that should be filed unde= r "effect" and not "cause"? )



[default@= unknown] use KS;
Authenticated to keyspace: KS
[default@KS] describe CF;
ColumnFamil= y: CF
Key Validation Class: org.apache.cassandra.db.marshal.BytesType <= br>Default column value validator: org.apache.cassandra.db.marshal.BytesTyp= e
Columns sorted by: org.apache.cassandra.db.marshal.BytesType
GC grace s= econds: 691200
Compaction min/max thresholds: 4/32
Read repair chan= ce: 0.1
DC Local Read repair chance: 0.0
Replicate on write: true <= br> Caching: ALL
Bloom Filter FP chance: 0.01
Built indexes: []
Com= paction Strategy: org.apache.cassandra.db.compaction.LeveledCompactionStrat= egy
Compaction Strategy Options:
sstable_size_in_mb: 5
Compress= ion Options:
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
=


Keyspace: KS
Read Count: 628950
Read Latency: 93.199211= 21869784 ms.
Write Count: 1219021
Write Latency: 0.1435238088597325= 4 ms.
Pending Tasks: 0
Column Family: CF
SSTable count: 22682
Space u= sed (live): 119771434915
Space used (total): 119771434915
Number of= Keys (estimate): 203837952
Memtable Columns Count: 13125
Memtable = Data Size: 33212827
Memtable Switch Count: 15
Read Count: 629009
Read Latency: 88.434 m= s.
Write Count: 1219038
Write Latency: 0.095 ms.
Pending Tasks:= 0
Bloom Filter False Positives: 37939419
Bloom Filter False Ratio:= 0.97928
Bloom Filter Space Used: 261572784
Compacted row minimum size: 104
= Compacted row maximum size: 263210
Compacted row mean size: 3041

I upgraded sstables after changing the FP chance


Thanks! =
Andras
<attachment.eml>


--bcaec55555fe941aa304d8f8a332--