Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 836E7D254 for ; Thu, 23 May 2013 14:15:56 +0000 (UTC) Received: (qmail 79983 invoked by uid 500); 23 May 2013 14:15:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 79958 invoked by uid 500); 23 May 2013 14:15:53 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 79938 invoked by uid 99); 23 May 2013 14:15:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 14:15:53 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [92.60.177.132] (HELO web1.alefhost.od.ua) (92.60.177.132) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 May 2013 14:15:47 +0000 Received: from [10.0.2.15] (unknown [78.26.128.196]) by web1.alefhost.od.ua (Postfix) with ESMTPSA id 695A3249F4 for ; Thu, 23 May 2013 17:15:20 +0300 (EEST) Message-ID: <519E20DE.8000803@4friends.od.ua> Date: Thu, 23 May 2013 16:59:58 +0300 From: Igor User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130329 Thunderbird/17.0.5 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Re: High performance disk io References: <00ad01ce56f1$f3199450$d94cbcf0$@struq.com> <519CD0FB.6030609@4friends.od.ua> <00be01ce56fa$82a72d50$87f587f0$@struq.com> In-Reply-To: <00be01ce56fa$82a72d50$87f587f0$@struq.com> Content-Type: multipart/alternative; boundary="------------030409090604080200020208" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. --------------030409090604080200020208 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hello Christopher, BTW, are you talking about 99th percentiles on client side, or about percentiles from cassandra histograms for CF on cassandra side? Thanks! On 05/22/2013 05:41 PM, Christopher Wirt wrote: > > Hi Igor, > > Yea same here, 15ms for 99^th percentile is our max. Currently getting > one or two ms for most CF. It goes up at peak times which is what we > want to avoid. > > We're using Cass 1.2.4 w/vnodes and our own barebones driver on top of > thrift. Needed to be .NET so Hector and Astyanax were not options. > > Do you use SSDs or multiple SSDs in any kind of configuration or RAID? > > Thanks > > Chris > > *From:*Igor [mailto:igor@4friends.od.ua] > *Sent:* 22 May 2013 15:07 > *To:* user@cassandra.apache.org > *Subject:* Re: High performance disk io > > Hello > > What level of read performance do you expect? We have limit 15 ms for > 99 percentile with average read latency near 0.9ms. For some CF 99 > percentile actually equals to 2ms, for other - to 10ms, this depends > on the data volume you read in each query. > > Tuning read performance involved cleaning up data model, tuning > cassandra.yaml, switching from Hector to astyanax, tuning OS parameters. > > On 05/22/2013 04:40 PM, Christopher Wirt wrote: > > Hello, > > We're looking at deploying a new ring where we want the best > possible read performance. > > We've setup a cluster with 6 nodes, replication level 3, 32Gb of > memory, 8Gb Heap, 800Mb keycache, each holding 40/50Gb of data on > a 200Gb SSD and 500Gb SATA for OS and commitlog > > Three column families > > ColFamily1 50% of the load and data > > ColFamily2 35% of the load and data > > ColFamily3 15% of the load and data > > At the moment we are still seeing around 20% disk utilisation and > occasionally as high as 40/50% on some nodes at peak time.. we are > conducting some semi live testing. > > CPU looks fine, memory is fine, keycache hit rate is about 80% > (could be better, so maybe we should be increasing the keycache size?) > > Anyway, we're looking into what we can do to improve this. > > One conversion we are having at the moment is around the SSD disk > setup.. > > We are considering moving to have 3 smaller SSD drives and > spreading the data across those. > > The possibilities are: > > -We have a RAID0 of the smaller SSDs and hope that improves > performance. > > Will this acutally yield better throughput? > > -We mount the SSDs to different directories and define multiple > data directories in Cassandra.yaml. > > Will not having a layer of RAID controller improve the throughput? > > -We mount the SSDs to different columns family directories and > have a single data directory declared in Cassandra.yaml. > > Think this is quite attractive idea. > > What are the drawbacks? System column families will be on the main > SATA? > > -We don't change anything and just keep upping our keycache. > > -Anything you guys can think of. > > Ideas and thoughts welcome. Thanks for your time and expertise. > > Chris > --------------030409090604080200020208 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Hello Christopher,

BTW, are you talking about 99th percentiles on client side, or about percentiles from cassandra histograms for CF on cassandra side?

Thanks!

On 05/22/2013 05:41 PM, Christopher Wirt wrote:

Hi Igor,

 

Yea same here, 15ms for 99th percentile is our max. Currently getting one or two ms for most CF. It goes up at peak times which is what we want to avoid.

 

We’re using Cass 1.2.4 w/vnodes and our own barebones driver on top of thrift. Needed to be .NET so Hector and Astyanax were not options.

 

Do you use SSDs or multiple SSDs in any kind of configuration or RAID?

 

Thanks

 

Chris

 

From: Igor [mailto:igor@4friends.od.ua]
Sent: 22 May 2013 15:07
To: user@cassandra.apache.org
Subject: Re: High performance disk io

 

Hello

What level of read performance do you expect? We have limit 15 ms for 99 percentile with average read latency near 0.9ms. For some CF 99 percentile actually equals to 2ms, for other - to 10ms, this depends on the data volume you read in each query.

Tuning read performance involved cleaning up data model, tuning cassandra.yaml, switching from Hector to astyanax, tuning OS parameters.

On 05/22/2013 04:40 PM, Christopher Wirt wrote:

Hello,

 

We’re looking at deploying a new ring where we want the best possible read performance.

 

We’ve setup a cluster with 6 nodes, replication level 3, 32Gb of memory, 8Gb Heap, 800Mb keycache, each holding 40/50Gb of data on a 200Gb SSD and 500Gb SATA for OS and commitlog

Three column families

ColFamily1 50% of the load and data

ColFamily2 35% of the load and data

ColFamily3 15% of the load and data

 

At the moment we are still seeing around 20% disk utilisation and occasionally as high as 40/50% on some nodes at peak time.. we are conducting some semi live testing.

CPU looks fine, memory is fine, keycache hit rate is about 80% (could be better, so maybe we should be increasing the keycache size?)

 

Anyway, we’re looking into what we can do to improve this.

 

One conversion we are having at the moment is around the SSD disk setup..

 

We are considering moving to have 3 smaller SSD drives and spreading the data across those.

 

The possibilities are:

-We have a RAID0 of the smaller SSDs and hope that improves performance.

Will this acutally yield better throughput?

 

-We mount the SSDs to different directories and define multiple data directories in Cassandra.yaml.

Will not having a layer of RAID controller improve the throughput?

 

-We mount the SSDs to different columns family directories and have a single data directory declared in Cassandra.yaml.

Think this is quite attractive idea.

What are the drawbacks? System column families will be on the main SATA?

 

-We don’t change anything and just keep upping our keycache.

-Anything you guys can think of.

 

Ideas and thoughts welcome. Thanks for your time and expertise.

 

Chris

 

 

 


--------------030409090604080200020208--