From user-return-37444-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Nov 7 04:34:18 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E962106F8 for ; Thu, 7 Nov 2013 04:34:18 +0000 (UTC) Received: (qmail 66580 invoked by uid 500); 7 Nov 2013 04:34:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 66475 invoked by uid 500); 7 Nov 2013 04:34:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 66467 invoked by uid 99); 7 Nov 2013 04:34:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Nov 2013 04:34:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.160.52] (HELO mail-pb0-f52.google.com) (209.85.160.52) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Nov 2013 04:34:05 +0000 Received: by mail-pb0-f52.google.com with SMTP id rr4so24892pbb.11 for ; Wed, 06 Nov 2013 20:33:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=jFtuYcIIMyzd2jt3MUq4RT1jRcvlzNrNodYLU87yOcc=; b=ikHBO5PcCWLee9uMflDGoffTDXmWjXnG/ALBeidDNu0VzR7WmPilHL+8ILTsMGBVvJ 9ZVkVKfqPmwdLxW3/8DT15+Rxr7PKb2ACVm8S1zVGP+vroi/6Hr8abIrr9ESSwhDqM/S vtrWdTwXM1kBGA/LPv1IMQgt5uKPM24ymvBQIlqNAt0qhQ/Z1Kvv9r52GL6S9peV04LO mISmsk30krE9AssaU0fn9wu6Y1i4NqM1bbvMzP0quPJ4ulr1Yb+N2eMpOZwQb6oMDQad 7JV/FoB9L3ixnc0/BJb6yiGjvZ58RJ7EzxdGpKXOAKJ5QwwvtNrLT49YeEyTmXuPm4o5 hBjA== X-Gm-Message-State: ALoCoQnKBEJsDEveZmC5eoI2ED/yBEzjksxntVF5X32i+uDm1z/T3OE0oY6HK8GE8kFE7xUw10rE X-Received: by 10.66.196.168 with SMTP id in8mr7515779pac.18.1383798823764; Wed, 06 Nov 2013 20:33:43 -0800 (PST) Received: from [172.16.1.20] ([203.86.207.101]) by mx.google.com with ESMTPSA id bp5sm1790999pbb.18.2013.11.06.20.33.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 06 Nov 2013 20:33:43 -0800 (PST) From: Aaron Morton Content-Type: multipart/alternative; boundary="Apple-Mail=_87520899-68AC-470C-9FD2-27B46505DC8A" Message-Id: <2224FB9C-6480-4577-9B24-FEA3ED4C50F8@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1816\)) Subject: Re: Config changes to leverage new hardware Date: Thu, 7 Nov 2013 17:33:43 +1300 References: <218c9fe89d68457db021e5559febc6db@SINPR03MB139.apcprd03.prod.outlook.com> To: Cassandra User In-Reply-To: <218c9fe89d68457db021e5559febc6db@SINPR03MB139.apcprd03.prod.outlook.com> X-Mailer: Apple Mail (2.1816) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_87520899-68AC-470C-9FD2-27B46505DC8A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 = soon. You will make more use of the extra memory moving to 1.2 as it moves = bloom filters and compression data off heap.=20 Also grab the TLAB setting from cassandra-env.sh in v1.2 > As of now, our performance tests (our application specific as well as = cassandra-stress) are not showing any significant difference in the = hardwares, which is a little disheartening, since the new hardware has a = lot more RAM and CPU. For reads or writes or both ?=20 Writes tend to scale with cores as long as the commit log can keep up.=20= Reads improve with disk IO and page cache size when the hot set is in = memory.=20 > Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 = disk used for commitlog and 3 disks RAID 0 for data) > New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, = eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for = data) Is the disk IO on the commit log volume keeping up ? You cranked up the concurrent writers and the commit log may not keep = up. You could put the commit log on the same RAID volume to see if that = improves writes.=20 > The config we tried modifying so far was concurrent_reads to (16 * = number of drives) and concurrent_writes to (8 * number of cores) as per=20= 256 write threads is a lot. Make sure the commit log can keep up, I = would put it back to 32, maybe try 64. Not sure the concurrent list for = the commit log will work well with that many threads.=20 May want to put the reads down as well.=20 It=92s easier to tune the system if you can provide some info on the = workload.=20 Cheers ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 7/11/2013, at 12:35 pm, Arindam Barua wrote: > =20 > We want to upgrade our Cassandra cluster to have newer hardware, and = were wondering if anyone has suggestions on Cassandra or linux config = changes that will prove to be beneficial. > As of now, our performance tests (our application specific as well as = cassandra-stress) are not showing any significant difference in the = hardwares, which is a little disheartening, since the new hardware has a = lot more RAM and CPU. > =20 > Old Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 = disk used for commitlog and 3 disks RAID 0 for data) > New Hardware: 32 cores (2 8-core with hyperthreading), 128 GB RAM, = eight 1-TB disks ( 1 disk used for commitlog and 7 disks RAID 0 for = data) > =20 > Most of the cassandra config currently is the default, and we are = using LeveledCompaction strategy. Default key cache, row cache turned = off. > The config we tried modifying so far was concurrent_reads to (16 * = number of drives) and concurrent_writes to (8 * number of cores) as per = recommendation in cassandra.yaml, but that didn=92t make much = difference. > We were hoping that at least the extra RAM in the new hardware will be = used for Linux file caching and hence an improvement in performance will = be observed. > =20 > Running Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 = soon. > =20 > Thanks, > Arindam --Apple-Mail=_87520899-68AC-470C-9FD2-27B46505DC8A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
Running Cassandra 1.1.5 currently, = but evaluating to upgrade to 1.2.11 = soon.
You will make more use of the extra = memory moving to 1.2 as it moves bloom filters and compression data off = heap. 

Also grab the TLAB setting from cassandra-env.sh in = v1.2

As of now, our performance tests (our application specific = as well as cassandra-stress) are not showing any significant difference = in the hardwares, which is a little disheartening, since the new = hardware has a lot more RAM and CPU.
For = reads or writes or both ? 

Writes tend to scale = with cores as long as the commit log can keep up. 
Reads improve with disk IO and page cache size = when the hot set is in memory. 

Old Hardware: 8 = cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk used for = commitlog and 3 disks RAID 0 for data)
New Hardware: 32 cores (2 8-core with = hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for = commitlog and 7 disks RAID 0 for data)
Is = the disk IO on the commit log volume keeping up ?
You cranked up the concurrent writers and the = commit log may not keep up. You could put the commit log on the same = RAID volume to see if that improves writes. 

The config we tried = modifying so far was concurrent_reads to (16 * number of drives) and = concurrent_writes to (8 * number of cores) as = per 
256 write threads is a lot. Make = sure the commit log can keep up, I would put it back to 32, maybe try = 64. Not sure the concurrent list for the commit log will work well with = that many threads. 

May want to put the reads down as = well. 

It=92s easier to tune the system if you can provide some = info on the workload. 

Cheers

http://www.thelastpickle.com

On 7/11/2013, at 12:35 pm, Arindam Barua <abarua@247-inc.com> = wrote:

 
We want to = upgrade our Cassandra cluster to have newer hardware, and were wondering = if anyone has suggestions on Cassandra or linux config changes that will = prove to be beneficial.
As of now, = our performance tests (our application specific as well as = cassandra-stress) are not showing any significant difference in the = hardwares, which is a little disheartening, since the new hardware has a = lot more RAM and CPU.
 
Old = Hardware: 8 cores (2 quad core), 32 GB RAM, four 1-TB disks ( 1 disk = used for commitlog and 3 disks RAID 0 for data)
New Hardware: 32 cores (2 8-core with = hyperthreading), 128 GB RAM, eight 1-TB disks ( 1 disk used for = commitlog and 7 disks RAID 0 for data)
 
Most = of the cassandra config currently is the default, and we are using = LeveledCompaction strategy. Default key cache, row cache turned = off.
The config we tried modifying = so far was concurrent_reads to (16 * number of drives) and = concurrent_writes to (8 * number of cores) as per recommendation in = cassandra.yaml, but that didn=92t make much = difference.
We were hoping that = at least the extra RAM in the new hardware will be used for Linux file = caching and hence an improvement in performance will be = observed.
 
Running = Cassandra 1.1.5 currently, but evaluating to upgrade to 1.2.11 = soon.
 
Thanks,
Arindam

= --Apple-Mail=_87520899-68AC-470C-9FD2-27B46505DC8A--