From user-return-35829-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Aug 8 08:58:50 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3595910A38 for ; Thu, 8 Aug 2013 08:58:50 +0000 (UTC) Received: (qmail 70954 invoked by uid 500); 8 Aug 2013 08:58:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 70735 invoked by uid 500); 8 Aug 2013 08:58:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 70727 invoked by uid 99); 8 Aug 2013 08:58:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Aug 2013 08:58:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chris.wirt@struq.com designates 74.125.83.47 as permitted sender) Received: from [74.125.83.47] (HELO mail-ee0-f47.google.com) (74.125.83.47) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Aug 2013 08:58:40 +0000 Received: by mail-ee0-f47.google.com with SMTP id d49so1352078eek.6 for ; Thu, 08 Aug 2013 01:58:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:from:to:references:in-reply-to:subject:date :message-id:mime-version:content-type:thread-index:content-language; bh=VqSEUugcM/5L9COiP6/gW6LanK1JOAaiVKogM4NxT9I=; b=KLMe5Ra3BcvloouNvcAbt5naokdGWz25DDHlOz9u8XO9W+9lkMIqKQEODTUJC6otoB JJsUnwUcTZVHIPaT+ZQvC4Y0NCx1OyAwIINj/fuO5UxYlejtJKLT9jAncZBhqCeZa+IM f1jUukJtIG6zH/ezvGA4+Y6fN3J11zLO7jz7BuPnX0W/tNTKipZGG1fj0+ikObRQcysR JhGGc+B//sz3FABbCpIvKo0yE8nyhnn9PxKLWKkY6tPd8TQpM2f4u1GtQerIFVxQTYF3 LFdFEFVOu3+k3GlaW5Xumv/lrmbvEL5W+2Cws/uTok7q72r37TR7erLY10JDF7KYVQFP Cpxw== X-Gm-Message-State: ALoCoQkuCtIRAjoQ8E608jakgum8BjQe+YVPz1gqAUepT4mP2weO6bdtOiuLwAKbBimz4/O3n8e3 X-Received: by 10.14.211.134 with SMTP id w6mr7252305eeo.26.1375952298456; Thu, 08 Aug 2013 01:58:18 -0700 (PDT) Received: from StevePereiraPC (46-65-46-16.zone16.bethere.co.uk. [46.65.46.16]) by mx.google.com with ESMTPSA id bp43sm17361700eeb.4.2013.08.08.01.58.16 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 08 Aug 2013 01:58:17 -0700 (PDT) From: "Christopher Wirt" To: References: <6E7CBD0D-6DD3-491F-9FE4-3E9256B03AC7@gmail.com> In-Reply-To: Subject: RE: lots of small nodes vs fewer big nodes Date: Thu, 8 Aug 2013 09:58:13 +0100 Message-ID: <007701ce9415$6aafc010$400f4030$@struq.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0078_01CE941D.CC769910" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQIbuxksYhOWPYy5wdGwiV+vhmx2SAHt/N+BAjCRNaOYz70cAA== Content-Language: en-gb X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_0078_01CE941D.CC769910 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I found using a JBOD SSD setup (one per data directory) to be faster than RAID. JBOD configuration will also allow a disk to fail and the remaining disks to continue serving reads if you set disk_failure_policy: best_effort. If you do go for a RAID controller watch out for any special read/write caching options. I found these also killed performance. From: Alain RODRIGUEZ [mailto:arodrime@gmail.com] Sent: 08 August 2013 09:30 To: user@cassandra.apache.org Subject: Re: lots of small nodes vs fewer big nodes I advise you to have 7GB RAM better (more would be better if you have a lot of data per node). But the real difference is made when you use SSDs or RAIDs of SSDs, since the bottleneck is the disk throughput in most cases. We are in the cloud, we tried a lot of configurations and were comfortable only with a nodes RAM > 7 GB (RAM >15 GB with lot of data / node) and see a real enhancement when we switched to SSDs (latency from 20-40 to 3-5 ms), even reducing the number of nodes from 18 to 3. This was quite impressive and I recommend SSD (or RAID of SSDs) since to anyone who can afford it. Alain 2013/8/7 Andrey Ilinykh You still have the same amount of RAM, so you cache the same amount of data. I don't think you gain much here. On the other side, maintenance procedures (compaction, repair) may hit your 2CPU box. I wouldn't do it. Thank you, Andrey On Wed, Aug 7, 2013 at 10:24 AM, Paul Ingalls wrote: Quick question about systems architecture. Would it be better to run 5 nodes with 7GB RAM and 4CPU's or 10 nodes with 3.5GB RAM and 2CPUS? I'm currently running the former, but am considering the latter. My goal would be to improve overall performance by spreading the IO across more disks. My currently cluster has low CPU utilization but does spend a good amount of time in iowait. Would moving to more smaller nodes help with that? Or would I run into trouble with the smaller ram and cpu? Thanks! Paul ------=_NextPart_000_0078_01CE941D.CC769910 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I found using a JBOD SSD setup (one per data directory) to be faster = than RAID. JBOD configuration will also allow a disk to fail and the = remaining disks to continue serving reads if you set = disk_failure_policy: best_effort.

 

If you do go for a RAID controller watch out for any special = read/write caching options. I found these also killed performance. =

 

 

From:= Alain = RODRIGUEZ [mailto:arodrime@gmail.com]
Sent: 08 August 2013 = 09:30
To: user@cassandra.apache.org
Subject: Re: = lots of small nodes vs fewer big nodes

 

I = advise you to have 7GB RAM better (more would be better if you have a = lot of data per node). But the real difference is made when you use SSDs = or RAIDs of SSDs, since the bottleneck is the disk throughput in most = cases.

 

We are in the cloud, we tried a lot of configurations = and were comfortable only with a nodes RAM > 7 GB (RAM >15 GB with = lot of data / node) and see a real enhancement when we switched to SSDs = (latency from 20-40 to 3-5 ms), even reducing the number of nodes from = 18 to 3. This was quite impressive and I recommend SSD (or RAID of SSDs) = since to anyone who can afford it.

 

Alain

 

 

 

2013/8/7 Andrey Ilinykh <ailinykh@gmail.com>

You still have the same amount of RAM, so you cache = the same amount of data. I don't think you gain much here. On the other = side, maintenance procedures (compaction, repair) may hit your 2CPU box. = I wouldn't do it.

 

Thank you,

  = Andrey

 

On Wed, Aug 7, 2013 at 10:24 AM, Paul Ingalls <paulingalls@gmail.com> wrote:

Quick question about systems = architecture.

Would it be better to run 5 nodes with 7GB RAM and = 4CPU's or 10 nodes with 3.5GB RAM and 2CPUS?

I'm currently = running the former, but am considering the latter.  My goal would = be to improve overall performance by spreading the IO across more disks. =  My currently cluster has low CPU utilization but does spend a good = amount of time in iowait.  Would moving to more smaller nodes help = with that?  Or would I run into trouble with the smaller ram and = cpu?

Thanks!

Paul

 

 

------=_NextPart_000_0078_01CE941D.CC769910--