Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 59360 invoked from network); 6 Mar 2010 07:13:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Mar 2010 07:13:39 -0000 Received: (qmail 27523 invoked by uid 500); 6 Mar 2010 07:13:23 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 27417 invoked by uid 500); 6 Mar 2010 07:13:23 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 27409 invoked by uid 99); 6 Mar 2010 07:13:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:13:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric.rosenberry@iovation.com designates 74.121.28.15 as permitted sender) Received: from [74.121.28.15] (HELO mail.iovation.com) (74.121.28.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:13:19 +0000 Received: from cpexch01.iovation.com ([10.12.2.21]) by cpexch01.iovation.com ([10.12.2.21]) with mapi; Fri, 5 Mar 2010 23:12:58 -0800 From: "Rosenberry, Eric" To: "cassandra-user@incubator.apache.org" Content-Class: urn:content-classes:message Date: Fri, 5 Mar 2010 23:12:56 -0800 Subject: Cassandra hardware - balancing CPU/memory/iops/disk space Thread-Topic: Cassandra hardware - balancing CPU/memory/iops/disk space Thread-Index: Acq8NWyXgOWF2CTqSl66gg1T5kPBbg== Message-ID: <139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3@cpexch01.iovation.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3cpexch01iovat_" MIME-Version: 1.0 --_000_139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3cpexch01iovat_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I am looking for advice from others that are further along in deploying Cas= sandra in production environments than we are. I want to know what you are= finding your bottlenecks to be. I would feel silly purchasing dual proces= sor quad core 2.93ghz Nehalem machines with 192 gigs of RAM just to find ou= t that the two local SATA disks kept all that CPU and RAM from being useful= (clearly that example would be a dumb). I need to spec out hardware for an "optimal" Cassandra node (though our rea= d/write characteristics are not yet fully defined so let's go with an "aver= age" configuration). My main concern is finding the right balance of: * Available CPU * RAM amount * RAM speed (think Nehalem architecture where memory comes in a few= speeds, though I doubt this is much of a concern as it is mainly dictated = by which processor you buy and how many slots you populate) * Total iops available (i.e. number of disks) * Total disk space available (depending on the ratio of iops/space = deciding on SAS vs. SATA and various rotational speeds) My current thinking is 1U boxes with four 3.5 inch disks since that seems t= o be a readily available config. One big question is should I go with a si= ngle processor Nehalem system to go with those four disks, or would two CPU= 's be useful, and also, how much RAM is appropriate to match? I am making = the assumption that Cassandra nodes are going to be disk bound as they must= do a random read to answer any given query (i.e. indexes in RAM, but all d= ata lives on disk?). The other big decision is what type of hard disks others are finding to pro= vide the optimal ratio of iops to available space? SAS or SATA? And what = rotational speed? Let me throw out here an actual hardware config and feel free to tell me th= e error of my ways: * A SuperMicro SuperServer 6016T-NTRF configured as follows: o 2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem architect= ure (this proc provides a lot of bang for the buck, faster procs get more e= xpensive quickly) o Qty 12, 4 gig 1066mhz DIMMS for a total of 48 gigs RAM (the 4 gig DIMMS= seem to be the price sweet spot) o Dual on board 1 gigabit NIC's (perhaps one for client connections and t= he other for cluster communication?) o Dual power supplies (I don't want to lose half my cluster due to a fail= ure on one power leg) o 4x 1TB SATA disks (this is a complete SWAG) o No RAID controller (all just single individual disks presented to the O= S) - Though is there any down side to using a RAID controller with RAID 0 (= perhaps one single disk for the log for sequential io's, and 3x disks in a = stripe for the random io's) o The on-board IPMI based OOB controller (so we can kick the boxes remote= ly if need be) * http://www.supermicro.com/products/system/1U/6016/SYS-6016T-NTRF.= cfm I can't help but think the above config has way too much RAM and CPU and no= t enough iops capacity. My understanding is that Cassandra does not cache = much in RAM though? Any thoughts are appreciated. Thanks. -Eric _______________________________________________________________ Eric Rosenberry Sr. Infrastructure Architect | Chief Bit Plumber iovation 111 SW Fifth Avenue Suite 3200 Portland, OR 97204 www.iovation.com The information contained in this email message may be privileged, confiden= tial and protected from disclosure. If you are not the intended recipient, = any dissemination, distribution or copying is strictly prohibited. If you t= hink that you have received this email message in error, please notify the = sender by reply email and delete the message and any attachments. --_000_139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3cpexch01iovat_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I am looking for advice from others that are further a= long in deploying Cassandra in production environments than we are.  I want= to know what you are finding your bottlenecks to be.  I would feel silly purchasing dual processor quad core 2.93ghz Nehalem machines with 192 gigs = of RAM just to find out that the two local SATA disks kept all that CPU and RA= M from being useful (clearly that example would be a dumb).

 

I need to spec out hardware for an “optimal̶= 1; Cassandra node (though our read/write characteristics are not yet fully def= ined so let’s go with an “average” configuration).<= /p>

 

My main concern is finding the right balance of:<= /o:p>

·&= nbsp;        Available CPU

·&= nbsp;        RAM amount

·&= nbsp;        RAM speed (think Nehalem architecture where memory comes in a few speeds, though I doubt this is much of a concern as i= t is mainly dictated by which processor you buy and how many slots you populate)=

·&= nbsp;        Total iops available (i.e. number of disks)<= o:p>

·&= nbsp;        Total disk space available (depending on the ratio of iops/space deciding on SAS vs. SATA and various rotational speeds)=

 

My current thinking is 1U boxes with four 3.5 inch dis= ks since that seems to be a readily available config.  One big question i= s should I go with a single processor Nehalem system to go with those four di= sks, or would two CPU’s be useful, and also, how much RAM is appropriate t= o match?  I am making the assumption that Cassandra nodes are going to b= e disk bound as they must do a random read to answer any given query (i.e. indexes in RAM, but all data lives on disk?).

 

The other big decision is what type of hard disks othe= rs are finding to provide the optimal ratio of iops to available space?  SAS = or SATA?  And what rotational speed?

 

Let me throw out here an actual hardware config and fe= el free to tell me the error of my ways:

·&= nbsp;        A SuperMicro SuperServer 6016T-NTRF configur= ed as follows:

o   2.26 ghz E5520 dual processor quad core hyperthreaded Nehalem architecture (this proc provides a lot of bang for th= e buck, faster procs get more expensive quickly)

o   Qty 12, 4 gig 1066mhz DIMMS for a total of 4= 8 gigs RAM (the 4 gig DIMMS seem to be the price sweet spot)

o   Dual on board 1 gigabit NIC’s (perhaps= one for client connections and the other for cluster communication?)=

o   Dual power supplies (I don’t want to l= ose half my cluster due to a failure on one power leg)

o   4x 1TB SATA disks (this is a complete SWAG)<= o:p>

o   No RAID controller (all just single individu= al disks presented to the OS) – Though is there any down side to using a RAID controller with RAID 0 (perhaps one single disk for the log for sequen= tial io’s, and 3x disks in a stripe for the random io’s)<= /p>

o   The on-board IPMI based OOB controller (so w= e can kick the boxes remotely if need be)

·&= nbsp;        http://www.supermicro.com/products/sys= tem/1U/6016/SYS-6016T-NTRF.cfm

 

I can’t help but think the above config has way = too much RAM and CPU and not enough iops capacity.  My understanding is th= at Cassandra does not cache much in RAM though?

 

Any thoughts are appreciated.  Thanks.=

 

-Eric

______________________________________________________________= _
Eric Rosenberry
Sr. Infrastructure Architect | Chief Bit Plumber

 

 
iovation<= BR>111 SW Fifth Avenue
Suite 3200
Portland, OR 97204
www.iovation.com
 
The information c= ontained in this email message may be privileged, confidential and protecte= d from disclosure. If you are not the intended recipient, any dissemination= , distribution or copying is strictly prohibited. If you think that you hav= e received this email message in error, please notify the sender by reply e= mail and delete the message and any attachments.
--_000_139EF7B8AE318D408CFCB51C899CAF3403C8D5E9F3cpexch01iovat_--