Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E08CFD82C for ; Wed, 4 Jul 2012 12:52:46 +0000 (UTC) Received: (qmail 16253 invoked by uid 500); 4 Jul 2012 12:52:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 16090 invoked by uid 500); 4 Jul 2012 12:52:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 16066 invoked by uid 99); 4 Jul 2012 12:52:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jul 2012 12:52:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of al3xdm@gmail.com designates 74.125.82.44 as permitted sender) Received: from [74.125.82.44] (HELO mail-wg0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jul 2012 12:52:38 +0000 Received: by wgbdr13 with SMTP id dr13so5855101wgb.25 for ; Wed, 04 Jul 2012 05:52:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4b9Gk05veF2/Kbbhe1RCljEVZ/5sGSibmjcqqJOfZcI=; b=1FbmmSd281lrf794KRE8bY5M0Ktm10ividWIpd0Ix2DMP4zfdk5CR5M47Md/u0G9wl snFezggp2ElxjcqABxVoc/aPXCnyNCSNXA1s2ReTAVVzMxVJa0Fj3p9rj4j5bx7mgzOk psge06BvayKwhuqxRS2/BYuO1wGyxNp24IyTYTjDQKSqEFvYRyFGFTtaDeHgDxx5zguQ /KYm9ED/ZiKw/p8ASM+aQavSktYBUI9mklVYGBL8d9e3qwufUyO3V71VdHvLZpEeHjaw ggiX2oVRfd0wZO5wtb0L4lb3bO5TIYsMjFxBT0X+pZaSsA7c80e+kVeJuIhxlQOEfQB3 Y2mw== MIME-Version: 1.0 Received: by 10.180.93.68 with SMTP id cs4mr33545179wib.14.1341406337147; Wed, 04 Jul 2012 05:52:17 -0700 (PDT) Received: by 10.194.38.6 with HTTP; Wed, 4 Jul 2012 05:52:17 -0700 (PDT) In-Reply-To: <47D9EE5E-2AA3-4B79-88B6-A5D8C8390B0A@gmail.com> References: <47D9EE5E-2AA3-4B79-88B6-A5D8C8390B0A@gmail.com> Date: Wed, 4 Jul 2012 13:52:17 +0100 Message-ID: Subject: Re: Expanding Cassandra on EC2 with consistency From: Alex Major To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043892bd39a3bb04c4008027 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043892bd39a3bb04c4008027 Content-Type: text/plain; charset=ISO-8859-1 Hi Dan, We run RF 2 on RAID0 EBS drives. The reason we use EBS over on-instance storage is two fold; Firstly we have a relatively small cluster ( 4 nodes ), so we're quite sensitive to any AWS issues (at the region level). If we had a larger cluster then we would definitely use ephemeral storage as ephemeral storage will provide much more consistent (*slightly higher*) throughput than EBS. I'm sure you've read a lot about how bad EBS performance is, but genuinely we see very little difference between EBS / ephemeral storage in terms of performance when in a RAID0 setup. Some numbers which are similar to our tests can be found here: http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html. The only way you'll know for yourself whether EBS is acceptable is running your own tests, but definitely take anything you read on blog posts etc about EBS performance with a pinch of salt. The only real disadvantage of EBS in our experience is that it's an additional moving part that will fail. You can find yourself in a position where EC2 is running normally but EBS is down, thus your nodes are down. The second reason, and the most important for us was that we didn't have time to build a good automated backup service for the ephemeral storage. The advantage of EBS is that the instance can fail and we can start a new one using the same drive, where-as data on ephemeral will get lost if the node is lost/shutdown. Using ephemeral storage would put us in a difficult position of loosing some data should a couple of instances fail for whatever reason (we don't loose instances often, however when we do we tend to loose several). If you run a larger cluster (imo 10+) then definitely use ephemeral as you shouldn't be very sensitive to loosing a node or two. The second point however doesn't hold as strong today as it did when we made our decision last year. Netflix recently open-sourced a really good tool ( https://github.com/Netflix/Priam/wiki/Backups ) which will automate the back-up of Cassandra data to S3. I'd definitely recommend checking it out to see if it will help you with AWS backups / restores, we're currently looking at rolling it out. Hope that helps, Alex. On Wed, Jul 4, 2012 at 2:56 AM, Dan Foody wrote: > Hi Alex, > > Can you share what replication factor you're running? > And, are you using ephemeral disks or EBS volumes? > > Thanks! > > - Dan > > > > On Jul 3, 2012, at 5:52 PM, Alex Major wrote: > > Hi Mike, > > We've run a small (4 node) cluster in the EU region since September last > year. We run across all 3 availability zones in the EU region, with 2 nodes > in one AZ and then a further node in each AZ. The latency difference > between running inside of and between AZ's has been minimal in our > experience. > > It's only when we've gone cross-region that there's been latency problem. > We temporarily ran a 9 node cluster across 3 regions, however even then > using local quoram the latency was better than the standard datacenter - > datacenter latency we're used to. > > EC2Snitch is definitely the way to go in favour of NTS in my opinion. NTS > was a pain to get setup with the internal (private) IP address setup, so > much so that we never got it safely replicating the data as we wanted. > > Alex. > > On Tue, Jul 3, 2012 at 2:16 PM, Michael Theroux wrote: > >> Hello, >> >> We are currently running a web application utilizing Cassandra on EC2. >> Given the recent outages experienced with Amazon, we want to consider >> expanding Cassandra across availability zones sooner rather than later. >> >> We are trying to determine the optimal way to deploy Cassandra in this >> deployment. We are researching the NetworkTopologyStrategy, and the >> EC2Snitch. We are also interested in providing a high level of read or >> write consistency, >> >> My understanding is that the EC2Snitch recognizes availability zones as >> racks, and regions as data-centers. This seems to be a common >> configuration. However, if we were to want to utilize queries with a READ >> or WRITE consistency of QUORUM, would there be a high possibility that the >> communication necessary to establish a quorum, across availability zones? >> >> My understanding is that the NetworkTopologyStrategy attempts to prefer >> replicas be stored on other racks within the datacenter, which would equate >> to other availability zones in EC2. This implies to me that in order to >> have the quorum of nodes necessary to achieve consistency, that Cassandra >> will communicate with nodes across availability zones. >> >> First, is my understanding correct? Second, given the high latency that >> can sometimes exists between availability zones, is this a problem, and >> instead we should treat availability zones as data centers? >> >> Ideally, we would be able to setup a situation where we could store >> replicas across availability zones in case of failure, but establish a high >> level of read or write consistency within a single availability zone. >> >> I appreciate your responses, >> Thanks, >> -Mike >> >> >> >> > > --f46d043892bd39a3bb04c4008027 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Dan,

We run RF 2 on RAID0 EBS drives.=A0The reason we= use EBS over on-instance storage is two fold;

Fir= stly we have a relatively small cluster ( 4 nodes ), so we're quite sen= sitive to any AWS issues (at the region level). If we had a larger cluster = then we would definitely use ephemeral storage as ephemeral storage will pr= ovide much more consistent (*slightly higher*) throughput than EBS. I'm= sure you've read a lot about how bad EBS performance is, but genuinely= we see very little difference between EBS / ephemeral storage in terms of = performance when in a RAID0 setup. Some numbers which are similar to our te= sts can be found here: http://stu.mp/2009/12/disk-io-and-throu= ghput-benchmarks-on-amazons-ec2.html . The only way you'll know for= yourself whether EBS is acceptable is running your own tests, but definite= ly take anything you read on blog posts etc about EBS performance with a pi= nch of salt. The only real disadvantage of EBS in our experience is that it= 's an additional moving part that will fail. You can find yourself in a= position where EC2 is running normally but EBS is down, thus your nodes ar= e down.

The second reason, and the most important for us was th= at we didn't have time to build a good automated backup service for the= ephemeral storage. The advantage of EBS is that the instance can fail and = we can start a new one using the same drive, where-as data on ephemeral wil= l get lost if the node is lost/shutdown. Using ephemeral storage would put = us in a difficult position of loosing some data should a couple of instance= s fail for whatever reason (we don't loose instances often, however whe= n we do we tend to loose several). If you run a larger cluster (imo 10+) th= en definitely use ephemeral as you shouldn't be very sensitive to loosi= ng a node or two.=A0

The second point however doesn't hold as strong tod= ay as it did when we made our decision last year. Netflix recently open-sou= rced a really good tool (=A0https://github.com/Netflix/Priam/wiki/Backups ) which will a= utomate the back-up of Cassandra data to S3. I'd definitely recommend c= hecking it out to see if it will help you with AWS backups / restores, we&#= 39;re currently looking at rolling it out.=A0

Hope that helps,

Alex.

On Wed, Jul 4, 2012 at 2:56 AM, Dan Foody= <dan.foody@gmail.com> wrote:
Hi Alex,=

Can you share what replication factor you're runnin= g?
And, are you using ephemeral disks or EBS volumes?

Thanks!

- Dan



On Jul 3, 2012, at 5:52 PM, Alex Major wrote:

Hi Mike,

We've run a small (4 no= de) cluster in the EU region since September last year. We run across all 3= =A0availability=A0zones in the EU region, with 2 nodes in one AZ and then a= further node in each AZ. The latency difference between running inside of = and between AZ's has been minimal in our experience.=A0

It's only when we've gone cross-region that the= re's been latency problem. We temporarily ran a 9 node cluster across 3= regions, however even then using local quoram the latency was better than = the standard datacenter - datacenter latency we're used to.

EC2Snitch is=A0definitely=A0the way to go in favour of = NTS in my=A0opinion. NTS was a pain to get setup with the internal (private= ) IP address setup, so much so that we never got it safely replicating the = data as we wanted.

Alex.

On Tue, = Jul 3, 2012 at 2:16 PM, Michael Theroux <mtheroux2@yahoo.com> wrote:
Hello,

We are currently running a web application utilizing Cassandra on EC2. =A0G= iven the recent outages experienced with Amazon, we want to consider expand= ing Cassandra across availability zones sooner rather than later.

We are trying to determine the optimal way to deploy Cassandra in this depl= oyment. =A0We are researching the NetworkTopologyStrategy, and the EC2Snitc= h. =A0We are also interested in providing a high level of read or write con= sistency,

My understanding is that the EC2Snitch recognizes availability zones as rac= ks, and regions as data-centers. =A0This seems to be a common configuration= . =A0However, if we were to want to utilize queries with a READ or WRITE co= nsistency of QUORUM, would there be a high possibility that the communicati= on necessary to establish a quorum, across availability zones?

My understanding is that the NetworkTopologyStrategy attempts to prefer rep= licas be stored on other racks within the datacenter, which would equate to= other availability zones in EC2. =A0This implies to me that in order to ha= ve the quorum of nodes necessary to achieve consistency, that Cassandra wil= l communicate with nodes across availability zones.

First, is my understanding correct? =A0Second, given the high latency that = can sometimes exists between availability zones, is this a problem, and ins= tead we should treat availability zones as data centers?

Ideally, we would be able to setup a situation where we could store replica= s across availability zones in case of failure, but establish a high level = of read or write consistency within a single availability zone.

I appreciate your responses,
Thanks,
-Mike






--f46d043892bd39a3bb04c4008027--