incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Major <al3...@gmail.com>
Subject Re: Expanding Cassandra on EC2 with consistency
Date Wed, 04 Jul 2012 12:52:17 GMT
Hi Dan,

We run RF 2 on RAID0 EBS drives. The reason we use EBS over on-instance
storage is two fold;

Firstly we have a relatively small cluster ( 4 nodes ), so we're quite
sensitive to any AWS issues (at the region level). If we had a larger
cluster then we would definitely use ephemeral storage as ephemeral storage
will provide much more consistent (*slightly higher*) throughput than EBS.
I'm sure you've read a lot about how bad EBS performance is, but genuinely
we see very little difference between EBS / ephemeral storage in terms of
performance when in a RAID0 setup. Some numbers which are similar to our
tests can be found here:
http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html.
The only way you'll know for yourself whether EBS is acceptable is
running your own tests, but definitely take anything you read on blog posts
etc about EBS performance with a pinch of salt. The only real disadvantage
of EBS in our experience is that it's an additional moving part that will
fail. You can find yourself in a position where EC2 is running normally but
EBS is down, thus your nodes are down.

The second reason, and the most important for us was that we didn't have
time to build a good automated backup service for the ephemeral storage.
The advantage of EBS is that the instance can fail and we can start a new
one using the same drive, where-as data on ephemeral will get lost if the
node is lost/shutdown. Using ephemeral storage would put us in a difficult
position of loosing some data should a couple of instances fail for
whatever reason (we don't loose instances often, however when we do we tend
to loose several). If you run a larger cluster (imo 10+) then definitely
use ephemeral as you shouldn't be very sensitive to loosing a node or two.

The second point however doesn't hold as strong today as it did when we
made our decision last year. Netflix recently open-sourced a really good
tool ( https://github.com/Netflix/Priam/wiki/Backups ) which will automate
the back-up of Cassandra data to S3. I'd definitely recommend checking it
out to see if it will help you with AWS backups / restores, we're currently
looking at rolling it out.

Hope that helps,

Alex.

On Wed, Jul 4, 2012 at 2:56 AM, Dan Foody <dan.foody@gmail.com> wrote:

> Hi Alex,
>
> Can you share what replication factor you're running?
> And, are you using ephemeral disks or EBS volumes?
>
> Thanks!
>
> - Dan
>
>
>
> On Jul 3, 2012, at 5:52 PM, Alex Major wrote:
>
> Hi Mike,
>
> We've run a small (4 node) cluster in the EU region since September last
> year. We run across all 3 availability zones in the EU region, with 2 nodes
> in one AZ and then a further node in each AZ. The latency difference
> between running inside of and between AZ's has been minimal in our
> experience.
>
> It's only when we've gone cross-region that there's been latency problem.
> We temporarily ran a 9 node cluster across 3 regions, however even then
> using local quoram the latency was better than the standard datacenter -
> datacenter latency we're used to.
>
> EC2Snitch is definitely the way to go in favour of NTS in my opinion. NTS
> was a pain to get setup with the internal (private) IP address setup, so
> much so that we never got it safely replicating the data as we wanted.
>
> Alex.
>
> On Tue, Jul 3, 2012 at 2:16 PM, Michael Theroux <mtheroux2@yahoo.com>wrote:
>
>> Hello,
>>
>> We are currently running a web application utilizing Cassandra on EC2.
>>  Given the recent outages experienced with Amazon, we want to consider
>> expanding Cassandra across availability zones sooner rather than later.
>>
>> We are trying to determine the optimal way to deploy Cassandra in this
>> deployment.  We are researching the NetworkTopologyStrategy, and the
>> EC2Snitch.  We are also interested in providing a high level of read or
>> write consistency,
>>
>> My understanding is that the EC2Snitch recognizes availability zones as
>> racks, and regions as data-centers.  This seems to be a common
>> configuration.  However, if we were to want to utilize queries with a READ
>> or WRITE consistency of QUORUM, would there be a high possibility that the
>> communication necessary to establish a quorum, across availability zones?
>>
>> My understanding is that the NetworkTopologyStrategy attempts to prefer
>> replicas be stored on other racks within the datacenter, which would equate
>> to other availability zones in EC2.  This implies to me that in order to
>> have the quorum of nodes necessary to achieve consistency, that Cassandra
>> will communicate with nodes across availability zones.
>>
>> First, is my understanding correct?  Second, given the high latency that
>> can sometimes exists between availability zones, is this a problem, and
>> instead we should treat availability zones as data centers?
>>
>> Ideally, we would be able to setup a situation where we could store
>> replicas across availability zones in case of failure, but establish a high
>> level of read or write consistency within a single availability zone.
>>
>> I appreciate your responses,
>> Thanks,
>> -Mike
>>
>>
>>
>>
>
>

Mime
View raw message