cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <ober...@civicscience.com>
Subject Re: advice for EC2 deployment
Date Tue, 26 Apr 2011 13:18:12 GMT
Thanks Aaron!

Unless no one on this list uses EC2, there were a few minor troubles end of
last week through the weekend which taught me a lot about obscure failure
modes in various applications I use :-)  My original post was trying to be
more redundant than fast, which has been by overall goal from even before
moving to Cassandra (my downtime from the EC2 madness was minimal, and due
to only having one single point of failure == the amazon load balancer).  My
secondary goal was  trying to make moving to a second region easier, but is
that is causing problems I can drop the idea.

I might be downplaying the cost of inter-AZ communication, but I've lived
with that for quite some time, for example my current setup of MySQL in
Master-Master replication is split over zones, and my webservers live in yet
different zones.  Maybe Cassandra is "chattier" than I'm used to?  (again,
I'm fairly new to cassandra)

Based on that article, the discussion, and the recent EC2 issues, it sounds
like it would be better to start with:
-6 nodes split in two AZs 3/3
-Configure replication to do 2 in one AZ and one in the other
(NetworkTopology treats AZs as racks, so does RF=3,us-east=3 make this
happen naturally?)
-What does LOCAL_QUORUM do in this case?  Is there a "rack quorum"?  Or does
the natural latencies of AZs make LOCAL_QUORUM behave like a rack quorum?

will

On Tue, Apr 26, 2011 at 1:14 AM, aaron morton <aaron@thelastpickle.com>wrote:

> For background see this article:
>
> http://www.datastax.com/dev/blog/deploying-cassandra-across-multiple-data-centers
>
>
> <http://www.datastax.com/dev/blog/deploying-cassandra-across-multiple-data-centers>And
> this recent discussion
> http://www.mail-archive.com/user@cassandra.apache.org/msg12502.html
>
> <http://www.mail-archive.com/user@cassandra.apache.org/msg12502.html>Issues
> that may be a concern:
> - lots of cross AZ latency in us-east, e.g. LOCAL_QUORUM ops must wait
> cross AZ . Also consider it during maintenance tasks, how much of a pain is
> it going to be to have latency between every node.
> - IMHO not having sufficient (by that I mean 3) replicas in a cassandra DC
> to handle a single node failure when working at Quorum reduces the utility
> of the DC. e.g. with a local RF of 2 in the west, the quorum is 2, and if
> you lose one node from the replica set you will not be able to use local
> QUORUM for keys in that range. Or consider a failure mode where the west is
> disconnected from the east.
>
> Could you start simple with 3 replicas in one AZ in us-east and 3 replicas
> in an AZ+Region ?  Then work through some failure scenarios.
>
> Hope that helps.
> Aaron
>
>
> On 22 Apr 2011, at 03:28, William Oberman wrote:
>
> Hi,
>
> My service is not yet ready to be fully multi-DC, due to how some of my
> legacy MySQL stuff works.  But, I wanted to get cassandra going ASAP and
> work towards multi-DC.  I have two main cassandra use cases: one where I can
> handle eventual consistency (and all of the writes/reads are currently ONE),
> and one where I can't (writes/reads are currently QUORUM).  My test cluster
> is currently 4 smalls all in us-east with RF=3 (more to prove I can
> clustering, than to have an exact production replica).  All of my unit
> tests, and "load tests" (again, not to prove true max load, but to more to
> tease out concurrency issues) are passing now.
>
> For production, I was thinking of doing:
> -4 cassandra larges in us-east (where I am now), once in each AZ
> -1 cassandra large in us-west (where I have nothing)
> For now, my data can fit into a single large's 2 disk ephemeral using
> RAID0, and I was then thinking of doing a RF=3 with us-east=2 and
> us-west=1.  If I do eventual consistency at ONE, and consistency at
> LOCAL_QUORUM, I was hoping:
> -eventual consistency ops would be really fast
> -consistent ops would be pretty fast (what does LOCAL_QUORUM do in this
> case?  return after 1 or 2 us-east nodes ack?)
> -us-west would contain a complete copy of my data, so it's a good
> eventually consistent "close to real time" backup  (assuming it can keep up
> over long periods of time, but I think it should)
> -eventually, when I'm ready to roll out in us-west I'll be able to change
> the replication settings and that server in us-west could help seed new
> cassandra instances faster than the ones in us-east
>
> Or am I missing something really fundamental about how cassandra works
> making this a terrible plan?  I should have plenty of time to get my
> multi-DC working before the instance in us-west fills up (but even then, I
> should be able to add instances over there to stall fairly trivially,
> right?).
>
> Thanks!
>
> will
>
>
>


-- 
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) oberman@civicscience.com

Mime
View raw message