cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott White <>
Subject Re: EC2, XFS, and ec2-consistent-snapshot with Cassandra
Date Wed, 14 Apr 2010 02:15:56 GMT
> I've implemented this with MySQL before, and it worked extremely well
> (miles beyond mysqldump or mysqlhotcopy).  On a given node, you sacrifice a
> short period of availability (less than 0.5 seconds) to get a full,
> consistent snapshot of your EBS volume that can be sent off to S3 in the
> background, after the filesystem has unlocked and disk activity has
> resumed.  Has anybody tried implementing this with a Cassandra cluster?
> What are the issues you ran into?

Yes I have implemented this and it works as one would expect. I would
recommend stopping one cassandra jvm at a time and fail fast so if anything
happens to one node, the worst case is that you have to swap out that one
node assuming you have replication set up but I haven't run into any issues.
With the instance taken down freezing the filesystem is a precautionary
measure in case it accidentally gets restarted, etc.

> How did it compare with using Cassandra's "nodetool snapshot"?

They both work well but serve different needs. EC2 snapshotting your nodes
makes sense if, for example, you run the entire cluster in the same
availability zone and you want to be able to restore from backup if the data
center has an outage. It's also handy for things like "cloning" your cluster
since you can create a new cluster of the same size and loading the data off
the ec2 snapshots. That being said, EC2 machines can be very unreliable in
terms of network latency, etc. so being tightly coupled to EC2 can be risky.
If you have a rack aware strategy with your cluster partitioned across
different availability zones, nodetool snapshot may be all you need.

> I think I could do this on a running node with a 0.5 second timeout.  The
> XFS docs state "Any process attempting to write to the frozen filesystem
> will block  waiting  for  the  filesystem  to  be unfrozen."  Having writes
> block on a node for <0.5s sounds like something the Cassandra would handle
> fine.

If you just stop one instance at a time, it's a non-issue if you have say
RF=3 and CL=quorum. Reads and writes will just be redirected to other up
nodes while the node is down.

> The Cassandra docs state "You can get an eventually consistent backup by
> flushing all nodes and snapshotting; no individual node's backup is
> guaranteed to be consistent but if you restore from that snapshot then
> clients will get eventually consistent behavior as usual."  This lead me to
> believe that as long as I have snapshot each node in the cluster within a
> reasonable window (say 2 hours), I'd be able to bring the entire cluster
> back with a guarantee that it is consistent up to the point where the
> snapshot window began.

Right, one thing to keep in mind is depending on how much data you are
storing on each box, the ec2 snapshot may not finish within 2 hours. Also if
you are using cassandra snapshot, you don't want to just keep snapshotting
without either removing or moving the data of previous snapshots off the
disk since the disk will fill up quickly.

View raw message