cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Cassandra 2 DC deployment
Date Fri, 15 Apr 2011 19:47:20 GMT
> You are right about the automatic fallback to ONE. Its quite possible, if 2 nodes die
for some reason I will have the same problem. So probably the right thing to do would be to
read/write at ONE only when we lose a DC by changing some manual configuration. Since we shouldn't
be losing DCs that often, this should be an acceptable change. So my follow up questions would
be -

Seems reasonable to have a human do it, since it seems that you really
want QUORUM - so presumably there is some kind of negative impact and
you don't want that sporadically happening every time there is a
hiccup. But of course I don't know the context.

> When would be the right time to start reading/writing at QUORUM again?

I'd say usually as soon as possible, but it will depend on details of
your situation. For example, if you have 2 DC:s with 5 nodes in one
and 1 node in another, and there is a partition - the DC with just one
node will start seeing "older" data (from the point of view of writes
done in the 1-node DC) if you start asking for quorum since a lot of
the time a quorum will be 4 nodes in the "other" DC. So if there is
interest in preferring the "local" dc's copy of the data after an
emergency fallback to CL.ONE, it may be detrimental to go QUORUM too

But this will depend on what your application is actually doing and
what is important to you.

> Should we be marking the 2 nodes in the lost DC as down?
> Should we be doing some administrative work on Cassandra before we start reading/writing
at QUORUM again?

Are you talking about permanently losing a DC then, rather than just a
transient partition? For non-permanent situations it seems
counter-productive to mark other DC's nodes as down. Oh and btw, keep
in mind you can choose to use LOCAL_QUORUM to get intra-site
consistency (rather than ONE).

As for administrative work: I can't answer in general since we're
talking about very special circumstances, but at least it's valid to
say that whenever you have some kind of issue that has caused
inconsistency, running 'nodetool repair' (perhaps earlier than the
standard weekly/whatever repair) is the most efficient way to achieve
consistency again.

/ Peter Schuller

View raw message