Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of dan.hendry.junk@gmail.com
 designates 209.85.216.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=b/rEAFLAk4c02ASKE70N+QyUwDV6sFMc9ViN0S/XSBLJyzaQg4xPGqHUh4TFQcB2xz
         88ygRHXDtbjwXLD6oijPqDnFjbPHdINPz633ScL8MIDuGM5XSEQ2/wPGjfeT7meZxazT
         eYzWbq6JIWkx4QGOVED4pWx9PJzeHixHx4Xts=
MIME-Version: 1.0
Date: Fri, 3 Dec 2010 22:19:52 -0500
Message-ID: <AANLkTimVb6nuMvP+i2G5eX2ecxR1vFi2n9nxaM31kT0F@mail.gmail.com>
Subject: Confused about consistency
From: Dan Hendry <dan.hendry.junk@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001485ea3c74d4d91004968d207b

--001485ea3c74d4d91004968d207b
Content-Type: text/plain; charset=ISO-8859-1

I am seeing fairly strange, behavior in my Cassandra cluster.

Setup
 - 3 nodes (lets call them nodes 1 2 and 3)
 - RF=2
 - A set of servers (producers) which which write data to the cluster at
consistency level ONE
 - A set of servers (consumers/processors) which read data from the cluster
at consistency level ALL
 - Cassandra 0.7 (recent out of the svn branch, post beta 3)
 - Clients use the pelops library

Situation:
 - Everything is humming along nicely
 - A Cassandra node (say 3) goes down (even with 24 GB of ram, OOM errors
are the bain of my existence)
 - Producers continue to happily write to the cluster but consumers start
complaining by throwing TimeOutExceptions and UnavailableExceptions.
 - I stagger out of bed in the middle of the night and restart Cassandra on
node 3.
 - The consumers stop complaining and get back to business but generate
garbage data for the period node 3 was down. Its almost like half the data
is missing half the time. (Again, I am reading at consistency level ALL).
 - I force the consumers to reprocess data for the period node 3 was down.
They generate accurate output which is different from the first time round.

To be explicit, what seems to be happening is first read at consistency ALL
gives "A,C,E" (for example) and the second read at consistency level ALL
gives "A,B,C,D,E". Is this a Cassandra bug? Is my knowledge of consistency
levels flawed? My understanding is that you could achieve strongly
consistent behavior by writing at ONE and reading at ALL.

After this experience, my theory (uneducated, untested, and
under-researched) is that "strong consistency" applies only to column
values, not the set of columns (or super-columns in this case) which make up
a row. Any thoughts?

--001485ea3c74d4d91004968d207b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am seeing fairly strange, behavior in my Cassandra cluster.<div><br></div=
><div>Setup</div><div>=A0- 3 nodes (lets call them nodes 1 2 and 3)</div><d=
iv><meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"=
>=A0- RF=3D2</div>
<div>=A0- A set of servers=A0(producers)=A0which which write data to the cl=
uster at consistency level ONE</div><div>=A0- A set of servers (consumers/p=
rocessors) which read data from the cluster at consistency level ALL</div><=
meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><di=
v>
=A0- Cassandra 0.7 (recent out of the svn branch, post beta 3)</div><div>=
=A0- Clients use the pelops library</div><div><br></div><div>Situation:</di=
v><div>=A0- Everything is humming along nicely</div><div>=A0- A Cassandra n=
ode (say 3) goes down (even with 24 GB of ram, OOM errors are the bain of m=
y existence)</div>
<div>=A0- Producers continue to happily write to the cluster but consumers =
start complaining by throwing TimeOutExceptions and UnavailableExceptions.<=
/div><div>=A0- I stagger out of bed in the middle of the night and restart =
Cassandra on node 3.</div>
<div>=A0- The consumers stop complaining and get back to business but gener=
ate garbage data for the period node 3 was down. Its almost like half the d=
ata is missing half the time. (Again, I am reading at consistency level ALL=
).</div>
<div>=A0- I force the consumers to reprocess data for the period node 3 was=
 down. They generate accurate output which is different from the first time=
 round.=A0</div><div><br></div><div>To be explicit, what seems to be happen=
ing is first read at consistency ALL gives &quot;A,C,E&quot; (for example) =
and the second read at consistency level ALL gives &quot;A,B,C,D,E&quot;. I=
s this a Cassandra bug? Is my knowledge of consistency levels flawed? My un=
derstanding is that you could achieve strongly consistent behavior by writi=
ng at ONE and reading at ALL.=A0</div>
<div><br></div><div>After this experience, my theory (uneducated, untested,=
 and under-researched) is that &quot;strong consistency&quot; applies only =
to column values, not the set of columns (or super-columns in this case) wh=
ich make up a row. Any thoughts?</div>

--001485ea3c74d4d91004968d207b--