cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Node Inconsistency
Date Mon, 10 Jan 2011 21:31:30 GMT
> above.  From looking at the data I'm guessing that the results from the 3
> nodes are correct and the results from the 2 nodes are old (the diff between
> the result sets is that the 54 is a subset of the 68).

If I interpret the thread correctly, those 2 that you say you believe
are old are the ones that are returning extra results. So, returning
results that should no longer be there. That suggests to me that you
are seeing old values that have since been deleted (unless you're
seeing some kind of arbitrary random data popping up).

Is it possible that you're not running 'nodetool repair' frequently
enough with respect to GCGraceSeconds? The negative results of that
would be that deletions are essentially forgotten if they don't reach
all nodes in time. Have you so far only seen incorrect results in the
form of additional data that should be deleted, or have you also seen
old versions of columns? (That refuse to self-heal, that is.)

Now, GCGraceSeconds/repair issues would not explain, to me at least,
why read-repair is not fixing the discrepancy. I don't think I saw you
explicitly confirm it, so I'll ask: Are you indeed running with read
repair turned on?

If it is turned off, and if all your discrepancies are in the form of
forgotten deletes, then GCGraceSeconds/repair seems like a likely
candidate cause.

/ Peter Schuller

View raw message