cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roland Otta <Roland.O...@willhaben.at>
Subject Re: Inconsistent data after adding a new DC and rebuilding
Date Tue, 11 Apr 2017 07:43:10 GMT
well .. thats pretty much the same we saw in our environment (cassandra 3.7).

in our case a full repair fixed the issues.
but no doubt .. it would be more satisfying to know the root cause for that issue

br,
roland


On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote:
In 3 out of 5 nodes of our new DC the rebuild process finished successfully. In the other
two nodes not (the process was hanging doing nothing) so we killed it, removed all data and
started again. This time finished successfully.

Here is the netstats output of one of the new newly added nodes:

Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 269142
Mismatch (Blocking): 169866
Mismatch (Background): 4
Pool Name                    Active   Pending      Completed   Dropped
Commands                        n/a         2       10031126      1935
Responses                       n/a        97       22565129       n/a


On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta <Roland.Otta@willhaben.at<mailto:Roland.Otta@willhaben.at>>
wrote:
Hi,

we have seen similar issues here.

have you verified that your rebuilds have been finished successfully? we have seen rebuilds
that stopped streaming and working but have not finished.
what does nodetool netstats output for your newly built up nodes?

br,
roland


On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
Hello,

We recently added a new datacenter to our cluster and run "nodetool rebuild -- <DC_current>"
in all 5 new nodes, one by one.

After this process finished we noticed there is data missing from the new datacenter, although
it exists on the current one.

How would that be possible? Should I maybe have run repair in all nodes of the current DC
before adding the new one?

Running Cassandra 2.1.15

Kind regards,
George





Mime
View raw message