cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Sigletos <sigle...@textkernel.nl>
Subject Re: Inconsistent data after adding a new DC and rebuilding
Date Tue, 11 Apr 2017 11:49:46 GMT
Thanks for your reply. Yes, it would be nice to know the root cause.

Now running a full repair. Hopefully this will solve the problem

On Tue, Apr 11, 2017 at 9:43 AM, Roland Otta <Roland.Otta@willhaben.at>
wrote:

> well .. thats pretty much the same we saw in our environment (cassandra
> 3.7).
>
> in our case a full repair fixed the issues.
> but no doubt .. it would be more satisfying to know the root cause for
> that issue
>
> br,
> roland
>
>
> On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote:
>
> In 3 out of 5 nodes of our new DC the rebuild process finished
> successfully. In the other two nodes not (the process was hanging doing
> nothing) so we killed it, removed all data and started again. This time
> finished successfully.
>
> Here is the netstats output of one of the new newly added nodes:
>
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 269142
> Mismatch (Blocking): 169866
> Mismatch (Background): 4
> Pool Name                    Active   Pending      Completed   Dropped
> Commands                        n/a         2       10031126      1935
> Responses                       n/a        97       22565129       n/a
>
>
> On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta <Roland.Otta@willhaben.at>
> wrote:
>
> Hi,
>
> we have seen similar issues here.
>
> have you verified that your rebuilds have been finished successfully? we
> have seen rebuilds that stopped streaming and working but have not finished.
> what does nodetool netstats output for your newly built up nodes?
>
> br,
> roland
>
>
> On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
>
> Hello,
>
> We recently added a new datacenter to our cluster and run "nodetool
> rebuild -- <DC_current>" in all 5 new nodes, one by one.
>
> After this process finished we noticed there is data missing from the new
> datacenter, although it exists on the current one.
>
> How would that be possible? Should I maybe have run repair in all nodes of
> the current DC before adding the new one?
>
> Running Cassandra 2.1.15
>
> Kind regards,
> George
>
>
>
>
>
>

Mime
View raw message