cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atul Saroha <atul.sar...@snapdeal.com>
Subject Re: [cassandra 3.6.] Nodetool Repair + tombstone behaviour
Date Thu, 29 Sep 2016 10:33:05 GMT
Hi,

We are not sure whether this issue is linked to that node or not. Our
application does frequent delete and insert.

May be our approach is not correct for nodetool repair. Yes, we generally
fire repair on all boxes at same time. Till now, it was manual with default
configuration ( command: "nodetool repair").
Yes, we saw validation error but that is linked to already running repair
of  same partition on other box for same partition range. We saw error
validation failed with some ip as repair in already running for the same
SSTable.
Just few days back, we had 2 DCs with 3 nodes each and replication was also
3. It means all data on each node.

On Thu, Sep 29, 2016 at 2:49 PM, Alexander Dejanovski <
alex@thelastpickle.com> wrote:

> Hi Atul,
>
> could you be more specific on how you are running repair ? What's the
> precise command line for that, does it run on several nodes at the same
> time, etc...
> What is your gc_grace_seconds ?
> Do you see errors in your logs that would be linked to repairs (Validation
> failure or failure to create a merkle tree)?
>
> You seem to mention a single node that went down but say the whole cluster
> seem to have zombie data.
> What is the connection you see between the node that went down and the
> fact that deleted data comes back to life ?
> What is your strategy for cyclic maintenance repair (schedule, command
> line or tool, etc...) ?
>
> Thanks,
>
> On Thu, Sep 29, 2016 at 10:40 AM Atul Saroha <atul.saroha@snapdeal.com>
> wrote:
>
>> Hi,
>>
>> We have seen a weird behaviour in cassandra 3.6.
>> Once our node was went down more than 10 hrs. After that, we had ran
>> Nodetool repair multiple times. But tombstone are not getting sync properly
>> over the cluster. On day- today basis, on expiry of every grace period,
>> deleted records start surfacing again in cassandra.
>>
>> It seems Nodetool repair in not syncing tomebstone across cluster.
>> FYI, we have 3 data centres now.
>>
>> Just want the help how to verify and debug this issue. Help will be
>> appreciated.
>>
>>
>> --
>> Regards,
>> Atul Saroha
>>
>> *Lead Software Engineer | CAMS*
>>
>> M: +91 8447784271
>> Plot #362, ASF Center - Tower A, 1st Floor, Sec-18,
>> Udyog Vihar Phase IV,Gurgaon, Haryana, India
>>
>> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 
Regards,
Atul Saroha

*Lead Software Engineer | CAMS*

M: +91 8447784271
Plot #362, ASF Center - Tower A, 1st Floor, Sec-18,
Udyog Vihar Phase IV,Gurgaon, Haryana, India

Mime
View raw message