cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Ortolani <>
Subject Re: Incremental repairs leading to unrepaired data
Date Wed, 10 Aug 2016 11:41:06 GMT
That's what I was thinking. Maybe GC pressure?
Some more details: during anticompaction I have some CFs exploding to 1K
SStables (to be back to ~200 upon completion).
HW specs should be quite good (12 cores/32 GB ram) but, I admit, still
relying on spinning disks, with ~150GB per node.
Current version is 3.0.8.

On Wed, Aug 10, 2016 at 12:36 PM, Paulo Motta <>

> That's pretty low already, but perhaps you should lower to see if it will
> improve the dropped mutations during anti-compaction (even if it increases
> repair time), otherwise the problem might be somewhere else. Generally
> dropped mutations is a signal of cluster overload, so if there's nothing
> else wrong perhaps you need to increase your capacity. What version are you
> in?
> 2016-08-10 8:21 GMT-03:00 Stefano Ortolani <>:
>> Not yet. Right now I have it set at 16.
>> Would halving it more or less double the repair time?
>> On Tue, Aug 9, 2016 at 7:58 PM, Paulo Motta <>
>> wrote:
>>> Anticompaction throttling can be done by setting the usual
>>> compaction_throughput_mb_per_sec knob on cassandra.yaml or via nodetool
>>> setcompactionthroughput. Did you try lowering that  and checking if that
>>> improves the dropped mutations?
>>> 2016-08-09 13:32 GMT-03:00 Stefano Ortolani <>:
>>>> Hi all,
>>>> I am running incremental repaird on a weekly basis (can't do it every
>>>> day as one single run takes 36 hours), and every time, I have at least one
>>>> node dropping mutations as part of the process (this almost always during
>>>> the anticompaction phase). Ironically this leads to a system where
>>>> repairing makes data consistent at the cost of making some other data not
>>>> consistent.
>>>> Does anybody know why this is happening?
>>>> My feeling is that this might be caused by anticompacting column
>>>> families with really wide rows and with many SStables. If that is the case,
>>>> any way I can throttle that?
>>>> Thanks!
>>>> Stefano

View raw message