cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Ortolani <ostef...@gmail.com>
Subject Re: Incremental repairs leading to unrepaired data
Date Wed, 10 Aug 2016 11:41:06 GMT
That's what I was thinking. Maybe GC pressure?
Some more details: during anticompaction I have some CFs exploding to 1K
SStables (to be back to ~200 upon completion).
HW specs should be quite good (12 cores/32 GB ram) but, I admit, still
relying on spinning disks, with ~150GB per node.
Current version is 3.0.8.


On Wed, Aug 10, 2016 at 12:36 PM, Paulo Motta <pauloricardomg@gmail.com>
wrote:

> That's pretty low already, but perhaps you should lower to see if it will
> improve the dropped mutations during anti-compaction (even if it increases
> repair time), otherwise the problem might be somewhere else. Generally
> dropped mutations is a signal of cluster overload, so if there's nothing
> else wrong perhaps you need to increase your capacity. What version are you
> in?
>
> 2016-08-10 8:21 GMT-03:00 Stefano Ortolani <ostefano@gmail.com>:
>
>> Not yet. Right now I have it set at 16.
>> Would halving it more or less double the repair time?
>>
>> On Tue, Aug 9, 2016 at 7:58 PM, Paulo Motta <pauloricardomg@gmail.com>
>> wrote:
>>
>>> Anticompaction throttling can be done by setting the usual
>>> compaction_throughput_mb_per_sec knob on cassandra.yaml or via nodetool
>>> setcompactionthroughput. Did you try lowering that  and checking if that
>>> improves the dropped mutations?
>>>
>>> 2016-08-09 13:32 GMT-03:00 Stefano Ortolani <ostefano@gmail.com>:
>>>
>>>> Hi all,
>>>>
>>>> I am running incremental repaird on a weekly basis (can't do it every
>>>> day as one single run takes 36 hours), and every time, I have at least one
>>>> node dropping mutations as part of the process (this almost always during
>>>> the anticompaction phase). Ironically this leads to a system where
>>>> repairing makes data consistent at the cost of making some other data not
>>>> consistent.
>>>>
>>>> Does anybody know why this is happening?
>>>>
>>>> My feeling is that this might be caused by anticompacting column
>>>> families with really wide rows and with many SStables. If that is the case,
>>>> any way I can throttle that?
>>>>
>>>> Thanks!
>>>> Stefano
>>>>
>>>
>>>
>>
>

Mime
View raw message