cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Increase compaction performance
Date Thu, 11 Feb 2016 16:38:52 GMT
Also, are you using incremental repairs (not sure about the available
options in Spotify Reaper) what command did you run ?

2016-02-11 17:33 GMT+01:00 Alain RODRIGUEZ <arodrime@gmail.com>:

> CPU load is fine, SSD disks below 30% utilization, no long GC pauses
>
>
>
> What is your current compaction throughput ?  The current value of
> 'concurrent_compactors' (cassandra.yaml or through JMX) ?
>
> nodetool getcompactionthroughput
>
> How to speed up compaction? Increased compaction throughput and concurrent
>> compactors but no change. Seems there is plenty idle resources but can't
>> force C* to use it.
>>
>
> You might want to try un-throttle the compaction throughput through:
>
> nodetool setcompactionsthroughput 0
>
> Choose a canari node. Monitor compaction pending and disk throughput (make
> sure server is ok too - CPU...)
>
> Some other information could be useful:
>
> What is your number of cores per machine and the compaction strategies for
> the 'most compacting' tables. What are write/update patterns, any TTL or
> tombstones ? Do you use a high number of vnodes ?
>
> Also what is your repair routine and your values for gc_grace_seconds ?
> When was your last repair and do you think your cluster is suffering of a
> high entropy ?
>
> You can lower the stream throughput to make sure nodes can cope with what
> repairs are feeding them.
>
> nodetool getstreamthroughput
> nodetool setstreamthroughput X
>
> C*heers,
>
> -----------------
> Alain Rodriguez
> France
>
> The Last Pickle
> http://www.thelastpickle.com
>
> 2016-02-11 16:55 GMT+01:00 Michał Łowicki <mlowicki@gmail.com>:
>
>> Hi,
>>
>> Using 2.1.12 across 3 DCs. Each DC has 8 nodes. Trying to run repair
>> using Cassandra Reaper but nodes after couple of hours are full of pending
>> compaction tasks (regular not the ones about validation)
>>
>> CPU load is fine, SSD disks below 30% utilization, no long GC pauses.
>>
>> How to speed up compaction? Increased compaction throughput and
>> concurrent compactors but no change. Seems there is plenty idle
>> resources but can't force C* to use it.
>>
>> Any clue where there might be a bottleneck?
>>
>>
>> --
>> BR,
>> Michał Łowicki
>>
>>
>

Mime
View raw message