Also, are you using incremental repairs (not sure about the available options in Spotify Reaper) what command did you run ?

2016-02-11 17:33 GMT+01:00 Alain RODRIGUEZ <arodrime@gmail.com>:
CPU load is fine, SSD disks below 30% utilization, no long GC pauses


What is your current compaction throughput ?  The current value of 'concurrent_compactors' (cassandra.yaml or through JMX) ?

nodetool getcompactionthroughput

How to speed up compaction? Increased compaction throughput and concurrent compactors but no change. Seems there is plenty idle resources but can't force C* to use it.

You might want to try un-throttle the compaction throughput through:

nodetool setcompactionsthroughput 0

Choose a canari node. Monitor compaction pending and disk throughput (make sure server is ok too - CPU...)

Some other information could be useful:

What is your number of cores per machine and the compaction strategies for the 'most compacting' tables. What are write/update patterns, any TTL or tombstones ? Do you use a high number of vnodes ?

Also what is your repair routine and your values for gc_grace_seconds ? When was your last repair and do you think your cluster is suffering of a high entropy ? 

You can lower the stream throughput to make sure nodes can cope with what repairs are feeding them.

nodetool getstreamthroughput
nodetool setstreamthroughput X

C*heers,

-----------------
Alain Rodriguez
France

The Last Pickle

2016-02-11 16:55 GMT+01:00 Michał Łowicki <mlowicki@gmail.com>:
Hi,

Using 2.1.12 across 3 DCs. Each DC has 8 nodes. Trying to run repair using Cassandra Reaper but nodes after couple of hours are full of pending compaction tasks (regular not the ones about validation)

CPU load is fine, SSD disks below 30% utilization, no long GC pauses.

How to speed up compaction? Increased compaction throughput and concurrent compactors but no change. Seems there is plenty idle resources but can't force C* to use it.

Any clue where there might be a bottleneck?


--
BR,
Michał Łowicki