cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabrice Facorat <fabrice.faco...@gmail.com>
Subject Re: SSTable count at 10K during repair (won't decrease)
Date Fri, 20 May 2016 17:14:47 GMT
Are you using repairParallelism = sequential or parallel ?

As said by Alain:
- try to decrease streamthroughput to avoid overflooding nodes with a lots
of (small) streamed sstables
- if you are using // repair, switch to sequential
- don't start too much repair simultaneously.
- Do you really need to use LCS for your tables ? LCS make the problem even
worse. Use it with parcimony ;)



2016-05-06 18:05 GMT+02:00 Jean-Francois Gosselin <jfgosselin@gmail.com>:

> - Cassandra 2.1.13
> - SSDs
> - LeveledCompactionStrategy
> - Range repair (not incremental) with Spotify's Reaper
> https://github.com/spotify/cassandra-reaper
>
> Problem : When we run a repair job sometimes the SSTable count goes to 10K
> on one of nodes (not always the same node). The Reaper is smart enough to
> postpone the repair on this node since the number of pending compactions is
> > 20 but number of SSTables stays around 10K.
> Even If I set the compactionthroughput 0 (disable throttling) the SSTable
> count stays around 10K.
>
> Workaround: If we abort the repair, and restart the node it quickly (in 15
> minutes) goes back to 200 SSTables ...
>
> Any suggestions as what I should look at ?
>
> When it occurs, I've noticed that nodetool compactionstats and cfstats (on
> the table with 10K SSTables) takes minutes to return with a result.
>
> I thought that the issue might be related to
> https://issues.apache.org/jira/browse/CASSANDRA-10766 as I see the
> MemtablePostFlush waiting on the countdown latch but the Pending
> MemtablePostFlush is going up and down according to tpstats.
>
> Complete stack trace : http://pastebin.com/K1r3CUff
>
> I took some tpstats (roughly every minutes). Only these pools are not at 0
> (Active/Pending).
>
> Pool Name                    Active   Pending      Completed   Blocked
>  All time blocked
> MemtableFlushWriter               2         2         139864         0
>             0
> MemtablePostFlush                 1        13         223714         0
>             0
> CompactionExecutor               10        10         804964         0
>             0
>
> MemtableFlushWriter               4         4         139889         0
>             0
> MemtablePostFlush                 1        12         223744         0
>             0
> CompactionExecutor               12        12         805365         0
>             0
>
> MemtableFlushWriter               5         5         139896         0
>             0
> MemtablePostFlush                 1        10         223755         0
>             0
> CompactionExecutor                9         9         805503         0
>             0
>
> MemtableFlushWriter               4         4         139907         0
>             0
> MemtablePostFlush                 1        13         223762         0
>             0
> CompactionExecutor                9         9         805703         0
>             0
>
>
> MemtableFlushWriter               5         5         139927         0
>             0
> MemtablePostFlush                 1        14         223783         0
>             0
> CompactionExecutor               10        10         805971         0
>             0
>
> MemtableFlushWriter               7         7         139956         0
>             0
> MemtablePostFlush                 1        23         223806         0
>             0
> CompactionExecutor               10        10         806428         0
>             0
>
> nodetool compactionstats shows pending tasks 66
>
> Keyspace: foo
>         Read Count: 6308735
>         Read Latency: 12.132909585836147 ms.
>         Write Count: 15394697
>         Write Latency: 0.09054346675351908 ms.
>         Pending Flushes: 15
>                 Table: bar
>                 SSTable count: 10326
>                 SSTables in each level: [10090/4, 10, 106/100, 112, 0, 0,
> 0, 0, 0]
>                 Space used (live): 69204087872
>                 Space used (total): 69206400092
>                 Space used by snapshots (total): 2708047105
>                 Off heap memory used (total): 35230672
>                 SSTable Compression Ratio: 0.339043411676821
>                 Number of keys (estimate): 1601158
>                 Memtable cell count: 86524
>                 Memtable data size: 6508214
>                 Memtable off heap memory used: 0
>                 Memtable switch count: 22719
>                 Local read count: 6310549
>                 Local read latency: 12.135 ms
>                 Local write count: 15397653
>                 Local write latency: 0.091 ms
>                 Pending flushes: 10
>                 Bloom filter false positives: 2282107
>                 Bloom filter false ratio: 0.38494
>                 Bloom filter space used: 3244792
>                 Bloom filter off heap memory used: 3162168
>                 Index summary off heap memory used: 3348360
>                 Compression metadata off heap memory used: 28720144
>                 Compacted partition minimum bytes: 87
>                 Compacted partition maximum bytes: 2816159
>                 Compacted partition mean bytes: 69860
>                 Average live cells per slice (last five minutes):
> 817.6059838850788
>                 Maximum live cells per slice (last five minutes): 5002.0
>                 Average tombstones per slice (last five minutes): 0.0
>                 Maximum tombstones per slice (last five minutes): 0.0
>
> Thanks
>
> J-F Gosselin
>
>


-- 
Close the World, Open the Net
http://www.linux-wizard.net

Mime
View raw message