cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Galbraith <david92galbra...@gmail.com>
Subject Disastrous profusion of SSTables
Date Thu, 26 Mar 2015 07:51:48 GMT
Hey! So I'm running Cassandra 2.1.2 and using the
SizeTieredCompactionStrategy. I'm doing about 3k writes/sec on a single
node. My read performance is terrible, all my queries just time out. So I
do nodetool cfstats:

    Read Count: 42071
    Read Latency: 67.47804242827601 ms.
    Write Count: 131964300
    Write Latency: 0.011721604274792501 ms.
    Pending Flushes: 0
        Table: metrics16513
        SSTable count: 641
        Space used (live): 6366740812
        Space used (total): 6366740812
        Space used by snapshots (total): 0
        SSTable Compression Ratio: 0.25272488401992765
        Memtable cell count: 0
        Memtable data size: 0
        Memtable switch count: 1016
        Local read count: 42071
        Local read latency: 67.479 ms
        Local write count: 131964300
        Local write latency: 0.012 ms
        Pending flushes: 0
        Bloom filter false positives: 994
        Bloom filter false ratio: 0.00000
        Bloom filter space used: 37840376
        Compacted partition minimum bytes: 104
        Compacted partition maximum bytes: 24601
        Compacted partition mean bytes: 255
        Average live cells per slice (last five minutes): 111.67243951154147
        Maximum live cells per slice (last five minutes): 1588.0
        Average tombstones per slice (last five minutes): 0.0
        Maximum tombstones per slice (last five minutes): 0.0

and nodetool cfhistograms:

Percentile  SSTables     Write Latency      Read Latency    Partition
Size        Cell Count
                              (micros)          (micros)
(bytes)
50%            46.00              6.99         154844.95
149                 1
75%           430.00              8.53        3518837.53
179                 1
95%           430.00             11.32        7252897.25
215                 2
98%           430.00             15.54       22103886.34
215                 3
99%           430.00             29.86       22290608.19
1597                50
Min             0.00              1.66             26.91
104                 0
Max           430.00         269795.38       27311364.89
24601               924

Gross!! There are 641 SSTables in there, and all my reads are hitting
hundreds of them and timing out. How could this possibly have happened, and
what can I do about it? Nodetool compactionstats says pending tasks: 0, by
the way. Thanks!

Mime
View raw message