cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Galbraith <david92galbra...@gmail.com>
Subject Re: Disastrous profusion of SSTables
Date Thu, 26 Mar 2015 20:30:35 GMT
It looks like it was CASSANDRA-8860, setting that cold reads to omit thing
down to zero took my SSTable count from 641 to 1 and made all my queries
work. Thank you!!

On Thu, Mar 26, 2015 at 4:55 AM, graham sanderson <graham@vast.com> wrote:

> you may be seeing
>
> https://issues.apache.org/jira/browse/CASSANDRA-8860
> https://issues.apache.org/jira/browse/CASSANDRA-8635
>
> related issues (which ends up with excessive numbers of sstables)
>
> we applied
>
> *diff --git
> a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
> b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactio*
> *index fbd715c..cbb8c8b 100644*
> *---
> a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java*
> *+++
> b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java*
> @@ -118,7 +118,11 @@ public class SizeTieredCompactionStrategy extends
> AbstractCompactionStrategy
>      static List<SSTableReader> filterColdSSTables(List<SSTableReader>
> sstables, double coldReadsToOmit, int minThreshold)
>      {
>          if (coldReadsToOmit == 0.0)
> +        {
> +            if (!sstables.isEmpty())
> +                logger.debug("Skipping cold sstable filter for list sized
> {} containing {}", sstables.size(), sstables.get(0).getFilename());
>              return sstables;
> +        }
>
>
>          // Sort the sstables by hotness (coldest-first). We first build a
> map because the hotness may change during the sort.
>          final Map<SSTableReader, Double> hotnessSnapshot =
> getHotnessMap(sstables);
> *diff --git
> a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
> b/src/java/org/apache/cassandra/db/compaction/SizeTieredCo*
> *index 84e7d61..c6c5f1b 100644*
> *---
> a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java*
> *+++
> b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java*
> @@ -26,7 +26,7 @@ public final class SizeTieredCompactionStrategyOptions
>      protected static final long DEFAULT_MIN_SSTABLE_SIZE = 50L * 1024L *
> 1024L;
>      protected static final double DEFAULT_BUCKET_LOW = 0.5;
>      protected static final double DEFAULT_BUCKET_HIGH = 1.5;
> -    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.05;
> +    protected static final double DEFAULT_COLD_READS_TO_OMIT = 0.0;
>      protected static final String MIN_SSTABLE_SIZE_KEY =
> "min_sstable_size";
>      protected static final String BUCKET_LOW_KEY = "bucket_low";
>      protected static final String BUCKET_HIGH_KEY = "bucket_high";
>
> to our 2.1.3, though the entire coldReadsToOmit is removed in 2.1.4
>
> Note you don’t have to patch your code, you can set the value on each
> table (we just have a lot and dynamically generated ones) - basically try
> setting coldReadsToOmit back to 0 which was the default in 2.0.x
>
> On Mar 26, 2015, at 3:56 AM, Anishek Agarwal <anishek@gmail.com> wrote:
>
> Are you frequently updating same rows ? What is the memtable flush size ?
> can you post the table create query here in please.
>
> On Thu, Mar 26, 2015 at 1:21 PM, Dave Galbraith <
> david92galbraith@gmail.com> wrote:
>
>> Hey! So I'm running Cassandra 2.1.2 and using the
>> SizeTieredCompactionStrategy. I'm doing about 3k writes/sec on a single
>> node. My read performance is terrible, all my queries just time out. So I
>> do nodetool cfstats:
>>
>>     Read Count: 42071
>>     Read Latency: 67.47804242827601 ms.
>>     Write Count: 131964300
>>     Write Latency: 0.011721604274792501 ms.
>>     Pending Flushes: 0
>>         Table: metrics16513
>>         SSTable count: 641
>>         Space used (live): 6366740812
>>         Space used (total): 6366740812
>>         Space used by snapshots (total): 0
>>         SSTable Compression Ratio: 0.25272488401992765
>>         Memtable cell count: 0
>>         Memtable data size: 0
>>         Memtable switch count: 1016
>>         Local read count: 42071
>>         Local read latency: 67.479 ms
>>         Local write count: 131964300
>>         Local write latency: 0.012 ms
>>         Pending flushes: 0
>>         Bloom filter false positives: 994
>>         Bloom filter false ratio: 0.00000
>>         Bloom filter space used: 37840376
>>         Compacted partition minimum bytes: 104
>>         Compacted partition maximum bytes: 24601
>>         Compacted partition mean bytes: 255
>>         Average live cells per slice (last five minutes):
>> 111.67243951154147
>>         Maximum live cells per slice (last five minutes): 1588.0
>>         Average tombstones per slice (last five minutes): 0.0
>>         Maximum tombstones per slice (last five minutes): 0.0
>>
>> and nodetool cfhistograms:
>>
>> Percentile  SSTables     Write Latency      Read Latency    Partition
>> Size        Cell Count
>>                               (micros)          (micros)
>> (bytes)
>> 50%            46.00              6.99         154844.95
>> 149                 1
>> 75%           430.00              8.53        3518837.53
>> 179                 1
>> 95%           430.00             11.32        7252897.25
>> 215                 2
>> 98%           430.00             15.54       22103886.34
>> 215                 3
>> 99%           430.00             29.86       22290608.19
>> 1597                50
>> Min             0.00              1.66             26.91
>> 104                 0
>> Max           430.00         269795.38       27311364.89
>> 24601               924
>>
>> Gross!! There are 641 SSTables in there, and all my reads are hitting
>> hundreds of them and timing out. How could this possibly have happened, and
>> what can I do about it? Nodetool compactionstats says pending tasks: 0, by
>> the way. Thanks!
>>
>
>
>

Mime
View raw message